On the nature of the novel paradigm of programming
Artifical Intelligence success stories have become so ubiquitous that it seems that every piece of software will soon incorporate AI in one way or another. But what does an AI program do? AI programs can learn and adapt with experience. The difference in building AI programs from building non-AI ones is bringing about a paradigm shift in our understanding of what programming is. In this article, I explore the essentials of the new paradigm and possible future routes it can take.
Karpathy, in his article “Software 2.0“, describes an AI-triggered paradigm shift. He imagines the “Programmer 2.0” as more involved with ‘editing and growing datasets’ than writing code, and he predicts a decline in demand for human developers. AI, he says, will find the right algorithm itself, given the right dataset. I argue otherwise. I believe that Software 2.0 will be less about data, but more about defining the proper constraints around the search space of possible algorithms. I argue that the demand for human developers will not weaken, but instead, its nature will change.
Divergence of Paradigms
An algorithm is a finite sequence of machine-interpretable instructions. In the old paradigm, the programmer’s goal was to knit together a sequence of instructions that constitute a single, well-defined algorithm. In the new paradigm, the programmer only specifies a template, which encodes multiple potential algorithms, and the machine learns the optimal algorithms using training data. The template maps to some region in the whole algorithmic space — the space of all possible algorithms.
The Taxonomy of Algorithmic Templates
Turing Machine (TM) is a universal template. It encodes in itself the space of all possible algorithms. What follows is that every conceivable algorithm is, in essence, a Turing Machine (TM) with a table of rules. The table of rules dictates the precise pathway of execution. A special Turing Machine called the Universal Turing Machine (UTM) can simulate the execution of any other Turing Machine, given a table of rules as an input.
Lambda Calculus (LC), developed by Alonso Church — Alan Turing’s supervisor — is another universal template, equivalent to the Turing Machine. The aesthetics of LC differ considerably from that of TM. TM formalizes computing as manipulation of symbols on a strip of tape according to a table of rules. LC formalizes it as the stacking…