The End of Programming

The end of classical Computer Science is coming, and most of us are dinosaurs waiting for the meteor to hit.

The Evolution of Classical Computer Science

We're witnessing a transformative shift in the landscape of Computer Science, signaling the end of traditional methodologies as we know them.
Reflecting on my journey, which began with programming personal computers like the Commodore VIC-20 and Apple ][e in the 1980s, and continued through my academic pursuits in Computer Science, including a PhD at Berkeley, I realize the predominant focus was on what I term “classical” CS: encompassing programming, algorithms, data structures, systems, and programming languages. In this paradigm, the pinnacle achievement was translating ideas into human-authored programs — typically in languages like Java, C++, or Python. Regardless of complexity, concepts in Classical CS, from database join algorithms to intricate protocols like Paxos consensus, were all encapsulated within human-readable, human-understandable code.

Navigating the AI Winter: Reflections on Traditional Algorithms

Back in the early ’90s when I was in college, the field of AI was going through a rough patch known as the AI Winter, and classical algorithms ruled the roost. I remember my research stint at Cornell with Dan Huttenlocher, a leading figure in computer vision (now the Dean of the MIT School of Computing). In Dan’s PhD-level computer vision course around 1995, we never delved into topics like deep learning or neural networks; our focus was on classical algorithms such as Canny edge detection, optical flow, and Hausdorff distances. Deep learning was just starting to emerge, far from being considered mainstream AI or CS.

Despite the passage of 30 years, the foundational principles of Computer Science remain rooted in data structures, algorithms, and programming. I'll be surprised if, in another three decades, or even a mere decade, we're still following the same approach in CS education. It's clear to me that the field of CS is on the cusp of significant changes, changes that many of us may not be fully prepared for:

Programming will be obsolete

The future of programming is on the brink of a radical transformation. I firmly believe that the traditional notion of "writing a program" is on its way out, with AI systems poised to replace most software as we know it. For the vast majority of applications, software will be trained rather than manually coded. Even for scenarios requiring seemingly straightforward programs, AI will step in to generate them, alleviating the need for human intervention.

This concept may sound far-fetched, but consider the evolution of Computer Science from its roots in Electrical Engineering. Early pioneers likely envisioned a future where deep knowledge of semiconductors, binary arithmetic, and microprocessor design would be essential for understanding software. However, in today's reality, the majority of software developers have little to no understanding of CPU mechanics or transistor physics. Similarly, I anticipate that future Computer Scientists will be so detached from traditional software concepts that tasks like reversing a linked list or implementing Quicksort will seem alien to them.

Current advancements in AI, such as coding assistants like CoPilot, merely scratch the surface of what's to come. It's evident to me that AI will ultimately take over the entire process of writing programs, relegating humans to a supervisory role at best. The rapid progress in AI content generation, such as the leap from DALL-E v1 to DALL-E v2 within a span of 15 months, highlights the underestimation of large AI model capabilities. What once seemed like science fiction is quickly becoming reality.

I'm not just talking about CoPilot replacing programmers; I envision a future where the concept of writing programs is entirely replaced by training models. Future CS students won't need to learn mundane tasks like adding nodes to binary trees or coding in C++. Such education will become obsolete, akin to teaching engineering students how to use a slide rule.

Engineers of the future will effortlessly deploy massive AI models, ready to tackle any task, with minimal human input. The focus will shift from manually coding algorithms to providing the right examples, training data, and evaluation methods. Powerful models capable of generalizing through few-shot learning will render massive datasets unnecessary in most cases. The process of "training" an AI model won't involve complex operations like running gradient descent loops in PyTorch; instead, it will be about teaching by example, letting the machine do the heavy lifting.

In this new era of Computer Science, machines will possess immense capabilities and knowledge, resembling an educational endeavor more than an engineering one. Unlike human children, these AI systems will be entrusted with critical tasks like piloting airplanes, managing power grids, and potentially governing entire nations. As the focus shifts towards educating intelligent machines rather than directly programming them, the traditional notion of programming will become obsolete.

How does all of this change how we think about the field of Computer Science?
How does all of this change how we think about the field of Computer Science?

The new atomic unit of computation becomes not a processor, memory, and I/O system implementing a von Neumann machine, but rather a massive, pre-trained, highly adaptive AI model. This is a seismic shift in the way we think about computation — not as a predictable, static process, governed by instruction sets, type systems, and notions of decidability. AI-based computation has long since crossed the Rubicon of being amenable to static analysis and formal proof. We are rapidly moving towards a world where the fundamental building blocks of computation are temperamental, mysterious, adaptive agents.

This shift is underscored by the fact that** nobody actually understands how large AI models work**. People are publishing research papers actually discovering new behaviors of existing large models, even though these systems have been “engineered” by humans. Large AI models are capable of doing things that they have not been explicitly trained to do, which should scare the shit out of Nick Bostrom and anyone else worried (rightfully) about an superintelligent AI running amok. We currently have no way, apart from empirical study, to determine the limits of current AI systems. As for future AI models that are orders of magnitude larger and more complex — good friggin’ luck!

The shift in focus from programs to models should be obvious to anyone who has read any modern machine learning papers. These papers barely mention the code or systems underlying their innovations; the building blocks of AI systems are much higher-level abstractions like attention layers, tokenizers, and datasets. A time traveller from even 20 years ago would have a hard time making sense of the three sentences in the (75-page-long!) GPT-3 paper that describe the actual software that was built for the model:

• We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer [CGRS19]. To study the dependence of ML performance on model size, we train 8 different sizes of model, ranging over three orders of magnitude from 125 million parameters to 175 billion parameters, with the last being the model we call GPT-3. Previous work [KMH+20] suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for downstream language tasks.

This shift in the underlying definition of computing presents a huge opportunity, and plenty of huge risks. Yet I think it’s time to accept that this is a very likely future, and evolve our thinking accordingly, rather than just sit here waiting for the meteor to hit.

Sidebar: What have I been up to lately?

My long-time readers will notice that I haven’t been blogging much lately. Sorry about that! I’ve been a bit busy.

About three years ago, I left Google for a startup that was acquired by Apple. I subsequently left Apple to lead engineering at another startup for a couple of years. In that time I learned a ton about startup life, building AI systems, and building teams. It was great.

A couple of months ago I jumped ship to start my own company. I’m now the co-founder and CEO of Fixie.ai, a stealth startup in the AI space, and the blog post above may or may not represent some of the thinking going into this new venture. We have a fantastic founding team and I’m really excited about what we’re going to be able to accomplish. Over the next few months I hope to share more about what we’re up to. Until then, hang tight!

Conclusion:

"In conclusion, the landscape of Computer Science is on the brink of a profound transformation. The traditional approach of manually writing programs is gradually giving way to AI-driven systems that learn and adapt on their own. This shift challenges conventional notions of programming and calls for a reevaluation of how we educate future generations of Computer Scientists.

As we embrace this new era, it's essential to recognize the potential and risks it presents. While AI-powered programming promises unprecedented efficiency and innovation, it also raises concerns about control, understanding, and the ethical implications of autonomous systems.

Ultimately, the future of Computer Science lies in educating intelligent machines, rather than merely programming them. This shift not only reshapes the field but also opens up new possibilities for collaboration between humans and AI.

Credit: This article is adapted from an original piece by Matt Welsh.