I discovered Structure and Interpretation of Computer Programs in my late teens and quickly moved on to Common Lisp, working my way up from a z80 macro assembler to various web frameworks, and fun projects like a CAM system. After 12 years where I didn't work with Lisp at all, I recently decided to go back, and I am delighted by what I found. This is a series of articles that articulate my thoughts about coming back to an old love, and document the very practical things I found along the way.
When I mention Lisp in this article, it will refer to either Scheme or Common Lisp, the two languages I have actually used. You can probably replace them with Emacs Lisp or Clojure or any other SEXP based language and follow along just as well. I hate it when people write LISP uppercase as if we were still using something from the 60ies, I'm going with Lisp to convey a modern touch.
In this first article, I will talk about what I missed most: working inside a language.
While many languages offer a REPL (read eval loop, i.e. a prompt you can use to execute statements), few adopt it as the central way to interact with your software system.
In Lisp projects, you write functions and modules and packages in files, as is usual in programming projects, but you always have the compiler running along, compiling what you write and giving you feedback on what you typed. In traditional IDEs, the IDEs understanding of the program is divorced from the execution environment (either by being implemented in the IDE itself, or being run in a separate Language Server Process). With Lisp, the compiler functions as LSP to help you interact with your code (go to definition, inspect, etc...), as quick prompt to run experiments, as debugger to trace / debug / instrument and interact with the actual running system, as shell to manage your packages, deployments, builds and runtime systems.
With Lisp, everything feels intimately (and robustly) connected.
Programming is about getting computers to do things for us. But computers only really care about instructions that their CPU can execute. Since our brains can't comprehend streams of assembly language, we created programming languages, coherent, readable ways of assembling words and concepts so that we can collaborate amongst humans on the one side, and have computers execute our ideas on the other.
These dialects are shaped by:
- frameworks and libraries used (i.e., we use react and redux)
- design patterns used (we use higher order components and context providers for a global store)
- code and naming conventions (we call our handlers onX, and our store actions are of the form verbObjectObject. we use immer for imperative-like store reducers)
Finding a satisfying API, syntax and naming conventions for concepts can be tremendously difficult. Impedance mismatch with the underlying programming languages can also mean that bugs are easier to make than they really should. When transpiling or using advanced meta programming techniques, the runtime errors are often hard to map back to the original code. Dialects still feel like dialects, modified, lived, bastardized versions of the underlying programming language.
Lisp languages don't really have much in way of syntax, as you usually write the program in terms of nested linked lists representing the abstract syntax tree. This gives you a much more simpler tool to not only create a programming dialect, but actually modify the underlying grammar to allow for a much more concise expression of useful concepts.
This is a two edged sword, as it is easy to create incoherent project languages with inscrutable grammatical extensions. A project usually needs at most one or two grammatical extensions to support its project language, and these are usually trivial (for example, an easy way to define state machine enums). In traditional languages, a clever closure pattern or some code generation will get you there just as well.
The beauty of Lisp however is during the ideation phase. It is very easy to try out different syntax ideas, move seamlessly between the meta and the practical level, run experiments in the REPL, massage syntax. This makes it possible to quickly home in on what fundamental concepts for the project are, and expressing them succinctly.
Over time, I forgot how easy it was to use Lisp to experiment with different approaches. Designing a concurrent task language in C++ takes many lines of code and a lot of careful thought. While you can sketch things out pretty quickly using macros and code generation, or by being well acquainted with C++ templates, you still wrestle with a lot of syntax and operational complexity.
In a Lisp language, you can experiment by writing a program as you wish you could write it, then implementing it in 3 macros and then running it, printing out ASTs in the REPL for debugging. Building a grammar for concurrent data streams is an afternoon project.