Back to Lisp Part 1 - Working Inside the Language

I discovered Structure and Interpretation of Computer Programs in my late teens and quickly moved on to Common Lisp, working my way up from a z80 macro assembler to various web frameworks, and fun projects like a CAM system. After 12 years where I didn't work with Lisp at all, I recently decided to go back, and I am delighted by what I found. This is a series of articles that articulate my thoughts about coming back to an old love, and document the very practical things I found along the way.

When I mention Lisp in this article, it will refer to either Scheme or Common Lisp, the two languages I have actually used. You can probably replace them with Emacs Lisp or Clojure or any other SEXP based language and follow along just as well. I hate it when people write LISP uppercase as if we were still using something from the 60ies, I'm going with Lisp to convey a modern touch.

In this first article, I will talk about what I missed most: working inside a language.

REPL programming is the foundation

While many languages offer a REPL (read eval loop, i.e. a prompt you can use to execute statements), few adopt it as the central way to interact with your software system.

Notebook-style interfaces are pretty common these days, (python notebooks, R studio and Wolfram Mathematica come to mind), but they are more about "sessions" and "documents", about exploring and dissecting data than large scale programming. Other systems often come with a debug console you can use to inspect and modify the system at runtime (think browser javascript console). And finally, scratch files that can quickly be executed are common in IDEs.

In Lisp projects, you write functions and modules and packages in files, as is usual in programming projects, but you always have the compiler running along, compiling what you write and giving you feedback on what you typed. In traditional IDEs, the IDEs understanding of the program is divorced from the execution environment (either by being implemented in the IDE itself, or being run in a separate Language Server Process). With Lisp, the compiler functions as LSP to help you interact with your code (go to definition, inspect, etc...), as quick prompt to run experiments, as debugger to trace / debug / instrument and interact with the actual running system, as shell to manage your packages, deployments, builds and runtime systems.

With Lisp, everything feels intimately (and robustly) connected.

Programming as language creation is explicit

Programming is about getting computers to do things for us. But computers only really care about instructions that their CPU can execute. Since our brains can't comprehend streams of assembly language, we created programming languages, coherent, readable ways of assembling words and concepts so that we can collaborate amongst humans on the one side, and have computers execute our ideas on the other.

Programming languages (as in python, javascript, etc...), libraries, frameworks, naming conventions, design patterns all contribute to create a "programming dialect" shared by people working on a project. This language allows us to express solutions to the problems we are trying to solve in a way that is both executable by a computer, and understandable by our colleagues. More targeted project languages are often called "Domain Specific Languages".

These dialects are shaped by:

• frameworks and libraries used (i.e., we use react and redux)
• design patterns used (we use higher order components and context providers for a global store)
• code and naming conventions (we call our handlers onX, and our store actions are of the form verbObjectObject. we use immer for imperative-like store reducers)

The programming language used can be more or less flexible in terms of syntax: domain-specific languages often have to carry syntactical baggage around to express certain concepts. For example, you might implement state machines by using classes for transitions, enums for states, and certain naming conventions for event dispatching. Javascript I think is so successful because it is a language that makes it easy to be creative and elegant with syntax.

Finding a satisfying API, syntax and naming conventions for concepts can be tremendously difficult. Impedance mismatch with the underlying programming languages can also mean that bugs are easier to make than they really should. When transpiling or using advanced meta programming techniques, the runtime errors are often hard to map back to the original code. Dialects still feel like dialects, modified, lived, bastardized versions of the underlying programming language.

Lisp languages don't really have much in way of syntax, as you usually write the program in terms of nested linked lists representing the abstract syntax tree. This gives you a much more simpler tool to not only create a programming dialect, but actually modify the underlying grammar to allow for a much more concise expression of useful concepts.

This is a two edged sword, as it is easy to create incoherent project languages with inscrutable grammatical extensions. A project usually needs at most one or two grammatical extensions to support its project language, and these are usually trivial (for example, an easy way to define state machine enums). In traditional languages, a clever closure pattern or some code generation will get you there just as well.

The beauty of Lisp however is during the ideation phase. It is very easy to try out different syntax ideas, move seamlessly between the meta and the practical level, run experiments in the REPL, massage syntax. This makes it possible to quickly home in on what fundamental concepts for the project are, and expressing them succinctly.

Javascript is an interesting language because it is very malleable. It allows us to use many clever tricks to make dialects that look almost like languages. Over the years, many things and patterns have been tried in Javascript: functional chains with lodash and underscore, functional reactive programming with redux and react hooks, effect programming with react hooks and many more I have never about. Of course, Javascript is also a widely used transpilation target, bringing interesting innovations to the core language while still maintaining close similarity. Source maps for example are an essential tool in making Javascript dialects useful in practice.

I missed experimenting with concepts at the language level

My past experience with Lisp has deeply engrained this way of "thinking in languages". Even when programming PHP, Javascript or C++, I will encounter patterns and ideas that I have explored in Lisp. While I can't transform the language I am working with, I can build a dialect that has sound grammatical foundations, because I built it as a "real" language in Lisp. I work with what I have (syntax tricks, naming conventions, API design) to make it as elegant and computationally airtight as possible.

Over time, I forgot how easy it was to use Lisp to experiment with different approaches. Designing a concurrent task language in C++ takes many lines of code and a lot of careful thought. While you can sketch things out pretty quickly using macros and code generation, or by being well acquainted with C++ templates, you still wrestle with a lot of syntax and operational complexity.

In a Lisp language, you can experiment by writing a program as you wish you could write it, then implementing it in 3 macros and then running it, printing out ASTs in the REPL for debugging. Building a grammar for concurrent data streams is an afternoon project.

Even though my main languages currently (PHP, Javascript and Golang) are not Lisps at all, I reconnected with this idea of creating languages, and will keep on experimenting with concepts that I can then port over once refined.