HTTP over unix sockets in Common Lisp

In this post we are going to take took at how we can send HTTP requests using Unix sockets in Common Lisp.

Motivation

Let's first begin to analyze the motivation behind this post. To understand that, first we need to find an answer to the question, how do we send HTTP request using unix sockets in Common Lisp?

I have already asked the same question in the Lisp Reddit Channel. This post is just an attempt to deep dive in to the answer to the problem at hand.

Basically all I want to do is to implement the following curl request in Lisp

curl --unix-socket /var/run/docker.sock http://localhost/v1.41/containers/json

cl-docker

Why do I have a requirement like this in the first place? A couple of weeks back I have written a post about Running Docker commands in Lisp REPL where I have mentioned that the proper way to build a Lisp SDK for Docker involves sending HTTP requests using unix sockets. Hence this post is an elaborate explanation of how the solution actually works.

Running docker commands from Common Lisp REPLs

Rajasegar Chandran ・ Sep 13 '22

#commonlisp #lisp #docker #productivity

Before diving in to the solution, let's first set some context around Unix sockets, how it is different from TCP/IP sockets in the first place.

Unix sockets

A UNIX socket, also known as UNIX Domain Socket, is an inter-process communication mechanism that allows bidirectional data exchange between processes running on the same machine. UNIX domain sockets know that they’re executing on the same system, so they can avoid some checks and operations (like routing); which makes them faster and lighter than IP sockets. So if you plan to communicate with processes on the same host, this is a better option than IP sockets.

UNIX domain sockets are subject to file system permissions, while TCP sockets can be controlled only on the packet filter level.

The solution

This the answer I got from the user Aidenn0 that fits perfectly for my requirements.

(let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :stream)))
  (sb-bsd-sockets:socket-connect socket "/var/run/docker.sock")
  (let ((stream (sb-bsd-sockets:socket-make-stream socket
                                                   :element-type '(unsigned-byte 8)
                                                   :input t
                                                   :output t
                                                   :buffering :none)))
    (let ((wrapped-stream (flexi-streams:make-flexi-stream (drakma::make-chunked-stream stream)
                                                           :external-format drakma::+latin-1+)))
      (drakma:http-request "http://localhost/v1.41/containers/json" :stream wrapped-stream))))

Let's see how the above code works.

First, create a new socket by making a new instance of sb-bsd-sockets:local-socket with the socket type set as :stream. And then make the socket connected to the local unix socket of Docker in /var/run/docker.sock.

Now create a new socket stream and wrap it with the flexi-streams.

FLEXI-STREAMS

FLEXI-STREAMS implements "virtual" bivalent streams that can be layered atop real binary or bivalent streams and that can be used to read and write character data in various single- or multi-octet encodings which can be changed on the fly. It also supplies in-memory binary streams which are similar to string streams.

Finally pass this stream to the HTTP client library like Drakma to make the HTTP request using the wrapped stream.

Drakma

Drakma is a full-featured HTTP client implemented in Common Lisp. It knows how to handle HTTP/1.1 chunking, persistent connections, re-usable sockets, SSL, continuable uploads, file uploads, cookies, and more.

This request will return a JSON response which can then be processed using any JSON library like yason or cl-json

And this is how I used the solution to build an SDK for Docker in Common Lisp through the cl-docker package.

(defun docker-api (url)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :stream)))
    (sb-bsd-sockets:socket-connect socket "/var/run/docker.sock")
    (let ((stream (sb-bsd-sockets:socket-make-stream socket                     
        :element-type '(unsigned-byte 8)
    :input t
    :output t
    :buffering :none)))
    (let ((wrapped-stream (flexi-streams:make-flexi-stream (drakma::make-chunked-stream stream)
    :external-format :utf-8)))
    (yason:parse (dex:get (docker-url url) :stream wrapped-stream :want-stream t) :object-as :alist)))))

FYI, I have used Dexador instead of Drakma to send HTTP requests by using the same stream object. The API of Dexador is pretty much similar to Drakma. And to parse the JSON response I use yason and returning the output as an Alist (association list).

References:

• Drakma
• Flexi-streams
• Unix domain socket
• Unix sockets vs TCP/IP sockets

Hope you enjoyed the post. I, myself had a lot of learning about sockets and streams, especially unix sockets while working on this problem. Please let me know for any queries/feedback in the comments section.