Serving Plug: Building an Elixir HTTP server from scratch

Welcome back to another edition of Elixir Alchemy! In our continued quest to find out what’s happening under the hood, we'll take a deep dive into HTTP servers in Elixir.

To understand how HTTP servers work, we'll implement a minimal example of one that can run a Plug application. We’ll learn about decoding requests and encoding responses, as well as how Plug interacts with the web server by building a minimal subset of an HTTP server that can accept HTTP requests and run a Plug application.

-> The HTTP server we’re building is for educational purposes. We won’t build a production-ready HTTP server. If you're looking for one that is, please try cowboy, which is the default choice of HTTP server in Elixir applications.

HTTP over TCP

We’ll start with the fundamentals. HTTP is a protocol that commonly uses TCP to transport requests and responses between an HTTP client—like a web browser—and a web server.

Erlang provides the :gen_tcp module that can be used to start a TCP socket that receives and transmits data. We'll use that library as the basis of our server.

# lib/http.ex
defmodule Http do
  require Logger

  def start_link(port: port) do
    {:ok, socket} = :gen_tcp.listen(port, active: false, packet: :http_bin, reuseaddr: true)
    Logger.info("Accepting connections on port #{port}")

    {:ok, spawn_link(Http, :accept, [socket])}
  end

  def accept(socket) do
    {:ok, request} = :gen_tcp.accept(socket)

    spawn(fn ->
      body = "Hello world! The time is #{Time.to_string(Time.utc_now())}"

      response = """
      HTTP/1.1 200\r
      Content-Type: text/html\r
      Content-Length: #{byte_size(body)}\r
      \r
      #{body}
      """

      send_response(request, response)
    end)

    accept(socket)
  end

  def send_response(socket, response) do
    :gen_tcp.send(socket, response)
    :gen_tcp.close(socket)
  end

  def child_spec(opts) do
    %{id: Http, start: {Http, :start_link, [opts]}}
  end
end

In this example, we’ve created a TCP server that responds to every request with the current time in an HTTP response.

We start a socket to listen to the passed in port in start_link/1, and we'll spawn accept/1 in a new process that waits until a request comes in over the socket by calling :gen_tcp.accept/1.

When it does, we put it into the request variable and create a response to send to the client. In this case, we'll be sending a response that shows the current time.

Building HTTP responses

HTTP/1.1 200\r\nContent-Type: text/html\r\n\r\n\Hello world! The time is 14:45:49.058045

An HTTP response contains a couple of parts:

• A status line, with the protocol version (HTTP/1.1) and a response code (200)
• A carriage return, followed by a line feed (\r\n) to split the status line from the rest of the response
• (Optional) header lines (Content-Type: text/html), separated by CRLFs
• A double CRLF, to separate the headers from the response body
• The response body that will be shown in the browser (Hello world! The time is 14:45:49.058045)

We pass the body we built to send_response/2, which takes care of sending the response over the socket and finally closing the socket connection.

We spawn a process for each request so the server can call accept/1 again to accept new requests. This way, we can respond to requests in parallel, instead of subsequent requests having to wait for previous ones to complete being processed.

Running the Server

Let's try it out. We'll use a supervisor to run our HTTP server to make sure it's restarted immediately when if it fails.

Our server implementation has a child_spec/1 function that specifies how it should be started. It states that it should call the Http.start/1 function with the passed options, which will return the newly spawned process' ID.

# lib/http.ex
defmodule Http do
  # ...

  def child_spec(opts) do
    %{id: Http, start: {Http, :start_link, [opts]}}
  end
end

Because of that, we can add it to the list of children managed by the supervisor in lib/http/application.ex.

# lib/http/application.ex
defmodule Http.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      {Http, port: 8080}
    ]

    opts = [strategy: :one_for_one, name: Http.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

We pass {Http, port: 8080} as one of our supervisor's children to start the server at port 8080 when the application is started.

$ mix run --no-halt
19:39:29.454 [info]  Accepting connections on port 8080

If we start the server and use our browser to send a request, we can see that it indeed returns the current time.

Plug

Now that we know how a web server works, let's take it to the next level. Our current implementation has the response hard-coded into the server. To allow our web server to run different apps, we'll move the app out into a separate Plug module.

# lib/current_time.ex
defmodule CurrentTime do
  import Plug.Conn

  def init(options), do: options

  def call(conn, _opts) do
    conn
    |> put_resp_content_type("text/html")
    |> send_resp(200, "Hello world! The time is #{Time.to_string(Time.utc_now())}")
  end
end

The CurrentTime module defines a call/2 function that takes the passed in %Plug.Conn struct. It then sets the response content type to "text/html" before sending the "Hello world!" message—together with the current time—back as a response.

Our new module behaves the same as the web server example, but it's detached from the web server. Because of Plug, we could swap out servers without having to change the application's code, and we can also change the application without having to touch the web server.

Writing a Plug Adapter

To make sure our web server can communicate with our web application, we need to build a %Plug.Conn{} struct to pass to CurrentTime.call/2. We'll also need to turn the sent response into a string that our web server can send back over the socket.

To do this, we'll create an adapter that handles the communication between our Plug app and our web server.

# lib/http/adapter.ex
defmodule Http.PlugAdapter do
  def dispatch(request, plug) do
    %Plug.Conn{
      adapter: {Http.PlugAdapter, request},
      owner: self()
    }
    |> plug.call([])
  end

  def send_resp(socket, status, headers, body) do
    response = "HTTP/1.1 #{status}\r\n#{headers(headers)}\r\n#{body}"

    Http.send_response(socket, response)
    {:ok, nil, socket}
  end

  def child_spec(plug: plug, port: port) do
    Http.child_spec(port: port, dispatch: &dispatch(&1, plug))
  end

  defp headers(headers) do
    Enum.reduce(headers, "", fn {key, value}, acc ->
      acc <> key <> ": " <> value <> "\n\r"
    end)
  end
end

Instead of responding directly from Http.accept/2, we'll use our adapter's dispatch/2 function to build a %Plug.Conn{} struct and pass that to our plug's call/2 function.

In the %Plug.Conn{}, we'll set the :adapter to link to our Adapter module, and then pass the socket the response that should be sent over. This ensures that the Plug app knows which module to call send_resp/4 on.

Our adapter's send_resp/4 takes the socket connection, the response status, a list of headers and a body, which are all prepared by the Plug application. It uses the passed in arguments to build the response and calls out to Http.send_response/2 that we've implemented before.

The child_spec/1 for our adapter returns the child_spec/1 for the Http module. This causes the web server to start when we supervise our adapter. We'll pass the dispatch function as the dispatch so that it can be called by our web server when it receives a response.

# lib/http/application.ex
defmodule Http.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      {Http.PlugAdapter, plug: CurrentTime, port: 8080}
    ]

    opts = [strategy: :one_for_one, name: Http.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Instead of starting Http in our application module, we'll start Http.PlugAdapter, which will take care of setting the plug, preparing the dispatch function and starting the web server.

# lib/http.ex
defmodule Http do
  require Logger

  def start_link(port: port, dispatch: dispatch) do
    {:ok, socket} = :gen_tcp.listen(port, active: false, packet: :http_bin, reuseaddr: true)
    Logger.info("Accepting connections on port #{port}")

    {:ok, spawn_link(Http, :accept, [socket, dispatch])}
  end

  def accept(socket, dispatch) do
    {:ok, request} = :gen_tcp.accept(socket)

    spawn(fn ->
      dispatch.(request)
    end)

    accept(socket, dispatch)
  end

  # ...
end

Since we now handle requests in our Plug, we can remove most of the code in Http.accept/2. The Http.start_link/2 function will now receive the dispatch function from the adapter, which is used to send the request to in Http.accept/2.

$ mix run --no-halt
19:39:29.454 [info]  Accepting connections on port 8080

Running the server again, everything still works exactly as before. However, our HTTP server, web application and Plug adapter are now three separate modules.

Swapping out Plug Applications

Because our server is now separate from our adapter and web application, we can swap the Plug out to run another application on the server. Let's give that a shot.

# mix.exs
defmodule Http.MixProject do
  # ...

  defp deps do
    [
      {:plug_octopus, github: "jeffkreeftmeijer/plug_octopus"},
      {:plug, "~> 1.7"}
    ]
  end
end

In our mix.exs file, we add :plug_octopus as a dependency.

# lib/http/application.ex
defmodule Http.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      {Http.PlugAdapter, plug: Plug.Octopus, port: 8080}
    ]

    opts = [strategy: :one_for_one, name: Http.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

We then swap CurrentTime for Plug.Octopus in our Http.Application module. Starting the server and visiting http://localhost:8080 now shows an octopus!

However, clicking the flip! and crash! buttons doesn't do anything and any URL that's called shows the same page. That's because we skipped over parsing the requests altogether. Since we don't read the requests, we'll always pass the same response back. Let's fix that.

Parsing Requests

To read requests, we'll need to read the response from the socket. Thanks to the http_bin option that we're passing when calling :gen_tcp.listen/2, the request is returned in a format we can pattern match on.

# lib/http.ex
defmodule Http do
  # ...

  def read_request(request, acc \\ %{headers: []}) do
    case :gen_tcp.recv(request, 0) do
      {:ok, {:http_request, :GET, {:abs_path, full_path}, _}} ->
        read_request(request, Map.put(acc, :full_path, full_path))

      {:ok, :http_eoh} ->
        acc

      {:ok, {:http_header, _, key, _, value}} ->
        read_request(
          request,
          Map.put(acc, :headers, [{String.downcase(to_string(key)), value} | acc.headers])
        )

      {:ok, line} ->
        read_request(request, acc)
    end
  end

  # ...
end

The Http.read_request/2 function takes a socket connection and will be called from the dispatch function. It will keep calling :gen_tcp.recv/2 to accept lines from the request until it receives an :http_eoh response, indicating the end of the requests headers.

We match on the :http_request line, which includes the full request path. We'll use that to extract the path and URL parameters later. We'll also match on all :http_header lines, which we convert to a list we can pass to our Plug application later.

# lib/http/adapter.ex
defmodule Http.PlugAdapter do
  def dispatch(request, plug) do
    %{full_path: full_path} = Http.read_request(request)

    %Plug.Conn{
      adapter: {Http.PlugAdapter, request},
      owner: self(),
      path_info: path_info(full_path),
      query_string: query_string(full_path)
    }
    |> plug.call([])
  end

  # ...

  defp headers(headers) do
    Enum.reduce(headers, "", fn {key, value}, acc ->
      acc <> key <> ": " <> value <> "\n\r"
    end)
  end

  defp path_info(full_path) do
    [path | _] = String.split(full_path, "?")
    path |> String.split("/") |> Enum.reject(&(&1 == ""))
  end

  defp query_string([_]), do: ""
  defp query_string([_, query_string]), do: query_string

  defp query_string(full_path) do
    full_path
    |> String.split("?")
    |> query_string
  end
end

We call Http.read_request/1 from Http.PlugAdapter.dispatch/2. Having the full_path, we can extract the path_info (a list of path segments), and query_string (all URL parameters after the "?"). We add these to the %Plug.Conn{} to have our Plug app handle the rest.

Restarting the server, we can now flip and crash the lobster.

A Minimal Web Server Example that flips and crashes

With everyone in place, and no screws on the floor, our project to look into HTTP servers in Elixir is done. We've implemented a web server that extracts data from requests and used it to send a request to a Plug application. It even has concurrency included: since each request spawns a separate process, our web server can handle multiple concurrent users.

There's more to HTTP servers than we've shown in this article, but we hope implementing one from the ground up gave you some insight into how a web server could work.

Check out the finished project if you'd like to review the code, and don't forget to subscribe to the mailing list if you'd like to read more Elixir Alchemy!