HTTP(S) proxy server in Crystal in less than 50 lines of code

The purpose of this tutorial is to implement a proxy server for HTTP and HTTPS in pure Crystal.

Handling of HTTP is a matter of parsing request, passing such request to the destination server, reading response, and passing it back to the client.

HTTPS is different as it'll use a technique called HTTP CONNECT tunneling. At first, the client sends a request using HTTP CONNECT method to set up the tunnel between the client and destination server. When such a tunnel consisting of two TCP connections is ready, the client starts a regular TLS handshake with the destination server to establish a secure connection and later send requests and receive responses.

All we need for that is a built-in Crystal HTTP server and client from HTTP module.

HTTP

To support HTTP we will use a built-in HTTP server and client. The role of the proxy is to handle HTTP requests, pass a request to the desired destination, and send a response back to the client.

HTTP CONNECT tunneling

If a client wants to use HTTPS to talk to the server. The client is aware of using a proxy. A simple HTTP request/response flow cannot be used since the client needs to establish a secure connection with the server (HTTPS). The technique which works is to use the HTTP CONNECT method.

In this mechanism, the client asks a proxy server to forward the TCP connection to the desired destination. The server then proceeds to make the connection on behalf of the client. Once the connection has been established by the server, the proxy server continues to proxy the TCP stream to and from the client. Only the initial connection request is HTTP - after that, the server simply proxies the established TCP connection.

Implementation

Keep in mind the presented code is not a production-grade solution.

require "http"

class HTTP::ProxyHandler
  include HTTP::Handler

  def call(context)
    case context.request.method
    when "CONNECT"
      handle_tunneling(context)
    else
      handle_http(context)
    end
  end

  private def handle_tunneling(context)
    host, port = context.request.resource.split(":", 2)
    upstream = TCPSocket.new(host, port)

    context.response.upgrade do |downstream|
      channel = Channel(Nil).new(2)

      downstream = downstream.as(TCPSocket)
      downstream.sync = true

      spawn do
        transfer(upstream, downstream, channel)
        transfer(downstream, upstream, channel)
      end

      2.times { channel.receive }
    end
  end

  private def transfer(destination, source, channel)
    spawn do
      IO.copy(destination, source)
    rescue ex
      Log.error(exception: ex) { "Unhandled exception on HTTP::ProxyHandler" }
    ensure
      channel.send(nil)
    end
  end

  private def handle_http(context)
    uri = URI.parse(context.request.resource)
    client = HTTP::Client.new(uri)
    response = client.exec(context.request)

    context.response.headers.merge!(response.headers)
    context.response.status_code = response.status_code
    context.response.puts(response.body)
  end
end

proxy_handler = HTTP::ProxyHandler.new
server = HTTP::Server.new([proxy_handler])
address = server.bind_tcp(8080)
puts "Listening on http://#{address}"
server.listen

Our server while getting a request will take one of two paths:

• handling HTTP
• handling HTTP CONNECT tunneling.

This is done with:

def call(context)
  case context.request.method
  when "CONNECT"
    handle_tunneling(context)
  else
    handle_http(context)
  end
end

Function to handle HTTP — handle_http is self-explanatory so let's focus on handling tunneling.

The first part of handle_tunneling is about setting connection to destination server:

host, port = context.request.resource.split(":", 2)
upstream = TCPSocket.new(host, port)

All the magic happens inside the upgrade handler block:

context.response.upgrade do |downstream|
  ...
end

Once we've two TCP connections (client → proxy, proxy → destination server) we need to set tunnel up:

transfer(upstream, downstream, channel)
transfer(downstream, upstream, channel)

Next, we spawning two fibers and waiting for data are copied in two directions: from the client to the destination server and backward.

A buffered channel of capacity 2 is used to communicate that both fibers ended. So execution goes to the main fiber.

More info about Crystal concurrency you can find in the official documentation

And finally, we run our proxy server.

proxy_handler = HTTP::ProxyHandler.new
server = HTTP::Server.new([proxy_handler])
address = server.bind_tcp(8080)
puts "Listening on http://#{address}"
server.listen

The above code will initialize a server with our proxy handler.
The port of the HTTP server is set by using the method bind_tcp on the object HTTP::Server (the port set to 8080).

For more information, check HTTP::Server documentation

Testing

To test our proxy you can use e.g. Chromium:

chromium-browser --proxy-server=http://localhost:8080

or curl:

curl -Lv --proxy http://localhost:8080 https://httpbin.org/get

Testing the Proxy speeds on my DigitalOcean instance:

Happy Crystalling!