OpenResty on NixOS for an API Gateway

#nixos #devops #api #openresty

Are you using an API gateway? Do you really need one? If you are using NixOS and feel comfortable with some Lua, you may want to consider OpenResty on NixOS as an API gateway.

An API Gateway is simply a service that acts as a reverse proxy for API requests to one or more upstream services. Modern API gateways streamline the management of many functionalities such as authentication, authorization, rate limiting, caching, logging, monitoring, etc. There are quite a few such proprietary and open-source API gateways available. Cloud service providers offer their own managed API gateways as well.

We have been using Apache APISIX for a while now. It is a high-performance, cloud-native API gateway solution. It also has a nice dashboard for managing APIs. However, I have been looking for a simpler and more portable solution for our use case. In particular, I want to be able manage the API gateway as a NixOS service so that the configuration can be tested and redeployed easily.

Two of the fairly popular open-source API gateways, Kong and Apache APISIX, are based on OpenResty that is mainly powered by Nginx and Lua. NixOS has first-class support for OpenResty. So, naturally, I decided to give it a try. The result was promising enough for us to start migrating our current APISIX deployments to OpenResty on NixOS.

In this post, I will show how to set up OpenResty as NixOS service along with a few rudimentary location examples.

We will test the NixOS configuration on a QEMU-based virtual machine. For this, I will use my short technical note on my blog, “Running NixOS Guests on QEMU”.

Configuration

Below is the configuration.nix file contents:

## file: ./configuration.nix
{ pkgs, ... }:

let
  thisVersion = "0.0.1";
in
{
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  networking.firewall.allowedTCPPorts = [ 22 80 ];

  users.users.root = {
    initialPassword = "hebele-hubele";
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJIQtEmoHu44pUDwX5GEw20JLmfZaI+xVXin74GI396z"
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILdd2ubdTn5LPsN0zaxylrpkQTW+1Vr/uWQaEQXoGkd3"
    ];
  };

  services.openssh.enable = true;

  services.nginx = {
    enable = true;
    package = pkgs.openresty;

    virtualHosts."localhost" = {
      default = true;

      locations."/" = {
        return = "200 'Hello World!'";
      };

      locations."/version" = {
        extraConfig = ''
          default_type application/json;

          content_by_lua_block {
            ngx.say("{\"version\":\"${thisVersion}\"}")
          }
        '';
      };

      locations."/proxy" = {
        proxyPass = "http://httpbin.org/";
      };

      locations."/proxy/rewrite" = {
        proxyPass = "http://httpbin.org/";
        extraConfig = ''
          header_filter_by_lua_block {
            if ngx.req.get_method() == "POST" and ngx.status == 200 then
              ngx.status = 201
            end
          }
        '';
      };
    };
  };

  system.stateVersion = "24.05";
}

Explanation

This configuration is pretty straightforward if you are familiar with NixOS and NixOS Nginx service options.

A few highlights:

Instead of using the default Nginx package, we set services.nginx.package to pkgs.openresty. This will install OpenResty instead of Nginx.
The / location is pretty much Nginx.
The /proxy location is proxying requests to httpbin.org, again a typical Nginx configuration option.
The /version location is using Lua to return a JSON response with the version number.
The /proxy/rewrite location is proxying requests to httpbin.org and changing the status code to 201 if the request method is POST.

The last one is when OpenResty shines. You can use Lua to manipulate the request and response.

Note that the last example is actaully a production fix for us. We have a legacy service that returns 200 for all successful requests. But, the more recent versions of the service client requires an 201 response for successful POST requests.

Create and Launch VM

Let’s build the virtual machine with the following command:

nix-build '<nixpkgs/nixos>' -A vm -I nixpkgs=channel:nixos-24.05 -I nixos-config=./configuration.nix

The built artifact can be found in the result symlink.

Then, launch the VM with run-nixos-vm script:

QEMU_KERNEL_PARAMS="console=ttyS0" QEMU_NET_OPTS="hostfwd=tcp:127.0.0.1:2222-:22,hostfwd=tcp:127.0.0.1:24680-:80" ./result/bin/run-nixos-vm -nographic; reset

… and test the configuration:

$ curl -D - http://localhost:24680
HTTP/1.1 200 OK
Server: openresty
Date: Thu, 01 Aug 2024 03:23:51 GMT
Content-Type: application/octet-stream
Content-Length: 12
Connection: keep-alive

Hello World!

Do we have a version?

$ curl -D - http://localhost:24680/version
HTTP/1.1 200 OK
Server: openresty
Date: Thu, 01 Aug 2024 03:24:33 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive

{"version":"0.0.1"}

Let’s proxy some requests:

$ curl -D - http://localhost:24680/proxy/anything/hello/world
HTTP/1.1 200 OK
Server: openresty
Date: Thu, 01 Aug 2024 03:25:29 GMT
Content-Type: application/json
Content-Length: 355
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-66ab002a-0ccb8633321d4b9563c6c599"
  },
  "json": null,
  "method": "GET",
  "origin": "123.123.123.123",
  "url": "http://httpbin.org/anything/hello/world"
}

Let’s rewrite the response status upon successful POST requests:

$ curl -D - -X POST http://localhost:24680/proxy/rewrite/anything
HTTP/1.1 201 Created
Server: openresty
Date: Thu, 01 Aug 2024 03:26:17 GMT
Content-Type: application/json
Content-Length: 344
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-66ab0059-53f1b550657f07c10284f9e6"
  },
  "json": null,
  "method": "POST",
  "origin": "123.123.123.123",
  "url": "http://httpbin.org/anything"
}

Note that the POST request to httpbin.org would return 200 status code:

curl -D - -X POST http://httpbin.org/anything
HTTP/1.1 200 OK
Date: Thu, 01 Aug 2024 03:27:18 GMT
Content-Type: application/json
Content-Length: 344
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-66ab0096-7a3aa51e62f480b749d819f9"
  },
  "json": null,
  "method": "POST",
  "origin": "123.123.123.123",
  "url": "http://httpbin.org/anything"
}

Final Remarks

I acknowledge the need for complex and expensive API gateway solutions for large and busy services. But, for small and medium-sized services, a simple API gateway can be sufficient.

OpenResty is a powerful tool that can be used for many purposes, potentially growing into a custom, in-house API gateway solution. Deploying OpenResty on NixOS gives us the ability to manage the configuration as code. This is an advantage for testing and reproducibility, and brings us one step closer to using OpenResty as an API gateway with endless possibilities.

Therefore, OpenResty on NixOS can be a good choice for both small and large service setups and teams. I like the idea that I invest into OpenResty and NixOS instead of a proprietary or particular solution.

But there can be some sharp edges in relation to NixOS. For example, I wanted to demonstrate how to use OpenResty for securing APIs with OpenID. Currently, both NixOS 24.05 and NixOS Unstable channels have broken lua-resty-openidc dependencies. It may not be a big deal for seasoned NixOS users, but it is a showstopper for newcomers and too much for a simple blog post.