Managing HTTP Headers in Pipy

When applications are running behind using some sort of proxy, there is often a need to make the applications aware that they are running behind a proxy, and often there is a need to manage request and response headers in the proxy. Pipy is an open source network stream processor and it handles the lower-level details and provides an event driven interface to developers. Pipy decodes incoming network streams into streams of events and makes them available to developers. Pipy's versatile nature allows it to be used in use cases like HTTP forward proxy, reverse proxy, socks proxy. In this article we will look at how to manage HTTP headers in Pipy and see how Pipy makes it easy to manage, modify, and add HTTP headers.

HTTP headers are the core part of HTTP requests and responses, as they allow client and server to pass additional information like information about the requested context, so that server can serve tailored responses. Response headers contain information; that doesn't relate to the content of the message, but provides detailed context of the response. An HTTP header is a key value pair separated by a colon (:) , where key is case-insensitive and white spaces between colon (:) and value are normally ignored.

Sample HTTP message showing a few request headers after a GET request

GET / HTTP/1.1
Host: flomesh.io
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:98.0) Gecko/20100101 Firefox/98.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1

Sample HTTP message showing a response and representation headers after a GET request

200 OK
Access-Control-Allow-Origin: *
Connection: Keep-Alive
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Server: pipy
etag: 1646987690802
transfer-encoding: chunked

For readers interested in learning more about Request/Response Header fields can refer to Section 5 & 7 of RFC7231 - HTTP Semantics and Contents

Pipy is a low level network stream processor where it decodes incoming discrete network bytes and makes them accesible via events. Refer to Pipy Concepts document for more details. Pipy Message is a container for a series of events that compose a whole message in an event stream. And via this container we can get access to request or response details like header, body etc. Pipy comes with built-in support for multiple protocols like HTTP, Dubbo, MQTT, and decoded head contents are stored in composite structure Message.head. Refer to Head object for more details.

With the above context and knowledge in hand, we can now wear our developer hats and get our hands dirty with some code.

Dumping request head

Let's write a simple Pipy script which will log request header details to console and return back HTTP request header as JSON object.

pipy()

.listen(8080)
  .demuxHTTP('req')

.pipeline('req')
  .handleMessageStart(
    msg => (
      console.log('Path:', msg.head.path, 'Headers:', msg.head.headers)
    )
  )
  .replaceMessage(
    msg => new Message(
    {
      headers: {
        'content-type': 'application/json'
      }
    },
      JSON.encode(msg.head))
  )

Testing above script via curl will yield output like:

$ curl -s http://localhost:8080 | jq .
{
  "protocol": "HTTP/1.1",
  "headers": {
    "host": "localhost:8080",
    "user-agent": "curl/7.77.0",
    "accept": "*/*"
  },
  "method": "GET",
  "path": "/"
}

And if you access http://localhost:8080 via a Firefox browser, you will see response like:

{

    "protocol": "HTTP/1.1",
    "headers": {
        "host": "localhost:8080",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:98.0) Gecko/20100101 Firefox/98.0",
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "accept-language": "en-US,en;q=0.5",
        "accept-encoding": "gzip, deflate",
        "connection": "keep-alive",
        "upgrade-insecure-requests": "1",
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "none",
        "sec-fetch-user": "?1"
    },
    "method": "GET",
    "path": "/"

}

With just a few lines of code we have made a simple network service which can be used as a debugging tool to aid us echo back all of the HTTP request header details our http client is making.

Adding/Updating HTTP Request/Response headers

When applications are running behind a proxy, it's sometimes required by applications that they get the details of the actual client who initiated the request. Pipy doesn't mutate any of the contents by default, but it provides an interface via which programmers can manage these details. So let's create a simple script which will add x-forwarded-for header to an incoming request, before passing it to the upstream server. And add two custom headers to response received from upstream server.

x-forwarded-for (XFF) is a standard proxy header which indicates the IP addresses that a request has flowed through on its way from the client to the server. A compliant proxy will append the IP address of the nearest client to the XFF list before proxying the request. Some examples of XFF are:

• x-forwarded-for: 50.0.0.1 (single client)
• x-forwarded-for: 50.0.0.1, 40.0.0.1 (external proxy hop)
• x-forwarded-for: 50.0.0.1, 10.0.0.1 (internal proxy hop)

Let's write a proxy with a few upstream endpoints, and we will add x-forwarded-for header to requests before sending them to upstream and we will add two custom headers to response received.

pipy({
  _router: new algo.URLRouter({
    '/hi/*': 'localhost:8080',
    '/echo': 'localhost:8081',
    '/ip/*': 'localhost:8082',
  }),

  _target: '',
})

.listen(8000)
  .demuxHTTP('request')

.pipeline('request')
  .handleMessageStart(
    msg => (
      _target = _router.find(
        msg.head.headers.host,
        msg.head.path,
      ),
      _target && (
        msg?.head?.headers?.['x-forwarded-for'] ? (
          msg.head.headers['x-forwarded-for'] = `${msg.head.headers['x-forwarded-for']}, ${__inbound.remoteAddress}`
        ) : (
          msg.head.headers['x-forwarded-for'] = __inbound.remoteAddress
        )
      )
    )
  )
  .link(
    'forward', () => Boolean(_target),
    '404'
  )

.pipeline('forward')
  .muxHTTP(
    'connection',
    () => _target
  )  
  .handleMessage(
    msg => (
      msg.head.headers["X-My-Header1"] = 'My header 1',
      msg.head.headers["X-My-Header2"] = 'My header 2'
    )
  )

.pipeline('connection')
  .connect(
    () => _target
  )

.pipeline('404')
  .replaceMessage(
    new Message({ status: 404 }, 'No route')
  )

Run this script in one terminal. Open another terminal and run our dump headers script. Now in new terminal execute a curl request:

$ curl -v http://localhost:8000/hi
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /hi HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.77.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< X-My-Header1: My header 1
< X-My-Header2: My header 2
< content-length: 206
< connection: keep-alive
<
* Connection #0 to host localhost left intact
{"protocol":"HTTP/1.1","headers":{"host":"localhost:8000","user-agent":"curl/7.77.0","accept":"*/*","x-forwarded-for":"127.0.0.1","content-length":"0","connection":"keep-alive"},"method":"GET","path":"/hi"}

We can validate from the output that request now contains x-forwarded-for header and response contains our added headers named X-My-Header1 and X-My-Header2.

As we have seen Pipy makes it quite trivial to work with HTTP headers and we can use this technique to log all incoming requests/responses, map any custom headers which are specific to our application needs with very little script.

Conclusion

Pipy is an open-source, extremely fast, and lightweight network traffic processor which can be used in a variety of use cases ranging from edge routers, load balancing & proxying (forward/reverse), API gateways, Static HTTP Servers, Service mesh sidecars, and many other applications. Pipy is in active development and maintained by full-time committers and contributors; though still an early version, it has been battle-tested and in production use by several commercial clients.

Step-by-step tutorials and documentation can be found on Pipy website or accessed via Pipy admin console web UI. The community is welcome to contribute to Pipy development, give it a try for their particular use-case, and provide their feedback and insights.