Intercepting RESTful Responses with Middleware

When we look at building HTTP servers, we typically jump to a golden example of a client and a server passing back and forth data for creating, updating, reading, and deleting resources (i.e. RESTful services). This could be text files, binary data (like .mov, and .mp4 files), and even pre-formatted data (JSON, XML, etc.) for future use by another program. Multiple clients may communicate with the one server at a time, and the server always provides consistent data without any need for intervention.

The golden example is contrived to simplify how we talk about the communication between clients and the server. More complex solutions, which tend to appear more frequently than most would think, are left out of the conversation. My particular focus in this article, is on the usage of third-party APIs in-house, and how to leverage the functionality that they provide, while still maintaining control over the responses clients receive.

Assuming that an in-house server is not simply a middleman for the communication between the client and a third-party API, there should be some sort of quality control over the responses clients receive. The standard Go net/http package provides some functionality to intercept incoming requests before they reach their intended target (Handler). However, there is no pre-baked method of intercepting outgoing responses before they reach the client.

Table of Contents

• Receiving Requests from Browser Clients
• Receiving Requests from Programmatic Clients
• Using third-party APIs
• Intercepting Requests
• Intercepting Responses
• Credits

Receiving Requests from Browser Clients

To save time, let us talk about the most used request method, “GET”. A GET request, does what is says on the tin, it retrieves some data from storage (e.g. a post from DEV, a profile from GitHub, a feed from Twitter, etc.). When you enter a URL into your browser, like https://foresthoffman.com, your browser attempts to GET it by default. That is how we read things online.

A server’s job is to fulfill client requests. So, when a browser makes a GET request for https://foresthoffman.com it generates a request that gets routed to that server and waits for a response. In my case, my website responds with an HTML file, which requires a few other external files, which in turn causes the browser to make more GET requests, until it has completely loaded the website.

For a server that receives GET requests programmatically, i.e. code talking to code, things get more interesting.

Receiving Requests from Programmatic Clients

Assume we have a server expecting GET requests to a static endpoint, programmatic clients might pull from this endpoint and parse our response however they want. Here is an example of a localhost server hosting an /example endpoint:

package main

import "net/http"

func exampleHandler() http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      w.WriteHeader(http.StatusOK)
      _, err := w.Write([]byte("hey"))
      if err != nil {
         panic(err)
      }
   })
}

func main() {
   http.Handle("/example", exampleHandler())
   err := http.ListenAndServe(":9001", nil)
   if err != nil {
      panic(err)
   }
}

A client may then GET this endpoint by curling:

curl -X "GET" http://localhost:9001/example

The response body should contain: hey. This by itself does not seem complex, but imagine if the response was JSON formatted according to this server response structure:

type Response struct {
   Message string `json:"message"`
}

The handler would have to change accordingly:

func exampleHandler() http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      respBytes, err := json.Marshal(Response{Message: "hey"})
      if err != nil {
         panic(err)
      }
      w.WriteHeader(http.StatusOK)
      _, err = w.Write(respBytes)
      if err != nil {
         panic(err)
      }
   })
}

The curl output should then be: {"message":"hey"}. Assuming the client can parse this JSON, a mirrored object structure could be created on the client-side. Cool!

Using third-party APIs

For some tasks, it makes more sense to rely on functionality supported by third-party APIs, e.g. Google Cloud, MongoDB, Tus.io, etc. This of course adds its own complexity to our server, since we do not control the responses provided by these third-party APIs. Let us add a “third-party” handler now.

func thirdPartyHandler() http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      respBytes, err := json.Marshal(struct{
         Timestamp string `json:"timestamp"`
      }{
         Timestamp: fmt.Sprint(time.Now().UnixNano()),
      })
      if err != nil {
         panic(err)
      }
      w.WriteHeader(http.StatusOK)
      _, err = w.Write(respBytes)
      if err != nil {
         panic(err)
      }
   })
}

This “third-party” handler takes a request, gets the current date and formats it as a string of the nanoseconds since January 1st 1970. While this timestamp is handy, it does not conform to the structure we have implemented for our own handler.

We can see the difference in the response structures after giving the third-party handler its own endpoint in main:

http.Handle("/third", thirdPartyHandler())

Now, when a client makes a GET request for the new endpoint, they will see: {"timestamp":"1610378453927873100"}. Of course, the timestamp is based on the time at which the request was made, so the timestamp will change.

We can see here that the handy third-party response does what we need, but could confuse clients, since the {"message":"hey"} and {"timestamp":"1610378453927873100"} structures do not line up. This is where HTTP middleware come into play.

Intercepting Requests

A “middleware”, at its core is a HTTP handler. It has an input, the request, and an optional output, the response. The output is optional, because middlewares are not intended to be the end target of a request. They are middlemen, so they normally modify the request and pass it along to the next middleware or handler.

They look like this:

func exampleMiddleware(h http.Handler) http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      h.ServeHTTP(w, r)
   })
}

This middleware returns a wrapping handler which calls any handler passed into it, so a new endpoint using this middleware could be created like so:

http.Handle("/fourth", exampleMiddleware(thirdPartyHandler()))

Curling this new endpoint should reveal the response we would expect from the /third endpoint:
{"timestamp":"1610381096848493300"}. So, here we have our example of intercepting an HTTP request. We did not modify the request at all, but we did pass it along to a different handler.

Let us explore modifying a request. This is very powerful, because it allows us to restrict access to certain handlers if need be.

Here is a middleware that expects to find a secret key in the URL parameters (e.g. website.com?key=corgis-are-cute):

func restrictMiddleware(h http.Handler) http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      if r.URL.Query().Get("super-secret-key") == "" {
         respBytes, err := json.Marshal(Response{Message: "key must not be empty"})
         if err != nil {
            panic(err)
         }
         w.WriteHeader(http.StatusBadRequest)
         _, err = w.Write(respBytes)
         if err != nil {
            panic(err)
         }
         return
      }
      h.ServeHTTP(w, r)
   })
}

This middleware’s new endpoint will wrap the first handler we created:

http.Handle("/restrict", restrictMiddleware(exampleHandler()))

Now, when a client curls this endpoint without a secret key, they will be met with: {"message":"key must not be empty"}.

To provide a key, clients must format their curl request like so:

curl -X "GET" http://localhost:9001/restrict?super-secret-key=secret

Then, clients will be able to see the expected response: {"message":"hey"}!

With this example, we can see that a server may restrict access to certain endpoints by preventing the target handler from ever handling the request.

Intercepting Responses

We have already seen how useful middlewares can be when processing requests, but when it comes to unexpectedly abrupt responses from third-party handlers, they cannot help us. At least, not by default.

In order to intercept a response from a chunk of code that we do not control, we must capture the response in memory while maintaining the response’s integrity. We must ensure that the response is not sent to the client before we are done with it. Otherwise, any modifications we may make will be useless.

By default, the http.ResponseWriter interface does not support reading, as it only features methods for writing. We can achieve the functionality we desire by creating a custom struct that implements the http.ResponseWriter interface. However, we do not want to completely throw away the features that the http.ResponseWriter provides, so we’ll have to include the response writer as a field in our custom struct like so:

type responseSkimmer struct {
   http.ResponseWriter
   body bytes.Buffer
   header http.Header
   status int
}

func (rs *responseSkimmer) Header() http.Header {
   return rs.header
}

func (rs *responseSkimmer) Write(b []byte) (int, error) {
   rs.body.Reset()
   return rs.body.Write(b)
}

func (rs *responseSkimmer) WriteHeader(code int) {
   rs.status = code
}

Next, we can implement this “skimmer” in a middleware that will receive the third-party handler as a parameter. This will showcase how we can capture the response from the third-party handler, manipulate it, and respond with the new body as though nothing happened.

func responseMiddleware(h http.Handler) http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      // Intercept response.
      skimmer := &responseSkimmer{
         ResponseWriter: w,
         body: bytes.Buffer{},
         header: http.Header{},
         status: 0,
      }
      h.ServeHTTP(skimmer, r)

      // Read.
      if len(skimmer.body.Bytes()) == 0 {
         skimmer.ResponseWriter.WriteHeader(http.StatusNoContent)
         _, err := skimmer.ResponseWriter.Write([]byte(""))
         if err != nil {
            panic(err)
         }
         return
      }

      // Transform.
      respBytes, err := json.Marshal(Response{Message: string(skimmer.body.Bytes())})
      if err != nil {
         panic(err)
      }

      // Write.
      skimmer.ResponseWriter.WriteHeader(skimmer.status)
      _, err = skimmer.ResponseWriter.Write(respBytes)
      if err != nil {
         panic(err)
      }
      return
   })
}

Lastly, we need to add an endpoint and pass along the new middleware and the existing handler.

http.Handle("/intercept", responseMiddleware(thirdPartyHandler()))

Curling the new endpoint we should see the same response we saw from the third-party handler endpoint, /third, but the difference is that it has been wrapped by the response structure we created for the /example endpoint: {"message":"{\"timestamp\":\"1610599055718152700\"}"}.

And there we have it. A middleware that works for outgoing responses, and not just incoming requests! Woo!

If you made it this far, thank you for reading! Hopefully, you found a useful nugget of knowledge here. Cheers!

Credits

Cover image by Florian Steciuk on Unsplash! :D