Serving compressed static assets with HTTP in Go 1.16

The upcoming Go 1.16 release introduces a new way to manage static assets with embed package and //go:embed compiler directives.

Before there were multiple 3rd party tools that achieved embedding by explicitly generating .go files with assets data organized as strings or byte slices.

New embed package improves dev experience by eliminating the need for an explicit step to convert assets to Go code. Embedding is performed transparently during go build.

Another advantage is that embedded data is placed into a read-only TEXT section of a binary and it does not occupy process residential memory unless necessary.

To use embed package for static assets you can create a .go file in directory with asset files:

package static

import "embed"

//go:embed *
var Assets embed.FS

and then use static.Assets in your application.

You can create http.Handler to serve assets with

http.FileServer(http.FS(static.Assets))

and mount it into your HTTP server.

This would work to serve your .js, .css, .png and other files out of the box with just one limitation: it won't help you with Content-Encoding of those assets.

Many web resources (html, css, js to name a few types) allow high compression ratio due to their textual nature. Serving those resources compressed can make a big difference on web application latency and bandwidth consumption.

For dynamic data compression is usually applied dynamically, your web application can have an HTTP middleware to compress responses or it can be done externally with another web server (for example nginx). Dynamic compression takes extra CPU and memory, but in case of relatively small and dynamic payloads that cost is usually justified by improved overall performance.

On the other hand static assets don't change during application life cycle and compressing them over and over again for each request is a waste of resource. A better solution is to compress them once and serve compressed copies to many requests.

Nginx has a ngx_http_gzip_static_module module to serve pre-compressed data using .gz file extensions to map compressed resources to requests.

I thought this approach can work well for Go embedded files too and implemented github.com/vearutop/statigz: a file server that can serve compressed version of an asset if it is available in embedded file system.

You need to use

statigz.FileServer(static.Assets)

instead of

http.FileServer(http.FS(static.Assets))

to enable statically compressed assets.

How it works?

Assume file server receives a request for script.js.

First, it checks if request accepts compressed encoding of response. If yes, it checks if script.js.gz (or script.js.br for brotli) is available in embedded file system and serves contents of script.js.gz.

If request does not accept available encodings (which should be unlikely in case of browsers), file server checks if script.js is available in file system and serves its content.

If script.js is not available, it checks for script.js.gz and decodes its content into response.

If neither script.js nor script.js.gz|br are available file server would respond with status 404 Not found.

So, you can embed only compressed resources and serve them to clients that accept compression and to those that don't.

Files are served with ETag that is a hash of their contents to allow cache on client side.

Brotli is a more efficient than gzip compression format, but it has limitations (see this and this) outside of HTTPS. At the same time its support adds ~260KB to binary size, so it is not enabled by default but available as an option.

// Add import "github.com/vearutop/statigz/brotli".
statigz.FileServer(static.Assets, brotli.AddEncoding)

Recommended way of embedding assets is to compress them before the build, so that binary includes *.gz or *.br files. This can be inconvenient in some cases, there is EncodeOnInit option to compress assets in runtime when creating file server.

statigz.FileServer(static.Assets, brotli.AddEncoding, statigz.EncodeOnInit)

Once compressed, assets will be served directly without additional dynamic compression.

Files with extensions .gz, .br, .gif, .jpg, .png, .webp are excluded from runtime encoding by default.

NOTE: Compressing assets in runtime can degrade startup performance and increase memory usage to prepare and store compressed data.