Deploying Static Sites with statit

In my earlier post You Don’t Want Atomic Deploys I explained why atomic deployments are a lie because the web is a fundamentally distributed system. You should read that first as this article uses the terminology explained there.

In that post I only gave a high-level overview of how to solve the problem. I have since created my own static site deployment tool that is now powering most of my sites (including this one). I’m going to give a quick overview of the tool and how it solves the problems.

Note: This tool was made by me and is mostly focused on deploying my sites following my conventions. However I did attempt to put most of my conventions behind flags to make it customizable. That being said the user-base is still very small so it may be missing customization options that would be needed to make it work for your site. For example S3 is the only target currently supported. Patches are definitely welcome for most customization options, please open a feature request to discuss.

I was previously deploying my sites to IPFS. I liked this because people could easily archive them and access them over IPFS even if my host was down. In order to make the site available over HTTP at my domain I was using Cloudflare’s CNAME gateway. This was really cool, you could just CNAME kevincox.ca to their domain and they would automatically serve /ipns/kevincox.ca. This worked great until one day they broke it with no explanation, so I needed to find another solution.

The solution I ended up switching to was uploading to S3 and using CloudFront as a CDN. This provides good performance and cheap pay-as-you-go pricing. I tend to spend just a couple of cents a month on S3 requests. It is quite possible that if my site gets more popular the costs may actually go down as I will be more likely to get more cache hits in the CDN layer.

There are many tools for uploading files to S3 but surprisingly few that support setting headers in a flexible way and nearly none that support cleanup of old files in a way that suits websites. So I made my own, statit. It is now in use by a big handful of my own sites, including this one!

Usage

Basic usage is quite simple:

statit --bucket=my-bucket deploy www/ --dry-run

Most of the defaults should be suitable for most sites. The most important thing is to configure your assets and entries. This is done with the --assets flag. Simply declare the paths that should be considered assets and everything else will be considered an entry. The default is a/** (everything inside the a/ directory). But if you keep your assets in different locations you can pass flags such as --assets 'scripts/**' --assets 'styles/**'.

statit will automatically guess an appropriate content-type based on the file extension, however if this is not suitable you can override it for specific files with the --content-type option. For example --content-type 'pgp.key' 'application/pgp-keys' can be used to see the proper headers for PGP keys. The first argument is a glob pattern just like the --assets flag.

In --dry-run mode statit will print a list of the modifications it would make. If it looks right remove the --dry-run to actually deploy the site.

For a complete list of options see statit’s README and statit --help

Method of Operation

What statit does is fairly simple, consisting of a few main steps.

Read Existing State

The first step is reading the existing state. This is important for skipping unnecessary operations (like already up-to-date files) and cleanup.

Process New Version

Next statit scans the input directory and collects information about the new version.

1. List files.
2. Record file contents.
3. Resolve file metadata such as headers and asset/entry status.

Step 3 is customizable through various command-line flags. For example the --cache flag can be used to adjust the caching settings for particular files.

At the end of this step we have a complete image of what the live site should look like when the deploy is complete.

Upload Assets

This is where the deployment really starts. The assets are uploaded to the bucket. New files will be publicly accessible but should be unreferenced at this point, so visitors will not yet experience any change.

Upload Entries

This is where the new site is made live! The entries are uploaded. At this point (caching aside) new visitors will see the new content. However, removed pages will still resolve to the old version.

Write Manifest

statit writes manifests to store deployment-related metadata. This allows for proper cleanup of old versions. Currently, the manifest essentially consists of a list of known files.

Cleanup

In one step statit performs two types of cleanup.

1. Remove deleted entries.
2. Remove obsolete assets.

Part 1 is important to ensure that removed pages are made immediately unavailable. This should result in your 404 page appearing where content was deleted.

Part 2 is a cleanup process that isn’t critical to site function but reduces storage usage. statit will remove any assets that have been unused for at least 7 days. This period should be the longest timeframe for which an out-of-date asset can be requested. For many sites this is your longest entry cache plus the max load time. So if you cache pages for 1h than 2h should be very safe. However for sites that lazily load content (such as single-page applications with lazy code loading) users can potentially be using an older version of your site for much longer. 7 days is chosen for now as it should be conservative enough for most use cases, it will be made configurable in the future.