Incremental Builds with Jamify

With Jamify, you can now fully leverage the new Gatsby Cloud incremental builds feature to build your sites even faster! We are proud to be the first to bring this exciting feature to your publication pipeline when sourced from a headless Ghost CMS.

Gatsby Cloud recently introduced a new Incremental Builds feature that makes publishing your site even faster! Today, I'm excited to announce best in class feature support for incremental builds in Jamify. We are proud to be the very first to bring Ghost CMS incremental build support to your publication pipeline!

This will allow you to make content changes and rebuild only the parts of your site that are affected by the change. With this feature, you can easily achieve sub-ten seconds build times following a content change. This makes a big difference especially for large sites, where the entire site had to be re-built for even the smallest data change.

One of the biggest obstacles towards a wide adoption of static site generators has always been build processing times. Incremental builds is a huge step forward towards removing a strong argument against static site builders. I am sure this technology leap will invigorate new momentum to Gastby and the Jamstack in general.

Content vs. Configuration Changes

Incremental Builds are currently only supported for content changes. If you publish a new post, update an existing post or add a new tag or author, and incremental build is triggered. However, if you make a configuration change such as installing a new plugin or changing a setting in gatsby-node.js, a full rebuild of your site is necessary.

On production sites, content changes are the frequent and most important ones, so publication authors will directly profit from reduced publication times!

Explore Incremental Builds

With Jamify, incremental builds are software enabled by default: zero configuration needed! All the prerequisites have been included in the Jamify starter, such as supporting the latest gatsby version as well as ensuring best in class CMS integration. Let's dive in and see incremental builds in action.

Jamify starter

To begin, you need to make a copy of the Jamify starter and save it into your personal Github account. You can either use a fork or make use of the template approach.

Open the Jamify starter project and press the Use this template button to make a copy.

You can choose any repository name, so I took incremental-builds. After pressing Create repository from template, you'll see a new repository with that name under your account.

It contains a self-contained software package for sourcing content from a headless Ghost CMS that transforms your content into a flaring fast React front-end that can be directly served from the edges of a Content Delivery Network (CDN).

The Jamify starter can be enhanced with plugins. Check out the list of available plugins for Jamify or simply write your own!

Gatsby Cloud

Now, you are ready to connect this repository to Gatsby Cloud. Go to the Gatsby Cloud dashboard and click on Create new site which walks you through the set up process:

This is a three step process. First choose I already have a Gatsby site, next, connect to your repository (e.g., github-user/incremental-builds) and leave the branch options as they are.

The final step is where you need to provide your Ghost Content API keys for both the Preview and the Build environments.

If you skip the last step, your project is still building and sourcing in content from a demo server. However, you need to be able to make changes to your content in order to test out the incremental build feature. Thus, you'll have a lot more fun, if you provide the keys of your own Ghost CMS here.

Initial Builds

After completing the last step, the build process is directly triggered, producing the standard demo site:

As you can see from the below screenshot, the total build process was less than a minute and took 55 seconds :

As this was the initial build that started from an empty cache, it's interesting to see how long it takes to rebuild the site. Use the Trigger build -> Build master branch option to trigger a rebuild:

In this case, the rebuild is _ 20% faster _ because previously processed images can be directly fetched from the cache.

Triggering incremental builds

It's important to understand that a rebuild is not an incremental build. An incremental build in Gatsby Cloud can only be triggered by webhooks!

Incremental Builds in Gatsby Cloud are available only on the professional plans. However, Gatsby offers a free 14-day trial period for new users on the professional plan. This should be sufficient for testing this stunning feature.

Gatsby Cloud automatically generates webhooks for every site. You just have to find them! Go to the Settings -> General -> Webhooks and copy the webhook under Preview Webhook to your clipboard:

You'll want to configure this webhook in your headless Ghost CMS, but for a first test you can directly trigger it from the command line by sending a POST request with curl:

$ curl -X POST https://webhook.gatsbyjs.com/hooks/data_source/publish/86992927-dffe-4481-ad6a-ebb0985d84d1

Use your own webhook and trigger an incremental build which is then labeled with an incremental badge:

Wow 🎉! A build in just three seconds. Take a smoothie and celebrate!

A sceptic would grumble: Nice build times, but nothing has changed on your site, no wonder it's so fast! Okay, sceptic. I hear you, let's do some real content edits...

Ghost CMS Content Edits

You need to define a trigger in your Ghost CMS, so content or site edits initiate a POST request to Gatsby Cloud webhook endpoint.

If you do not yet run a headless Ghost CMS, check out the tutorial that describes how to set up Ghost CMS on Hetzner Cloud.

In the Ghost Admin panel, go to Integrations -> Add custom integration and give it a name, e.g. Gatsby Cloud. Next add a new webhook:

give it a name, choose Site changed (rebuild) as an event and paste the previously copied Gatsby Cloud webhook URL into the target field.

If you inspect the list of events that Ghost provides, you may be tempted to choose more fine grained triggers, such as Post published, Post updated and others. Unfortunately, most of these triggers didn't work (tested with Ghost version 3.18.1), so the Site changed (rebuild) is currently the only useful option.

In-Page Content Edit

It's time to make your first content edit. Let's start simple and change the headline of the welcome post (left before, right after the edit):

To trigger the incremental build press the Update button on the top right corner:

This incremental build was ready in just six seconds :

On successive runs, I observed some variance in the 4 to 8 seconds range.

I experienced stability issues with incremental builds on the production branch. Sporadically, a subsequent incremental build would not change the site, despite showing a successful build status. I therefore switched to the preview branch, which always showed consistent results.

Unpublishing an article

A common use case is to take an article out of your site. Again, this change takes about five seconds to build:

Publishing a new article

Reversing the last step is equivalent to publishing a new article. As this is also a change with just on page, you can expect comparable build times, in this case 4 seconds :

Changing Settings

Finally, we are going to make a change to the site settings by changing the blog title, which takes just three seconds to rebuild!

I encourage you to play a little with different type of edits and especially test out common content changes, such as adding or removing tags and authors or changing a a feature or inline image.

I performed a wide range of different edits and was getting consistent incremental build times in the three to fifteen seconds range: this is truly amazing.

Conditional page builds

While I want to give full credit to Gatsby for pulling out this game changing incremental build feature, a lot of people I spoke with were rather disappointed when they heard that incremental builds are exclusively available on Gatsby Cloud. Here is quote from a Gatsby team member, in the context of describing incremental builds:

It's important to clarify that in order to achieve the results that we have, we need to own the CI/CD pipeline. This isn't something that can be packaged in our open-source repo.

— Josh Comeau

While I fully understand that Gatsby needs to protect their technological investments in Gatsby Cloud, I really don't buy into this argument. When you build locally or on your own server, you also own the pipeline including the cache and the full build history, so what is it exactly that cannot be packaged as open-source?

I really hope Gatsby does not loose credibility in the open source community because of these apparent marketing communications and that they will eventually provide more technical details into the incremental build process so that all parts that are portable can be integrated into the open source tool-chain.

Fortunately, small portions of the content-diffing methods that are part of incremental builds have already been provided as an experimental feature called conditional page builds.

Locally built conditional page builds

In the following I am going to share with you my latest test results based on this experimental feature. As before, I am going to build the Jamify starter demo site. You can easily repeat these steps on your own machine. First clone the starter:

$ git clone https://github.com/styxlab/gatsby-starter-try-ghost.git jamify-conditional-page-builds

and change into the newly created directory:

$ cd jamify-conditional-page-builds

Resolve all package dependencies by issuing the yarn. This may take a couple of seconds to complete but only needs to be called once.

[jamify-conditional-page-builds]$ yarn

Note that the conditional page build feature is only available for the build command, so it does not affect the development mode.

You enable this feature by setting the lengthy environment variable GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES to true. The build command is then called as follows:

GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages

The option --log-pages shows a summary at the end of the build log with details about the pages that were deleted, updated or created.

First experimental feature test

Now, it's time to build the demo site for the first time:

[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
...
Done in 38.91s.

This is the initial build time starting from an empty cache. You can see that all feature pictures are pulled in and subsequently processed with the transformer-sharp tools. Also notice the routes to all pages are listed at the end of the log. The build time of roughly 40 seconds is comparable with the measured times on Gatsby Cloud for exactly the same project site. You can preview this site by calling gatsby serve and visiting http://localhost:9000:

Let's conditionally rebuild this site without making any content changes:

[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
...
Done in 20.83s.

This is a factor 2 improvement compared to the initial build, but can that be attributed to the experimental feature?

In order to check that, let's repeat the previous two build commands, this time without conditional page builds. First, clear the cache:

[jamify-conditional-page-builds]$ yarn clean

Next, use the gatsby build command directly, without any additional parameters:

[jamify-conditional-page-builds]$ gatsby build
...
Done in 36.04s.

and a second time, with the cache populated:

[jamify-conditional-page-builds]$ gatsby build
...
Done in 21.91s.

As the benchmarks suggest, the factor two improvement between first and second is solely due to caching and cannot be attributed to the experimental build feature!

CMS page edits

While the conditional page build feature did not make any difference in the previous test, let's check if that feature makes a difference in a more practical use case.

Therefore, we are going to make the same page edit test that we previously made on Gatsby Cloud. To refresh initial test conditions with conditional page builds enabled, please call the following three commands:

[jamify-conditional-page-builds]$ yarn clean
[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages

In the headless Ghost CMS, you can now change the title of the welcome post:

Make sure to press the Ctrl+S or the Update button in the top right corner in order to publish the changes:

[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
...
Done in 21.83s.

Well... not so impressive. As you can see from the logs, all pages are rebuild instead of the expected one page.

I've made a couple of more tests and was able to see that under special conditions, only one page is recreated. For example, when adding a Ghost page with no relations to other pages (using different tags and authors), unpublishing it would only delete this one page.

Conclusions on conditional page builds

For the time being, the experimental page build feature does not improve the build process more significantly than what the standard Gatsby cache mechanism provides. Gatsby Cloud incremental builds are boosting performance by a factor of 5 on top of what the standard caching gives you, a level that is currently out of reach with the conditional page build feature.

Incremental builds on Netlify?

Shortly after incremental builds were announced on Gatsby Cloud, Netlify was quick to jump on the band wagon and publish a blog article on enabling Gatsby incremental build support on Netlify.

I was really thrilled to see a little competition on accelerating Gatsby build processing times and was eager to test it out. I'm going to repeat my tests here, so you can follow along and don't just need to take my word for it.

Set-up

To enable this feature on Netlify, you need to install a build plugin. If you inspect the netlify.toml file that is shipped with the Jamify Starter, this is already configured for you, so this plugin will be automatically installed during the build process.

# ./netlify.toml
[build]
  command = "gatsby build"
  publish = "demo/public"

[[plugins]]
  package = "netlify-plugin-gatsby-cache"

As always, I'm starting from scratch. First you need a copy of the Jamify starter template, and give it a unique name such as netlify-incremental:

Once the template is copied to your repo, clone it so you can make some changes locally:

$ git clone https://github.com/<your-github-user>/netlify-incremental.git
$ cd netlify-incremental

Due to a current bug described in an article entitled Incremental build time increases, we must switch from yarn to npm package management. You must therefore delete the yarn.lock in your starter project and use npm install to generate a package-lock.json instead:

[netlify-incremental]$ rm -rf yarn.lock
[netlify-incremental]$ npm install

Next, change the build command in the netlify.toml file:

# ./netlify.toml
[build]
  command = "GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages"
  publish = "demo/public"

[[plugins]]
  package = "netlify-plugin-gatsby-cache"

Stage and check-in the changes with git:

[netlify-incremental]$ git commit -a -m "switch to npm"
[netlify-incremental]$ git push

Deploy to Netlify

First make sure to have their cli tool installed:

$ sudo npm -g install netlify-cli

You can now deploy the site from the command line:

[netlify-incremental]$ netlify init

Choose Create & configure a new site, your team, optionally a site name, Authorize with GitHub through app.netlify.com and use the defaults for build command and deploy directory. The latter two settings are overwritten by your netlify.toml file.

Netlify build times

Switch to your new site on Netlify with

[netlify-incremental]$ netlify open

As you can see, the initial build took approximately three minutes. This time includes bootstrapping the build environment, so a second build should be faster. Let's test that with issuing a webhook trigger:

curl -X POST https://api.netlify.com/build_hooks/5ee4c313b2863fd798b7411a

A re-build is utilizing the cache and is therefore much faster, it only takes less than a minute to complete. I made a couple of more test with netlify-plugin-gatsby-cache and with the experimental page build setting GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true enabled. Here are the results:

When compared to builds with the standard gatsby develop command (netlify-plugin-gatsby-cache enabled!) the picture is almost the same:

Conclusions on Netlify builds

What Netlify praises as incremental builds should really be called cashed builds — it has nothing to to with the incremental build feature on Gatsby Cloud where you can achieve sub-ten seconds build times.

Summary

With Jamify, you can try the Gatsby Cloud Incremental Builds feature today and achieve stunning sub-ten seconds build times on simple CMS page edits. As content changes are the most frequent changes on a live site, this is really great news for publishers.

While the preview branch showed extremely stable results, subsequent incremental builds on the production branch sometimes silently refused to apply expected content changes. I hope, these annoyances will vanish once incremental builds becomes more mature.

Conditional page builds turned out to be extremely experimental and I did not see any substantial build time reductions: neither when built locally nor when built on Netlify. On the positive side, caching always reduces build times for subsequent builds. That's not a surprise, but something to keep in mind when developing plugins for Gatsby.

With Gatsby Cloud being the first to show what's possible when it comes to accelerating build times, I am confident that other vendors will follow soon. We are just at the beginning of a new Jamstack revolution!

Do you want early access to Blogody, the brand new blogging platform that I am creating? Just sign-up on the new Blogody landing page and be among the first to get notified!

This post was originally published at jamify.org on June 06, 2020.