Webhooks - Day 3 of the #25DaysOfServerless Challenge

This article is part of #25DaysOfServerless. New challenges will be published every day from Microsoft Cloud Advocates throughout the month of December. Find out more about how Microsoft Azure enables your Serverless functions.

Have an idea or a solution? Share your thoughts on Twitter!

APIs are what made it possible to build distributed architecture for the web. Think about it. You can right just the right amount of small lines of code and really do powerful things. Maybe you want to ask Stripe to collect payment for you, or you want to tell Auth0 to confirm that the user is your customer. Perhaps you want to notify your user via email, or schedule a calendar event for a sales follow up.

When last did you write code that did any of those yourself from the ground up? Probably, never. But yet, you have a well tested and durable API services. Services that have gone through the tests of time and are standing tall. All you need to do is 1..2..3, plug and play!

It goes both ways though — instead of asking Stripe to process payment or pushing data to Github, you want them to notify your endpoint when something happens that wasn't triggered by your website. This means you did not make a request for it. Hence you are not expecting any response.

Consider a user cancelling a paid subscription or that a commit has been made to a repo that your website pulls content from. You probably want to follow up with the user for the former, or maybe flush your cache to remove fresh content from Github.

The latter was our case when building the 25DaysOfServerless website.

Webhook Calls are the Opposite of API Calls

If your website or server makes a call to an API endpoint for data, that is an API call. In the case where a service provider calls your own custom API endpoint to send data, that's a webhook.

For 25DaysOfServerless website, we stored all the challenges on Github and fetched them using access tokens and Github API. To reduce workload for the frontend, we decided to process the markdown on the serverless API. But things got ugly, real quick.

Process means we had to parse markdown, upload images in the markdown to a CDN, download the images back from the CDN, before sending to the client. Basically, when you ask for a challenge on the website, we are making two additional requests plus markdown parsing. Each request took over 8 seconds to process.

The first thing that came to mind is to cache — we did, and we went for everyone's favorite, Redis.

Caching brought down the total response time for each challenge to below 2 seconds, but there was a price to pay.

async function processRequest(week, day) {
  const challengeFromRedis = await redisGet(client, pathToChallenge);

  if (challengeFromRedis) {
    console.log('challenge is cahced');
    return { content: challengeFromRedis };
  } else {
    console.log('challenge not in cache, yet');
    const response = await fetchChallenge(week, day);

    // Process markdown
    const decodedReadme = decodeContent(response.data.content);
    const markedContent = await parseMarkdown(decodedReadme, week, day);
    const content = Object.assign(response.data, { content: markedContent });

    // Add to redis cache
    await redisSet(client, pathToChallenge, content.content);

    // Return processed markdown
    return { content: content.content };
  }
}

As you can see, we are checking if the challenge is cached. If it is cached, we return the cached value. If it is not cached, we add the challenge to the cache and return the challenge.

I took a considerable role in developing this and felt so good about bringing that response time down until reality knocked on my door the next morning. When my colleagues added or updated challenges on Github, guess what happens?

The users kept seeing the cached value — rookie mistake, huh?

I sat in one corner thinking of all the magic I could pull of to keep the cache updated. I thought of midnight cron jobs that went to Github and checked if there was a new commit so it could flush the cache. Felt like a great workaround until our next stand up where I shared my problem and my manger, Simona dropped the best answer hot out of the oven on us.

Webhooks!

All we need to do is ask Github to send a post request to a URL we give it when a commit/push is made. That way, when we receive that request, we can run a serverless function that clears the cache.

Creating a Github Webhook

This was as easy as A B C.

Go to the settings of the repo you want to add a hook, make sure you have admin privileges, and just stick the hook URL in there (I'll tell you more about this URL in a minute)

Make sure to enable to push event and save the webhook. Edit a file commit, Github will call that URL for you with a post method and some data.

Let me show you how to create a URL for yourself using a serverless function.

Creating a Webhook URL

A webhook URL is a regular endpoint. The only thing unique about it is that it knows it could receive payload via a post request and that it can access that data from req.body.

• Create and deploy a serverless function with just VS Code.
• Create a function that the webhook can call.

Here's an example of a function I had:

const redis = require('redis');
const promisify = require('util').promisify;

module.exports = async function(context, req) {
  // Connect to your redis client
  const client = redis.createClient(6380, process.env['REDIS_CACHE_HOSTNAME'], {
    auth_pass: process.env['REDIS_CACHE_KEY'],
    tls: { servername: process.env['REDIS_CACHE_HOSTNAME'] }
  });

  const flushAsync = promisify(client.flushdb).bind(client);

  // Clear cache
  await flushAsync();

  context.res = {
    body: 'Cache cleared'
  };
};

Want to submit your solution to this challenge? Build a solution locally and then submit an issue. If your solution doesn't involve code you can record a short video and submit it as a link in the issue description. Make sure to tell us which challenge the solution is for. We're excited to see what you build! Do you have comments or questions? Add them to the comments area below.

Watch for surprises all during December as we celebrate 25 Days of Serverless. Stay tuned here on dev.to as we feature challenges and solutions! Sign up for a free account on Azure to get ready for the challenges!