A friend of mine shared this link with me the other day, about CVE-2022-43412. The issue disclosed in that CVE is that Jenkins Generic Webhook Trigger Plugin was using a non-constant time comparison function when checking whether the provided and expected webhook token are equal. In addition to the aforementioned CVE, there's also CVE-2022-43411 which is the same issue, but with the Jenkins Gitlab plugin.

In this blog post we will explain how the attack works, how to avoid similar problems, and what we do at Svix to protect our customers.

What is the issue?

The issue described in this CVE uses a timing-based side-channel to execute an oracle attack against the hash verification. Or in more plain words: this attack uses the fact that the comparison may take different times based on the content, to construct a valid signature even without knowing the key.

I know this may sound a bit like magic, because it kinda is. Though let's demystify it by using a more simple example, and go from there.

What's a timing based oracle attack?

Let's assume it's not a webhook, but rather a website, and the website is password protect. The developer who built this website is new to security, so they implemented the security as follows:

const password = 'Secret123'
if (user_provided_password != password) {
  return 'Wrong password!'
}

Now let's take a deeper look at the comparison in the condition. Because we used a high level function we can just use != between strings, but what goes on behind the scenes is that it actually compares them character by character (well, it's actually more complex than that, but let's assume that for simplicity).

So a naive implementation may look like this:

// This function assumes the strings are the same length for simplicity
function is_equal(a, b) {
    for (let i = 0; i < a.length; i++) {
        if (a[i] != b[i]) {
            // Can stop iteration, strings are not equal!
            return false;
        }
    }

    return true;
}

For the sake of example, let's assume our computer is very slow, and doing a[i] != b[i] takes a second each time.

Looking at the above then, we can see that if the user passes Bad123123 as the password, it'll try to compare the first characters (B vs S), and because they are unequal, the comparison will stop early so the whole function will take 1 second. Though if we pass Secret123 the whole function will take 9 seconds, as it compared all of the characters.

This means that we found an observable change of behavior based on the password. Now let's use that to our advantage.

We know that this function is slower the more characters are correct. So we can just start feeding it with characters one after the other and measure the time it took. We try A and get 1 second, B and get one second, all the way to S, where it all of a sudden takes two seconds. This lets us find the first character.

Now we can do the same for the second character. We pass Sa which takes 2 seconds, and then Sb, which takes two seconds, all the way to Se which then takes 3 seconds, so our password starts with Se!

We can repeat this process until we found the full password.

What's the correct way to solve it then? Well to make comparison constant time!

// This function assumes the strings are the same length for simplicity
function is_equal_constant(a, b) {
    let ret = true;

    for (let i = 0; i < a.length; i++) {
        // Putting the && ret at the end, to avoid short-circuiting
        ret = (a[i] != b[i]) && ret;
    }

    return true;
}

With this function the comparison always takes the same time, which protects us from timing attacks!

Note: do not use the function above in production as it may not work as expected. Optimizing compilers may change the behavior, so it's important to use purpose-built comparison functions that are provided by your system libraries which already protect against that.

How does it apply to webhooks?

Webhooks don't really have a password, but they have signatures, which in a way are dynamic passwords. With webhooks you pass a specific payload, e.g:

{
  "foo": "bar"
}

and sign it with a secret key in a way that only someone with the key could have signed it. The signature (e.g. Vh082qUqkhY7WiBWktRKRbP6/c+EYoqtHi/dMULaTKc=) is then passed with the payload, and the receiver can verify the sender has the secret key, and thus that the webhook can be trusted.

An attacker may want to send a malicious webhook, but without the secret key, the signature will not match, and the webhook will be rejected.

The aforementioned timing attack can't help us with recovering the secret key used for the signature, but it can help us find a valid signature!

So similarly to how we managed to find the password, we can find a valid signature for the malicious payload, and then just use this valid signature as a proof we know the key and get the webhook validated. Even though we never actually learned the key!

How does Svix solve it?

Security is hard. The reason for that is that security issues are not always obvious. If you have a bug in your code, you will usually notice it while testing it. Though with security, you don't always know what to test for.

This issue, unfortunately can't be mitigated on the sender side (Svix), it has to be mitigated on the client side. So we can't blanket solve it on our end.

We have however created a set of open source webhook signature libraries, that make it easy both to sign and verify webhook signatures correctly. Svix customers can offer them to their customers to ensure they have a great developer experience and stay secure. Additionally, non Svix customers can also reuse them, both for signing, and verification to ensure that they offer their customers an easy and secure experience.

For more content like this, make sure to follow us on Twitter, Github or RSS for the latest updates for the Svix webhook service, or join the discussion on our community Slack.