Zero Downtime Secret Rotation for Webhooks

Before we talk about zero downtime secret rotations for webhooks, we need to explain what are webhook secrets and what they are used for.

Because of the way webhooks work, attackers can impersonate services by simply sending a fake webhook to an endpoint. Think about it: it's just an HTTP POST from an unknown source. In order to prevent it, Svix, and many popular webhook providers sign every webhook and its metadata with a unique secret key. This is called a webhook secret.

You can read more on how Svix signs webhooks in the verifying webhooks section of the docs, and we also previously wrote a post about securely verifying signature hashes in case you find this interesting.

Secret rotation

The secret, as the name implies, needs to be secret in order to work effectively. If the secret is ever compromised, or suspected to be compromised, it needs to be rotated (replaced by a new one) in order to ensure the security of the signature scheme.

Rotating the webhook secret is a simple operation, usually it can be done by just a click of a button. The problem is that the moment you rotate the key, webhooks will be signed with the new key, which means that your endpoints verifying the signature using the old key will fail to verify the webhooks. This is because the key used to sign and the key used to verify are now different.

This is easy to solve, you just need to update the verifying endpoint to use the new key and everything will work again. The problem is that webhooks will fail from when you rotate, until the secret is updated.

Zero downtime secret rotation

The solution to this problem is simple: sign the webhooks with both the old key, and the new key for a set period of time to give your webhook consumers enough time to update their endpoints to use the new key.

You may ask yourself: isn't this a security issue? The key is compromised, an evil attacker may have access to it. While that's true, and why you need to rotate the secret in the first place, signing the payload with the compromised secret is not actually a security issue. This is because it doesn't give an attacker any advantage. The compromise of a key can let an attacker create fake payloads and sign them to trick the verifiers, but having a legitimate payload signed doesn't cause an issue.

OK, so how do we do it? As we said above the solution is just to support signing the same message with multiple keys. This can be done in many different ways, but the easiest way (and what Svix does), is to pass multiple space-delimited signatures in the webhook-signature header like so:

v1,g0hM9SsE+OTPJTGt/tmIKtSyZlE3uFJELVlNIOLJ1OE= v1,bm9ldHUjKzFob2VudXRob2VodWUzMjRvdWVvdW9ldQo=

Then, your consumer can just try and match each signature and as long as one of them matches, consider it a success. Here's an example JavaScript snippet that does it:

const signatures = signatureHeader.split(' ')

for (const signature of signatures) {
  if (timingSafeEqual(signature, expectedSignature)) {
    return true // Verification succeeded
  }

  return false // Verification failed
}

As we saw, making sure your webhook secret can be safely rotated with zero downtime is fairly easy, you just need to make sure to design your signature scheme to support it in the first place. Alternatively, you can just use the Svix service which takes care of everything for you, or use any of the Svix libraries to implement secure signing and verification.

For more content like this, make sure to follow us on Twitter, Github or RSS for the latest updates for the Svix webhook service, or join the discussion on our community Slack.