Reliable Webhook Delivery Anywhere: No Open Ports, No Delayed Messages

Webhooks were created as a way to enable asynchronous server to server communications. When something happens on server A, they send a webhook to server B, and server B can then react on this event.

This works great when it's two internet-facing services that need to communicate, as it usually is the case, but what happens when the webhook recipient is not internet-facing? The whole model breaks.

Another implied requirement of the above relationship is that the server would not only be be internet-facing, but also available and responding to requests. If the server is down, the webhooks can't be delivered.

In this post we'll show how to utilize Svix Ingest and Polling Endpoints for durable webhook deliveries behind firewalls and to unreliable endpoints. This is especially relevant for home labs and services running on corporate networks where servers are not internet-facing and often have less stringent uptime requirements.

Making webhook endpoints internet-facing

Making webhook endpoints internet-facing is not always desirable. For example, you may be operating in highly sensitive or compliance-heavy requirements where company policies don't allow for it. Additionally, exposing an endpoint to the internet increases the potential surface area for a variety of attacks such as DoS or even exploiting the endpoint or the web server.

Though if that's what you're after you can use tools like ngrok or Svix Play (the svix listen command) to setup a webhook forwarder that can tunnel your webhook requests to your server even behind a firewall.

Reliably dealing with server downtime

Internet-facing servers usually come with stringent uptime requirements. If you're exposing an API (either internal or external) to the internet, even a little bit of downtime leads to a degraded experience for your customers. This is not the case with hobbyist home labs, like a self hosted Gitlab instance, or internal enterprise services. Both of these may have maintenance windows, be rebooted often, or even turned off at night or weekends to save power.

Webhooks were designed with reliability in mind, so most webhook implementations (including Svix) include automatic retries following an exponential backoff. While this is a great solution for short lived issues, it was not intended to handle servers being offline for long periods of time.

A better solution would be to use a combination of Svix Ingest and Polling Endpoints to solve both problems.

We previously wrote a tutorial on how to turn any webhook into a polling API. We will be using exactly the same techniques shown there to receive webhooks from any source and reliably ingest them into our own system.

Once done, we can have something like this in our code:

import requests

url = "https://api.us.svix.com/api/v1/app/app_2sJi19JCkIFjTDkM9ZrAnu8vgaV/poller/poll_ity"
headers = {
    "Content-Type": "application/json",
    "Accept": "application/json",
    # In a real codebase the token will be securely stored
    "Authorization": "Bearer sk_poll_UAIljBiBWMbuenwK_6LCGytnNufYv40B.us"
}

# Load iterator from persistent storage
iterator = load_iterator()

while True:
    params = {"iterator": iterator} if iterator is not None else {}
    response = requests.get(url, headers=headers, params=params)
    if not response.ok:
        sleep(5)
        continue

    payload = response.json()
    # Do something with the webhook
    process_webhooks(payload["data"])

    # Persist iterator
    iterator = payload["iterator"]
    save_iterator(iterator)

    if payload["done"]:
        # If there's no more data to fetch, sleep for a bit.
        sleep(1)

This is it. This is all you need in order to reliably receive webhooks from any service even if the recipient endpoint is not internet accessible (e.g. behind a firewall), is unreliable, or otherwise often unavailable.

What's next?

While the code above is very short and easy to reason about, the developer experience can still be significantly improved. For example, in the code sample above the iterator is persisted locally, which is slightly annoying and causes extra work on the receiver end. To solve this, we will be releasing automatic server side iterator management in the next couple of weeks, which will enable us to remove all of the state handling from the receiver end.

Additionally, we plan on integrating this functionality into both Svix Bridge and Svix CLI which will let people delete this code altogether.

Got any other interesting use-cases for this functionality or have any suggestions for additional improvements? We would love to hear. Please email us at contact@svix.com and let us know!

For more content like this, make sure to follow us on Twitter, Github, RSS, or our newsletter for the latest updates for the Svix webhook service, or join the discussion on our community Slack.