Turn Any Webhook into a Polling API

While webhooks are a great way to get real-time API notifications, there are situations where polling for events is preferred.

In this blog post we will show how to use two of our recent launches Svix Ingest and Polling Endpoints to transform webhooks from any provider into a polling endpoint.

When is polling better?

There are a few examples where polling for events works better than webhooks.

One example is when testing webhooks locally. It's much easier to poll for events than exposing a public HTTP endpoint (even with tools like Svix Play ngrok).

Another example is when you don't care about getting events in real-time and prefer getting them all at once at the end of a day. One place where this use-case comes up, is when you need to store all of the events for compliance reasons. In that case batching and saving them all at once is easier and more cost efficient.

There are more examples where polling may work better, but without further ado, let's jump in to how to actually achieve this!

Setting up a webhook receiver

In order to be able to poll the endpoints, we first need to send them to Svix Ingest. The first step is to go the Svix Ingest page on the dashboard and create a new source.

After choosing a name for the source, you will get back a URL. You can then use the URL as an endpoint destination for the webhook provider you would like to use. In the following screenshot we chose Github as the source type, but you can use any provider, including just generic URLS.

Once the endpoint has been created on the provider's end (Github) and webhooks are flowing to Svix Ingest, we can move onto the next step.

Polling for events

The next step is setting up a polling endpoint which we can use to poll the API. Svix Ingest also supports forwarding the event to multiple webhook endpoints in addition to polling. You can read more about it on the Svix Ingest page.

To create a polling endpoint, click on the Ingest Source we created in the previous step, go to destinations and click "Add Endpoint". Then choose "Polling Endpoint" from the selection box on the top right corner like so:

Once created we will get a new polling URL that we can use for polling. In order to use it we first create an API key to be used with this endpoint from the "Create API Key" button. We will then be presented with an API key. Please save it somewhere as it will only be presented once.

You can poll this endpoint using the API key like so:

curl \
  -X GET "https://api.us.svix.com/api/v1/app/app_2sJi19JCkIFjTDkM9ZrAnu8vgaV/poller/poll_ity" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer sk_poll_UAIljBiBWMbuenwK_6LCGytnNufYv40B.us'

When you first call the Polling Endpoint, it will return an empty array and an iterator.

{
  "data": [],
  "iterator": "eyJvZmZzZXQiOi05MjIzMzcyMDM2ODU0Nzc1ODA4LCJhZnRlciI6bnVsbH0",
  "done": false
}

Use the iterator in the next call to the Polling Endpoint to return all messages received since the iterator was created. Messages will be returned in the order they were received.

curl \
  -X GET "https://api.us.svix.com/api/v1/app/app_2sJi19JCkIFjTDkM9ZrAnu8vgaV/poller/poll_ity?iterator=eyJvZmZzZXQiOi05MjIzMzcyMDM2ODU0Nzc1ODA4LCJhZnRlciI6bnVsbH0" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer sk_poll_UAIljBiBWMbuenwK_6LCGytnNufYv40B.us'

Please note, that calls without an iterator set always return an empty response starting from after the most recent call, so make sure to store the iterator on your side in order to be able to get all of the messages.

That's it! We have now turned our Github webhooks into something we can poll. You can do the same with any other webhook provider.

If you're using a webhook provider that is already using Svix, you can ask them to enable polling-endpoints on their integration. They can do it with the click of a button.

For more content like this, make sure to follow us on Twitter, Github, RSS, or our newsletter for the latest updates for the Svix webhook service, or join the discussion on our community Slack.