The What, Why, and How of Payload Transformations

If you don't already know, we've had a new feature available in beta for some time now called Transformations. I was one of the primary contributors to this feature, and am hoping to explain to you what they are, why they're useful, and how to use them.

Later, in another post, I'll be doing a deeper dive into how transformations work under the hood for those of you interested in the implementation.

A scenario to start

Say you're a Svix customer who offers an e-commerce software suite. Your customers are small, online businesses who sell and send out physical goods. You use Svix to notify these customers of important events: a new order has been made, a support ticket has been sent, a charge-back has occurred, etc.

All of these events need to be routed to various places in specific formats. This can be done by defining event type filters per endpoint. Then you, the company offering this e-commerce system, can dispatch an order.created event, a ticket.created event, or an order.chargeback event.

However, what do you do when your customers need the same event sent in different formats? For example, your customer may need the order.created event sent to several places.

They may have an application tracking inventory and another application keeping statistics on the types of products sold.

But, they might also need its details sent via email, in a human readable format, to their warehouse, such that the product is shipped promptly.

The naive solution is to offer more event types. Even then, the default webhook format may not match your customers' exact needs. There could be too many combinations of formats and events to manage as a Svix customer. And then, you may not want to create event types just for one of your customers who wants to do something a bit different.

With transformations, it's so much easier for your customers to make these customizations.

Introducing transformations

In brief, transformations allow your consumers to apply changes to messages before they're dispatched to any given endpoint. These changes -- or transformations -- are defined via JavaScript code that runs per message dispatched for any endpoint where the feature is enabled.

In the scenario presented above, a customer can very easily alter the content of each order.created message per destination using a short JS function. Say that the email dispatch of order details can be made with an HTTP request: then your customer, with no work on your end, can just define a transformation to reorganize the payload into that of the email service's required schema.

Let's get this out of the way -- the cost

Good news: Transformations are free and unlimited.

Bad news: Transformations are free and unlimited for now.

Since the feature is in beta, we don't want to charge any additional costs for something that may not work entirely as expected. But because this is quite a computationally expensive feature to offer, transformations will incur some additional cost once we're confident the feature is production-ready. The pricing has not been finalized, but we intend for it to be very reasonable.

Limitations

We have added a few restrictions in order to ensure that all customer data is secure and isolated, as well as some limitations that ensure that everyone can use the feature.

For one, precautions exist so you cannot access any:

• Files
• Environment variables
• Networking
• Generally any other I/O

This shouldn't come up in non-malicious use of the feature very often, but it's definitely a needed restriction.

Additionally, there are hard limits on RAM use and processing time, so no loops that run a million times. We wanted to make sure nobody can starve other customers' access to this feature.

Finally, you may not use async JavaScript in these scripts. No changes you make should be IO bound, though, so we hope that's not too much of a problem.

In the future these restrictions may be eased, but only after I can ensure nothing is easily abused.

Beside these three points, you can manipulate the webhook dispatched in practically any way you'd ever want.

How to use them

You can see the docs on this feature here for a more concise explanation, but I have some examples that may help you understand the feature.

First of all, transformations have to be explicitly enabled per environment. You can get there via the Settings page on the dashboard. Under the Settings section, enter the General Settings page. From there, you can simply flip the Enable Transformations switch to turn it on or off.

From the app portal, assuming your organization allows transformations as mentioned above, your customers can enter the Endpoints section, click on any endpoint, then enter the Advanced tab. In this tab, you can enable and edit a transformation for that one endpoint.

This brings up an entire JavaScript editor with validation and test events so consumers can ensure their transformations work just how they want the transformations to work.

From this editor you are given the following template:

/**
 * @param webhook the webhook object
 * @param webhook.method destination method. Allowed values: "POST", "PUT"
 * @param webhook.url current destination address
 * @param webhook.eventType current webhook Event Type
 * @param webhook.payload JSON payload
 */
function handler(webhook) {
  // modify the webhook object...

  // and return it
  return webhook
}

Every time a webhook is sent through an endpoint which has a script defined, the handler function in the script is called. This function is expected to take one argument -- called webhook in the example above. As it states in the comment, this webhook variable has the following members:

• method -- The HTTP method the webhook should be sent with. It can be either POST or PUT, but this defaults to POST.
• url -- The URL to send the webhook too. This can be any arbitrary URL that meets the requirements of the environment (so no plain HTTP requests if HTTPS Only Endpoints is enabled in the general settings page).
• eventType -- If you're familiar at all with Svix, you know that each message is associated with an "event type" -- an arbitrary identifier that is used to mark the type of webhook. You can change it here to further modify the final payload.
• payload -- Finally, the actual message contents can be programmatically manipulated. The payload, however, be a valid JSON value both before and after transformations are applied.

The webhook value given to the handler is mutable, and it should be returned after the requisite alterations have been made.

Examples

Example 1

Say you're a client of the hypothetical e-commerce service mentioned earlier. You want to send a notification to some service every time an order has been paid for so you can know what product to package and ship.

But there's a catch: this service expects that the product ID be encoded in the URL's path instead of in the payload.

Well, luckily, with transformations, you don't need your own server to forward requests that need this tiny adjustment.

function handler(webhook) {
    if (webhook.eventType === "invoice.paid") {
        const product = webhook.payload.product_id;
        delete webhook.payload.product_id;

        webhook.url = 'https://a-valid.url/product/' + product + '/purchased/';
    }

    return webhook
]

Example 2

Need more invasive manipulations to the payload? Consider this example: You're again the client of the hypothetical e-commerce service. This time, you want a message sent via a company-wide chat app to a channel that sends a message every time you get an order, but it also has to scrub PII from the data.

The format this chat application requires may be vastly different from the schema of the webhook the e-commerce service sends. Say the default webhook looks something like:

{
  "product_id": "prod_abc123",
  "quantity": 7,
  "customer": {
    "name": "Jane Doe",
    "address_1": "123 Fake St",
    "address_2": null,
    "city": "RealCityName",
    "state": "IAmRunningOutOfFakeNames",
    "zip_code": "12345-6789"
  }
}

But then the chat application requires this format:

{
  "channel": "some-id",
  "message": "some-message"
}

Transformations are to the rescue again! Combined with our custom headers feature allowing you to send any API tokens needed with the request, you could use the following code:

function handler(webhook) {
  if (webhook.eventType === 'invoice.paid') {
    const quantity = webhook.payload.quantity
    const first_name = webhook.payload.customer.name.split(' ')[0]

    webhook.payload = {
      channel: 'we-got-an-order',
      message: first_name + ' just ordered ' + quantity + ' products!',
    }
  }

  return webhook
}

TL;DR

Transformations allow your customers to modify select aspects of a webhook, such as the payload and URL, before it's dispatched by writing short JavaScript functions,which can be written in the app portal or updated with the Svix API.

Transformations are an excellent feature for interfacing with different applications, enabling your customers to send events directly to an external service's API in whatever format the service requires.

This can vastly speed up your customers' workflow. No longer do they have to hack together a functional web server just to receive a webhook and forward it after making minute alterations. Instead, they can just write one function and forget about the rest.