A new user signs up for an email list. A new prospect is added to a CRM. A new customer starts a subscription.
If you are integrating with multiple services, these are the type of events you need sent to your application to get the right information to your users in real-time. But how can your application efficiently and effectively respond to these events in real-time? The answer lies in the power of webhooks.
Webhooks provide a powerful way for applications to communicate in real-time. They react to events, such as a new user signing up or a customer starting a subscription, enabling your system to respond promptly and efficiently. Despite their potency, handling webhooks comes with challenges, from verifying their source securely to managing high volumes without straining server resources.
Here, we will delve into the essentials of handling webhooks, covering everything from understanding their basic components, implementing secure authentication, and scaling your application to accommodate a heavy influx of webhooks. But let’s start with the fundamentals–what are webhooks and why can’t you just use APIs for this?
With APIs, you ask for data, you receive that data. With webhooks, you receive data as it changes.
Webhooks are a type of "user-defined HTTP callback"—they provide real-time information to other applications. Essentially, they are a way for an application to provide other applications with real-time information, making it easy for users to receive notifications or data updates immediately when specific events occur.
Here's a technical breakdown of how webhooks work:
Contrast that with APIs. APIs follow a synchronous request-response model–the client requests data from the server and the server responds. But the API model doesn’t work well with real-time events or data. You would have to continually poll the API to get the updated data, which would tie up bandwidth and resources (and get you rate-limited). Depending on how much data you’re dealing with, you could also potentially miss events.
Webhooks are asynchronous and event-driven. With webhooks, the server sends data to the client automatically when a specified event occurs. This makes webhooks more efficient when you need to receive data as soon as it changes or events occur.
The downside of webhooks is that webhooks can be more complex to implement and maintain. Whereas with APIs it is the service you’re communicating with that sets up the endpoint, with webhooks, you are setting up the endpoint and then giving that URL to the service so they can send data to it. This also means that all the implementation details–error handling, authentication, securing requests–is done to you.
So given that, what needs to go into your webhook?
Here’s how some of these look in a real-life application–Vessel. WE have webhooks integrated into our product to send two types of events to users: STATUSCHANGE and OBJECTCHANGE.
The headers we send to a user are:
In the body, or payload, of our webhook we have an eventId, the type of event, and then a data object:
Remember, these components might not all be present or could vary in every webhook implementation, but these are the general parts you can expect to encounter.
Let’s start with a basic example. Let's assume we are receiving a webhook from a service that triggers whenever a new user signs up on their platform. Our job is to take the data from the webhook and process it, in this case, we'll just log it.
We’ll set up a basic server with express in a file called app.js:
Here, we define an async function as the handler for the /webhook route. When a POST request is made to this route, the handler function is triggered.
Inside the handler function, we get the body of the request, which is the data sent by the webhook, and pass it to the processWebhookData function. This function simulates some async processing using a setTimeout.
If everything goes well, the server sends back a 200 status code. If there's an error, we catch it and return a 500 status code.
We can run this using:
How are we going to test this? We need a service to send us some data. Here, we’ll use GitHub as it is extremely easy to set up webhooks in GitHub. Head to your page on GitHub, go to settings, and then to webhooks. You’ll have a page like this:
Click on “Add webhook” and you’ll get to a page like this:
The most important field here is “Payload URL”. This is the URL of your application. Now, you’re probably just using localhost to serve your app now which is only available to your local network. Webhooks require your server to be publicly accessible over the internet so that the webhook provider can send POST requests to it.
To do that with localhost we can use ngrok. ngrok exposes your localhost with a custom URL for testing situations just like this. Follow the instructions to get set up with ngrok and then run it locally to get your URL. It is that URL you are going to add to “Payload URL,” appended with /webhook. So your URL will look something like this:
We’ll add this to GitHub, and change a couple of the settings. We want the data format set to urlencoded, and we want all events to trigger this webhook. We don’t have a secret set up so we’ll leave that blank.
Click ‘Add webhook,’ and head back to your terminal running your application. You should see data sent from GitHub:
Congratulations, you’ve got a working webhook.
The above example is good for testing. But if you wanted to use it as part of an integration, it is missing some functionality. Namely:
Let’s start with those first two bullet points: security and data handling.
Authenticating webhooks is a little different to authenticating APIs. Effectively, we want to know both a) the message is coming from the source we intend, and b) the message itself is authentic. A common option is HMAC, which stands for Hash-based Message Authentication Code.
HMAC combines a secret key with the message data, hashes the result with a hash function, combines that hash with the secret key again, and then applies the hash function a second time. The output hash is unique to the input data and the secret key. Thus, even a small change in either results in a significant difference in the output hash.
In the context of webhooks, HMAC is often used to validate that the incoming HTTP request is indeed from the expected sender and hasn't been tampered with during transmission. The sender will generate a signature (HMAC) of the payload using a secret key, and include this signature in the header or body of the request. The receiver will generate its own HMAC of the received payload using its own copy of the secret key, and compare this to the received HMAC. If they match, the webhook is valid and originated from the expected sender.
Data handling is an easier prospect. The best option is not to apply any logic within the ‘webhook’ function, but to use the webhook as an entry point into your application and then route data depending on what has been sent. You can do this either based on the payload itself, or via an added header signifying the event type.
Here’s an example of a webhook endpoint with both HMAC security and data routing:
Here, we apply the bodyParser middleware to parse incoming JSON payloads. We also save the raw body to req.rawBody to verify the signature later. When a POST request is received at the /webhook endpoint, we extract the signature and event type from the headers and the payload from the body. We then generate a hash of the received raw body using the same algorithm and secret as the sender, and compare it to the received signature. If the signature matches, we print the event type and payload to the console.
We then use the Fetch API to send a POST request to another server to notify it about the received data. We send a 200 OK status code back to the webhook provider to acknowledge that we've received and processed the webhook.
Here is where testing (and documentation) becomes important. Most providers aren’t going to have an x-signature or an x-event header–they are going to be called something else. For instance, with HubSpot, these are X-HubSpot-Signature and eventType, respectively. You have to tailor your webhook to fit the payload the provider is going to send. The good news is that these are often well-documented and larger providers are going to provide you with a lot of information to make the parsing and logic within your application easier to build.
Good testing tools here are services like RequestBin or webhook.site that give you a public URL to add to a provider so you can see the data they send without having to set up your own server. You can get an understanding of the payloads, the data types and structures, and headers sent and then tailor your own endpoints to fit the providers.
Scaling can be a problem with webhooks for a few reasons:
To handle these scaling issues, you'll need to implement scalable architectures and patterns, such as using a queue and worker pattern for asynchronous processing, using auto-scaling to adjust resources based on demand, implementing robust error handling and retry mechanisms, and carefully managing access to shared resources.
We can add queues to our endpoint using bull and redis:
Here we create a new Bull queue and add incoming webhook events to the queue. The webhookQueue.process() function is where you define how to process jobs in the queue. In this case, it sends a POST request to another server with the webhook data.
Potentially, you’ll also have to scale horizontally, adding webhooks for many third-party integrations. In that case solutions such as Vessel can allow you to handle many integrations together without building separate logic for each service. This can make using webhooks significantly easier to implement at scale.
Webhooks are a powerful tool for creating real-time, responsive applications. They enable systems to communicate and react to events as they happen, bridging the gap between different services and systems in a scalable and efficient way.
But handling webhooks comes with its own set of challenges. Implementing webhooks is not a one-size-fits-all process. Your application might have unique needs and challenges, and the solutions should be tailored to those needs. However, with the principles and patterns we've discussed here—like async processing, rate limiting, and scaling—you're well-equipped to design and implement an effective webhook handling system.
Keep in mind that the ultimate goal of using webhooks is to create seamless integrations and real-time interactivity that enhance the user experience. With careful planning, thoughtful implementation, and thorough testing, webhooks can become an integral part of your application's toolkit, driving engagement and supporting the growth of your platform.