Working around HubSpot's limited Webhooks

Webhooks are an API pattern for subscribing to events in a system. Rather than repeatedly polling the HubSpot REST API's list of Contacts, you can define a webhook subscription that tells HubSpot where to deliver news of a Contacts changing. Webhooks provide some much-needed relief from HubSpot's tight API rate limits, especially if your application needs to react quickly to changes (and would therefore need to poll frequently to achieve the same update-turnaround).

Of course, webhooks have their drawbacks. We've written about those before. Though they might let you run your application on ephemeral infrastructure, your app does need to be constantly available: different webhook APIs make different commitments to retries, but most will eventually give up trying to deliver events.¹ If your application uses webhooks to maintain an up-to-date partial state --- e.g. if it's receiving contact.propertyChange events to maintain a collection of Contacts --- you'll need to periodically poll to correct for any events you've missed.

The particular drawback of HubSpot's Webhooks is that there just aren't enough of them.

Each HubSpot webhook subscription receives events of one "subscription type:" events pertaining to one data life cycle event (creation, deletion, or propertyChange) for one CRM object type: contact, company, or deal.

*.propertyChange subscriptions need to specify which properties they care about. You can create a subscription for each property on an object, but you can't generically specify "any property" with a wildcard expression. This is just one of several other limitations we'll address later in this post.

Suppose you're building a tool that caches the email addresses of every Contact in your HubSpot CRM. Once you've created a HubSpot app — and noted its ID — you'd register your webhook receiver (announcing it and specifying its limits) by sending a PUT request to /webhooks/v3/{appId}/settings:

curl --request PUT \
  --url https://api.hubapi.com/webhooks/v3/{appId}/settings \
  --header 'content-type: application/json' \
  --data '{
  "targetUrl": "https://www.example.com/hubspot/target",
  "throttling": {
    "maxConcurrentRequests": 10,
    "period": "SECONDLY"
  }
}'

Once your receiver is registered, you can create your contact.propertyChange subscription with a POST request to /webhooks/v3/{appId}/subscriptions:

curl --request POST \
  --url https://api.hubapi.com/webhooks/v3/{appId}/subscriptions \
  --header 'content-type: application/json' \
  --data '{
  "eventType": "contact.propertyChange",
  "propertyName": "email",
  "active": true
}'

This works like a charm for simple applications. For more sophisticated HubSpot integrations, and for engineers syncing HubSpot data en masse to Postgres, it's insufficient.

What the Webhooks API won't give you

If you need to listen for updates to any property on an object, it's a problem that each webhook subscription only tracks updates to one.

If you're excitedly realizing that there's a property, hs_lastmodifieddate, updated in conjunction with any other property, I have more bad news: HubSpot already thought of that. *.propertyChange subscriptions on hs_lastmodifieddate and num_unique_conversion_events are forbidden.

On the bright side, at least the Webhooks API lets you dynamically create subscriptions. You can create one for every property, but keep in mind the object property lists are dynamic:

• HubSpot might introduce properties without breaking the existing API.

• Your team might introduce custom properties, tailoring your CRM to your business.

Each object's properties are introspectable via GET requests to /crm/v3/schemas/{objectType}. For example, you might fetch a JSON definition of all the fields on a Contact (both HubSpot-specified and custom):

curl --request GET \
  --url 'https://api.hubapi.com/crm/v3/schemas/contact'

In theory, that means you could poll the schemas and dynamically provision *.propertyChange webhook subscriptions for each property listed. As pseudocode:

On some polling interval:
  # List current subscriptions.
  GET /webhooks/v3/{appId}/subscriptions

  For each type in {Contact, Company, Deal}:
    # Get all current properties.
    GET /schemas/{type}
    For each property without a current subscription:
      # Create a new `{type}.propertyChange` subscription.
      POST /webhooks/v3/{appId}/subscriptions

There are two problems with this approach.

1. There are two delays each time you discover a new property: your polling interval, and HubSpot's five-minute webhook settings propagation time. In that period before the new webhook goes live, you're missing update events; that drift has to be backstopped with polling.

2. HubSpot imposes a hard limit on the number of Webhook subscriptions you can define: 1000. If you need to subscribe to more than 1000 properties, you're hosed.²

If those aren't concerns, you have a working HubSpot Webhooks solution for Contacts, Companies, and Deals.

If you need to track updates to obects besides those three, you know the disappointment that's coming next — it's probably what brought you to this post. HubSpot's Webhooks API only supports Contacts, Companies, and Deals. It doesn't support custom objects. It doesn't support any of the other built-in CRM types (Feedback Submissions, Line Items, Products, Tickets, or Quotes). It doesn't support Owners, the CRM-specific user model your team uses to assign customer relationships, or the Pipelines managing deal-flow for Sales Hub Professional and Enterprise accounts.

If your HubSpot integration counts on data from any of the above, you'll have to do without the HubSpot Webhooks API.

Why Workflows won't cut it

A handful of forum posts suggest driving webhook calls for custom objects using HubSpot Workflows, a business automation toolkit. Beware tumbling into this rabbit hole: for several reasons, Workflows aren't a substitute for proper Webhooks API support.

Object-based Workflows handle a greater breadth of object types than the Webhooks API, but support for custom objects is restricted to HubSpot Enterprise customers and the "Trigger a webhook" Action is restricted to Operations Hub Professional and Enterprise customers.

"Enrollment" governs the applicability of a Workflow to an object. Think of a Workflow's enrollment trigger as a filter condition: depending on the condition, HubSpot evaluates the enrollment trigger periodically or whenever a candidate object is updated to decide whether to drop the object into the workflow. You can set an enrollment trigger on a type's last modified date such that an object is enrolled anytime that date is set.

Crucially, objects will only pass through the workflow once unless they're re-enrolled by a separately-defined re-enrollment trigger. Just like the Webhooks API forbade subscribing to hs_lastmodifieddate, HubSpot prohibits re-enrollment by last modified date.

You can take the dynamic-provisioning approach I discussed for Webhooks API subscriptions, but the same pitfalls apply here.³ HubSpot hasn't designed Workflows as a Webhooks API alternative; you press it into service at your peril!

Homebrewing events, not webhooks

Why does anyone reach for HubSpot's Webhooks API in the first place? You need to know when objects change, and HubSpot's rate limits prevent you from the simple approach: constantly paginating through all of your HubSpot data as fast as possible. That simple approach would be viable if only the rate limits weren't an issue.

The key is optimized polling: search the CRM. Rather than paginating through all your HubSpot data, you can use the /crm/v3/objects/{object}/search endpoints to selectively fetch records that've changed. There's an endpoint for every CRM object type: Companies, Contacts, Deals, Feedback Submissions, Products, Tickets, Line Items, Quotes, and any custom object your team can dream up.

The trick is to hold onto a timestamp to serve as a cursor, to search for objects that've been updated since that cursor using filterGroups, and to update the cursor after each search:

# Initialize the cursor: Unix epoch (ms).
cursor_ts = 0

On some polling interval:
  For each type in {your object types}:
    # Fetch objects updated since the cursor timestamp.
    POST /crm/v3/objects/{type}/search
  Set the cursor to the earlier of
   ├ the most recent update timestamp
   └ the time of the first request

To fetch objects updated since the cursor timestamp, define a filter on hs_lastmodifieddate in your POST request body:

{
  "filterGroups":[
    {
      "filters":[
        {
          "propertyName": "hs_lastmodifieddate",
          "operator": "GTE",
          // Cursor value.
          "value": 1656009606234
        }
      ]
    }
  ]
}

Why not just set the cursor to the current time? Requests take time, so there's a chance updates occur in HubSpot between when you've sent data and when you receive and process it. Since the search endpoint always yields the most recent version of each object, you can safely manipulate the cursor to introduce an overlap in your search periods, which ensures no update falls through the cracks.

As with any polling strategy, this use of HubSpot's CRM Search endpoints has pitfalls.

1. Carefully determine (or dynamically set) a polling interval that won't exhaust your HubSpot API rate limits.

2. Since this fetches current object states rather than events, you'll miss intermediate updates that're overwritten between searches. This system works well for keeping data fresh, but it won't provide a full history of its changes.

   

Nonetheless, you've solved several issues inherent in webhook architectures. Storing a cursor means your tool refreshes on its own terms, so you don't have to fear missing events or sudden cascades of traffic. You're capturing every property on every HubSpot resource exposed by their REST API, but (mercifully) your API utilization is a function of the number of records updated per polling interval rather than the number of records total.

Of course, these APIs are subject to change. If you want to work with HubSpot webhooks, we recommend reading their latest documentation.

Alternatively, try Sequin. We've managed all of this complexity — the cursors, rate limits, monitoring for drift, massaging HubSpot's schemas into Postgres tables, and so on — so you don't have to read one of these blog posts ever again. Get started by exploring your HubSpot data in a free demo DB!

1. HubSpot retries webhook requests ten times over 24 hours if your handler is unconnectable, takes longer than five seconds to respond, or responds with a client or server error status code (4XX or 5XX). Down for more than 24 hours? No events for you! ↩

   If the HubSpot Webhooks API itself has an outage, all bets are off.

2. By our count, there are 547 default properties between Contacts, Companies, and Deals. That leaves 453 Webhook subscriptions for custom properties. ↩

   Object type	Default property count
   Contact	274
   Company	158
   Deal	115
   Sum	547

3. The 250-trigger limit for Workflows is even tighter than the 1000-subscription limit in the Webhooks API; it's not even enough to re-enroll on every Contact field. ↩