DEV Community

loading...

Firebase function retries with pubsub

danielsc profile image Daniel Schreiber ・2 min read

Ever tried to use exponential backoff for pubsub triggered firebase functions? It’s not straight forward, but I’ll walk you through.

Why?

Decoupling cloud/firebase functions from the trigger/invocation via pubsub message topics is a best practice to improve reliability and resilience. Especially for the latter you’d want to have a sensible retry setup, so that transient errors are retried. This retry can be configured in the pubsub subscription that triggers the function.

What is the problem?

Consider this function:

export const myFunction = functions.region('europe-west1')
    .runWith({failurePolicy: {retry: {}}})
    .pubsub.topic('my-topic')
    .onPublish(...)
Enter fullscreen mode Exit fullscreen mode

When deploying the function using the firebase cli, the subscription is recreated with standard retry settings. Using the firebase cli, it is currently not possible to define exponential backoff for the retry.

Note that the default settings can have a severe impact on your cloud bill. When retries are enabled (failurePolicy: {retry: {}}), the standard configuration retries immediately and for 7 days! So for non-transient errors or problems with a longer recovery time, you’ll pay for many invitations..

What is the solution?

After each function deployment, you need to update the subscription accordingly. The following configures retries with exponential backoff for at most 2 hours and delays between 10 seconds and 10 minutes:

# regular deployment:
firebase deploy --only functions --force
# update pubsub subscription for retries with exponential backoff:
gcloud pubsub subscriptions update \
 projects/$GCP_PROJECT_ID/subscriptions/gcf-myFunction-europe-west1-my-topic \
 --min-retry-delay=10s \
 --max-retry-delay=10m \
 --message-retention-duration=2h
Enter fullscreen mode Exit fullscreen mode

Note that --force is needed in CI setup to deploy functions with retries.

What are the alternatives?

You could as well

  • manually update the subscription (not recommend)
  • move from firebase cli to a direct deployment on GCP (eg via terraform)

If I missed something or you have other solutions, please comment!

Discussion (4)

pic
Editor guide
Collapse
reidab profile image
Reid Beels

Thanks! This pointed me in the right direction.

I found that by using the built-in $GCLOUD_PROJECT env var, you can add this as a postdeploy hook in firebase.json to ensure it runs with every deploy, like so:

{
  "functions": {
    "postdeploy": [
      "gcloud pubsub subscriptions update projects/$GCLOUD_PROJECT/subscriptions/gcf-myFunction-europe-west1-my-topic --min-retry-delay=10s --max-retry-delay=10m --message-retention-duration=2h"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode
Collapse
ranguna profile image
Luis Pais

This was spot on my friend, thank you very much for the article!

Now all that's left is a good way to setup DLQs and we are golden 👌

Collapse
danielsc profile image
Daniel Schreiber Author

This should be possible with some more gcloud commands - would really be interested in a small write up! :-)

Collapse
ranguna profile image
Luis Pais

Hahaha nice baton pass!
I'll see what I can come up with, it might not be as straight forward since we will be creating a new topic and subscription (+ permissions and all that fun stuff), but it should work.
Don't expect it any time soon though xP