DEV Community

Daniel Schreiber
Daniel Schreiber

Posted on

Firebase function retries with pubsub

#firebase #pubsub #gcp

Ever tried to use exponential backoff for pubsub triggered firebase functions? It’s not straight forward, but I’ll walk you through.

Why?

Decoupling cloud/firebase functions from the trigger/invocation via pubsub message topics is a best practice to improve reliability and resilience. Especially for the latter you’d want to have a sensible retry setup, so that transient errors are retried. This retry can be configured in the pubsub subscription that triggers the function.

What is the problem?

Consider this function:

export const myFunction = functions.region('europe-west1')
    .runWith({failurePolicy: {retry: {}}})
    .pubsub.topic('my-topic')
    .onPublish(...)
Enter fullscreen mode Exit fullscreen mode

When deploying the function using the firebase cli, the subscription is recreated with standard retry settings. Using the firebase cli, it is currently not possible to define exponential backoff for the retry.

Note that the default settings can have a severe impact on your cloud bill. When retries are enabled (failurePolicy: {retry: {}}), the standard configuration retries immediately and for 7 days! So for non-transient errors or problems with a longer recovery time, you’ll pay for many invitations..

What is the solution?

After each function deployment, you need to update the subscription accordingly. The following configures retries with exponential backoff for at most 2 hours and delays between 10 seconds and 10 minutes:

# regular deployment:
firebase deploy --only functions --force
# update pubsub subscription for retries with exponential backoff:
gcloud pubsub subscriptions update \
 projects/$GCP_PROJECT_ID/subscriptions/gcf-myFunction-europe-west1-my-topic \
 --min-retry-delay=10s \
 --max-retry-delay=10m \
 --message-retention-duration=2h
Enter fullscreen mode Exit fullscreen mode

Note that --force is needed in CI setup to deploy functions with retries.

What are the alternatives?

You could as well

  • manually update the subscription (not recommend)
  • move from firebase cli to a direct deployment on GCP (eg via terraform)

If I missed something or you have other solutions, please comment!

Top comments (4)

Collapse
 
reidab profile image
Reid Beels •

Thanks! This pointed me in the right direction.

I found that by using the built-in $GCLOUD_PROJECT env var, you can add this as a postdeploy hook in firebase.json to ensure it runs with every deploy, like so:

{
  "functions": {
    "postdeploy": [
      "gcloud pubsub subscriptions update projects/$GCLOUD_PROJECT/subscriptions/gcf-myFunction-europe-west1-my-topic --min-retry-delay=10s --max-retry-delay=10m --message-retention-duration=2h"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
ranguna profile image
Luis Pais •

This was spot on my friend, thank you very much for the article!

Now all that's left is a good way to setup DLQs and we are golden 👌

Collapse
 
danielsc profile image
Daniel Schreiber •

This should be possible with some more gcloud commands - would really be interested in a small write up! :-)

Collapse
 
ranguna profile image
Luis Pais •

Hahaha nice baton pass!
I'll see what I can come up with, it might not be as straight forward since we will be creating a new topic and subscription (+ permissions and all that fun stuff), but it should work.
Don't expect it any time soon though xP

Read next

sarwarasik profile image

Firebase push Notification Error: ⚠️ Auth error from APNS or Web Push Service

Sarwar Hossain -

jader_lima_b72a63be5bbddc profile image

Creating a data pipeline using Dataproc workflow templates and cloud Schedule

Jader Lima -

thebuildguy profile image

How to paginate queries in Firestore?

Tulsi Prasad -

chauhoangminhnguyen profile image

Deploy ReactJS application to Firebase in 5 minutes

chauhoangminhnguyen -