DEV Community

Daniel Schreiber
Daniel Schreiber

Posted on

Firebase function retries with pubsub

#firebase #pubsub #gcp

Ever tried to use exponential backoff for pubsub triggered firebase functions? It’s not straight forward, but I’ll walk you through.

Why?

Decoupling cloud/firebase functions from the trigger/invocation via pubsub message topics is a best practice to improve reliability and resilience. Especially for the latter you’d want to have a sensible retry setup, so that transient errors are retried. This retry can be configured in the pubsub subscription that triggers the function.

What is the problem?

Consider this function:

export const myFunction = functions.region('europe-west1')
    .runWith({failurePolicy: {retry: {}}})
    .pubsub.topic('my-topic')
    .onPublish(...)
Enter fullscreen mode Exit fullscreen mode

When deploying the function using the firebase cli, the subscription is recreated with standard retry settings. Using the firebase cli, it is currently not possible to define exponential backoff for the retry.

Note that the default settings can have a severe impact on your cloud bill. When retries are enabled (failurePolicy: {retry: {}}), the standard configuration retries immediately and for 7 days! So for non-transient errors or problems with a longer recovery time, you’ll pay for many invitations..

What is the solution?

After each function deployment, you need to update the subscription accordingly. The following configures retries with exponential backoff for at most 2 hours and delays between 10 seconds and 10 minutes:

# regular deployment:
firebase deploy --only functions --force
# update pubsub subscription for retries with exponential backoff:
gcloud pubsub subscriptions update \
 projects/$GCP_PROJECT_ID/subscriptions/gcf-myFunction-europe-west1-my-topic \
 --min-retry-delay=10s \
 --max-retry-delay=10m \
 --message-retention-duration=2h
Enter fullscreen mode Exit fullscreen mode

Note that --force is needed in CI setup to deploy functions with retries.

What are the alternatives?

You could as well

  • manually update the subscription (not recommend)
  • move from firebase cli to a direct deployment on GCP (eg via terraform)

If I missed something or you have other solutions, please comment!

Top comments (4)

Collapse
 
reidab profile image
Reid Beels •

Thanks! This pointed me in the right direction.

I found that by using the built-in $GCLOUD_PROJECT env var, you can add this as a postdeploy hook in firebase.json to ensure it runs with every deploy, like so:

{
  "functions": {
    "postdeploy": [
      "gcloud pubsub subscriptions update projects/$GCLOUD_PROJECT/subscriptions/gcf-myFunction-europe-west1-my-topic --min-retry-delay=10s --max-retry-delay=10m --message-retention-duration=2h"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
ranguna profile image
Luis Pais •

This was spot on my friend, thank you very much for the article!

Now all that's left is a good way to setup DLQs and we are golden 👌

Collapse
 
danielsc profile image
Daniel Schreiber •

This should be possible with some more gcloud commands - would really be interested in a small write up! :-)

Collapse
 
ranguna profile image
Luis Pais •

Hahaha nice baton pass!
I'll see what I can come up with, it might not be as straight forward since we will be creating a new topic and subscription (+ permissions and all that fun stuff), but it should work.
Don't expect it any time soon though xP

Read next

marcsamuel profile image

Top Cloud Security Risks and Challenges

Marc Samuel Pabilona -

ahmedraza_fyntune profile image

Php Reflection

Ahmedraza Fyntune -

mikeyoung44 profile image

AI and the Black Box Problem: How Machine Learning Challenges Mathematical Proof Verification

Mike Young -

learn_with_santosh profile image

Waterfall Model: Simple Breakdown

Santosh Shelar -