Background processing with Google Cloud Functions

Last reviewed : July 2024

Introduction

Everything I've talked about so far in this series has been focused on online activities delivered through an Internet browser. But once things start to get serious and your system begins to take off, there are lots of things you may want to do for which online operation is either inconvenient or downright undesirable.

"Management Information" is a good example - you'll probably want to summarise your accumulated data in a variety of useful ways. While you'll almost certainly want to view this summary online, it would not be sensible to use an online routine to build it. What you need here is some sort of "background processing" capability.

There are also situations where an online transaction will give rise to some sort of "consequential event". An example might be a requirement to send a "welcome" email to a new customer. Email is fiddly stuff and the online routine that signs up a user has got enough to do without getting involved with email as well. Much better to defer the task to a background general-purpose "mail-handler".

The Google "Cloud function" system provides you with a way of creating software modules to deliver such background tasks. Each of these modules takes the form of a self-contained chunk of code, written in Javascript and uploaded to the Cloud via the "deploy" mechanism. They can be built and tested locally using the emulator system.

Depending on the way in which these modules are constructed, they may either be allocated a URL by the deploy procedure, allowing you to launch them from a button or registered with a trigger that enables them to be "fired" by events that occur elsewhere in the system (for example, by the creation of a new document in a particular collection). This is a seriously powerful arrangement. Google refer to the modules as "Cloud functions" because they generally achieve their results by "exporting" a function specifying their actions.

To use them you'll have to upgrade your project to the "Blaze" plan. I know I've previously said that using Firebase is completely free, but this is only true if you stay within certain limits. Cloud Functions take you outside these. However, if you're worried that you might rack up a large bill, fret not, as you can always set a budget limit. My own budget limit is set at £2 and I've still to exceed this. You won't incur any significant costs until your project is generating serious volumes of activity. Also, if at any time you feel uncomfortable with the pay-as-you-go Blaze plan, you can always downgrade it back to the free Spark plan.

Ready to give it a go?  Follow this recipe to create a demo "Cloud function" in the fir-expts-app project that I introduced earlier:

Step 1: Use the CLI to initialise cloud function settings in your project

firebase init functions

One of the first questions that the CLI will ask you is:

What language would you like to use to write Cloud Functions?

It will currently offer you two options - Javascript and Typescript. Typescript has many advantages because of its superior syntax and error-handling capabilities. However, while I'd love to recommend it, I think that, if you're new to these technologies, Javascript would be a better choice. This is because, at the end of the day, your Cloud functions will run in Javascript anyway, and while a CLI deploy command will automatically convert an index.ts file (ie a function written in Typescript) into the equivalent Javascript, this is not the case when you're working with the emulators. You're quite likely to spend a lot of time using these and it's quite easy to get in a mess when running conversions manually. Unless you feel really confident, I suggest you select Javascript.

Back with the CLI, its next question is :

Do you want to use ESLint to catch probable bugs and enforce style?

Eslint will check your code for a few things that may otherwise get missed. But again, if you're just getting started with function development, a bunch of messages warning you about poor code style and similar exotic failings are simply going to be a distraction. So, my recommendation is that you decline this offer too - you can always (and probably should) set up eslint checking later with another init run.

Now answer "yes" to the question:

Do you want to install dependencies with npm now?

and wait for the appearance of a "Firebase initialization complete!" message to assure you that this stage has been successfully completed.

If you now examine your project, you'll find that a functions folder has been added at root. Inside this, alongside many other bits and pieces (these will be particularly numerous if you've opted to use Typescript as this adds additional complications) you'll find an index.js file. This contains a demo Cloud function. Google intend that this particular Function should be triggered by an HTTPS call - the simplest kind. Here's the content of index.js:

const functions = require("firebase-functions/v1");

exports.helloWorld = functions.https.onRequest((request, response) => {
    response.send("Hello from Firebase!");
});

[Last time I tried this (Nov, 2022), Google had actually commented these code lines out. Remove these comment marks for now so that the code can run - this will be really useful for you in a moment as index.js provides a great opportunity to see how everything works.

The use of the /v1 variant of firebase-functions library declares that the code has declined to use the new default firebase-functions library. This default (know as "second generation") employs an exotic Cloud Run configuration and is probably best avoided while you're still finding your feet]

Let's concentrate on the exports.helloWorld statement. The most important features of any Cloud Function are, firstly, the declaration of its "trigger" - the "event" that is to "fires" the Cloud Function and, secondly, the actions that are then to be performed.

The exports.helloWorld statement delivers both of these objectives in one compact expression. In this particular case, since Google intends that this Cloud Function should be triggered by a browser HTTPS call, they've built the Cloud Function around the SDK's https.onRequest method. They've then "loaded" this with a function that simply displays a message, "Hello from Firebase!," on the browser screen.

Finally, Google's exports.helloWorld statement allocates a "name" to the function - "helloWorld" in this case - and "exports" this for external use - more on this in a moment

You need to know more about https.onRequest. This function lets you respond to incoming web requests. When a request occurs, the https.onRequest method makes its request and response parameters available to allow you to interact with the incoming data and to return a response. For example, a parameter supplied by the addition of ?parameter=myParam to the function's url could be recovered by referencing request.query.parameter in the function's code. See Google's Call functions via HTTP requests document for further information.

You also need to know what's behind all this exports. business (and, for that matter, the preceding require statement). You'll surely have guessed by now that the Cloud Function code is being configured as a module. This is because when a function runs in the Cloud, it does so within Node.js, and code executed within Node does so as a module. The index.js code needs to obtain access to the firebase SDK functions and a require statement is the way that it does this. The const functions = require("firebase-functions/v1") statement at the head of the index.js file declares a functions variable and loads the SDK into it.

Because the Cloud Function is a module, its internal variables are invisible unless they are explicitly revealed by an exports. statement. In the demo function, exports.helloWorld makes the Helloworld Cloud function available to the Firebase Cloud control software.

I think it's worth mentioning that this is one area in which Typescript would have made things much neater. Typescript would have allowed you to use the familiar import and export statements you'd have used in a webapp module. However I'm sure you'll soon get used to the Node.js conventions and you'll also find that this is the pattern followed in Google's Firebase documentation.

Step 2 : Test your cloud function

You could in principle now go straight ahead and use the CLI deploy command to upload the demo function into the Cloud, but this wouldn't generally be a good idea. Remember, your project is on a billable Blaze account now, and while you can be pretty sure that this demo function will run without causing you any financial embarrassment this won't always be the case. Testing with the emulators, by contrast, is free. You'll also see, in a moment or two, that the emulators are really simple to use and, additionally, will save you a lot of time and trouble because they allow you to test amended code without first explicitly deploying it.

First off, if you've not used the emulators before, initialise them as follows:

firebase init emulators

and now launch them

firebase emulators:start

This should produce output something like the following:

i  emulators: Starting emulators: functions, firestore, hosting
!  functions: The following emulators are not running, calls to these services from the Functions emulator will affect production: auth, database, pubsub, storage
i  firestore: Firestore Emulator logging to firestore-debug.log
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5000
!  ui: Emulator UI unable to start on port 4000, starting on 4002 instead.
i  ui: Emulator UI logging to ui-debug.log
i  functions: Watching "C:\Users\mjoyc\Dropbox\Versioned Source\VSCodeProjects\firexptsapp\functions" for Cloud Functions...
+  functions[us-central1-helloWorld]: http function initialized (http://localhost:5001/fir-expts-app/us-central1/helloWorld).

???????????????????????????????????????????????????????????????
? ?  All emulators ready! It is now safe to connect your app. ?
? i  View Emulator UI at http://localhost:4002                ?
???????????????????????????????????????????????????????????????

????????????????????????????????????????????????????????????????
? Emulator  ? Host:Port      ? View in Emulator UI             ?
????????????????????????????????????????????????????????????????
? Functions ? localhost:5001 ? http://localhost:4002/functions ?
????????????????????????????????????????????????????????????????
? Firestore ? localhost:8080 ? http://localhost:4002/firestore ?
????????????????????????????????????????????????????????????????
? Hosting   ? localhost:5000 ? n/a                             ?
????????????????????????????????????????????????????????????????
  Emulator Hub running at localhost:4400
  Other reserved ports: 4500

Notice the +  functions[us-central1-helloWorld]: http function initialized (http://localhost:5001/fir-expts-app/us-central1/helloWorld)?

Copy and paste this URL into your browser to get the expected output:

Phew!

Step 3: Write a "serious" function

The demo "shopping list" application I've been using in this series lets users create "shopping lists" detailing purchase intentions. I propose to instrument this with an emailSpecialOffer function that emails users with a "Special Offer" if they add "Bread Buns" to their shopping list - a supremely silly thing to do, but bear with me ....

The emailSpecialOffer function needs to fire whenever a document is created in the userShoppingLists collection. For this purpose, I'll use the SDK's functions.firestore.document().onCreate() function:

Here's a first-cut of the complete Cloud function. For now, I've just added this code to the Google-supplied functions/index.js file introduced above. But see note 3 in the Postscript section below for advice on how you might organise filing arrangements for a whole library of functions.

const functions = require("firebase-functions/v1");
const admin = require("firebase-admin");
admin.initializeApp();

exports.emailSpecialOffer = functions.firestore.document('/userShoppingLists/{documentId}')
    .onCreate((snapShot, context) => {
        if (snapShot.data().userPurchase == "Bread Buns") {
            console.log("User " + snapShot.data().userEmail + " bought special offer item");
        } else {
            console.log("User " + snapShot.data().userEmail + " bought standard item " + snapShot.data().userPurchase);
        }
        return true;
    });

Note that, because I'm now going to be referencing a Firestore collection, I need to import firebase-admin functions and call initializeApp() to authenticate the function. The firebase-admin SDK is a special set of functions used specifically for Cloud function work (see the Firebase Admin SDK for details). Note that I don't need to supply credentials - according to Google's Run functions locally document "Cloud Firestore triggers are automatically granted permission because they are running in a trusted environment". In fact, if I was intending to use Firestore functions on document other than the one that fires the trigger, I'd strictly only need the admin.initializeApp() statement - but it's probably best to get used to using it as a standard practice when working with Firestore.

The body of the function checks for "special offer" purchases. Just now, it simply sends log messages to the console (more on this in a moment) so we can see whether things are roughly working. I'll add the email code later - there's too much else to think about right now.

So, start the emulators again:

firebase emulators:start

To obtain output looking something like the following:

i  emulators: Starting emulators: functions, firestore, hosting
!  emulators: It seems that you are running multiple instances of the emulator suite for project fir-expts-app. This may result in unexpected behavior.
!  functions: The following emulators are not running, calls to these services from the Functions emulator will affect production: auth, database, pubsub, storage
i  firestore: Firestore Emulator logging to firestore-debug.log
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5000
i  ui: Emulator UI logging to ui-debug.log
i  functions: Watching "C:\Users\mjoyc\Dropbox\Versioned Source\VSCodeProjects\firexptsapp\functions" for Cloud Functions...
+  functions[us-central1-emailSpecialOffer]: firestore function initialized.

???????????????????????????????????????????????????????????????
? ?  All emulators ready! It is now safe to connect your app. ?
? i  View Emulator UI at http://localhost:4002                ?
???????????????????????????????????????????????????????????????

????????????????????????????????????????????????????????????????
? Emulator  ? Host:Port      ? View in Emulator UI             ?
????????????????????????????????????????????????????????????????
? Functions ? localhost:5001 ? http://localhost:4002/functions ?
????????????????????????????????????????????????????????????????
? Firestore ? localhost:8080 ? http://localhost:4002/firestore ?
????????????????????????????????????????????????????????????????
? Hosting   ? localhost:5000 ? n/a                             ?
????????????????????????????????????????????????????????????????
  Emulator Hub running at localhost:4400
  Other reserved ports: 4500

So, the emulators are running. How do we make them do something useful? Well, the encouraging +  functions[us-central1-emailSpecialOffer]: firestore function initialized. message included in the CLI output suggests that the emulators at least know about emailSpecialOffer. Our function is triggered by the creation of a document in a userShoppingLists collection. So, let's create a document and see what happens.

Open the emulator UI by keying its http://localhost:4002 URL into the browser and use this to start the Firestore emulator. As described in my previous Using the Firebase emulators post, the userShoppingLists collection doesn't exist yet. OK - just create it. And now add a document with fields userEmail and userPurchase containing whatever data takes your fancy - it doesn't really matter for now.

OK. Done that and nothing appears to have happened. How do you know if the function fired? Well, you'll recall that the function is supposed to post some console messages if it runs. Where on earth are these going to appear? Go back to the Emulator UI's "overview" page and click "view logs" in the "Functions emulator panel". You should see output something like the following:

18:20:00 I  functions Watching "C:\Users\mjoyc\Dropbox\Versioned Source\VSCodeProjects\firexptsapp\functions" for Cloud Functions...
18:20:01 I  functions firestore function initialized.
18:23:00 I  function[us-central1-emailSpecialOffer] Beginning execution of "us-central1-emailSpecialOffer"
18:23:00 I  function[us-central1-emailSpecialOffer] User aa@gmail.com bought standard item firelighters
18:23:00 I  function[us-central1-emailSpecialOffer] Finished "us-central1-emailSpecialOffer" in ~1s

Wow - it worked - "aa@gmail.com" and "firelighters" were the values I put into my userShoppingLists document when creating this example. What I really like about this setup too is that if the function doesn't work, full details of the problems are posted in the logs. And then, when I've corrected my code, all I have to do is re-save it. and return to the emulators. When I create another document, I'll find that the logs are now referencing the updated function. I don't need to restart the emulators to "register" the correction.

This is a huge time saver. By contrast, when you eventually come to deploy a function for live, you'll find that this is a really slow and laborious procedure!

For the mailer procedure I've used Postmark, my favourite emailer, and installed their SDK into the functions folder as follows:

npm install npm install postmark

Here's the completed function :

const functions = require("firebase-functions/v1");
const admin = require("firebase-admin");
const postmark = require("postmark");
admin.initializeApp();

exports.emailSpecialOffer = functions.firestore.document('/userShoppingLists/{documentId}')
    .onCreate((snapShot, context) => {
        if (snapShot.data().userPurchase == "Bread Buns") {
            console.log("User " + snapShot.data().userEmail + " bought special offer item");

            var serverToken = "_my Postmark API server token_";
            var client = new postmark.ServerClient(serverToken);

            try {
                client.sendEmail({
                    "From": "_my validated email despatch address_",
                    "To": snapShot.data().userEmail,
                    "Subject": "Unmissable Special Offer",
                    "TextBody": "_special offer details formatted as html_"
                });
                return true;

            } catch (error) {
                console.log("Error : " + error.ErrorCode + " : " + error.Message);
                return false;
            }

        } else {
            console.log("User " + snapShot.data().userEmail + " bought standard item " + snapShot.data().userPurchase);
            return true;
        }

    });

The reason you need to install Postmark into your 'functions' folder is that when you eventually deploy your code into the Cloud, Firebase will effectively want to 'build' your function index.js and needs to know what components go into it. Installing into the functions folder ensures that its package.json file contains full information about what's included in the folder. Think of the 'functions' folder as a 'project within a project", where your index.js is the only bit that you actually own.

The return statements sprinkled throughout the Cloud function's payload function code are there because there is a requirement that the payload function must always return either a value or a promise. If you neglect to do this, it seems that your function will timeout (and charge you accordingly!)

Please take not of the word "validated" in the despatch email address field. Sadly, the antisocial activities of spam mailers means that organisations like Postmark have to work quite hard to maintain their reputations. How you assure your mailer that your email despatch address is reputable is a deeply technical issue and I'm afraid that it's likely that this is one area where you will probably need to pay for a trusted, commercial ISP.

Postscript

I think this just about wraps things up for this post. But here are one or two final points you might find useful:

1. In case you've not already noticed this, I'd like to point out that, unless you explicitly want to, you don't actually have to run your application in order to test your functions. Also note that you don't use webpack on Cloud functions.

2. To move the function into Production without disturbing any other elements of a Live system you would use the following command:
firebase deploy --only functions

As of Feb 2024, you should note that this command now expects to find a "functions configuration object" in your project's firebase.json. If you're getting "Cannot understand what targets to deploy/serve" errors, add the following to firebase.json (https://github.com/firebase/firebase-tools/issues/6672 refers)

{
 "functions": [
   {
     "source": "functions",
     "codebase": "default",
     "ignore": [
       "node_modules",
       ".git",
       "firebase-debug.log",
       "firebase-debug.*.log"
     ]
   }
 ]
}

1. You might be wondering how you would organise yourself to instrument your application with a whole library of functions. The answer is that you can implement them all in a single functions/index.js file, but management might become an issue if there were too many  - see Organise Multiple Functions for advice.

2. Having thought about things a little, you might wonder why you need to run your Postmark in a function at all. Don't they have an API like Firestore's that would allow you to make mailing requests direct from your Javascript? The answer is that no they don't because this would potentially expose your Postmark key. So, there's no alternative to using a Firebase function here. If you're also wondering whether an https function trigger might be more convenient than a database-initiated oncreate, you'll find that the parameterisation requirements of url calls make these a really nasty way of passing input data. Google has provided the onCall function to overcome this problem (see the next post in this series - 3.4 Getting serious with Firebase V9 - Cloud Storage: Code Patterns for File Upload, Reference, Delete, Fetch and Download - for an example. That said, https functions are a good way of building functions that are designed to be called by the Cloud Scheduler - backup scripts, for example.

3. You might also wonder how you would go about writing a function that performed some sort of traditional grunt background database-processing. Generating reports and management information would be good examples. The answer here, is that you would code these using pretty much the same pattern of Firestore functions you've been using so far. But there's one important difference. So far, all the code you've seen has been designed to run in a Firestore v9 webapp. Your functions, however, run in a Node.js environment and here things are all subtly different. What you need to do is go back to Google's examples and check the "Node.js" header rather than the "Web" heading for each. See the end of the 4.2 post for an example featuring a document deletion routine.

4. This post has really just scratched the surface of what Cloud functions can do for you and what you need to know in order to use them safely in a serious application. For more information I recommend the Google video series starting at How do Cloud Functions work? | Get to know Cloud Firestore #11.

Other posts in this series

If you've found this post interesting and would like to find out more about Firebase you might find it worthwhile having a look at the Index to this series.