DEV Community

loading...
Cover image for How to Use the Firestore Export API Before it is Released

How to Use the Firestore Export API Before it is Released

christianalexander profile image Christian Alexander ・3 min read

TL;DR

Check out FirestoreRestore.
It uses a brand new API to backup and restore your Firestore database!

Background

There have been many requests for Google to add backup and restore functionality to their Cloud Firestore offering. Until this week, the only method for backing up a Firestore database has been to run utilities that traverse the tree as an admin client, saving each document's JSON representation on disk.

Discovery

This morning, I opened the Google APIs Explorer and found two new methods listed under Firestore v1beta1: exportDocuments and importDocuments.

APIs Explorer Screenshot

Exploration

As the API documentation states, firestore.projects.databases.exportDocuments is able to export Firestore data to Google Cloud Storage.

To test this out, I opened up the Firebase Console and created a new project. In order for there to be data to back up, I created a user document from the console.

Creating a document

In the same project, I opened the Google Cloud Storage Console and created a bucket in which to store my backups.

From the Google APIs Explorer's exportDocuments page, I entered the Firebase project name, as well as a path to the storage bucket, and hit Execute.

Execute!

The following is the API's response:

{
 "name": "projects/firestore-restore/databases/(default)/operations/<REDACTED>",
 "metadata": {
  "@type": "type.googleapis.com/google.firestore.admin.v1beta1.ExportDocumentsMetadata",
  "startTime": "2018-07-18T23:20:26.535130Z",
  "operationState": "PROCESSING",
  "outputUriPrefix": "gs://firestore-restore/backups/2018-07-18"
 }
}
Enter fullscreen mode Exit fullscreen mode

A few moments later, I opened the bucket. Sure enough, the data was present!

Backup in the bucket

Restore

Now, to test the restore functionality. I deleted the contents of the Firestore database, and ran importDocuments with the same parameters as the export request, and sure enough, the data came back!

Is the Backup Complete?

Unfortunately, there is currently no "documented" way to check the status of a Firestore export in progress. Luckily, with a little URL guessing, I was able to find the endpoint for the operation resource. It was there all along; the name field in the export response is the status path!

A Temporary Tool: FirestoreRestore

As I don't know when Google will officially support and build tooling for backing up and restoring Cloud Firestore databases, I developed a simple command line utility to interact with the service.

It requires the creation of a service account with the "Cloud Datastore Import Export Admin" role. With a JSON key for the service account, this tool is able to perform a backup and restore.

Check out the README for more info πŸ˜€

Speculation

Google will likely launch backup and restore functionality within the Firebase UI. It'll probably expose the per-collection backup capability that is present in the API. It'll probably be much easier to use than some command line program you found on the internet.

Until then, enjoy the CLI!

Discussion (6)

pic
Editor guide
Collapse
dustinsoftware profile image
masters.πŸ€”

Thanks for the write up. How hard would it be to leverage this API to do offsite backups (in case you wanted to migrate somewhere else for instance)

Collapse
christianalexander profile image
Christian Alexander Author

Thanks for the question! It should be trivial to download the export files from the storage bucket and store them elsewhere. I'm not sure if the API would support an external destination directly (S3, for instance), but my guess is that gs:// URLs are the only supported destinations.

Migration may be a bigger issue. It appears that the files generated by this API are in the same format as Google Cloud Datastore entity exports, which appear to be proprietary. The decision to use Firebase comes with a ton of vendor lock-in, so this is not surprising.

If anyone is looking to export their data for use in another system, one of the tree traversal scripts found on NPM would likely be their best bet. Most of them are capable of producing JSON files that could be consumed by other systems.

Collapse
bpkennedy profile image
Brian Kennedy

The bucket doesn't support downloading folders (which is what this CLI produces in your bucket), however the google storage transfer service supports transferring to A3 and I think some Url of your own, too.

Collapse
bpkennedy profile image
Brian Kennedy

Thanks so much for sharing this. I also used this in tandem with the google storage transfer service to move backups to other projects (a la dev/staging). It's working great!

Collapse
newmesiss profile image
Luis Ruiz

Do you know if it is now possible to download the entire folder created when exporting? to save it on my computer

Collapse
christianalexander profile image
Christian Alexander Author

The files should be downloadable from the Google Cloud Storage bucket. I'm not sure about the file format or how to use the data after downloading.