loading...

Using FaunaDB for Live Counters in deletemydata.io

webhorcrux profile image Praveen ・8 min read

When users browse the web, they want to know that the content they are reading is reliable and trustworthy. In 2009, facebook launched the “share” button for publishers which showed analytics on articles for how many times the article has been shared on facebook. This was quickly followed by twitter with their own “tweet” button in 2010. And for the next few years, several more “share” buttons popped up like one from Pinterest for “pinning”. The key reason for publishers to adopt these “share” buttons is to provide a sense of trust back to the visitor on the site that others have read and found the content useful. It is the herd mentality that if many before you have read this then something must be right here.

deletemydata.io aims to offer a single reliable place on the web to find out how to delete anything. In order to increase reliability and the trust factor among visitors, I wanted to adopt the same growth strategy - show a live counter of users who have found the content valuable. And they tell content is useful by clicking a simple Yes/ No button for Was this helpful? At the bottom of the page.

In this article, I’m going to share how I implemented this live counter using my existing tech stack leveraging FaunaDB in a simple and efficient manner. To start with, it will help to understand deletemydata.io's tech stack first.

deletemydata.io tech stack

The three pillars of my stack are:

  • Netlify
  • React-Static
  • Imgix - Hosting images

Netlify is the best platform I’ve seen to date to build websites if you are a developer. React-Static is a static site generator that adopts JAMStack principles. JAMStack is the terminology used for pre-rendering files and serving them via a CDN without the need for having a backend server. It has a lot of advantages to the traditional way of using servers to build and render the same page over and over again.

Options For Implementing A Counter

There are several ways to implement a live counter. Some of the common ways are:

  • Using the facebook share toolbar i mentioned above.
  • Using redis and update

With the fb share toolbar, it's extremely simple to implement. But you don’t have control over the design of the toolbar itself and you’d need to share data of your visitors with facebook. This opens my site up to support GDPR and CCPA legislations for a user.

With redis, you’ll have control over the design unlike the toolbar. When you are setting up a new system, it's a lot of time consuming operational work - evaluating between GCP and AWS., opening up the system to internet access, adding a security layer on top is not trivial so that it's not abused etc.

There was this third option I ran into that was more friendly to my existing stack - FaunaDB. Although it was something I wasn’t familiar with early, on reading about Netlify’s add-on support for FaunaDB and its support for temporality natively, I decided it was worth looking into.

What's temporality

Temporality is the concept of offering retention for a piece of data. FaunaDB offers this functionality by supporting ttl (time to live) for each document you create. So now the collection is simply a journal with time stamped entries leveraging FaunaDB’s native ability to enforce retention on it. Each entry would look similar to the one below:

id record ttl
1 {pageid: xxxxxx} 30 days

The record above is the document that would be added. id and ttl are illustrated just to show how temporality would work.

I’d like to simply provide the ability to tell my users - how many before you have found the info reliable and have deleted this account in the last month. So, if for each page on the site, I have entries for users who find the page useful with an entry timestamp and combined that with a retention period of a month, I technically should be able to get # of users who have found this page useful in the last month.

This support was quite important for deletemydata.io. Any time you have content on a site, it’s important to keep it relevant. As a new user it gives me more confidence in the site when I know that the information is not stale. In this case, we share metrics showing that several others have deleted the same account you are looking for very recently.

With the options considered, FaunaDB had more to offer than the rest. It showed a lot of promise for me to quickly try out a POC to validate.

Getting Started with Netlify and Fauna

There are several resources out there for integrating FaunaDB with your netlify app. I’ll link the ones I used at the bottom.

Step 1: Install netlify cli

npm install netlify-cli -g

Step 2: Create fauna db instance for the site

netlify addons:create fauna

Add-on "fauna" created for nifty-8fce5d

Step 3: Create account with fauna

netlify addons:auth fauna

Opening fauna add-on admin URL:
https://dashboard.fauna.com/#auth={auth_token}

Fauna has an Oauth integration with Netlify. This is nice since you don’t have to create another account and can just sign in with Netlify.

Fuana-Netlify Oauth Integration

Once you authorize it, netlify will “import” a db for you into your fauna account.

netlify import db into fauna

Although nothing is present in your db, you’ll have keys setup for access via the netlify app.

secrest keys in netlify app

These keys will be injected automatically by netlify as an addon env variable. You can see this when you netlify dev command in your local environment.

netlify dev
◈ Netlify Dev ◈
◈ Injected addon env var:  FAUNADB_ADMIN_SECRET
◈ Injected addon env var:  FAUNADB_SERVER_SECRET
◈ Injected addon env var:  FAUNADB_CLIENT_SECRET

Before diving too deep into how we’ll write code in netlify to talk to fauna, I wanted to get a feel for fauna as a language, semantics and how this would look like in fauna. Fauna shell was an awesome place for that. You can see the fauna shell in the dashboard on the web or have it run locally.

Architecture

Each page in deletemydata.io has a unique slug. For a live counter, we want to have a counter per page that will also take into account that each counter lasts only for 30 days. 30 days is arbitrary, not too short to keep the counter values low and not too long to give the user a bad impression for stale content.

While I share with you the structure of the db in fauna, I’ll also share how I used the fauna shell to create and test this at the same time.

Data Model

data model

Single table with a single field - pageid containing a ttl of 30days for each entry.

We’ll call this collection (or table) deletes:

CreateCollection({ name: "deletes" }

We want the client to do two things:

  • Create an entry
  • Fetch the count

If you want to add a document into the collection, it's pretty straightforward. But we also want to have a ttl on this entry so that temporality can take effect

Create(Collection("deletes"), {
  data: {
    pageid: "test-1"
  },
  ttl: TimeAdd(Now(), 30, "days")
}

Each entry is a counter by itself. This will suffice for the first use case assuming FaunaDB adheres to its ttl for the document.

In order to support, fetching per pageid, we need to create an index for faster lookup.

CreateIndex({
  name: "deletes_by_pageid",
  source: Collection("deletes"),
  unique: false,
  terms: [{ field: ["data", "pageid"] }]
})

So now when you issue a count query for the pageid, we’ll get back the count of existing documents that match this value

Count(Match(Index("deletes_by_pageid"), "test-1"))

Note that using a count function is a risky proposition since if you have a large set of documents you could exceed the transaction limit of 30 seconds. It works a good starting point given that all documents are short lived to only be alive for 30 days.

After a few tests on documents with shortened ttls, there was enough confidence that this would work for this use case. One concern that might come with this is how fast it will count since we are creating a new document for each page id per feedback (this is how we know a user has found this valuable). But since we have an index on this document for the field, lookups were quite fast.

Netlify function

Now that we were able to test how things will look like with FaunaDB, I moved to implement the same with the app. In netlify, per JAMStack principles, although you don’t have a backend, you have access to run serverless lambda functions that your client can call.

Creation Flow

Here’s how the data flow for creation looked like

User ===> Clicks feedback-YES ===> Call deletes-create ===> Create a document

Client Code:

const faunadb = require('faunadb')

/* configure faunaDB Client with our secret */
const q = faunadb.query
const client = new faunadb.Client({
  secret: process.env.FAUNADB_SERVER_SECRET
})

/* export our lambda function as named "handler" export */
exports.handler = (event, context, callback) => {
  /* parse the string body into a useable JS object */
  const data = JSON.parse(event.body)
  console.log('Function `deletes-create` invoked', data)
  const item = {
    data: data,
    ttl: q.TimeAdd(q.Now(), 30, "days")
  }
  /* construct the fauna query */
  return client
    .query(q.Create(q.Collection("deletes"), item))
    .then(response => {
          console.log('success', response)
          /* Success! return the response with statusCode 200 */
          return callback(null,{
                statusCode: 200,
                body: JSON.stringify(response)
          })
    })
    .catch(error => {
          console.log('error', error)
          /* Error! return the error with statusCode 400 */
          return callback(null,{
                statusCode: 400,
                body: JSON.stringify(error)
          })
    })
}

Counter Flow

During rendering, the page will make a call to fetch the count from fauna.

Client Code:

const faunadb = require('faunadb')

const q = faunadb.query
const client = new faunadb.Client({
  secret: process.env.FAUNADB_SERVER_SECRET
})

/* export our lambda function as named "handler" export */
exports.handler = (event, context, callback) => {
  /* parse the string body into a useable JS object */
  console.log("Function `deletes-count` invoked")

  if(event && event.queryStringParameters && event.queryStringParameters.pageid) {
    /* construct the fauna query */
    return client.query(q.Count(q.Match(q.Index("deletes_by_pageid"), event.queryStringParameters.pageid)))
    .then((response) => {
          console.log("success", response)
          /* Success! return the response with statusCode 200 */
         return callback(null, {
               statusCode: 200,
               body: JSON.stringify(response)
         })
    }).catch((error) => {
         console.log("error", error)
        /* Error! return the error with statusCode 400 */
        return callback(null, {
              statusCode: 400,
              body: JSON.stringify(error)
        })
    })
  }
  return callback(null, {
    statusCode: 400,
    body: JSON.stringify("No query parameter pageid found")
  })
}

Production

Since the launch of the two functions, the response times are under 20ms for both creation and count queries. Several pages have already been counted several hundred times as being relevant by users. Here is a video of this in production: https://www.youtube.com/watch?v=AdTN0KYNz4A

Conclusion

FaunaDB is incredibly easy to use with netlify and simple to integrate with. It just took little over half a day to get this to production with ease. I’m sure that this architecture will need to evolve as pages gain traction to keep meeting strict SLAs. One way to do that will be to pre-aggregate values and store them. I’m surprised that this is able to perform as well without a cache in front of it. It’s awesome to see databases support temporality natively out of the box. It is such a time saver that goes great with such an expressive, easy to read programming language.

Resources:

Discussion

pic
Editor guide