DEV Community

Cover image for Core FQL concepts part 2: Temporality in FaunaDB
Fauna, Inc. profile image Fauna
Fauna for Fauna, Inc.

Posted on • Originally published at fauna.com

Core FQL concepts part 2: Temporality in FaunaDB

#serverless #fauna

Author: Pier Bover
Publish Date: Sep 16, 2020

Today we're going to explore one of FaunaDB's most unique features: its temporal capabilities.

This series assumes you have a grasp on the basics. If you're new to FaunaDB and/or FQL here's my introductory series on FQL.

In this article:

  • Document history
  • Querying document history
  • Filtering history events
  • Travelling back in time
  • Recovering documents
  • Careful with the spacetime continuum
  • Use cases and considerations

Document history

One of the main temporal features of FaunaDB is being able to record every change you make to a document. When this is configured, new document snapshots are created for every change and added to the document's history.

By default, when creating a collection we get 30 days of minimum document history:

CreateCollection({name: "MyCollection"})

{
ref: Collection("MyCollection"),
ts: 1598891726415000,
history_days: 30,
name: "MyCollection"
}
Enter fullscreen mode Exit fullscreen mode

History will last for at least 30 days with the default history_days value, but there are no guarantees when the deletion occurs as FaunaDB periodically reclaims expired snapshots.

We can change the minimum default to any number of days that fits our use case:

CreateCollection({
  name: "CollectionWithAYearOfHistory",
  history_days: 365
})
Enter fullscreen mode Exit fullscreen mode

If we wanted to retain all history until the end of times we'd just use a null value:

CreateCollection({
  name: "CollectionForever",
  history_days: null
})
Enter fullscreen mode Exit fullscreen mode

It's also possible to expire all history by default by passing a 0 value:

CreateCollection({
  name: "CollectionNoHistory",
  history_days: 0
})
Enter fullscreen mode Exit fullscreen mode

Difference with ttl_days

When creating a new collection it's also possible to configure ttl_days (time-to-live days). This setting is not related to its history. Instead, it refers to how long you'd like to keep the current or active version of a document alive. This is most useful for ephemeral data such as sessions, caches, and so on.

Check the documentation for more details.

Security implications

It should be noted that having permissions to read a document doesn't automatically grant permission to read or modify its history. Your users will need dedicated permissions to be able to interact with the history of a document.

For more info on authorization in FaunaDB check my previous article.

Querying document history

Before we go on, let's create some data:

> CreateCollection({name: "Lasers"})

> Create(
  Collection('Lasers'),
  {
    data:{
      color:"BLUE",
      crystal: "KYBER"
    }
  }
)

{
  ref: Ref(Collection("Lasers"), "275571113732342290"),
  ts: 1599063943620000,
  data: {
    color: "BLUE",
    crystal: "KYBER"
  }
}
Enter fullscreen mode Exit fullscreen mode

Let's also modify the document we just created to get some history:

> Update(
  Ref(Collection("Lasers"), "275571113732342290"),
  {data:{color:"GREEN"}}
)

> Delete(
  Ref(Collection("Lasers"), "275571113732342290")
)
Enter fullscreen mode Exit fullscreen mode

We can query a document's history by using the Events() function. Here's an example of what this history looks like:

> Paginate(
  Events(Ref(Collection("Lasers"), "275571113732342290"))
)

{
  data: [
    {
      ts: 1599063943620000,
      action: "create",
      document: Ref(Collection("Lasers"), "275571113732342290"),
      data: {
        color: "BLUE",
        crystal: "KYBER"
      }
    },
    {
      ts: 1599064033700000,
      action: "update",
      document: Ref(Collection("Lasers"), "275571113732342290"),
      data: {
        color: "GREEN"
      }
    },
    {
      ts: 1599064043550000,
      action: "delete",
      document: Ref(Collection("Lasers"), "275571113732342290"),
      data: null
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

As you can see, Events() takes a document reference and returns a list of all the changes we've performed on that document. An important point is that this history is available even after the document has been deleted.

Instead of using Events(), we could also use the options of Paginate() to retrieve the history of multiple documents at once:

> Paginate(
  Documents(Collection("Lasers")),
  {events: true}
)

{
  data: [
    {
      ts: 1599063943620000,
      action: "add",
      document: Ref(Collection("Lasers"), "275571113732342290")
    },
    {
      ts: 1599064043550000,
      action: "remove",
      document: Ref(Collection("Lasers"), "275571113732342290")
    },
    // etc...
  ]
}
Enter fullscreen mode Exit fullscreen mode

In this example, we are getting all the history of all the documents in the Lasers collection by combining Documents() with {events: true}.

This approach is extremely powerful as you could, for instance, combine multiple indexes to fetch the history of documents based on complex filters, or even from documents from multiple collections. Definitely check out the Paginate() documentation for more info on this.

Filtering history events

When querying the history of a document by using Events(), we get an array which can be manipulated just like any other array in FaunaDB.

For example here's how we could just get the update events:

> Filter(
  Select(
    ["data"],
    Paginate(Events(Ref(Collection("Lasers"), "275571113732342290")))
  ),
  Lambda(
    "event",
    Equals(
      Select(["action"], Var("event")),
      "update"
    )
  )
)

[
  {
    ts: 1599064033700000,
    action: "update",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: {
      color: "GREEN"
    }
  }
]
Enter fullscreen mode Exit fullscreen mode

We could also filter events by date:

> Filter(
  Select(
    ["data"],
    Paginate(Events(Ref(Collection("Lasers"), "275571113732342290")))
  ),
  Lambda(
    "event",
    Equals(
      ToDate(
        Epoch(
          Select(["ts"], Var("event")),
          "microseconds"
        )
      ),
      Date('2020-09-02')
    )
  )
)

[
  {
    ts: 1599063943620000,
    action: "create",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: {
      color: "BLUE",
      crystal: "KYBER"
    }
  },
  {
    ts: 1599064033700000,
    action: "update",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: {
      color: "GREEN"
    }
  },
  {
    ts: 1599064043550000,
    action: "delete",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: null
  }
]
Enter fullscreen mode Exit fullscreen mode

Quick note: We explored working with dates and times in the previous article. As a quick reminder, we're converting an integer to a Timestamp using Epoch(), and then that timestamp to a Date, to be able to compare it with Date('2020-09-02').

We could also use the Paginate() options to get the events after or before a certain date:

> Paginate(
  Documents(Collection("Lasers")),
  {
    events: true,
    before: Date("2020-09-03")
  }
)

> Paginate(
  Documents(Collection("Lasers")),
  {
    events: true,
    after: Date("2020-09-03")
  }
)
Enter fullscreen mode Exit fullscreen mode

Travelling back in time

The document history tells us that something happened at some point in time. What if we wanted to know the complete state of a document in the past? For example, after an update event or even in between history events?

We can use the At() function to do exactly that:

> At(
  1599064033700000,
  Get(Ref(Collection("Lasers"), "275571113732342290"))
)


{
  ref: Ref(Collection("Lasers"), "275571113732342290"),
  ts: 1599064033700000,
  data: {
    color: "GREEN",
    crystal: "KYBER"
  }
}
Enter fullscreen mode Exit fullscreen mode

Much like a time machine, At() allows us to execute FQL queries in the past. Here we're using the 1599064033700000 timestamp from the update event to get the full document when we updated the color to GREEN.

At() sets the point in time for all reads in FQL for that query. We can use indexes, filter data, etc. It's exactly like querying FaunaDB at that time.

It doesn't work exactly like a time machine though. We can't really write changes in the past with it. If we try to use At() to make any changes at any other time than Now() we will get an error.

Let's create a new document to try this out:

> Create(
  Collection('Lasers'),
  {data:{color:"RED"}}
)

{
  ref: Ref(Collection("Lasers"), "275572558214988288"),
  ts: 1599065321160000,
  data: {
    color: "RED"
  }
}
Enter fullscreen mode Exit fullscreen mode

And now let's add a couple of milliseconds to the creation timestamp and try to update the document in the past:

> At(
  1599065321170000,
  Update(
    Ref(Collection("Lasers"), "275572558214988288"),
    {data: {color: "PURPLE"}}
  )
)

Error: [
  {
    "position": [
      "expr"
    ],
    "code": "invalid write time",
    "description": "Cannot write outside of snapshot time."
  }
]
Enter fullscreen mode Exit fullscreen mode

Altering the spacetime continuum

While we cannot use At() to alter the past, it is possible to use Insert() and Remove() to modify the events in a document's history.

To demonstrate this, let's create a new document first:

> Create(
  Collection('Lasers'),
  {
    data:{
      color:"ORANGE",
      crystal: "QUARTZIUM"
    }
  }
)

{
  ref: Ref(Collection("Lasers"), "275678520646042112"),
  ts: 1599166374745000,
  data: {
    color: "ORANGE",
    crystal: "QUARTZIUM"
  }
}
Enter fullscreen mode Exit fullscreen mode

Now let's see what happens if we insert a create event 1 second before we create that document:

> Insert(
  Ref(Collection("Lasers"), "275678520646042112"),
  TimeSubtract(Epoch(1599166374745000, "microseconds"), 1, "second"),
  "create",
  {
    data: {
      color: "YELLOW",
      crystal: "QUARTZIUM"
    }
  }
)

{
  ts: 1599166373745000,
  action: "create",
  document: Ref(Collection("Lasers"), "275678520646042112"),
  data: {
    color: "YELLOW",
    crystal: "QUARTZIUM"
  }
}
Enter fullscreen mode Exit fullscreen mode

Keep in mind that the history is related to a document reference. Be careful not to use the wrong reference with Insert() and Remove(), otherwise these changes will have unintended consequences.

Let's see how the insertion of the create event affected the history of the document:

> Paginate(Events(Ref(Collection("Lasers"), "275678520646042112")))

{
  data: [
    {
      ts: 1599166373745000,
      action: "create",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "YELLOW",
        crystal: "QUARTZIUM"
      }
    },
    {
      ts: 1599166374745000,
      action: "update",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "ORANGE"
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

The new create event we just inserted is where we expect it to be, one second before the initial document creation, but look what happened with the last event in the array. It used to be a create event but it has been now converted to an update event.

In any case, if we now get the latest version of the document we can see it is unchanged:

> Get(Ref(Collection("Lasers"), "275678520646042112"))

{
  ref: Ref(Collection("Lasers"), "275678520646042112"),
  ts: 1599166374745000,
  data: {
    color: "ORANGE",
    crystal: "QUARTZIUM"
  }
}
Enter fullscreen mode Exit fullscreen mode

Recovering deleted documents

I'm sure I'm not the only one that has wanted to go back in time to restore something that was deleted by accident. FaunaDB offers a couple of ways to do this.

Removing the delete event

We've seen that deleting a document creates a delete event. Could we use Remove() to undo that?

First let's delete our orange laser:

> Delete(Ref(Collection("Lasers"), "275678520646042112"))
Enter fullscreen mode Exit fullscreen mode

Now let's get all the delete events for this document:

> Filter(
  Select(
    ["data"],
    Paginate(Events(Ref(Collection("Lasers"), "275678520646042112")))
  ),
  Lambda(
    "event",
    Equals(Select(["action"], Var("event")),"delete")
  )
)

[
  {
    ts: 1599177868380000,
    action: "delete",
    document: Ref(Collection("Lasers"), "275678520646042112"),
    data: null
  }
]
Enter fullscreen mode Exit fullscreen mode

Cool, so let's use Remove() to remove the delete event:

> Remove(
  Ref(Collection("Lasers"), "275678520646042112"),
  1599178388870000,
  "delete"
)
Enter fullscreen mode Exit fullscreen mode

We can confirm the removal of the delete event by checking its history:

> Paginate(Events(Ref(Collection("Lasers"), "275678520646042112")))

{
  data: [
    {
      ts: 1599166373745000,
      action: "create",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "YELLOW",
        crystal: "QUARTZIUM"
      }
    },
    {
      ts: 1599166374745000,
      action: "update",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "ORANGE"
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

We now can even Get() the document as usual:

Get(Ref(Collection("Lasers"), "275678520646042112"))

{
  ref: Ref(Collection("Lasers"), "275678520646042112"),
  ts: 1599166374745000,
  data: {
    color: "ORANGE",
    crystal: "QUARTZIUM"
  }
}
Enter fullscreen mode Exit fullscreen mode

Bringing data back from the past

Another option to recover data is basically reading it from the past using At() and inserting it again back into the present.

Let's create a new laser to start with a fresh document:

> Create(
  Collection('Lasers'),
  {
    data:{
      color:"BLUE"
    }
  }
)

{
  ref: Ref(Collection("Lasers"), "275752290343715347"),
  ts: 1599236727185000,
  data: {
    color: "BLUE"
  }
}
Enter fullscreen mode Exit fullscreen mode

Let's delete our document and check its history:

> Delete(Ref(Collection("Lasers"), "275752290343715347"))

> Paginate(Events(Ref(Collection("Lasers"), "275752290343715347")))

{
  data: [
    {
      ts: 1599236727185000,
      action: "create",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: {
        color: "BLUE"
      }
    },
    {
      ts: 1599236793325000,
      action: "delete",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: null
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

We could now use the timestamp of the create event with At() to read it:

> At(
  1599236727185000,
  Get(Ref(Collection("Lasers"), "275752290343715347"))
)

{
  ref: Ref(Collection("Lasers"), "275752290343715347"),
  ts: 1599236727185000,
  data: {
    color: "BLUE"
  }
}
Enter fullscreen mode Exit fullscreen mode

And finally restore the data before deletion, by simply creating a document with the same reference and data:

> Let(
  {
    deletedDoc: At(
      1599236727185000,
      Get(Ref(Collection("Lasers"), "275752290343715347"))
    ),
    ref: Select(["ref"], Var("deletedDoc")),
    data: Select(["data"], Var("deletedDoc"))
  },
  Create(
    Var("ref"),
    { data: Var("data") }
  )
)

{
  ref: Ref(Collection("Lasers"), "275752290343715347"),
  ts: 1599237181566000,
  data: {
    color: "BLUE"
  }
}
Enter fullscreen mode Exit fullscreen mode

Quick note: Let() not only allows us to format output objects but also execute any output query like we're doing here after having collected the necessary data.

Let's check what happened to the document's history:

Paginate(Events(Ref(Collection("Lasers"), "275752290343715347")))

{
  data: [
    {
      ts: 1599236727185000,
      action: "create",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: {
        color: "BLUE"
      }
    },
    {
      ts: 1599236793325000,
      action: "delete",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: null
    },
    {
      ts: 1599237181566000,
      action: "create",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: {
        color: "BLUE"
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

As you can see, even though we created a new document, the new create event has been appended to the previous history for that reference. This is happening because the history is actually associated with a document reference.

Use cases and considerations

Now that we've seen an overview of the temporal features in FaunaDB let's examine a couple of common use cases in more detail.

Safe delete

It's very common in databases without history or versioning to follow a couple of patterns to prevent data loss. One of those patterns marks data as deleted but keeps it around in its original place. This practice is usually called "soft delete". Another alternative is to have a "recycle bin", that's a table or a collection where you move the data to be deleted instead of actually deleting it.

Both of these patterns have pros and cons, and you can certainly implement them with FaunaDB. Although, unless you need specific features, the simplest option is certainly to delete documents as usual and use the temporal features we just saw to keep deleted data around for as long as you need it.

There are two major points to consider though:

  1. Fauna stores every single document change in its history, which occupies storage space. It might be overkill for simple safe delete at scale if you don't need document versioning.
  2. Since querying the history returns an array of events, you won't have as much flexibility as you would have by storing the deleted documents in a dedicated collection. For example, by using a "recycle bin" collection approach, you could use indexes and filter deleted documents by user, deletion reason, etc.

Version control

Another major use case for the temporal features of FaunaDB is storing and keeping track of content changes. Let's say you're working on SpaceDocs, the next app for collaborative writing of spacey documents. You obviously need a version history feature to keep track of changes.

Storage shouldn't be a concern here, since you would be storing all of the versions anyway, say in a DocVersions collection. But, again, if you used the history to keep older versions around you wouldn't be able to use common FQL features such as indexes. Maybe this would limit the kind of features you can build in the future.

Again, it all depends on your use case.

Conclusion

So that's it for today. Hopefully you learned something valuable!

In the following article of the series, we will continue our space adventure by checking on aggregation queries and other features.

If you have any questions don't hesitate to hit me up on Twitter: @pierb

Oldest comments (0)

Read next

abhirockzz profile image

[20 Days of DynamoDB] Day 20 - Converting between Go and DynamoDB types

Abhishek Gupta -

rexben001 profile image

Understanding Transactions in Amazon DynamoDB

Benjamin Ajewole -

shosenwales profile image

Building scalable apps with serverless PostgreSQL & Django

Shuaib Oseni -

abhirockzz profile image

[20 Days of DynamoDB] Day 11 - Using pagination with Query API

Abhishek Gupta -