loading...
Lambda Store

Much Swifter Than FaunaDB: Lambda Store - Serverless Redis

mattiabi profile image Mattia Bianchi ・10 min read

In our previous performance comparison post, Swifter Than DynamoDB: Lambda Store - Serverless Redis, we had compared latency characteristics of DynamoDB and Lambda Store using different types of operations, namely; insertion, retrieval, update and overwrite/replace.

In this post I will compare latency performance of Lambda Store with another notable serverless database; FaunaDB. Note that, this comparison is only for serverless offering of FaunaDB, please don't confuse with their on-premise deployments or Enterprise plan.

ℹ️ All benchmark code is open at my Github repo. I am open to all suggestions, reviews and even pull-requests to the benchmark code.

Before We Begin...

I am aware of the differences in architecture and consistency guarantees between FaunaDB and Redis. FaunaDB is based on the Calvin and supports distributed ACID transactions and strong isolation levels such as serializability, which Redis lacks of. Also, it's not very clear from their documentation but due to my understanding their serverless offering replicates the data between different regions of AWS and/or GCP. Lambda Store currently supports replication in the single region of AWS where Redis database is created, does not support multi-region replications yet.

These strong consistency features of FaunaDB come with significant latency cost which it's observed in the benchmark results below.
While building Lambda Store, initially we preferred low latency over stronger consistency, but we are planning to add configurable & stronger consistency modes.

Benchmark

In the benchmark, I used a single type of struct with a bunch fields, Product.

type Product struct {
    Id           int     `json:"pid"`
    CreationTime int64   `json:"creation_time"`
    UpdateTime   int64   `json:"update_time"`
    Name         string  `json:"name"`
    Description  string  `json:"description"`
    Category     string  `json:"category"`
    Count        uint64  `json:"count"`
    Price        float64 `json:"price"`
    Seller       string  `json:"seller"`
    SellerId     int     `json:"seller_id"`
    Country      string  `json:"country"`
    City         string  `json:"city"`
}

Scenarios

I ran four different test scenarios and with different parallelism settings and measured latency histogram of both products.
In the all tests, each parallel goroutine executes the benchmark method continuously in a loop for a minute.
I recorded the latency of each request in a histogram. Latency is measured as the elapsed time between sending a request and receiving its response.

Test scenarios are:

  • Read Test: Some number of Product objects are inserted initially. Then all fields of these objects are read.
  • Insertion Test: Unique Product objects are inserted to an initially empty database.
  • Overwrite Test: Some number of Product objects are inserted initially. Then all fields of these objects are overwritten.
  • Update Test: Some number of Product objects are inserted initially. Then only a few fields of these objects are updated.

⚡Number of throughput values are not strictly comparable, because we are testing different systems with different client implementations. Still, it makes sense to compare them, since there are very significant differences in most of the scenarios.

Setup & Environment

  • AWS eu-west-1 Europe (Ireland) region was used to execute client application and for Lambda Store. FaunaDB does not provide any configuration parameter to define region settings.
  • Machine type to run benchmark code was m6g.xlarge.
  • Operating system was Ubuntu Server 18.04 LTS 64-bit Arm version.
  • Go runtime was 1.14.3.
  • FaunaDB Go Driver version was v2.0.0.
  • Go Redis client version was 7.2.0.
  • Benchmarks were run with 1, 10 and 100 parallel goroutines.

⚡ In FaunaDB, Id field of the Product struct is indexed as unique to be able to access without knowing ref of FaunaDB.

Results

ℹ️ You can find all of the benchmark histogram files under results directory at my Github repo.

Read Benchmark

In the read benchmark, 1000 Product objects are inserted initially. Then all fields of randomly selected objects are read continuously. At FaunaDB document is accessed using the unique index. At Lambda Store all fields of a Redis hash are read.

Lambda Store has significantly lower latency and higher throughput values than FaunaDB in all parallelism settings.

Parallelism 1

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     657.919   0.50000000     45500            7430.143   0.50000000     2535
     664.063   0.75000000     68683           10067.967   0.75000000     3795
     670.719   0.87500000     79003           13443.071   0.87500000     4428
     681.471   0.93750000     84645           19005.439   0.93750000     4744
     738.815   0.99218750     89462          112263.167   0.99218750     5021
    1922.047   0.99902344     90071          378535.935   0.99902344     5056
    9035.775   1.00000000     90159          934805.503   1.00000000     5060

read Throughput - Parallelism-1

Parallelism 10

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     672.767   0.50000000    440205            9404.415   0.50000000    20516
     689.151   0.75000000    650423           13172.735   0.75000000    30771
     719.359   0.87500000    756731           18366.463   0.87500000    35894
     748.543   0.93750000    810550           30687.231   0.93750000    38455
    1042.943   0.99218750    857523          124321.791   0.99218750    40696
    2996.223   0.99902344    863426          467664.895   0.99902344    40977
   19775.487   1.00000000    864270          978845.695   1.00000000    41016

read Throughput - Parallelism-10

Parallelism 100

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

    3018.751   0.50000000    963478           84672.511   0.50000000    30501
    3311.615   0.75000000   1441871          104267.775   0.75000000    45679
    3801.087   0.87500000   1680127          123142.143   0.87500000    53286
    4485.119   0.93750000   1799943          151126.015   0.93750000    57084
    8278.015   0.99218750   1904865          457965.567   0.99218750    60413
   14934.015   0.99902344   1917978         1233125.375   0.99902344    60830
   35422.207   1.00000000   1919849         1272971.263   1.00000000    60888

read Throughput - Parallelism-100

Throughput Comparison

These are the number of completed read requests in a minute.

Lambda Store FaunaDB
P-1 90159 5060
P-10 864270 41016
P-100 1919849 60888

Insertion Benchmark

In the insertion benchmark, unique Product objects are inserted to an initially empty database.
At FaunaDB a new document is inserted to a collection for each request. At Lambda Store a new Redis hash is inserted to the Redis database.

ℹ️ Lambda Store has significantly higher throughput than FaunaDB in all parallelism settings. With the increase of parallelism, insertion latency of Lambda Store starts to increase after p99.9 percentile. When parallelism is 100, after p99.9 FaunaDB has better latency values but its throughput is not even close to Lambda Store. We already investigated this issue after DynamoDB benchmark and figured out that, while inserting new entries, there is an O(log n) cost which causes big delays under very high load. We have a solution to remove that cost, but it's not deployed to production yet.

Parallelism 1

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     676.863   0.50000000     41323          180355.071   0.50000000      152
     688.127   0.75000000     60858          190316.543   0.75000000      229
     709.119   0.87500000     70511          222822.399   0.87500000      266
     729.087   0.93750000     75454          287309.823   0.93750000      285
    2779.135   0.99218750     79834          505675.775   0.99218750      302
    7696.383   0.99902344     80382          557842.431   0.99609375      303
   19644.415   1.00000000     80460          616038.399   1.00000000      304

Insert Throughput - Parallelism-1

Parallelism 10

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     712.191   0.50000000    180766          172228.607   0.50000000     1535
     883.199   0.75000000    270599          221904.895   0.75000000     2303
    1261.567   0.87500000    315691          239206.399   0.87500000     2687
    1785.855   0.93750000    338244          286785.535   0.93750000     2879
   14344.191   0.99218750    357962          511967.231   0.99218750     3047
  181927.935   0.99902344    360428          661127.167   0.99902344     3068
 1911554.047   1.00000000    360779          714604.543   1.00000000     3070

Insert Throughput - Parallelism-10

Parallelism 100

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

    7737.343   0.50000000    185095          179175.423   0.50000000     14916
   11362.303   0.75000000    277640          197525.503   0.75000000     22340
   20529.151   0.87500000    323881          231211.007   0.87500000     26053
   32604.159   0.93750000    346976          277348.351   0.93750000     27902
  283901.951   0.99218750    367212          495976.447   0.99218750     29530
  664272.895   0.99902344    369743          664272.895   0.99902344     29733
 2031091.711   1.00000000    370100          835190.783   1.00000000     29762

Insert Throughput - Parallelism-100

Throughput Comparison

These are the number of completed insertion requests in a minute.

Lambda Store FaunaDB
P-1 80460 304
P-10 360779 3070
P-100 370100 29762

Overwrite Benchmark

In the overwrite benchmark, 1000 Product objects are inserted initially. Then all fields of randomly selected objects are overwritten continuously. At FaunaDB an existing document inside the collection is replaced completely for each request. At Lambda Store all fields of a Redis hash are updated with the new values.

Lambda Store has significantly lower latency and higher throughput values than FaunaDB in all parallelism settings.

Parallelism 1

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     670.207   0.50000000     42203          182583.295   0.50000000      150
     680.447   0.75000000     62829          193200.127   0.75000000      223
     697.343   0.87500000     72946          243269.631   0.87500000      260
     716.799   0.93750000     78117          314834.943   0.93750000      279
    1756.159   0.99218750     82642          469237.759   0.99218750      295
    4820.991   0.99902344     83211          529268.735   0.99609375      296
   16121.855   1.00000000     83292          944766.975   1.00000000      297

Overwrite Throughput - Parallelism-1

Parallelism 10

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     697.343   0.50000000    331951          153747.455   0.50000000      1640
     771.583   0.75000000    496116          163053.567   0.75000000      2454
     973.823   0.87500000    578297          203161.599   0.87500000      2863
    1715.199   0.93750000    619581          283115.519   0.93750000      3067
    3962.879   0.99218750    655717          884998.143   0.99218750      3246
    8732.671   0.99902344    660233         1523580.927   0.99902344      3268
   28622.847   1.00000000    660878         1799356.415   1.00000000      3271

Overwrite Throughput - Parallelism-10

Parallelism 100

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

    5992.447   0.50000000    417784          177471.487   0.50000000     14990
    7618.559   0.75000000    626341          197001.215   0.75000000     22477
    9592.831   0.87500000    730713          223346.687   0.87500000     26207
   14442.495   0.93750000    782845          293339.135   0.93750000     28075
   29048.831   0.99218750    828512          569901.055   0.99218750     29709
   40534.015   0.99902344    834216          871366.655   0.99902344     29912
   63045.631   1.00000000    835029         1138753.535   1.00000000     29941

Overwrite Throughput - Parallelism-100

Throughput Comparison

These are the number of completed overwrite requests in a minute.

Lambda Store FaunaDB
P-1 83292 297
P-10 660878 3271
P-100 835029 29941

Update Benchmark

In the update benchmark, 1000 Product objects are inserted initially. Then a few fields of randomly selected objects
are updated continuously. At FaunaDB fields of the document are updated for each request. At Lambda Store fields of a Redis hash are updated.

Lambda Store has significantly lower latency and higher throughput values than FaunaDB in all parallelism settings.

Parallelism 1

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     659.455   0.50000000     44178          251789.311   0.50000000      105
     667.647   0.75000000     66050          296747.007   0.75000000      183
     677.375   0.87500000     76796          350224.383   0.87500000      183
     692.735   0.93750000     82390          472907.775   0.93750000      196
     972.799   0.99218750     87057          754450.431   0.99218750      208
    5734.399   0.99902344     87658          924844.031   0.99609375      209
  206045.183   1.00000000     87742          924844.031   1.00000000      209

update Throughput - Parallelism-1

Parallelism 10

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count

     657.407   0.50000000    398300          184942.591   0.50000000     1241
     681.983   0.75000000    593651          241958.911   0.75000000     1858
     723.455   0.87500000    692694          363069.439   0.87500000     2167
     826.367   0.93750000    741556          539492.351   0.93750000     2322
    4118.527   0.99218750    784800          963117.055   0.99218750     2457
    7258.111   0.99902344    790202         1436549.119   0.99902344     2474
   15990.783   1.00000000    790974         1584398.335   1.00000000     2476

update Throughput - Parallelism-10

Parallelism 100

            =Lambda Store=                                =FaunaDB=

 Latency(μs)   Percentile     Count         Latency(μs)   Percentile     Count


    3966.975   0.50000000    649433          231342.079   0.50000000      6652
    5631.999   0.75000000    974519          530055.167   0.75000000      9981
    6868.991   0.87500000   1136378          824180.735   0.87500000     11641
    8208.383   0.93750000   1217459         1097859.071   0.93750000     12473
   20807.679   0.99218750   1288478         1800404.991   0.99218750     13200
   31997.951   0.99902344   1297340         2283798.527   0.99902344     13291
   72155.135   1.00000000   1298606         3024093.183   1.00000000     13303

update Throughput - Parallelism-100

Throughput Comparison

These are the number of completed update requests in a minute.

Lambda Store FaunaDB
P-1 87742 209
P-10 790974 2476
P-100 1298606 13303

Conclusion

At Lambda Store, our goal is to provide a low latency serverless Redis database. We currently support replication inside only a single region, we do not have multi-region replications yet, and that's an async replication similar to what Redis offers by default. We are thinking and working on stronger and optional consistency modes, but our purpose is NOT to build distributed transactions similar to FaunaDB.

FaunaDB supports multi-region replication and distributed ACID transactions. But that comes with a significant cost and their read & write latencies suffer from that. Their status page shows write latency stats from three regions. Those values are generally 140 ~ 200 ms range. But it's not clear where this latency value comes from, which percentile value is that.

Both Lambda Store and FaunaDB have cons and pros. It's a trade-off between two services, between stronger consistency & safety and lower latency & availability.

Discussion

pic
Editor guide