MongoDB Performance 3.4 vs 4.4 vs 5.0 vs 6.0

We've been using MongoDB v3.4.24 for quite a long time. And ever since MongoDB v6 got released, we have wanted to benchmark all these new versions to see if they fit our needs. We did such a benchmark years ago against MongoDB v4, but it turned out that MongoDB v4 was worse than MongoDB v3 in terms of performance. This time, we'll do a more detailed benchmark on MongoDB 3.4.24, 4.4.16, 5.0.12, and 6.0.1.

We ran the benchmark with https://github.com/mongodb/mongo-perf.

To better observe system resource usage and minimize the noises and randomness of the results, we've separated the testing server and client into 2 VMs with dedicated CPUs:

Server benchmark

Here is the python script for running the benchmark (change 6.0.1 to the version that is being tested, change 10.1.0.2 to the internal IP of the testing server, and change -t 1 to -t 4 to run with four threads):

#!/usr/bin/bash
tests=("simple_insert" "simple_multi_update" "simple_query" "simple_remove" "simple_text" "simple_update" "simple_update_mms" "simple_geo")
for i in {1..10}; do
  echo "==== Round $i ===="
  for t in "${tests[@]}"; do
    python3 -u benchrun.py -f testcases/$t.js --host 10.1.0.2 -t 1 --trialTime 1 --out 6.0.1-$t-$i.json
  done
done

We ran the tests ten times for each server, cut out the lowest and the highest value, and averaged the rest. We didn't run through all tests provided by the repo, but that's enough for our use case.

Here are the results for one thread -t 1 (higher is better, the winner for each test is highlighted in red):

The results for the four threads -t 4 look similar on winner distributions.

We are also attaching the system resource usage of the server with -t 4 if someone is interested.

It is disappointing that MongoDB v3.4.24 leads in almost all write operation tests (insert, update and delete) and most aggregation tests. v4.4.16 wins in the rest of the aggregation tests. v5.0.12 is the biggest loser. And finally, 6.0.1 wins for all queries.

There's no reason for us to migrate to v4.4.16 and v5.0.12. As for v6.0.1, although it received an impressive query performance bump, probably related to the Slot-Based Query Execution Engine, its write operation performance was only around 60% of v3.4.24, which was the worst among all four versions. We won't risk trading writing performance with querying.

Driver benchmark

We've done further tests on MongoDB ruby drivers (mongo gem, versions v2.13.0 and v2.18.1) by inserting a certain number of documents and measuring the duration. Here's the script for testing:

gem 'mongo', '=2.13.0'
require 'mongo'
require 'digest'
require 'benchmark'

Mongo::Logger.logger.level = Logger::WARN
client = Mongo::Client.new('mongodb://10.1.0.2:27017/test')

puts Benchmark.measure {
  100_000.times do |index|
    client[:users].insert_one(name: Digest::MD5.hexdigest(index.to_s))
  end
}

Results: (mongo v2.18.1 doesn't support MongoDB v3.4.24)

We can see that mongo v2.13.0 is slightly better than v2.18.1 on insert time, which is frustrating. Also, inserting performance degrades from v3.4.24 to v6.0.1.

Out-of-memory test on $near

Our final test was an OOM-Killer situation that occurred in our v3.4.24 setup. It failed once we concurrently fed 5k $near queries to the server.

Here is the script used for testing:

gem 'mongo', '=2.13.0'
require 'mongo'
require 'timeout'

Mongo::Logger.logger.level = Logger::WARN
client = Mongo::Client.new('mongodb://10.1.0.2:27017/test', max_pool_size: 5000)

col = client[:locations]

5000.times.map do
  Thread.new do
    col.find({
      :gps => {
        "$near" => {
          "$geometry": { type: "Point",  coordinates: [ rand(-90.000000000...90.000000000), rand(-180.000000000...180.000000000) ] },
        }
      }
    }).to_a
  end
end.each(&:join)

The location collection was seeded with 10k entries.

Unfortunately, all four versions (v3.4.24, v4.4.16, v5.0.12, v6.0.1) still got OOM-killed within 30 seconds after running the script.

There's an issue still unresolved, claiming that $near is using unbounded memory.

We've also tried to stop the script right before the server crashes. Hoping that it would stop at a high memory consumption. We wanted to see if the extra memory would be released just like with GC. But sadly, it didn't happen (we waited for 3 hours).

Conclusion

To sum up, MongoDB had not put its focus on performance optimization, at least for write operations. Writing performance became even worse on more recent versions. The only thing encouraging is that MongoDB v6 has introduced a new query engine that boosts query performance.

It is also frustrating to find out that newer versions of mongo ruby driver have poorer writing performance than older versions.

The memory leak issue regarding the $near query has not been resolved in any of the tested versions.

These results ultimately killed our enthusiasm for upgrading MongoDB at this time. We will continue to monitor and test the performance of newer versions.