Selecting the right NoSQL document database

When it comes to choosing a platform for a new application there is a lot to consider at different levels. For the CTO or Architect the focus is on providing the required scale, resiliency, maintainability, elasticity. And all of it within a certain price range. For the developers, the preference goes to elegant solutions, well integrated with the preferred technology stack and with smooth (pain-free) development experience.

All these factors come into play for selecting the right document-oriented NoSQL database. And for ones using the .NET technology stack or Azure customers, the options may include Cosmos DB and RavenDB. Hope that the below comparison between the two popular NoSQL databases will help to make the decision.

The CTO/Architect point of view

Ploughing your way through all the marketing buzzwords you can quickly determine that both databases have high performance, availability, fully managed and allow unlimited scaling (limited only by your budget). It means the devil is in the details.

Hosting

The hosting story for Cosmos DB is quite short – Azure only. In exchange it gives a very good integration with the Azure infrastructure.

RavenDB works on the three major cloud providers: AWS, Azure and Google Cloud (currently in beta). Also on Docker and as a service on Windows, Linux and MacOS (not to mention a Raspberry Pi option). Though, the cloud instance is managed independently from the cloud provider's portal (at own RavenDB Cloud portal), which comes with independent authentication, billing and lack of native automation tools.

It all depends on what you value more – platform independence or Azure integration.

Pricing

RavenDB has a wide variety of options – a simple table for offline licenses and several pricing tiers for hosting in the cloud allowing some tweaking of the hardware (CPU cores and priority, RAM, storage). The cloud production tiers provide 3 options: Basic with a burstable instance and predefined CPU credits; Standard with pre-allocated resources dedicated to the cluster; Performance – a custom-tailored solution. Either way, at the end you get a flat price with minor extras like Internet traffic costs.

Cosmos DB pricing is monotonously based on provisioning throughput expressed in Request Units per second (RU/s) plus storage cost. The throughput can be shared across the whole database or be container-based (e.g. per collection, partitioning, table or graph). Keep in mind that the provisioned RU/s are used not only for reads and writes, but also for building indexes. Price optimisation is well documented and in spite of some complexity, it provides high flexibility in tuning the resources and applying changes any time through Cosmos DB’s REST API.

Due to completely different approaches, it is hard to answer the main question – "which one is cheaper?" Perhaps, we could compare Cosmos DB prices with RavenDB's 'Basic' production tier, but it would highly depend on the database design, which can be completely different between the two vendors to gain performance/price benefits. Subjectively, the overall cost would be on par between the two. Cosmos DB gives many options for cost optimisation, but all price benefits can be overturned by the development costs of balancing the containers (more on this later) and some automation for dynamically changing the throughput.

Free options

Good news is that both vendors give something for free.

RavenDB has a free Community license for using locally as a service with some minor constraints and it is absolutely sufficient for testing and development purposes. In addition, there is a very modest free cloud version on AWS (forgive its slowness, it's free) and also an online testing environment to quickly fiddle with a DB, that would get scraped overnight.

Cosmos DB gives a free 30-day trial to start in the cloud. As for local development/testing, they've got a free Azure Cosmos DB local emulator to run on Windows (no Linux/MacOS) without creating an Azure subscription. Though it has some limitations (like not all consistency levels) it's a good way to spin off a local environment for developers. Alternatively, there is a free cloud version with 400 RU/s and 5 GB of storage.

Security

RavenDB provides a standard set including a certificate-based authentication, database encryption, permissions, audit, IP firewall, etc. When Cosmos DB has a bit more enhanced set, adding more granular user permissions, geo-fencing and role-based access control integrated with Active Directory.

I would question a necessity in so advanced permission settings and other security measures at the database level in the age of microservices. It is common practice of having one database per one bounded context accessible only via one application/service (again, one per bounded context). So, the database would allow connection from the single source only and the authorisation/authentication concerns would be delegated to the app... But in case you need advanced settings, use Cosmos DB.

Backups & Replication

When it comes to backups, RavenDB does what you need – full and incremental backups, on schedule and on demand, and a straightforward restoration procedure (link). The DB administrator is in full control of the backups (including location and retention time) and can restore the DB to a previous point in time by a single click. It ticks all the boxes of the usual requirements.

But Cosmos DB does it ... differently. It hides away the backups and does it automatically every 4 hours, but only the latest 2 backups are stored. To restore, "you should contact Azure support within 8 hours so that the Azure Cosmos DB team can help you restore the data from the backups"... Why?!! It may satisfy some amateur users, but for enterprise it sounds less than optimal.

There are two additional options to manage your own backups, but they come with various constraints and can be considered rather as a workaround than the traditional backup/restore solution.

Regarding replication, RavenDB gives several options to replicate fully or partially to another Raven database or even a SQL database (with a pre-defined data transformation). Another interesting option is delayed external replication to quickly "go back in time" and undo a faulty change (can be a short-term alternative to a backup). Plus, all cloud production instances work in clusters (3 nodes by default) to provide redundancy, increased availability and fault tolerance.

Cosmos DB again opts for hiding the concern from the consumers and transparently replicates the data to all the regions associated with your Cosmos account providing eventually consistent reads (link).

Despite all the efforts of the Cosmos DB's marketing team to steer all talks about backups to replication, it can't be a replacement. Backups is a vital safety cushion to protect developers from own mistakes (to revert 'bad' changes) and the owner of the database (e.g. for legal reasons or internal investigations).

This feature is on the road map, but with no ETA. Meanwhile, lack of the traditional backup/restore functionality in Cosmos DB can be a real dealbreaker and swing the pendulum in favour of RavenDB for many.

Resilience to high load

The database may run smoothly until something (or someone) starts challenging its resilience by throwing a high number of requests. Whether it caused by a high number of customers or a malicious DDoS attacker, the system must cope with it.

Cosmos DB handles high load gracefully. When the database or container is exceeding the provisioned throughput limit, the server preemptively ends the request with HTTP status code 429 ("Too Many Requests") providing a timeout for a retry. The native SDKs handle it nicely, spitting an exception on exceeding the number of retry attempts. Alternatively, Cosmos DB provides an Autopilot mode (in preview) for autoscaling provisioned throughput based on workload patterns, but the RU/s under the autopilot are 50% more expensive than manually provisioned.

All RavenDB's production instances in the cloud are run in clusters where each node is self-sufficient. It gives a decent auto recovery from failures, which take some nodes down (video). On the top of that, handling a peak load depends on the pricing tier. For a less expensive 'Basic' production tier you get a burstable instance with a certain budget of CPU credit (like provisioned throughput in Cosmos DB), where the requests get declined on exceeding the budget. For a more expensive 'Standard' tier you get a reserved cluster with pre-allocated resources, where in case of maxed out CPU and RAM you get slower responses and potential timeouts on the requests.

Seems all good here. Either database would be highly resilient with several strategies in its sleeves.

Migration Path

Both vendors have solutions for customers migrating from competitors.

At first glance, it may appear that Cosmos DB goes over the top to make the transition as smooth as possible by providing a compatible API and hundreds of articles/videos to guide potential customers to Azure. Apart from the "native" Core (SQL) API, it provides MongoDB, Cassandra, Azure Table and Gremlin (graph) APIs. On creating a new DB you get to choose one, but you cannot change it later (only via migrating to another DB). Microsoft keeps the Cassandra API up-to-date, but for some reason support for MongoDB API stopped at v3.6 with no plans for later versions.

RavenDB in its turn gives a way to import data from Cosmos DB / MongoDB and SQL databases (even supports exporting it back to SQL). Sure, it will require a big re-write of the code, but no one migrates to another DB just to change the label (unless you're a believer in amazing cost-reduction stories), likely it's done to leverage some functionality unavailable in the initial database.

Both databases won't disappoint Graph DB users (like Neo4j or OrientDB). Cosmos DB's Gremlin (graph) APIs supports Apache TinkerPop's Gremlin language to simplify migration. Then RavenDB provides integrated graph support in its native RQL language, which appears to be more readable and intuitive, but will suit greenfield projects only.

Total score for the CTO/Architect

If you value the Azure integration and costs of developing a 'special' backup solution and the throughput optimisation don't scare you, then go with Cosmos DB.

If you need cloud-independence or full control over your data and backups, then RavenDB is your choice.

The Developer point of view

The first thing developers may notice is both vendors support a SQL-like language (SELECT operations only) and JavaScript with a handful of extension methods to run on the server-side (ECMAScript v5.1 in RavenDB and v6 in Cosmos DB). Absence of the traditional SQL-like DELETE, UPDATE and INSERT is compensated by corresponding special commands for individual entries, when bulk operations require custom stored procedures in Cosmos DB and SQL-looking set-based operations in RavenDB.

Next one would be consistency. While NoSQL is all about eventual consistency, Cosmos DB and RavenDB are flexible enough to provide some flavours of strong (immediate) consistency (different levels/scopes in Cosmos DB, or more explicit waiting on writing/reading in RavenDB). It is a good start.

Difference between the two databases becomes more noticeable when developers take one step further and start applying the DDD skills to reflect in NoSQL all carefully designed entities, value objects and aggregates. Convenience of transferring the domain model to the database and ease of maintenance is a big factor affecting overall time/costs of development. Let's use this metric to assess the databases.

Indexes and Query language

Adding new objects is never an issue – all modern NoSQL platforms provide good integration with all major development environments. All of them are ACID compliant with full support of transactions (more on it for Cosmos DB and RavenDB). Problems start arise when

• querying data not by the identifier of a document, but various filtering conditions;
• performing bulk operations (e.g. deleting/updating multiple entries at once).

In both cases indexes come into play and they are handled in a special unique way by each NoSQL vendor.

Cosmos DB

Admittedly, indexes is not the strongest part of Cosmos DB. It does the basics and supports 3 kinds of indexes:

• Range for >, <, =, IN clauses, equality match for arrays, string prefix matches and also sorting (single field only).
• Spatial for points, lines and polygons.
• Composite for sorting/filtering on a combination of fields.

Here is the first pain. Sorting/filtering on multiple fields works only against the composite index that comes with a significant constraint – the sequence and order of properties matters:

• The sequence of properties in the index has to match the sequence in the ORDER BY clause.
• The ascending/descending order has to either match the order in the ORDER BY clause or be fully reversed on all the properties.

The same is applied to the WHERE clause and a combination of WHERE and ORDER BY.

And that's not it, hold tight. Queries with COUNT do not use indexes and perform a full scan (ticket). Try to use GROUP BY with an ORDER BY clause or subqueries – not supported. Need cross-document JOINs? Again, not an option (the only workaround goes via the Gremlin API). Want to use a string comparison? Mind it is always case-sensitive (it's recommended to store a "canonicalized" form). Bulk operations via stored procedures? Remember that the execution can be silently throttled on hitting the limit of provisioned throughput (can run several times to make sure it processes all the records).

Developers will have to steer around such 'features' for most of the usual enterprise applications. Though in some cases, lack of functionality is covered by a decent Azure integration. For example, Cosmos DB does not support full text index/search out-of-the-box, but Azure Cognitive Search comes to the rescue (for additional cost, of course).

RavenDB

RavenDB's philosophy is different – do as much data transformation as possible in the index, then query against that index. And RavenDB's indexes support almost everything you may need:

• The traditional Map indexes and Map-Reduce indexes for complex aggregations of data.
• Single- and Multi-collection indexes, where data can be pulled from several collections. Less need in JOINs in the query, data from other collections get resolved in the index (though, you still can JOIN in the query).
• Tokenized text for full-text search supporting suggestions, boosting and term vectors. RavenDB supports Lucene out-of-the-box with various analysers and with the document size limit of 2GB it eliminates a need in ElasticSearch (that is built over Lucene).
• Spatial indexes for retrieving data based on spatial coordinates with specified precision.
• JavaScript in indexes for complex data transformation.
• Dynamic fields to allow querying on fields unknown at index creation time.
• Indexing hierarchical data for resolving properties of recursively nested objects.

It may appear unusual that JOINs and GROUP BYs are handled in the indexes, but it makes sense for minimising querying costs. Anyway, it is a small price for not stumbling over the index related Cosmos DB constraints. Apart from that, the SELECT operates as expected – just define the querying properties (either from the collection or the index), add some last minute value transformations (can use JavaScript), sorting (no Cosmos DB-like gotchas here), paging, etc.

Both vendors advertise auto-indexing to ease the learning curve. Cosmos DB indexes of all the properties by default, then RavenDB may generate indexes in runtime for a given query. It may look nice and convenient, but maintenance of unused indexed properties and indexes (e.g. updating them on each change in the collection) will cost extra throughput/IOPS. Therefore, in a real enterprise environment you would not leave indexes unattended and opt to define them explicitly.

High-level language integration

Both vendors provide SDKs for .NET, Java, Node.js and Python. And both favour the .NET developers more (better documentation, more packages and updates), so let's compare their .NET integration as the most indicative.

Cosmos DB

The provided SDK is just an unsophisticated wrapper over the API. For a while, even the .NET developers did not have anything better from Microsoft and it sparked a raise of Cosmonaut (open source library) to fill the gap. The situation has barely changed after release of Entity Framework Core 3 last year.

Surprisingly, Cosmonaut hasn't been ousted by EF Core 3. In spite of being based on an older version of the SDK (a SDK 3 version is in preview), Cosmonaut is a competitive tool providing handy extensions, decent documentation and better control over the containers (features like Collection Sharing for storing different types in one collection). And the EF Core team fairly admits limitations of their product:

Many of these limitations are a result of limitations in the underlying Cosmos database engine and are not specific to EF. But most simply haven't been implemented yet.

As an example, EF Core 3 Cosmos DB provider does not support aggregate operators (Count, Max, etc.). It seems that Count is plagued with problems starting with Cosmos DB itself not using indexes when counting and draining the RUs, so the best option for EF Core 3 is a client-side counting:

query.Select(_ => 1).AsEnumerable().Count()

Such limitations do affect structuring the data (entities and aggregates) and add more pressure on the back-end code to handle extra logic.

When it comes to automated tests, the best solution is provided by a third-party package for EF Core (EfCore.TestSupport) and works on the top of Cosmos DB Emulator, but due to the emulator's constraints it will run on Windows only (can also use Windows containers). Alternatively, it would require mocking DB-related interfaces (e.g. see this example).

Query execution metrics are available in the pure SDK, but accessing them from EF Core or Cosmonaut is not a trivial task.

Integration with other popular .NET packages is quite limited for both, EF Core and Cosmonaut. For example, Dates are treated as strings and there are
no NodaTime support.

RevenDB

RavenDB is designed to describe persistence models and indexes in the code (the Code First approach). Considering very powerful and flexible indexes, ability to maintain them in the code becomes crucial.

Aggregates defined in a high-level language are neatly transferred to the collections. Take for example polymorphic data that can be handled in two ways:

• storing all derived types in one collection;
• storing derived types in dedicated collections, but having a multi-map index combining data from them.

In both cases you can query on a derived or base type/interface and the decision which option to choose would be based on the most common scenario (either querying independently each type or all together). This feature is not flashy, but such subtle options let the developer to structure the data in the most logical way and not getting bogged in bypassing limitations.

The .NET package is native, supports all the features (including a simple way to fetch the query statistics) and with a decent documentation.

The testing framework ticks all the boxes – a simple package (TestDriver) runs queries against a real in-memory database (leveraging the RavenDB embedded engine), works on all platforms and provides integration with .NET and Java.

Community / Tech support

And the last but not the least – community support. It plays a big role when it comes to leaning or resolving technical issues.

Cosmos DB is backed by a giant corporation and just the Microsoft label by itself can lead to higher popularity and bigger community. Imbalance in the number of blog posts and articles between the two (Google gives 100K search results for RavenDB and 500K for Cosmos DB) is enough for winning someone over via social conformity.

But apart from that, when it comes to learning and troubleshooting both of them look very similar:

• Documentation is at approximately the same level (Cosmos DB docs, RavenDB docs). The Cosmos DB's documentation is mostly focused on the administration aspects, then RavenDB – on coding.
• Number of books is neglectable for both (Cosmos DB - 4, RavenDB - 3).
• Number of StackOverflow posts is quite small for both (2,000+ for RavenDB and 4,000+ for Cosmos DB)
• Both have prompt and free tech support – on MSDN for Cosmos DB and on Google Groups for RavenDB. And for both platforms it is the main place for resolving ongoing problems. Small addition – RavenDB provides an online testing environment for reproducing and demonstrating issues to the tech support.

Total Developer score

No doubt Cosmos DB creates more buzz in the technical circles, mostly targetting less experienced in NoSQL audience. But from the developer's perspective, Cosmos DB is falling far behind RavenDB. Overall and espesially in the .NET environment.

RavenDB is far from ideal, but it gives reliable tools, so developers can focus on achieving the goal rather than finding workarounds.