DEV Community

Discussion on: Best distributed KV database?

Collapse
 
jgaskins profile image
Jamie Gaskins

If you truly need to scale to hundreds of terabytes, you can use any S3-like cloud object storage provider (actual S3, Google Cloud Storage, DigitalOcean Spaces, etc) as a KV store with virtually unlimited scale.

If you want something you can run locally: NATS KV store. NodeJS driver. Running the server only requires two commands: brew install nats-server and nats-server -js. Caveat: values must be under 1MB each in the server's default config or if you plan on using it with a global cluster like NGS.

NATS KV store was modeled after etcd which, last I checked, does not meet your "easy to setup" criterion.

If you don't actually need hundreds of terabytes (and I'd be curious why you think you do), you can use Redis. It's the single most popular KV store in existence, there's a driver for nearly every language, and it can still theoretically scale to hundreds of terabytes if your keys are evenly distributed. It theoretically supports up to 16384 nodes in a single cluster, so if you have that many nodes each with 512GB of RAM, you can store roughly 8 PB of data in it.