"...Our primary database (MySQL) simply cannot keep up..." MySQL runs some of the biggest data sets in the world. Find a good DBA. Clustering, federation, caching.
Using Redis as a caching layer is a great idea, but as a persistent storage media; I would not recommend that. Sure, it can save to disk, has replication, complex object types, etc. But Redis is, by its nature, a cache, not persistent storage.
Thanks for the suggestion! I'll clarify the use case just a bit. The problem is not with read performance from MySQL, but with write performance to it. The data we're storing here is transient by its very nature, with each record changing several times per minute. It's also not data that we'd care much about losing (we persist as much as we need to in a down-sampled fashion to other storage mediums). So even if we could tune MySQL to accept writes fast enough (which would be expensive if we need to hire a new full time employee to figure it out!) we don't need most of the features it gives us.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
"...Our primary database (MySQL) simply cannot keep up..." MySQL runs some of the biggest data sets in the world. Find a good DBA. Clustering, federation, caching.
Using Redis as a caching layer is a great idea, but as a persistent storage media; I would not recommend that. Sure, it can save to disk, has replication, complex object types, etc. But Redis is, by its nature, a cache, not persistent storage.
Right tool for the right job.
Thanks for the suggestion! I'll clarify the use case just a bit. The problem is not with read performance from MySQL, but with write performance to it. The data we're storing here is transient by its very nature, with each record changing several times per minute. It's also not data that we'd care much about losing (we persist as much as we need to in a down-sampled fashion to other storage mediums). So even if we could tune MySQL to accept writes fast enough (which would be expensive if we need to hire a new full time employee to figure it out!) we don't need most of the features it gives us.