Have you explored saving the request data in Apache Kafka? As you mentioned in the post, the request data is mostly immutable which seems like a good use-case for Kafka. Kafka allows creating topics with retention which can be a combination of size and time e.g maximum of 7 days or 10 GB whichever gets hit first. You can also use KSQL for querying the produced data and creating stuff like constantly updated materialized views on the latest requests.
On the flip side, it will mean at least having some JVM based software (kafka and zookeeper) in the stack. Also not sure if OCaml has a mature Kafka client library.
Have you explored saving the request data in Apache Kafka? As you mentioned in the post, the request data is mostly immutable which seems like a good use-case for Kafka. Kafka allows creating topics with retention which can be a combination of size and time e.g maximum of 7 days or 10 GB whichever gets hit first. You can also use KSQL for querying the produced data and creating stuff like constantly updated materialized views on the latest requests.
On the flip side, it will mean at least having some JVM based software (kafka and zookeeper) in the stack. Also not sure if OCaml has a mature Kafka client library.
That's a really interesting approach, seems like it would solve all of the problems I'm looking at. Thanks for the suggestion!