At Humyn.ai, our challenges sometimes involve distributing large datasets to our innovator community. For a recent challenge, we estimated ~$25 in charges for data being downloaded from S3, per participant, prompting investigation into other options.
Recently Cloudflare made their object storage product R2 generally available, advertising zero cost egress. The positioning of the product, taking direct aim at the relatively expensive cost of outgoing bandwidth from the other major cloud providers.
Our experience trialing R2 for disseminating large datasets follows.
Cost savings
In this case, the cost savings were potentially significant, but R2 does not provide any metrics on how much data was downloaded from a given bucket, however for anyone with a history of billing for a comparable product like S3, it should be easy to evaluate the impact of switching on the bottom line.
Supported clients
R2 (and apparently a number of other providers) implement the S3 API, allowing S3 clients to interact with R2 directly. I found that while the aws-cli worked for smaller files, there was an incompatibility with larger files using multi-part uploads.
I found a post from a Cloudflare engineer, that confirmed it was a known issue, but the configuration options in the aws cli to reduce concurrency did not seem to work.
In the end I switched to rclone with a limited concurrency of 1, which worked for files in the ~250GB range.
Features
Both pre-signed links and public buckets were supported in R2, which were enough for our short term needs.
S3 has extremely flexible IAM, amongst a wide range of other features and it was a little surprising that the tokens required to authenticate with R2 were global account level tokens, which couldn't be scoped per bucket.
With R2 being such a new product, I wouldn't be surprised if it continued to chip away at the advanced use cases supported by S3.
Overall our experience with R2 was positive and we'll be keeping an open mind on how it can be leveraged in our stack moving forward.
Top comments (0)