Backing up Amazon S3 data
As millions of organizations adopt the simplicity of Amazon S3 to store and manage business-critical data, the need for high-performance backup and recovery is becoming increasingly clear. The applications, data lakes, and machine learning models that run on Amazon S3 are often powered by critical and sensitive data that needs to be backed up on a continuous basis, and retrievable near-instantly. Clumio has pioneered backup and recovery for Amazon S3 (we even wrote a book on it) for 2 years now, and it’s exciting to see other players join the market.
Using AWS Backup for S3
AWS Backup recently announced a performance improvement for S3 that speeds up their backup times for buckets containing over 300 million objects. The improvement additionally allows backup of buckets containing more than their previous upper limit of 3 billion objects. Below is the average expected performance with these improvements as published in the AWS Backup Developer Guide, and we have added a column calculating the objects per second figure for each. Restore performance metrics were notably absent from the AWS Backup published results.
As published in Amazon S3 Backup Developer Guide
|
| Number of objects | Bucket size | Estimated time to complete initial backup | Objects per second (calculated by Clumio) |
| 135 million | 425 GB | 31 hours | 1,210 |
| 670 million | 800 TB | 38 hours | 4,898 |
| 5 billion | 6 PB | 100 hours | 13,889 |
| 7.5 billion | 370 TB | 180 hours | 11,574 |
Clumio’s performance results
We thought this might be a good opportunity to share some results of Amazon S3 backups with Clumio. Architected as a serverless data processing pipeline, speed and scale of Amazon S3 backups have been the primary focus at Clumio since we started supporting it in 2021.
What makes a backup fast
As you can see, object count and overall data volume both play a role in the backup time of a given bucket. This is illustrated by the two examples of buckets containing about 24 billion objects, but at very different sizes, resulting in very different backup times. Generally speaking, the object count should be your first consideration in estimating backup time, with data volume following. While other factors will always influence backup speed, you can count on Clumio’s backup speed being relatively predictable from the highest to the lowest object counts and data volumes. Notice that Clumio’s performance does not reduce for the lowest object count example, and also scales to well above 20 billion objects per bucket, resulting in exceptional backup speed regardless of the customer’s S3 environment. Indeed, we have customers backing up buckets that are a few GB to tens of PB.
Enabling high performance backups without affecting application performance
In addition to being fast, Clumio also detects when buckets being backed up are in demand, and throttles the backup processing to allow the primary application to perform optimally even while backup is in process. This is especially important during large seeding backups, which can get large enough to take up significant resources and API quotas. This means if an application wasn’t making any calls to a given bucket, its backup time could be as much as 50% faster. However, that’s not a typical scenario. Many applications, especially those with a global user base, need to run continuously. It’s important to preserve their functionality, especially for large buckets whose initial backups will take several days.
What about restoring data?
While backup speed is an important benchmark, restore speed is even more important. After all, the point of backups is to restore data quickly when you need it most. Even though the AWS Backup performance results didn’t showcase their restore metrics, let’s take a look at the restore speeds that you can expect with Clumio for Amazon S3 data.
S3 restores most commonly go to new buckets, which start with lower API rate limits. AWS scales up the API limits automatically as more objects are created in the destination bucket. This delayed scaling impacts the rate at which objects are created in the destination bucket during the initial phases when API limits are low. The restore process can be accelerated by working with AWS to proactively increase the limits before starting.
Achieving breakthrough zero-RTO restore performance with Clumio Instant Access
For large S3 buckets, waiting for a full restore of your backup to resume operations might be too disruptive. On the other hand, maintaining a warm failover (such as a replicated bucket) of your data all the time can be expensive (and doesn’t offer point in time recovery, air gap, and other features of backups). Clumio’s Instant Access feature exposes an S3 endpoint for applications to directly access data from backups without having to restore the objects at all. This reduces the RTO to minutes, even for buckets containing tens of petabytes.
Your keys to a fast and performant S3 backups
Hopefully this has been a helpful exploration of backup and restore speeds for Amazon S3. When assessing a backup solution for Amazon S3, be sure to test restore performance! If you’d like to learn how Clumio’s performance can help you better manage your data estate, request a personalized demo.
The post A closer look at speed and scale results when backing up Amazon S3 appeared first on Clumio.
Top comments (0)