shrimp is an Amazon S3 uploader for humans, with powerful interactive features.
I want to expand on the blog post that I wrote when I announced shrimp; Introducing shrimp and s3sha256sum. While shrimp is a great tool to have in your toolbox, there are cases where it is not the best tool for the job. Let's compare it to the AWS CLI, and also dive deeper on the key challenges that shrimp overcomes.
| AWS CLI | shrimp | Notes | |
|---|---|---|---|
| Upload files to Amazon S3 | ✅ | ✅ | |
| Download files from Amazon S3 | ✅ | ❌ | I've been considering implementing downloads in shrimp. Leave a comment below if this is something you need. |
| Synchronize directory | ✅ | ❌ | This might be the biggest blocker with regards to uploading a lot of files with shrimp. Definitely want this in a future version. |
| Pause upload | ❌ | ✅ | Just hit the space key when uploading with shrimp to pause the upload. |
| Resume interrupted upload | ❌ | ✅ | This is the primary reason why I couldn't use the AWS CLI to upload my backups to S3. Even worse, the AWS CLI will abort your multi-part upload when you interrupt it, destroying any chance of resuming it. |
| Retry in case of errors | ❌ | ✅ | If you experience network issues during an upload, the AWS CLI will terminate with an error. You'll have to wrap it in a script to handle the error if you want it to recover automatically. shrimp automatically retries and resumes an upload when any error occurs. |
| Bandwidth limiter | ✅ | ✅ | The AWS CLI supports it through the configuration file (no command line argument). |
| Adjust bandwidth limiter during upload | ❌ | ✅ | The AWS CLI cannot adjust the bandwidth limiter during the upload. And since you can't resume the upload there isn't any workaround. shrimp can even automatically adjust the bandwidth limiter based on the current day and time. |
| Supports alternative checksum algorithms | ❌ | ✅ | For some inexplicable reason, the AWS CLI still hasn't been updated to upload files using the alternative checksum feature launched in February 2022. Using alternative checksums also speeds up the upload since the checksum does not need to be computed before the upload starts. |
| Smart single/multi-part selection | ✅ | ❌ | shrimp always uses a multi-part upload to do its job. Ideally it should just use a single request for small files. I want to implement this even though it might increase complexity of the code. |
| Upload multiple parts in parallel | ✅ | ❌ | For simplicity, I chose to not even attempt concurrent part uploads in shrimp. You might think you need this even when you don't. I don't think it will significantly increase the throughput in most cases. I have used shrimp on a very fast connection and I easily reached 50 MB/s. |
| Advanced MFA management | ❌ | ✅ | If your credentials require MFA and you use the AWS CLI to upload a very large file, and the upload doesn't finish in the validity time of the MFA token, then your upload will not complete. This can make it very hard to upload huge files without sacrificing security. shrimp will prompt you again for another MFA code when needed. It also supports an advanced --mfa-secret option that can be used to make shrimp automatically generate the codes. |
The lack of some of these features in the AWS CLI were key in convincing me that I had to write shrimp, specifically:
- Resume interrupted upload - This is paramount when you need to upload hundreds of gigabytes to Glacier.
- Adjust bandwidth limiter during upload - This is essential when you need to upload large files over a residential internet connection. You want to let it rip overnight and then go slower when it would otherwise ruin your regular internet activities.
If the AWS CLI had supported these features then perhaps I would never have written shrimp.
I want to implement some of the missing features, like uploading multiple files similar to aws s3 sync. Please comment below if you need a specific feature, I'd be very interested to hear about your use case.
Hopefully shrimp can close the gap and become an even more valuable tool in everyone's toolbox. I would also love to see the AWS CLI learn some new tricks, like support for alternative checksum algorithms.
This blog post was also posted to GitHub: https://github.com/stefansundin/shrimp/discussions/5
Top comments (0)