DEV Community

Unlock the Power of Enhanced Filtering with S3 Batch Operations

What is Amazon S3 Batch Operations?

  • It allows you to perform batch operations on large sets of objects in Amazon S3.
  • You can select objects based on prefixes, tags, and metadata.
  • Actions that can be performed include copying, archiving, applying access controls, and adding object tags.
  • The goal is to simplify and automate repetitive tasks.


 

Any news on Batch Operations?

  • You can now manage objects within an entire bucket, based on prefix, suffix, creation date, or storage class.
  • You can quickly apply the operation to all the matching objects.


 

Benefit of Batch Operations

  • You can easily perform one-time or batch workloads:
  • Copying objects between staging and production buckets
  • Invoking an AWS Lambda function to convert file types
  • Restoring archived backups from S3 Glacier storage


 

Create a replication rule

Set the filtering: file name prefix

Image description

Image description

Image description

Image description


 

Create a CSV inventory configuration for batch operation

Manifest.csv contains all objects' names for the batch operation

Image description

Image description

Image description

Image description

Image description

Image description

generative-ai-cities,aws_team_1.png
generative-ai-cities,aws_team_2.png
generative-ai-cities,aws_team_3.png
generative-ai-cities,aws_team_1_large.png
generative-ai-cities,aws_team_2_large.png
generative-ai-cities,aws_team_3_large.png
Enter fullscreen mode Exit fullscreen mode


 

Create an S3 batch operation using a CSV inventory configuration

Select "copy" to replicate objects to the destination bucket.

Image description

Image description

Image description

Image description


 

Create an S3 batch operation using an S3 replication configuration

Filter objects by creation date
You can specify filters to reduce the scope of replicated objects.
These filters work in conjunction with existing filters in your replication configuration.
If no filters are specified, all objects defined by the replication configuration will be replicated.

Image description

Image description

Image description

Image description

Image description

Image description


 

Full visibility

Monitor the running time and percentage of objects completed.
Receive a detailed completion report with the status of each object.

Image description

Image description

Image description

Image description

Image description

Image description


 

Reference

Amazon S3 Batch Operations now manages buckets or prefixes in a single step

Replication configuration

Replication configuration hand-on

Top comments (2)

Collapse
 
kennc profile image
Kenn C

Thanks for your valuable sharing. It really helpful for me to do machine learning though S3 Batch Operations.

Collapse
 
danc profile image
Danny Chan

Happy learning 😀