Let’s discover Amazon S3 Batch Operations

2 min readAug 17, 2023

Imagine you want to copy over some of the objects between your buckets or replacing some meta information of your S3 objects at a large scale.

You’d have to go through each of the objects and do it? Or you can use the S3 Batch Operations feature.

S3 Batch Operations is an Amazon S3 data management feature that lets you manage billions of objects at scale with just a few clicks in the Amazon S3 Management Console or a single API request.

Using this you can do stuff like -

✅Modify S3 Object metadata or properties

✅Copy objects between S3 buckets

✅Encrypt unencrypted S3 Objects

✅Modify ACLs, Tags

✅Restore from S3 Glacier

✅Invoke a Lambda function to perform any Custom action on each S3 object

But, how do I get the list of all the objects I need to perform the Batch Operation on?

Use S3 Inventory —

Amazon S3 Inventory can provide you with a comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that lists your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string).

But this one returns a large dataset of all the objects in the bucket. What if I want to copy only a selected few?

Use S3 Select feature —

You can use structured query language (SQL) statements to filter the contents of an Amazon S3 object and retrieve only the subset of data that you need.

By using Amazon S3 Select to filter this data, you can reduce the amount of data that Amazon S3 transfers, which reduces the cost and latency to retrieve this data.

This feature works on objects stored in CSV, JSON, or Apache Parquet format!