S3 Batch Operations: Retag or Restore Millions of Objects Without Writing a Loop

The problem: a million-object to-do list

At some point every team with a large S3 footprint hits the same wall: you need to touch a huge number of objects — add a tag for cost allocation, restore an archive prefix from Glacier, copy objects to a new bucket with a different encryption key — and the obvious answer is a script that lists the bucket and loops over the keys.

That script works fine at 10,000 objects. At 10 million it becomes a small distributed-systems project: pagination, throttling retries, checkpointing so a crash doesn't restart from zero, and some way to answer "which objects failed?" afterwards. S3 Batch Operations exists to take that undifferentiated work off your plate. You hand it a manifest of object keys and one operation, and it runs the job asynchronously, at scale, with a per-object completion report.

How a batch job is structured

A job has four parts:

• A manifest — a CSV (or an S3 Inventory report) stored in S3, listing bucket, key, and optionally version ID for every object to process. If versioning matters to you, include version IDs; otherwise the job operates on the current version of each key.
• An operation — one of: Copy, ReplaceTagging, DeleteTagging, ReplaceAcl (legacy), InitiateRestore (Glacier/Deep Archive), Object Lock retention/legal hold, or LambdaInvoke for anything custom.
• An IAM role — the job assumes this role to read the manifest and act on the objects. This is where most first attempts fail: the role needs s3:GetObject on the manifest bucket and whatever permissions the operation requires on the target objects.
• A completion report — delivered to an S3 prefix you choose, listing every task as succeeded or failed with an error code. This is your audit trail.

Jobs run asynchronously and scale to millions of objects. You track progress through describe-job or the AWS/S3 CloudWatch metrics for the job.

Worked example: restore a Glacier prefix for an audit

Say compliance asks for every object under s3://finance-archive/ledger/2021/, which was transitioned to Glacier years ago. Restoring objects one by one via the console is not an option.

First, build a manifest. The practical route is an S3 Inventory report filtered to the prefix, but for a small verification run, write a CSV by hand:

finance-archive,ledger/2021/jan.parquet
finance-archive,ledger/2021/feb.parquet
finance-archive,ledger/2021/mar.parquet

Upload it to a bucket the job role can read, then create the job from any machine with the AWS CLI and permission to call s3control:CreateJob (typically an admin or data-platform role):

aws s3control create-job \
  --account-id <your-account-id> \
  --operation '{"S3InitiateRestoreObject": {"ExpirationInDays": 7, "GlacierJobTier": "STANDARD"}}' \
  --manifest '{"Spec": {"Format": "S3BatchOperations_CSV_20180820", "Fields": ["Bucket", "Key"]}, "Location": {"ObjectArn": "arn:aws:s3:::<manifest-bucket>/restore-2021.csv", "ETag": "<manifest-etag>"}}' \
  --report '{"Bucket": "arn:aws:s3:::<report-bucket>", "Prefix": "batch-reports/", "Format": "Report_CSV_20180820", "Enabled": true, "ReportScope": "AllTasks"}' \
  --priority 10 \
  --role-arn arn:aws:iam::<your-account-id>:role/<batch-ops-role> \
  --no-confirmation-required

Replace the placeholders with your account ID, bucket ARNs, the manifest's ETag (from aws s3api head-object), and the job role ARN. The job starts in a confirmation state unless you pass --no-confirmation-required — omitting that flag is actually a useful safety net for large jobs, since it forces a human to review the scope before anything runs.

Check progress:

aws s3control describe-job --account-id <your-account-id> --job-id <job-id>

Then open the completion report in the report bucket. Every manifest row should appear with a succeeded status; failures include an error code like RestoreAlreadyInProgress, which is usually benign. Remember the restore is temporary — ExpirationInDays: 7 means the copies vanish after a week, and the restore itself takes hours at the STANDARD tier.

When LambdaInvoke is the right operation

The built-in operations cover the common cases, but LambdaInvoke turns Batch Operations into a generic fan-out engine: the manifest drives the work, your function does whatever per-object logic you need — transcoding, PII scanning, re-encrypting with a new KMS key. Batch Operations invokes the function with the object details and records success or failure based on your response. The key discipline is to make the function idempotent, because a failed task may be retried and you don't want a retry to double-apply a change.

Trade-offs and limits worth knowing

• Job concurrency cap. Each account is limited to 2,500 active batch jobs. Submitting beyond that gets throttled, so pipelines that create jobs per-upload need a queue in front.
• Silent permission failures. If the job role can't read the manifest, the job fails without touching a single object. Always verify with a three-row manifest before pointing a job at a million-row inventory report.
• Versioning. Without version IDs in the manifest, you operate on current versions only. For compliance workflows (Object Lock retention, legal holds), pin the version IDs.
• Cost shape. You pay per object processed plus the underlying request costs (e.g., Glacier retrieval fees for restores). There is no extra data-transfer charge beyond normal S3 rates, but a restore job over a large archive can still be a meaningful bill — estimate before confirming.

The takeaway

If your runbook contains a script that paginates ListObjectsV2 and loops, Batch Operations probably replaces it with something faster, auditable, and safer. Start small: hand-write a three-object manifest, run the operation, and read the completion report end to end. Once you've seen the report answer "did every object get processed?" without you writing a line of retry logic, the loop script will feel like a relic.