How to tag all objects in a S3 bucket using AWS CLI?
Introduction
AWS S3 supports the ability to tag objects. Each tag is a key-value pair. You can associate up to 10 tags with an object.
Some of the benefits of tagging objects in S3 are:
- Categorizing your storage
- Fine-grained access control: You can grant IAM user permissions to read-only objects with specific tags
- Fine-grained lifecycle management: You can specify a tag-based filter in a lifecycle rule.
AWS s3api
supports Object Tagging via the put-object-tagging
command. However, this command only supports tagging one object at a time. In this article, we will look at how you can tag all objects within a S3 bucket.
Table of contents
List all objects in the bucket
First, we will use the list-object
command to filter and list all the objects we want to tag.
aws s3api list-objects --bucket my-test-bucket --prefix my-prefix/
This command will return all the objects in the bucket my-test-bucket
with my-prefix/
prefix.
Output:
{
"Contents": [
{
"Key": "prefix/my-file.json",
"LastModified": "2019-12-03T19:03:26+00:00",
"ETag": "\"c416452ad7f219e0bac6adc89d3a9285\"",
"Size": 8440,
"StorageClass": "STANDARD",
...
},
....
]
}
Retrieve object key
Next, we will use the --query
flag to only retrieve the object key as text. Let’s modify the command as follows:
aws s3api list-objects --bucket my-test-bucket --prefix my-prefix/ --query 'Contents[].[Key]' --output text
Things to note:
'Contents[].[Key]'
: We only want theKey
field so we are filtering on that field.--output text
: We want the output in a text format so its easier to pipe that output
Output:
prefix/my-file.json
prefix/report.csv
Add tags to objects
Now, we will use the output of the previous command and pipe it to the xargs
command. xargs
supports reading streams of data from standard input and pass it as input for another command. You can see examples of how to use xargs here.
We will use the put-object-tagging
command to tag individual objects.
This is what the complete command looks like:
aws s3api list-objects --bucket my-test-bucket --prefix my-prefix/ --query 'Contents[].[Key]' --output text | xargs -n 1 aws s3api put-object-tagging --bucket my-test-bucket --tagging 'TagSet=[{Key=foo,Value=bar}]' --key
Things to note:
- We are adding the tag
foo=bar
to our objects --key
gets passed as input to the xargs command- We can parallelise the command by using
-P
flag
We can verify that our objects were tagged by running the following command:
aws s3api get-object-tagging --bucket my-test-bucket --key prefix/my-file.json
{
"TagSet": [
{
"Key": "foo",
"Value": "bar"
}
]
}
Conclusion
You can use the following command to tag multiple objects at once:
aws s3api list-objects --bucket my-test-bucket --prefix my-prefix/ --query 'Contents[].[Key]' --output text | xargs -n 1 aws s3api put-object-tagging --bucket my-test-bucket --tagging 'TagSet=[{Key=foo,Value=bar}]' --key