AWS S3 CLI: How to use wildcards and filters to copy files?
Introduction
AWS S3 CLI doesn’t support regular expressions with the path parameter. To use regular expressions with the AWS S3 CLI, you can use the --include
and --exclude
parameters to filter the files that you want to copy. These options allow you to specify a regex pattern to filter the files. AWS will only return the files that match the pattern.
In this tutorial, we will look at how filters work and how we can use them to copy files that match a pattern.
Exclude & Include Filters
The --exclude
and --include
parameters perform pattern matching that either excludes or includes a file or object from the result. The following pattern symbols are supported.
*: Matches everything
?: Matches any single character
[sequence]: Matches any character in sequence
[!sequence]: Matches any character not in sequence
It is important to note the the order of the parameter matters. Whichever parameter is used first will be applied first. By default, all files are included. If you want to exclude a file, you must use the --exclude
parameter first. Using just --include
is not sufficient based on AWS documentation:
Note that, by default, all files are included. This means that providing only an --include filter will not change what files are transferred. --include will only re-include files that have been excluded from an --exclude filter. If you only want to upload files with a particular extension, you need to first exclude all files, then re-include the files with the particular extension. This command will upload only files ending with .jpg:
--exclude "*" --include "*.txt"
: In this case, all files will be excluded first. Then, only the files that match the pattern *.txt
will be included.
--include "*.txt" --exclude "*"
: In this case, all files will be excluded.
Examples
The following example will copy all files that match the pattern *.html
:
aws s3 cp s3://bucket/ s3://bucket/ --exclude "*" --include "*.html"
The following example will recursively copy only the files that match the pattern 2016-08*
:
aws s3 cp s3://data/ . --recursive --exclude "*" --include "2016-08*"`
Conclusion
In this tutorial, we looked at how we can use the --include
and --exclude
parameters to filter the files that we want to copy. We also looked at how the order of the parameters matter. We can use these parameters to copy files that match a pattern.