Introduction

AWS S3 CLI doesn’t support regular expressions with the path parameter. To use regular expressions with the AWS S3 CLI, you can use the --include and --exclude parameters to filter the files that you want to copy. These options allow you to specify a regex pattern to filter the files. AWS will only return the files that match the pattern.

In this tutorial, we will look at how filters work and how we can use them to copy files that match a pattern.

Exclude & Include Filters

The --exclude and --include parameters perform pattern matching that either excludes or includes a file or object from the result. The following pattern symbols are supported.

*: Matches everything
?: Matches any single character
[sequence]: Matches any character in sequence
[!sequence]: Matches any character not in sequence

It is important to note the the order of the parameter matters. Whichever parameter is used first will be applied first. By default, all files are included. If you want to exclude a file, you must use the --exclude parameter first. Using just --include is not sufficient based on AWS documentation:

Note that, by default, all files are included. This means that providing only an --include filter will not change what files are transferred. --include will only re-include files that have been excluded from an --exclude filter. If you only want to upload files with a particular extension, you need to first exclude all files, then re-include the files with the particular extension. This command will upload only files ending with .jpg:

--exclude "*" --include "*.txt": In this case, all files will be excluded first. Then, only the files that match the pattern *.txt will be included.

--include "*.txt" --exclude "*": In this case, all files will be excluded.

Examples

The following example will copy all files that match the pattern *.html:

aws s3 cp s3://bucket/ s3://bucket/ --exclude "*" --include "*.html"

The following example will recursively copy only the files that match the pattern 2016-08*:

aws s3 cp s3://data/ . --recursive --exclude "*" --include "2016-08*"`

Conclusion

In this tutorial, we looked at how we can use the --include and --exclude parameters to filter the files that we want to copy. We also looked at how the order of the parameters matter. We can use these parameters to copy files that match a pattern.