How to use Boto3 to upload files to an S3 Bucket?
Introduction
AWS Boto3 is the Python SDK for AWS. Boto3 can be used to directly interact with AWS resources from Python scripts.
Boto3’s S3 API has 3 different methods that can be used to upload files to an S3 bucket.
In this tutorial, we will look at these methods and understand the differences between them.
Table of contents
Prerequisites
- Python3
- Boto3: Boto3 can be installed using pip:
pip install boto3
- AWS Credentials: If you haven’t setup your AWS credentials before, this resource from AWS is helpful.
upload_file
The upload_file
method uploads a file to an S3 object.
The function signature of the method is:
upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)
Parameters
The parameters are:
Filename (str) -- The path to the file to upload.
Bucket (str) -- The name of the bucket to upload to.
Key (str) -- The name of the key to upload to.
ExtraArgs (dict) -- Extra arguments that may be passed to the client operation. For allowed upload arguments see boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS.
Callback (function) -- A method which takes a number of bytes transferred to be periodically called during the upload.
Config (boto3.s3.transfer.TransferConfig) -- The transfer configuration to be used when performing the transfer.
upload_file
reads a file from your file system and uploads it to S3. It supports Multipart Uploads. If you try to upload a file that is above a certain threshold, the file is uploaded in multiple parts.
Example of how to use this method:
import boto3
client = boto3.client("s3")
client.upload_file("path/to/file.txt", "your-bucket", "path/to/key.txt")
upload_fileobj
The function signature is:
upload_fileobj(Fileobj, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)
Parameters
Fileobj (a file-like object) -- A file-like object to upload. At a minimum, it must implement the read method, and must return bytes.
Bucket (str) -- The name of the bucket to upload to.
Key (str) -- The name of the key to upload to.
ExtraArgs (dict) -- Extra arguments that may be passed to the client operation. For allowed upload arguments see boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS.
Callback (function) -- A method which takes a number of bytes transferred to be periodically called during the upload.
Config (boto3.s3.transfer.TransferConfig) -- The transfer configuration to be used when performing the upload.
upload_fileobj
is similar to upload_file
.
The major difference between the two methods is that upload_fileobj
takes a file-like object as input instead of a filename. The file-like object must implement the read method and return bytes.
The file object doesn’t need to be stored on the local disk either. It may be represented as a file object in RAM.
Example:
File object in memory:
fo = io.BytesIO(b'my data stored as file object in RAM')
client.upload_fileobj(fo, bucket, key)
File stored on disk:
with open('filename', 'rb') as data:
client.upload_fileobj(data, 'mybucket', 'mykey')
put_object
put_object
maps directly to the low level S3 API. It doesn’t support multipart uploads.
The function signature is defined here.
Some of the important parameters are:
Body (bytes or seekable file-like object) -- Object data.
Bucket (string) -- The bucket name to which the PUT action was initiated.
Key (string) -- Object key for which the PUT action was initiated.
Example:
with open(filename, "rb") as f:
client.put_object(
Bucket=bucket,
Body=f,
Key=key
)