Introduction

AWS Boto3 is the Python SDK for AWS. Boto3 can be used to directly interact with AWS resources from Python scripts.

Some AWS operations limit the number of results returned in a single response. For example, the list_buckets operation returns at most 1000 buckets. If there are more than 1000 buckets, the response will contain a IsTruncated field with value True and a NextMarker field with a value that can be used to retrieve the next page of results.

Boto3 provides a paginator to handle this. A paginator is an iterator that will automatically paginate results for you. You can use a paginator to iterate over the results of an operation.

In this article, we will look at how to use Boto3 to paginate results from AWS operations.

Table of contents

Prerequisites

  • Python3
  • Boto3: Boto3 can be installed using pip: pip install boto3
  • AWS Credentials: If you haven’t setup AWS credentials before, this resource from AWS is helpful.

What is a Paginator?

A paginator is an iterator that will automatically paginate results for you. You can use a paginator to iterate over the results of an operation.

Pagination is used to limit the number of results returned in a single response. This helps reduce the network traffic and the amount of data that needs to be processed.

How to create a new paginator?

Paginators are created using the get_paginator method of the client. The get_paginator method takes the name of the operation as an argument.

For example, you can create a paginator for the list_buckets S3 operation using the following code:


import boto3

s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects')

How to use a paginator?

Once you have created a paginator, you can use it to iterate over the results of an operation. The paginate method of the paginator is used to iterate over the results of an operation. The paginate method takes the same arguments as the operation.

For example, you can use the paginate method to iterate over the results of the list_buckets operation:


import boto3

s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects')

for page in paginator.paginate(Bucket="my-bucket"):
    print(page)

How to customize the paginator?

The paginate method of the paginator accepts a PaginationConfig object as an argument. The PaginationConfig object can be used to customize the paginator.

For example, you can use the PaginationConfig object to customize the number of results returned in a single response:


import boto3

s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects')

for page in paginator.paginate(Bucket="my-bucket", PaginationConfig={'PageSize': 10}):
    print(page)

The different options available in the PaginationConfig object are:

  • MaxItems: The maximum number of items to return. If more items exist than the specified MaxItems, the NextToken response element is present and has a value (isn’t null). Include that value as the NextToken request parameter in the next call to the operation to get the next part of the results. Note that the service might return fewer results than the MaxItems value.
  • PageSize: The number of results to return in each page. The service might return fewer results than the specified PageSize value. If the service returns any results, it should return at least one.
  • StartingToken: A token to specify where to start paginating. This is the NextToken from a previous response.

Examples

Example 1: List all objects in an S3 bucket

In this example, we will use the list_objects_v2 operation to list all objects in an S3 bucket.


import boto3
s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects_v2')

for page in paginator.paginate(Bucket="www.learnaws.org", PaginationConfig={"PageSize":10}):
    print([c["Key"] for c in page["Contents"]])

Example 2: List all IAM roles

In this example, we will use the list_roles operation to list all IAM roles in the current account.


import boto3

iam = boto3.client('iam')
paginator = iam.get_paginator('list_roles')

for page in paginator.paginate():
    print([c["RoleName"] for c in page["Roles"]])

Example 3: List all EC2 instances

In this example, we will use the describe_instances operation to list all EC2 instances in the current account.


import boto3

ec2 = boto3.client('ec2')
paginator = ec2.get_paginator('describe_instances')

for page in paginator.paginate():
    print([c["InstanceId"] for c in page["Reservations"][0]["Instances"]])

Conclusion

In this article, we looked at how to use Boto3 to paginate results from AWS operations. We also looked at some examples of how to use paginators.