What is SQS?

Amazon Simple Queue Service (SQS) is a managed, message-queue service that enables us to build scalable and reliable systems. Queues allow services to be decoupled. They can communicate with each other asynchronously and are especially useful when the throughput of the producing service is different from the throughput of the consuming service.

How does it work?

/assets/img/sqs-fifo-deep-dive/sqs.png

SQS provides a message-queue service. To use SQS, you need the following components:

  • Producer(s): Producers are responsible for sending messages to a particular queue. Messages are stored in the queue when they are sent by the producer.
  • Consumer(s): Consumers are responsible for retrieving and processing the messages from a particular queue. Messages must be deleted by the consumer after processing to ensure they aren’t processed by any other consumers.

Standard

Standard queues are the default type of queue provided by SQS

FIFO

FIFO (First-in First-Out) queues are designed for applications where the order of events is critical or where de-duplication of messages is required.

Key Differences between Standard and FIFO queues

  Standard FIFO
Message Order Best-effort ordering Order is maintained (first-in-first-out)
Throughput Nearly unlimited 3000 messages per second with batching or 300 messages per second without batching
Delivery At-least once Exactly once

Message ordering

A standard queue tries to preserve the order of messages (best-effort), but there is a possibility of a message being delivered out of order.

In a FIFO queue, messages are grouped into “Message groups” and all messages within a message group are sent and received in strict order. A message group is required to send and receive a message from a FIFO queue.

Message Delivery

A standard queue guarantees at-least once delivery of a message. There are scenarios when a message could be received by a consumer more than once. It is recommended that applications using a Standard queue build their applications to be idempotent.

A FIFO queue, on the other hand, guarantees exactly-once message delivery. FIFO queues also provide different ways to configure message de-duplication.

  • Content-based de-duplication: uses a SHA-256 hash to generate the message de-duplication ID using the body of the message
  • Explicitly provide the message de-duplication ID

Throughput

Standard queues have no restriction on the amount of messages that can be produced or consumed, thus providing a nearly unlimited throughput.

FIFO queues, however, have strict thresholds on the throughput. FIFO queues can process at most 3000 messages per second with batching or 300 messages per second without batching.

When to use a FIFO queue?

FIFO queues are useful for applications that depend on the order of the events and / or need message de-duplication to built into the queue itself.

Examples of such applications could be:

  • Processing user-entered inputs in the right order (as entered)
  • Sending notifications to a customer where order is critical
  • Integrating with a third-party system where events need to be processed in-order

Limitations of a FIFO queue

Performance

The overhead of supporting exactly-once and in-order messaging impacts the performance of FIFO queues. This limitation could potentially become a bottleneck in your application. A more detailed analysis of the performance limitations is discussed in this blog post.

Message Ordering

FIFO queues guarantee message ordering based on the order in which they are received by SQS. Thus, its not possible to have total global ordering if you have multiple producers.

Message de-duplication

FIFO queues support content-based de-duplication. This is useful in the case where the producer might accidentally send a message more than once. FIFO de-duplicates messages over a 5 minute window which is not configurable. Any applications that rely on message de-duplication should be aware of this restriction.

Conclusion

SQS FIFO queues provide a way to ensure messages are delivered in-order and exactly-once. However, these features come at the cost of some performance limitations. SQS FIFO queues should ideally be used only for use-cases where these features essential otherwise the Standard queues would be a better fit.