Message Queues: Work Handed Off for Later

A message queue holds messages that describe work to be done, until a worker takes each one. Producers put messages in; consumers, often called workers, take them out. Each message is taken by one consumer, and stays in the queue until that consumer confirms it has finished, so a crash halfway through does not lose it.

The life of one message. A producer sends it, and the queue stores it. A free worker receives it, and for a time no other worker can see it. The worker does the job, then sends an acknowledgement, and the queue deletes the message. If no acknowledgement arrives within the time limit, because the worker crashed, the message becomes visible again and another worker takes it.

An analogy. A queue is the ticket rail in a restaurant kitchen: orders are clipped on in the order they arrive, and whichever cook is free takes the next ticket. The analogy breaks at a dropped ticket: a cook who drops one loses the order, while a queue hands an unacknowledged message to someone else.

Why use one. The producer does not wait for the work, and does not need to know which worker does it, or how many workers there are. To process faster, add workers; each takes messages from the same queue. When a burst of messages arrives, they wait in the queue, and the workers keep their own steady pace.

What it does not promise. Most queues deliver each message at least once, not exactly once, so a worker can occasionally receive one twice. Many do not keep strict order between messages either, unless asked to.

Products. Amazon Simple Queue Service (SQS) and RabbitMQ are widely used message queues.

So use a queue when work can wait and should be done once by whichever worker is free, and scale the work by adding workers rather than by speeding up the producer.