Load Balancers: Spreading Requests Across Servers

A load balancer receives every request for a service and sends each one to a single healthy copy of the server behind it. Two things decide where a request goes: health checks, which decide which copies may receive traffic at all, and an algorithm, which picks one of those copies for this request.

An analogy, and where it breaks. A load balancer works like a restaurant host who seats each arriving party with one of several waiters, and skips a waiter who has gone home. The analogy breaks on one point: a host can see which waiter is rushed off their feet, while a balancer using the simplest algorithm does not look at how busy a server is at all.

Health checks. Every few seconds the balancer sends each server a small request, such as GET /health. A server that fails several in a row is taken out of rotation; one that passes again is put back. A failed server stops receiving users' requests within seconds, without anyone acting.

Algorithms. Round robin sends requests to each server in turn, and is the default in NGINX. Least connections sends each request to the server with the fewest requests in progress. IP hash turns the client's address into a number and always picks the same server for the same client. AWS's Application Load Balancer offers round robin and least outstanding requests; its Network Load Balancer hashes each connection and keeps it on one server for the connection's life.

What it relies on. A load balancer can send a request anywhere only if every server can answer it, which is why the servers behind one are kept stateless. It also becomes a part every request passes through, so it needs a spare of its own, or a managed service that runs on several machines.

So think of a load balancer as two decisions made on every request: which servers are healthy, and which of them is next.