Single Points of Failure

A single point of failure is any part of a system that has no spare, so that when it stops, the whole system stops with it. Adding more copies of one part does not remove the single points elsewhere, and the one that remains decides how available the whole system is.

The symptom. A shop runs ten application servers so that a failure cannot take it down. One evening the whole site is unreachable for forty minutes. All ten servers were healthy throughout.

The trigger. The team counted servers and forgot to count the load balancer in front of them. It was one machine, and when it failed there was no path from any reader to any server.

The mechanism. A request needs every part on its path to be up. Spares in one part help only that part; the system is no more available than its weakest part with no spare. The ten servers made server failures harmless, and did nothing about the one machine every request passed through first.

The cost. Forty minutes of downtime is most of a year's budget at 99.99%, spent on a part that cost the least to duplicate.

The remedy. Walk the request path and, at each part, ask what happens if it stops. Every part whose answer is "the site goes down" needs a spare that takes over without a person: a second load balancer, or a managed one that runs on several machines already, and a replica the database can fail over to. In the diagram above, the replica covers the database; nothing covers the load balancer.

So before adding a tenth copy of anything, find the part that still has only one: availability is set by the parts you did not duplicate.