Practice: Diagnosing Ten Failures at Scale

Recognising a failure from its symptom is practised here on ten incident reports, none of which a test suite would catch. For each, name the failure and one fix before opening the answers.

The reports.

1. The home page cache entry expires at 09:00 and the database is overwhelmed for a minute.
2. After a ten-second outage, every client reconnects in the same instant, and the service falls over again.
3. One shard of the cache sits at 100% while the other nine idle; one celebrity's profile is on it.
4. An orders page that took 20 ms with ten orders takes 20 seconds for a customer with ten thousand.
5. The search service slows down; clients retry, traffic triples, and search stops entirely.
6. An application server runs fine for months, then crashes with out-of-memory, again and again.
7. One slow report query runs at lunchtime, and every other request hangs waiting for a database connection.
8. A user saves their new address, refreshes, and sees the old one.
9. A network glitch, and one customer is charged twice for one order.
10. A queue grows all day and the broker runs out of memory at 18:00.