Card 07/ 08
Practice: Naming the Failure in Four Incident Reports
Blocked threads, busy retries, one starved worker and intermittent timeouts: name each and its fix.
Deadlock (threads blocked forever in a cycle), livelock (threads busy but making no progress) and starvation (one thread never getting its turn) are told apart by processor use and by whether any thread is making progress. Each description below comes from a different incident report. Name the failure in each and say what would fix it, all four before you open the answer.
| What was observed | |
|---|---|
| 1 | Eight request threads BLOCKED, processor at two per cent, no progress for an hour |
| 2 | Processor at ninety per cent, two threads logging "retrying" thousands of times a second |
| 3 | Nine of ten workers finishing normally; one has not completed a task since start-up |
| 4 | Threads recover after five seconds, transfers fail under load, no hang |
The four failures, and what fixes each
| Failure | Fix | |
|---|---|---|
| 1 | Deadlock | A global lock ordering, applied everywhere both locks are taken |
| 2 | Livelock | Randomised backoff, or one thread that decides rather than both yielding |
| 3 | Starvation | A fair lock, or work-stealing so no thread depends on winning a race |
| 4 | Deadlock, with a timeout on top | Order the locks; keep the timeout as the net |
Processor use separates the first two, which is why it is the first thing to look at. Idle and stuck is a cycle of locks; busy and stuck is threads reacting to each other.
The fourth is the one that gets closed as a different bug. Intermittent failures under load with no hang look like a capacity problem, and they are the same cycle with a safety net under it.
← → move through sections and cards