Health Checks That Pass While the Application Fails

A health check proves only what it touches. A check that asks for a page needing nothing but the web server will pass while the database behind it is unreachable, and the load balancer will keep sending users to a server that fails every real request.

The symptom. Users see errors on every page that loads data. The load balancer's dashboard shows all three servers healthy and in rotation, and nothing has been taken out.

The trigger. One server has lost its connection to the database, perhaps after a network change or a pool of connections that ran out. Its health check is GET /health, which returns a fixed 200 OK without reading anything.

The mechanism. A load balancer knows nothing about your application beyond the answer to its check. A 200 means "healthy" to it, whatever the code behind that 200 did. The check measured only whether the process was alive.

The cost. A third of users fail for as long as the problem lasts, while the one automatic defence in place reports that all is well.

The remedy. Make the check touch what a real request needs, cheaply: one trivial query to the database, a ping to the cache. Keep it fast and give it a timeout, because a check that hangs counts as a failure. Take care with a shared dependency: if the database itself is down, a deep check takes every server out at once, so some teams fail the check only on problems local to the server.

So decide what "healthy" means for a real request, and make the check prove that, not merely that the process answers.