Unbounded Caches and the Slow Memory Leak

A cache with no size limit and no eviction keeps every key it is ever given, so it grows with every distinct value read and never shrinks. It behaves like a slow memory leak: the process works for weeks, then runs out of memory and is killed, with no change in traffic to explain it.

The symptom. Every application server's memory climbs steadily over three weeks. Traffic is flat. Then the servers are killed by the operating system for using too much memory, restart, and the climb begins again.

The trigger. A developer added a cache inside the application to speed up product lookups: a map from product id to product, filled on every miss. It had no maximum size and no expiry. In testing it held a few hundred products.

The mechanism. Each distinct key adds an entry, and nothing ever removes one. Traffic is flat, but the set of different products, users or search terms read over three weeks keeps growing, so memory grows with it, and the process looks healthy right up to the day it is killed.

The cost. Every server fails on a cycle, often at the same time if they were deployed together, and each restart throws away the whole cache, sending a wave of misses to the database.

The remedy. Give every cache a maximum size, in items or bytes, and an eviction rule such as LRU, and give entries a TTL so stale ones leave on their own. Watch the cache's size as a metric, so growth shows up long before the limit does.

So never create a cache without deciding its size: a cache with no limit is a memory leak that has not failed yet.