A ConcurrentHashMap Cached Embeddings to Save Pennies — and Nearly Killed an AI Chatbot at Launch
In-process caching of AI model outputs looks cheap and fast until it turns the JVM heap into a session database. The resulting GC pauses hit every request, not just the cached ones, and the fix is architectural — externalize state to Redis — not a JVM flag.
A production stress test with 1000 concurrent users sent P99 latency from 2 seconds to 5 seconds, with the latency graph spiking like an EKG. GC logs revealed Full GC pauses lasting 2.5 seconds and reclaiming only 400 MB out of 3.2 GB. A jmap histogram showed 200 MB tied up in float arrays — the locally cached embedding vectors stored per session inside a ConcurrentHashMap. Each 1024-dimensional float array cost 4 KB, and 1000 concurrent sessions pushed 130 MB straight into the old generation, creating a vicious cycle of heap exhaustion and low-efficiency Full GCs. MAT analysis confirmed the reference chain: SessionManager.sessions → ConversationContext → embeddingCache → float[1024], all GC-root reachable and never reclaimed while sessions lived. The emergency fix capped the cache and doubled the heap to 8 GB, but the real solution moved session state to Redis with 30-minute TTL auto-expiry, eliminating Full GCs entirely and bringing P99 back down to 2.2 seconds.
The core mistake was treating the JVM heap as a database for session state — a design shortcut that works at low concurrency and collapses under load.
Embedding caching saved fractions of a cent per API call but cost seconds of latency for every user, a trade-off that only becomes visible during stress testing.
ConcurrentHashMap's thread safety and zero network overhead seduced the developer into ignoring its fundamental limitation: objects it holds are GC roots and won't be collected until explicitly removed.
Redis TTL auto-expiry eliminated an entire class of hand-written cleanup code (scheduled tasks, removeIf loops) that is error-prone and easy to get wrong.
The four-step tuning method — baseline, analyze, tune, verify — treats JVM optimization as a data-driven experiment rather than parameter memorization.