A RAG Q&A Went from 8 Seconds to 2 Minutes — Redis Was the Silent Killer
Python's redis-py `socket_connect_timeout` is not a hard deadline — `ping()` and other operations can hang well beyond it. Any service that calls Redis multiple times per request without a circuit breaker will multiply that hang into minutes of stall, turning a cache from an accelerator into a denial-of-service.
A self-built knowledge-base Q&A system degraded overnight from 8-second responses to 2–5-minute page freezes. The Vue frontend was ruled out early; the real stall was on the backend. Five crude timing print statements inside the `ask()` function revealed that Redis cache lookups were consuming 10–21 seconds each, and a single request was hitting Redis four separate times — cache lookup, session read, session save, and cache save — multiplying the hang.
Temporarily replacing all Redis calls with `return None` dropped total latency to 16 seconds, confirming Redis as the sole culprit. The root cause was twofold: Python's redis-py `socket_connect_timeout` does not constrain `ping()`, so a connection attempt to a stopped Redis container hung for tens of seconds before failing, and the application had no circuit breaker to stop retrying on every request.
The fix added a global 60-second circuit-breaker flag in `redis_cache.py`. Once a connection fails, all Redis calls are skipped for 60 seconds. After also starting the Redis container, a cache hit returned in 314ms.
redis-py's `socket_connect_timeout` is misleading: it does not bound `ping()`, so a stopped container can hang a request for tens of seconds regardless of the configured timeout.
A cache that retries on every request without a circuit breaker becomes a multiplier of failure — four Redis calls per request turned a 10-second hang into a 40+ second stall.
The cheapest performance instrumentation — raw print statements with timestamps — beat intuition and profiling tools for this class of problem because it showed exactly where wall-clock time was disappearing.
The "turn it off" test (stubbing all Redis calls to return None) is a high-signal, low-effort diagnostic that should be in every developer's toolkit for dependency-induced latency.