跪拜 Guibai
← Back to the summary

A RAG Q&A Went from 8 Seconds to 2 Minutes — Redis Was the Silent Killer

An API Latency Investigation: From 2 Minutes to 314ms in 5 Steps

I built a knowledge base Q&A system. When it went live, it worked fine — 8 seconds per answer. Then one day it suddenly took 2–5 minutes, the page froze completely, and even Ctrl+C wouldn't respond. Users said, "It used to be fast." I spent two days investigating and finally squeezed a single query down to 314ms. Five steps, each narrowing the suspect list.

1. Symptoms: 8 Seconds Became 2 Minutes

After the Vue frontend went live, a single Q&A took 2–5 minutes. The old Jinja2 version was just as slow. The page froze entirely, and Ctrl+C in the terminal did nothing — meaning the backend request was stuck, not frontend rendering.

2. Step One: Rule Out the Vue CDN

Vue loaded via CDN, so the browser blocked on script download. Accessing unpkg.com from within China is extremely slow, so I suspected it.

Switched to the npmmirror domestic mirror — still slow. Do the math: even if script loading is slow, it should time out after a few dozen seconds. A 5-minute freeze means the problem is definitely in the backend.

As a side cleanup, I downloaded Vue locally into static/, eliminating the CDN dependency entirely — not the root cause, but worth doing.

3. Step Two: The Model Name Was a Trigger, Not the Root Cause

While previously debugging a DeepSeek 400 error, I discovered the model name had changed from deepseek-chat to deepseek-v4-pro. Switching to v4-pro made the API work again, but I didn't notice v4-pro was much slower.

Switching to deepseek-v4-flash cut API time to 5–16s (originally about 28s). It helped, but it couldn't explain 2 minutes — even if the API took 16s, there was still over a minute unaccounted for.

4. Step Three: Add Timing Logs, Lock onto Redis

I inserted print statements inside ask() to output the time each step took (debug output at the time, removed after the fix):

[Timing] Redis cache lookup: 10.3s     ← Suspicious!
[Timing] ChromaDB retrieval: 11.1s     ← Actually only 0.8s (including wait)
[Timing] Before DeepSeek call: 32.7s   ← Where did the 21 seconds in between go?
[Timing] DeepSeek returned: 45.1s

Five lines of print beat an hour of blind guessing. ChromaDB retrieval actually took only 0.8s, but the "Redis cache lookup" entry alone swallowed 10–21 seconds, and there was a mysterious extra 20 seconds between ChromaDB and DeepSeek — that was yet another Redis call.

5. Step Four: Temporarily Disable Redis, Confirmed

Replaced all Redis calls with empty functions that returned None, skipping Redis entirely.

Result: ChromaDB 0.6s, DeepSeek 16s, total 16s. Redis was the culprit.

6. Step Five: Fix Redis Hanging + Start Redis

The root cause had two layers:

Layer one: socket timeout is unreliable. The code set socket_connect_timeout, but ping() does not respect this parameter. When the Redis container wasn't running, a single connection hung for 10–21 seconds before throwing an error.

Layer two: a single request touched Redis 4 times. I traced through the source in ask() (rag_engine.py): cache lookup, read session, save session, save cache — four call sites. When Redis hung, every single one waited out the full timeout.

The fix: add a global circuit-breaker flag — if it can't connect, lock for 60 seconds and skip Redis entirely during that window. Here is the actual source from redis_cache.py (verbatim):

# Global state: avoid retrying Redis on every request
_redis_client = None
_redis_unavailable_until = 0.0  # Timestamp — don't retry before this


def _get_client():
    """Lazy-load Redis connection — if it can't connect, don't retry for 60s"""
    global _redis_client, _redis_unavailable_until

    now = _time.time()
    if now < _redis_unavailable_until:
        return None  # Recently failed, skip

    if _redis_client is not None:
        try:
            _redis_client.ping()
            return _redis_client
        except Exception:
            _redis_client = None

    try:
        import redis
        host = os.getenv("REDIS_HOST", "localhost")
        port = int(os.getenv("REDIS_PORT", "6379"))
        db = int(os.getenv("REDIS_DB", "0"))
        password = os.getenv("REDIS_PASSWORD", None)

        _redis_client = redis.Redis(
            host=host, port=port, db=db, password=password,
            socket_connect_timeout=0.5, socket_timeout=0.5,
        )
        _redis_client.ping()
        _redis_unavailable_until = 0.0  # Connected, reset flag
        logger.info(f"Redis connected: {host}:{port}")
        return _redis_client
    except Exception:
        logger.info("Redis not connected — won't retry for 60s")
        _redis_unavailable_until = now + 60  # Skip connection logic for 60s
        return None

Redis itself also wasn't running, so I quickly ran docker start redis-test to start it. After the fix: second hit on the same question, cache hit → 314ms.

7. Timeline Comparison

Stage Request Latency Root Cause
Original state (old deepseek-chat) ~8s Normal
Model name expired + switched to v4-pro 2–5 min Redis hanging 10–21s per call × multiple calls + API 28s
Switched to v4-flash 1.9 min Redis hanging still unresolved, API down to 16s
Temporarily disabled Redis 16s Only DeepSeek API time remains
Redis fixed + started 5s (first) / 314ms (cache hit) Normal

8. Lessons

  1. An external dependency failure must not drag down the main flow. Redis is a cache — it should be "speed up when available, skip when unavailable." Retrying for 21 seconds on a failed connection completely defeats the purpose of adding a cache.
  2. socket_connect_timeout is unreliable. The Python redis library's timeout parameter does not guarantee all operations complete within the timeout. Application-layer defense is more reliable than library parameters — if it can't connect, lock for N seconds.
  3. Every external dependency is a latency source. A single function called Redis 4 times, and each call could hang. "Helper" features like caching, logging, and monitoring can all drag down the main flow at critical moments.
  4. Timing logs are the fastest investigation tool. Print the time each step takes, and the bottleneck is visible immediately.
  5. Turn it off, and investigation becomes simple. Suspect Redis? Set the return value to None and confirm instantly. Fix it, then turn it back on.

9. Quick Troubleshooting Reference

Question Answer
Why socket timeout is unreliable Some redis-py operations (e.g., ping) are not strictly bound by socket timeout and can still hang beyond it
How to ensure a failed cache doesn't drag down the main flow Circuit breaker: if it can't connect, lock for N seconds and skip the cache entirely during that window
First step when investigating a performance issue Add per-step timing logs; use data to locate the problem, don't rely on guesswork

One-Sentence Summary

A Q&A query went from 8 seconds to 2 minutes, and in the end it was the cache dragging down the main flow. Write code treating every external dependency as "might fail" — speed up when available, skip when unavailable, circuit-break for 60 seconds, and never let the cache hold the main flow hostage.

Next post: why my knowledge base searches accurately — the precision gap after switching from English embeddings to a Chinese model, and how the V3 incremental index works.