RAG endpoint burns 8 CPU cores while chat uses 0.16: reranker is the real cost
For teams containerizing RAG services, this pinpoints reranking as the dominant CPU cost and shows that a lexical fallback can cut compute by 12x. The measurement methodology lesson (warmup, repeated sampling, sufficient sample size) applies to any side-by-side benchmark of inference endpoints.
A Dockerized Spring Boot service exposes two endpoints: /api/chat, which forwards to DeepSeek, and /api/rag/ask, which runs embeddings and a cross-encoder reranker. Load tests with 50 requests show chat scales linearly with concurrency (QPS doubles, latency flat) because the bottleneck is the external model's network round-trip. RAG, on the other hand, peaks at 797-866% CPU (about 8 cores on a 10-core machine), and nearly all of that goes to the bge-reranker-base model running under ONNX Runtime's default CPU execution provider.
Replacing the cross-encoder with a purely lexical reranker (bigram matching plus weighted vector scores) drops CPU to 66-74%, a 12x reduction. Document embeddings are computed once at startup, so per-request embedding cost is tiny. The cross-encoder re-ranks 5 candidates per query, truncating each to 512 tokens, which saturates all 10 ONNX intra-op threads. The author also reveals a measurement trap: the first benchmark used only 6-12 requests and a single docker stats snapshot, leading to a false conclusion that embedding itself cost 4.9 cores. Reproducing with warmup and 50 requests corrected the number to 0.7 cores.
Practical deployment details include a multi-stage Dockerfile that caches Maven dependencies, an exec-form entrypoint so SIGTERM reaches the JVM, a named volume for model caches (including the DJL cache that defaults to ~/.djl.ai), and a compose file that binds only to 127.0.0.1. The .env file is auto-injected by compose and can override defaults silently. ONNX Runtime's default intra-op thread count equals the host core count, which can hurt small containers; the author leaves tuning that parameter as an open question.
The 12x CPU gap between cross-encoder and lexical reranking is a strong argument for evaluating whether semantic reranking is worth its compute cost in containerized RAG, especially when downstream generation latency already dominates end-to-end response time.
The author's initial measurement mistake is a lesson for anyone benchmarking inference: a single snapshot of docker stats can land on a startup spike or a steady-state dip, and without warmup, the first request's ONNX initialization inflates CPU readings dramatically.
The claim that 'reranker is the CPU bottleneck' is explicitly scoped to a CPU-only ONNX Runtime on Apple Silicon with a 512-token rerank window; changing any of those variables (GPU provider, smaller model, shorter window) could shift the bottleneck entirely.
ONNX Runtime's default thread-per-core behavior means a high-core-count host can monopolize all cores for a single request, which is a trap for people deploying to small containers without tuning intra-op threads.
The .env auto-injection gotcha is a subtle deployment hazard: Compose's environment merging can silently override your intended defaults, leading to unexpectedly loading a 266 MB reranker.
The same endpoint measured a 7x difference, and the root cause is the measurement method, which is a hard truth to swallow. Averaging over a 5-minute window is a sound approach; the load test data finally adds up.