Backend · MySQL · SQL

A Layer-by-Layer Guide to Diagnosing API Timeouts Without Guessing

By 云技纵横 · Aug 12, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Timeout incidents routinely waste hours as teams blame the network, the database, or the code without evidence. A repeatable, layer-by-layer checklist turns a multi-team argument into a 15-minute forensic exercise and prevents the common mistake of masking a slow database with bigger thread pools.

Summary

When an API times out, the instinct to restart or blindly expand pools often destroys the evidence and pushes pressure downstream. A more reliable approach follows the request path from the outside in, answering one question at each layer: did the request arrive, and how long did it stay?

The diagnostic sequence starts with a curl command to separate client-side from server-side latency. Nginx access logs enriched with upstream timing variables then reveal whether the connection to the application was refused, established slowly, or established quickly but followed by a long wait for response headers. Inside the JVM, three consecutive thread dumps pinpoint whether Tomcat threads are blocked waiting for a HikariCP connection, stuck on a socket read, contending for a lock, or genuinely CPU-busy.

At the connection pool layer, four numbers tell the story: active, idle, pending, and max. A pool at maximum with a growing pending queue confirms congestion, but the root cause is further down — slow SQL, oversized transactions, or remote calls inside transactions. MySQL's process list and Performance Schema queries then distinguish between long-running statements and sessions waiting on locks, guiding the final step of checking execution plans or identifying blocking transactions.

Takeaways

— Start every timeout investigation with a curl command that captures connect, starttransfer, and total times to separate client-side from server-side latency.

— Add upstream timing variables to Nginx access logs: a low uct with a high uht means the connection reached the app but no response header came back.

— Three consecutive Java thread dumps are more reliable than a single snapshot; threads stuck at the same call across all three dumps indicate a persistent wait point.

— Tomcat threads blocked on HikariPool.getConnection() point to connection pool exhaustion, not necessarily a database problem.

— A HikariCP pool with active=max and pending>0 confirms congestion, but the fix is finding why connections aren't returning — not just increasing max.

— Use MySQL's Performance Schema events_statements_current to find long-running SQL and check for NO_INDEX_USED before running EXPLAIN.

— Never expand thread pools, connection pools, or timeouts before identifying the actual bottleneck; doing so just moves pressure to the next layer.

— Preserve Nginx logs, thread dumps, connection pool metrics, and MySQL session lists before restarting anything.

Conclusions

The diagnostic sequence is deliberately ordered by proximity to evidence: Nginx logs are always available, thread dumps require JVM access, and MySQL queries need database credentials, so starting from the edge avoids wasted context-switching.

Status code 499 is widely misunderstood as an Nginx problem when it often just means the client's timeout budget was smaller than the server's response time — a mismatch in timeout configuration across layers.

The advice to capture three thread dumps rather than one is underappreciated. A single dump can mislead by catching a thread in a transient state; repetition filters noise.

Connection pool exhaustion is a symptom, not a diagnosis. The article correctly frames it as a congestion indicator that forces the next question: what is holding the connections?

EXPLAIN ANALYZE runs the actual query, which makes it dangerous on production write paths or large tables — a caveat many guides omit.

Concepts & terms

Nginx upstream timing variables

Custom log fields — upstream_connect_time, upstream_header_time, upstream_response_time — that measure how long Nginx spends connecting to a backend and waiting for headers, separating network latency from application latency.

HikariCP pool metrics (active, idle, pending, max)

Four counters that describe connection pool health: active connections in use, idle connections available, threads waiting for a connection (pending), and the configured maximum pool size.

Nginx 499 status code

A non-standard Nginx code indicating the client closed the connection before the server responded, often caused by a client-side timeout shorter than the server's processing time.

Performance Schema events_statements_current

A MySQL Performance Schema table that shows currently executing SQL statements with timing, rows examined, and index usage, providing more detail than SHOW PROCESSLIST.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗