A Layer-by-Layer Guide to Diagnosing API Timeouts Without Guessing

Online Interface Suddenly Times Out: How to Determine If It's Stuck at Nginx, Thread Pool, Connection Pool, or SQL?

When an interface times out, Nginx says the upstream is unresponsive, there are no exceptions in the Java logs, and the database CPU isn't high.

Everyone thinks their own layer is fine, and in the end, the only option is to restart the application.

Actually, you don't need to go through all the logs. By looking for 4 sets of evidence along the request chain, you can usually determine which layer the request is stuck at in about ten minutes.

1. First, Understand This Request Chain

A typical Spring Boot interface goes through at least the following layers:

Client
  ↓
Nginx
  ↓
Tomcat Request Thread
  ↓
HikariCP Database Connection Pool
  ↓
MySQL Executes SQL

If queuing occurs at any of these layers, all the user might see are two words:

Timeout

But the fixes are completely different:

Nginx cannot connect to the application       -> Check address, port, network, and instance status
Tomcat request threads exhausted              -> Check what threads are executing or waiting for
HikariCP connection pool exhausted            -> Check why connections are not being returned for a long time
MySQL execution slows down                    -> Check slow SQL, lock waits, and execution plans

If you immediately restart, expand the thread pool, or increase timeout values, you can easily wipe out the evidence and potentially pass the pressure further downstream.

The correct approach is not "check everything," but rather:

From the outside in, answer only one question at a time: Did the request reach this layer? If it did, how long did it stay here?

2. Step 1: First Determine Whether the Timeout Occurred on the Caller Side or the Server Side

Start by initiating a request from the same entry point as the user:

curl -v \
  --connect-timeout 2 \
  --max-time 10 \
  -o /dev/null \
  -s \
  -w 'code=%{http_code} connect=%{time_connect}s start=%{time_starttransfer}s total=%{time_total}s\n' \
  'https://api.example.com/orders/10001'

Example output:

code=504 connect=0.018s start=3.006s total=3.006s

This data at least indicates:

The TCP/TLS connection was established quickly.
The real wait occurred after the connection was established.
A 504 was returned by the gateway around the 3-second mark.

If connect itself is high, prioritize checking DNS, network, load balancer, and the entry port.

If connect is fast but starttransfer is high, it means the request entered the server-side chain but simply did not receive a response header in time.

Don't just test once from your own machine. Ideally, test simultaneously from an internal network machine, the Nginx node, and the network where the application resides to distinguish between a single caller issue and a common server-side problem.

3. Step 2: Use Nginx Timing to Determine If the Upstream Accepted the Request

If the Nginx access log only has a status code, its diagnostic value is very limited.

It is recommended to add a log format that includes upstream timing:

log_format upstream_timing
    '$request_id $remote_addr "$request" status=$status '
    'rt=$request_time uct=$upstream_connect_time '
    'uht=$upstream_header_time urt=$upstream_response_time '
    'upstream=$upstream_addr';

access_log /var/log/nginx/access.log upstream_timing;

When troubleshooting, check:

tail -n 200 /var/log/nginx/access.log
tail -n 100 /var/log/nginx/error.log

Focus on these four values:

rt   = Total time Nginx spent processing the request
uct  = Time Nginx spent establishing a connection with the upstream
uht  = Time Nginx waited to receive the response header from the upstream
urt  = Response time of Nginx's interaction with the upstream

Scenario 1: `uct` is empty, error log shows connection refused

connect() failed (111: Connection refused) while connecting to upstream

The request did not reach Java. Prioritize checking:

Upstream address and port
Whether the application is listening
Container network and service name
Whether the instance is restarting

Scenario 2: `uct` is very small, but `uht` is large

For example:

status=504 rt=3.001 uct=0.001 uht=3.000 urt=3.000

This indicates that Nginx connected to Spring Boot quickly, but the upstream did not return a response header for a long time.

The focus of investigation should shift into the Java application, not continue adjusting Nginx network configurations.

Scenario 3: Status code is 499

499 is a common non-standard Nginx status code, indicating the client closed the connection before Nginx returned a response.

It doesn't necessarily mean there's a problem with Nginx; it could be that the client had a 3-second timeout, but the server returned the response at the 5th second.

In this case, continue looking at upstream timing and application logs to find out who was slower than the client's timeout budget.

Note that proxy_connect_timeout controls the timeout for establishing a connection with the upstream; proxy_read_timeout controls the allowed wait time between two reads of upstream data, and is not equivalent to the total execution time of the entire interface.

4. Step 3: Check What the Tomcat Request Threads Are Actually Doing

If Nginx has successfully connected to the upstream, the next step is not to guess about CPU, GC, or SQL.

Look directly at the Java request threads.

First, find the process:

jps -l

Assume the PID is 18472. Capture the thread stack three times consecutively:

jcmd 18472 Thread.print -l > /tmp/thread-1.log
sleep 5
jcmd 18472 Thread.print -l > /tmp/thread-2.log
sleep 5
jcmd 18472 Thread.print -l > /tmp/thread-3.log

Don't just capture once. If the same threads are stuck at the same location across three consecutive thread stacks, it's a stronger indicator of a persistent wait point.

Focus on searching for Tomcat request threads:

grep -n 'http-nio-.*-exec' /tmp/thread-1.log

Common results can be interpreted as follows.

1. Many threads stuck at `HikariPool.getConnection()`

at com.zaxxer.hikari.pool.HikariPool.getConnection(...)
at com.zaxxer.hikari.HikariDataSource.getConnection(...)

This indicates request threads are waiting for a database connection. The next step is to check HikariCP.

2. Many threads stuck at Socket read

at java.net.Socket$SocketInputStream.read(...)

Combine this with the upper call stack to determine if it's waiting for Redis, an HTTP downstream service, the database, or another network service.

3. Many threads stuck on a lock

java.lang.Thread.State: BLOCKED
- waiting to lock <0x...>

This indicates potential lock contention within the application. You need to find the thread holding the same lock, rather than increasing the number of Tomcat threads.

4. Many threads in a long-term RUNNABLE state

If the same business logic, serialization, encryption, or loop logic repeatedly appears in multiple stacks, and the process CPU is elevated, then investigate along the CPU hotspot.

Spring Boot Actuator generates http.server.requests metrics for web requests by default. After integrating monitoring tools like Prometheus in a production environment, you can simultaneously check interface P95/P99, request volume, and exception rates; /actuator/metrics is more suitable for diagnosing which metrics are currently registered and is not recommended as a production monitoring backend.

5. Step 4: Use 4 Numbers to Determine If It's the Database Connection Pool

If the thread stack is stuck at HikariCP, immediately check:

active   Number of connections currently in use
idle     Number of idle connections
pending  Number of threads currently waiting for a connection
max      Maximum connection pool size

A typical connection pool exhaustion scenario looks like this:

active  = 30
idle    = 0
pending = 74
max     = 30

It means:

All 30 connections are occupied.
No connections can be lent out immediately.
74 request threads are waiting.

When the HikariCP connection pool reaches maximumPoolSize and there are no idle connections, getConnection() waits up to connectionTimeout before throwing an exception.

Therefore, you often see in the logs:

Connection is not available, request timed out after 30000ms

But a full connection pool is not the root cause; it's just the congestion point.

You still need to answer: Why aren't the connections being returned?

Slow SQL execution
Transaction scope is too large
Remote interfaces are called within a transaction
Lock waits occurring in the database
Code not closing connections correctly
Insufficient available connections on the database itself

At this point, directly increasing the connection pool from 30 to 100 might just send 100 slow SQL queries to the database simultaneously.

The correct action is to first record the connection pool metrics and thread stacks, then go into MySQL to find the sessions that are occupying the connections.

6. Step 5: Confirm Whether MySQL Is Executing Slowly or Waiting for Locks

The most direct entry point:

SHOW FULL PROCESSLIST;

Focus on:

Time     How long the current state has lasted
State    Whether it's executing, sending data, or waiting for a lock
Info     The currently executing SQL

The official MySQL 8.4 documentation recommends prioritizing the Performance Schema-based processlist implementation. You can also query current statement events:

SELECT
    t.PROCESSLIST_ID,
    t.PROCESSLIST_USER,
    t.PROCESSLIST_HOST,
    ROUND(es.TIMER_WAIT / 1000000000000, 3) AS seconds,
    es.ROWS_EXAMINED,
    es.NO_INDEX_USED,
    es.SQL_TEXT
FROM performance_schema.events_statements_current es
JOIN performance_schema.threads t
  ON t.THREAD_ID = es.THREAD_ID
WHERE es.SQL_TEXT IS NOT NULL
  AND es.END_EVENT_ID IS NULL
ORDER BY es.TIMER_WAIT DESC;

If many sessions are executing the same SQL, continue to check the execution plan:

EXPLAIN
SELECT ...;

Focus on:

Access type
Index actually used
Estimated number of rows scanned
Whether temporary tables or extra sorting appear

If the session state points to a lock wait, first identify the blocker and the blocked party, then decide whether to kill the session, roll back the transaction, or fix the business code.

EXPLAIN ANALYZE actually executes the statement. For write SQL, large table queries, or during production peaks, do not run it directly just to see the plan; first evaluate the impact in a safe environment.

7. 5-Minute Quick Judgment Table

On-Site Evidence	Where the Request Might Be Stuck	Next Step
Nginx `connection refused`	Upstream address, port, or instance	Check listening, container network, and service status
Nginx `uct` high	Establishing upstream connection	Check network, DNS, connection, and instance load
Nginx `uct` low, `uht` high	Java or Java's downstream	Enter the application and capture thread stacks
Many Tomcat threads waiting on HikariCP	Database connection pool	Check active, idle, pending, max
Many Tomcat threads waiting on Socket	Redis, HTTP, or database network calls	Locate the specific client based on the full call stack
HikariCP `active=max`, `pending>0`	Connection pool is queuing	Check slow SQL, long transactions, lock waits, and leaks
MySQL same type of SQL executing for a long time	SQL or index	Check execution plan, rows scanned, and data distribution
Many MySQL sessions waiting for locks	Long transactions or lock conflicts	Locate the blocking session and transaction scope

The value of this table is not to replace a full investigation, but to first decide which layer you should spend your time on.

8. Three Actions Most Likely to Make an Incident Worse

1. Directly expanding all pools

The Tomcat thread pool, database connection pool, and downstream connection pools form a chain.

Only increasing the upstream pool might allow more requests to simultaneously squeeze into an already overloaded downstream.

2. Changing all timeouts to 60 seconds

Bigger timeouts don't mean more stability.

If the caller is only willing to wait 3 seconds, but Nginx and Java allow the downstream to execute for 60 seconds, the user has already left while the server side is still consuming threads and connections.

Timeouts should be set layer by layer, starting from the entire call chain's budget, and unlimited retries should be avoided.

3. Restarting before preserving the scene

A restart clears threads, connections, and some temporary states.

If the business allows, at least save the following first:

Incident time and affected interfaces
Nginx access/error log
Three thread stack snapshots
Connection pool metrics
MySQL current sessions
Recent deployments and configuration changes

Preserve the evidence before mitigating the loss, so the post-mortem review doesn't end with just "it was fine after the restart."

9. The Troubleshooting Sequence I Will Follow

[ ] Use curl to record connect, starttransfer, total
[ ] Check Nginx status codes, error logs, and upstream timing
[ ] Confirm whether the request reached Spring Boot
[ ] Capture 3 consecutive Java thread stacks
[ ] Determine if Tomcat threads are running, waiting for locks, or waiting for downstream
[ ] Check HikariCP active, idle, pending, max
[ ] Query MySQL current sessions and lock waits
[ ] Check execution plans for suspicious SQL
[ ] Verify recent deployments, configurations, and traffic changes
[ ] After preserving the scene, decide on rate limiting, instance removal, rollback, or restart

The entire troubleshooting mainline can be condensed into one sentence:

Nginx timing determines the direction, thread stacks find the wait point, resource pools confirm congestion, and MySQL continues to trace the root cause.

Summary

When an interface times out, don't start by asking "Is the database slow?"

First, answer these four questions in order:

Did Nginx connect to the application?
Where are the Tomcat request threads stuck?
Has the connection pool started queuing?
Is MySQL executing or waiting for locks?

As long as there is verifiable evidence at each layer, troubleshooting won't turn into multiple teams guessing at each other.

More importantly, don't treat "increase threads, increase connection pool, increase timeout" as a universal fix.

Pools and timeouts only control how pressure queues up; the real root cause could still be slow SQL, long transactions, lock contention, remote calls, or incorrect capacity design.

If you are dealing with interface timeouts, Nginx 504s, thread pool queuing, connection pool exhaustion, or slow SQL, you can follow me and send me sanitized error logs, thread stacks, and key configurations. I can help take a look at simple issues and initial diagnoses for free. For production information, please first hide domain names, IPs, passwords, Tokens, database accounts, and business data.