A 2 AM CPU 100% Alert That Ended at a GC Storm: A Three-Layer Java Diagnosis Runbook
Troubleshooting a Production CPU 100% Incident
At 2:14 AM, the monitoring alert went off: order service CPU usage hit 100%, and API response time spiked from 50ms to 3s. This post recaps a complete investigation journey—from the monitoring dashboard to a single line of code, with every command used, every piece of evidence evaluated, and every pitfall encountered, all laid out in full.
0. The Alert Hits: Build a Framework First, Then Type Commands
A newcomer's first reaction to a CPU alert is to run top; an old hand's first move is to establish an action framework. Because troubleshooting isn't a pile of commands, it's a chain of "what to do next" decisions. Without a skeleton, you'll get lost in command output.
This investigation followed a six-step framework throughout:
| Step | Action | Question Answered |
|---|---|---|
| 1 | Confirm the symptom | Is it really 100%? How large is the blast radius? |
| 2 | Stop the bleeding | Is the impact spreading? Should we pull traffic / rate-limit first? |
| 3 | Preserve the scene | Before restarting, have we saved the evidence? |
| 4 | Layer-by-layer localization | Monitoring side → machine side → code line, narrowing down each layer |
| 5 | Verify recovery | After the fix, did the metrics return to normal? |
| 6 | Retrospective | Why did it happen? Why didn't we catch it? |
Everything that follows is an expansion of step 4, "layer-by-layer localization." Layering is the central idea of this post:
The first layer (monitoring side) answers "which machine has the problem"—horizontal narrowing; the second layer (machine side) answers "where did the CPU go"—establishing causality; the third layer (the four-link chain) answers "which line of code is burning CPU"—vertical localization. Each layer has its own tools and judgment criteria, and you cannot skip between layers—skipping the horizontal approach to start with vertical will be like looking for a needle in a haystack across dozens of machines.
1. Layer One - Monitoring Side: Define the Impact First, Stay Off the Machine
1.1 Why the First Thing Is to Look at the Dashboard, Not SSH into the Machine
Because the same alert can have completely different root-cause distributions:
- If only 1 machine has high CPU → the problem lies in that machine's local state (a special request, local cache, local resource contention)
- If all 8 machines are high → the problem is in their commonalities (code, config, shared dependencies, traffic ingress)—it's impossible for local states on 8 machines to be coincidentally abnormal at the same time
So step one: overlay the CPU curves of all instances on the monitoring dashboard onto a single chart, and answer a three-way question.
1.2 Branch One: All 8 High → Look for Commonalities, Three Checks in Order
First check: align the change timeline. Does the CPU rise point align with a release, a config push, or a traffic switch? If it aligns → rollback is the priority, do not hunt for root cause online—first restore the system to the pre-change state, then investigate why offline. This corresponds to "stop the bleeding" in the six-step framework.
Second check: align QPS vs CPU curves. These are two easily confused causal patterns:
- QPS flat, CPU rising → per-request cost has gone up (code hotspot, GC, slower dependencies)
- QPS and CPU rising together → capacity issue, buy time by scaling out, then optimize later
Third check: inspect common dependencies. The only things shared by 8 machines are four: code, config, common dependencies (cache/DB/downstream services), and traffic ingress. Check in order: whether cache hit rate plummeted (cache breakdown penetrating to DB), whether downstream services slowed and filled the thread pool, and whether slow SQL is tying up the connection pool.
1.3 Branch Two: Some Machines High → Look for Differences Between Instances
If 3/8 are high, the problem isn't commonality, it's the differences on those few:
- Load balancer strategy: sticky sessions, long-lived connection reuse, or uneven hashing can pin hotspot traffic onto a few instances
- Gray release batch: did the problem machines just happen to be the batch that got the gray release (batch 3)?
- Data shape: are these instances handling special data (a large customer, a big batch)?
1.4 Branch Three: Single Machine High → Pull Traffic, Degenerate to a Single-Machine Investigation
This is the path the case in this post took: only one order-service instance was spiking CPU. Two steps:
Step one, pull traffic—remove that instance from the load balancer and redirect requests to the remaining machines. Pulling traffic isn't "fixing" the problem; it's protecting the scene: while business recovers, that machine's state is frozen, and you can investigate slowly without choosing between "business keeps deteriorating" and "preserving the scene."
Step two, gather monitoring-side evidence before going to the machine. Focus on the percentile values of API response time:
avg flat + p99 spiking = periodic short pauses (typical shape of GC STW)
avg and p99 rising together = chronic slowdown (CPU contention / thread-pool queuing)
The value of this judgment: you can preselect half the direction without going to the machine. A p99 long tail indicates "intermittent stalls," and in the Java world, the most classic creator of intermittent stalls is GC pause (STW, Stop The World). Going to the machine with this hypothesis is an order of magnitude more efficient than going empty-handed.
1.5 A Side Note: APM First
In production, the first line of attack isn't top, it's APM (Application Performance Management). Built on three pillars—Metrics, Tracing, Logging—it lifts you from the "single-machine view" to a "full-trace view": slow-call clustering can tell you directly which method is the hotspot, and each instance's CPU curve is on the screen.
The four-link chain (Chapter 3) is the second line when APM is absent or inconclusive—when APM can only tell you "this machine's CPU is high" but not "why," only then do you go to the machine.
2. Layer Two - Machine Side: top Reconnaissance and Causality Analysis
The monitoring side confirmed: it's a single-machine problem. SSH in, the first command is always top.
2.1 Reading top Line by Line: Every Field Is Evidence
Here's the real snapshot from that incident (data anonymized, process names altered):
top - 14:20:31 up 545 days, 11:04, 2 users, load average: 2.74, 2.80, 2.57
Tasks: 1359 total, 1 running, 1357 sleeping, 0 stopped, 1 zombie
Cpu(s): 26.4%us, 0.9%sy, 0.0%ni, 72.6%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9501 admin 20 0 38.6g 5.0g 16m S 1424.9 2.0 910502:37 java
59740 admin 20 0 16.3g 7.9g 20m S 218.2 3.1 51317:13 java
6944 admin 20 0 11.2g 17.6g 18m S 32.9 7.0 38814:22 java
First line - load average: the system-level "congestion index."
| Term | Meaning |
|---|---|
| load average | The 1/5/15-minute average of runnable + uninterruptible tasks |
Don't read it as "over the core count means danger"; look at the trend: three numbers descending = pressure ebbing; ascending = pressure building; sustained high = chronic overload. This is a 64-core machine, so load 2.74 means the system overall is very idle—this "contradiction" becomes useful later.
Second line - Cpu(s): the eight destinations of CPU time.
| Field | Full Name | Meaning | Debugging Value |
|---|---|---|---|
| us | user | Executing code in user space | Business logic, JVM bytecode execution all live here |
| sy | system | Working in kernel space | System calls, thread scheduling/context switches, locks |
| ni | nice | Low-priority processes | Who changed process priority |
| id | idle | Idle | Baseline for how busy the system is |
| wa | iowait | Waiting for disk I/O | Disk bottleneck signal |
| hi/si | Hardware/Software interrupts | NIC and other hardware interrupts | Network storms, interrupt imbalance |
| st | steal | Stolen by the hypervisor | Virtualized neighbor contention |
Note the %CPU scale : 100% = one core. A process at 1424.9% = burning about 14 cores.
Third block - key columns in the process table:
| Column | Meaning |
|---|---|
| S | State: R running, S sleeping, D uninterruptible sleep (in I/O), Z zombie |
| TIME+ | Cumulative CPU time—distinct from %CPU's instantaneous value; watch its growth rate |
| RES | Resident memory actually in physical RAM (excluding swap) |
2.2 "Extreme" Is Relative to a Baseline, Not an Absolute Number
To judge whether us 26.4% is high, don't consult a dictionary; compare against this machine's historical baseline: this machine's us has always hovered in the 20–30% range (because the business process normally uses ~200%), so 26.4% itself isn't abnormal. What's truly abnormal is the composition:
System idle 72.6% (overall idle) + single process 1424.9% ÷ 64 cores ≈ 22% of total compute
Conclusion: the system isn't busy, but a single process stands out—the problem collapses to inside PID 9501. At this point the system row's us/sy distribution has no diagnostic power, because the system's us is largely contributed by 9501 itself.
2.3 Causality Analysis: CPU High and Slow API—Which Is Cause, Which Is Effect?
There are two symptoms: "CPU 100%" and "API slow." They're not parallel—which is the cause determines which direction you investigate. The judgment isn't a single field but a combination of "CPU where it went + thread state":
- CPU is the culprit: high us + a few threads RUNNABLE and crunching hard → infinite loop / hot code is eating CPU, and slow APIs are a side effect of resource starvation. Investigate the four-link chain (Chapter 3).
- Slowness is the culprit: high sy + many threads WAITING + abnormally high total thread count → slow APIs cause request buildup, buildup causes threads to wake and switch frenetically, and CPU is spent on kernel context switching. Verify with:
vmstat 1 # watch cs (context switches per second)
# normal machines run from a few thousand to tens of thousands; hundreds of thousands to millions = switching storm confirmed
- Memory is the culprit: GC threads active + API p99 long tail → both CPU and slow API are symptoms; switch to GC investigation (Chapter 4).
Why does "slowness is the culprit" burn CPU? The full mechanism chain:
Slow API (slow DB/downstream) → request buildup → thread pool saturated → new requests queue
→ many threads blocked on I/O (WAITING) → when I/O completes, threads get woken → kernel context switches
→ the more threads, the more frequent wake/switch → sy spikes
In one sentence: slowness doesn't directly burn CPU; it's the buildup and switching caused by slowness that burns CPU.
2.4 The Key Read in This Case: Three Numbers Contradict Each Other
Return to the screenshot in 2.1 and align three groups of numbers:
| Metric | Value | Meaning |
|---|---|---|
| 9501's %CPU | 1424.9% | Instant sample: about 14 cores at full load |
| load average | 2.74 | Over the past 1 minute, only ~3 tasks are waiting for CPU on average |
| Tasks running | 1 | At the snapshot instant, only 1 task is in running state |
Instant 14 cores, average 3, instant 1—three numbers contradict each other. The only explanation: this process's CPU usage is pulsed—bursts, then falls back. A stable infinite loop (business hotspot) would keep all three numbers consistently high.
Typical sources of pulsed consumption: the concurrent collection cycles of the CMS garbage collector—during the concurrent marking phase, a gang of GC threads burns CPU, then falls back when the cycle ends. Scheduled tasks also have this shape, but the order service has no cron-style tasks. With the hypothesis of "suspected periodic GC burst," move to the third layer.
2.5 Do a Quick Exclusion: hi/si/st
When us isn't high but the overall is 100%, first rule out three things that aren't "your code's fault":
| Field | What high means | Action |
|---|---|---|
| hi/si | NIC interrupt storm (large traffic, multi-queue imbalance) | Look at traffic source, irqbalance tuning |
| st | VM stolen by host neighbor | Ask ops to migrate; not a code problem |
| ni | Low-priority process was nice-raised | Check who changed the priority |
In this case all three are 0.0, clean exclusion.
3. Layer Three - Single-Machine Localization: The Four-Link Chain, from Process to Code Line
After the machine side narrowed to PID 9501, vertical localization begins. This layer is the most classic Java debugging routine, a chain of four steps:
3.1 First Link: top to Find the Process
We already know from the system line that java process 9501 is eating CPU, but for scripted troubleshooting, ps is better:
ps -eo pid,pcpu,args --sort=-pcpu | head
# sorted by CPU descending; output isn't limited by terminal window line count
A tip: top -b batch mode is by default limited by the window's line count, so it may not show the target process—for scripted sampling, ps beats top.
3.2 Second Link: top -Hp to Find the Thread
top aggregates by process; top -H expands by thread:
top -Hp 9501
# sorted by %CPU descending; the thread burning the most CPU is on top
Two practical details:
Detail one: %CPU is a sampled value and will jitter. It's the difference between two refreshes, so it's noisy. A more stable piece of evidence is the growth of TIME+ : watch for 10 seconds; the thread whose cumulative CPU time grows fastest is the real hotspot.
Detail two: Java thread names may not display. Under older JDK versions, the COMMAND column for threads may show all as java (custom thread names don't fall through to the OS layer). Don't panic; continue through links three and four, and the thread identity will be revealed in jstack.
In this case, in the top -Hp 9501 output, the threads with the highest %CPU had these names:
TID USER %CPU COMMAND
9583 admin 21.3 java # that 21.3% is just the sampling instant; look at the aggregate of multiple threads
9576 admin 21.1 java
...
3.3 Third Link: printf '%x' to Convert to Hexadecimal
Once you have the thread ID (TID), do something that looks odd—convert it to hex:
printf '%x\n' 9583
# output: 256f
Why must you convert? Because the same thread has two numbers from two perspectives:
top -Hpgives you the OS-perspective thread ID: the real thread that the Linux kernel schedules, in decimal- jstack output shows the JVM-perspective thread ID
nid(Native thread ID): the same thread, in hexadecimal
A Java thread is a two-layer structure: the OS thread is the entity the kernel schedules, the JVM thread is the logical unit HotSpot manages, and nid is the anchor that bridges the two. The decimal-to-hex step is precisely how you make the "hot thread from the OS view" match the "code stack from the JVM view." If this identity didn't exist, this method wouldn't exist either.
3.4 Fourth Link: jstack to Reverse-Lookup the Code Line
# dump to disk first, then search (terminal scrollback will truncate tens of thousands of lines)
sudo -u admin /opt/java/bin/jstack 9501 > /tmp/stack_9501.log 2>&1
# use the hex nid for a precise reverse lookup (with the prefix, to avoid false matches on address strings)
grep -A 30 'nid=0x256f' /tmp/stack_9501.log
The three essentials of dumping to disk (jstack output can easily run to tens of thousands of lines):
| Essential | Command | Purpose |
|---|---|---|
| Dump | jstack pid > file.log |
Keeps the terminal from scrolling + preserves the scene as evidence |
| Search | grep -A 30 'nid=0x...' file.log |
Outputs only the target thread's stack, a few dozen lines |
| Categorize | grep 'java.lang.Thread.State' file.log | sort | uniq -c |
Distribution of thread states: how many running, how many waiting |
Why not just grep '256f'? Because jstack output is full of hex—stack frame addresses, lock object addresses can all contain ...256f.... The nid=0x prefix is an exact match that can only hit the thread header line.
The grep result in this case—the real culprit shows its face for the first time:
"Gang worker#10 (Parallel CMS Threads)" os_prio=0 tid=0x00007f974c368800 nid=0x256f runnable
"Gang worker#4 (Parallel CMS Threads)" os_prio=0 tid=0x00007f974c327800 nid=0x2568 runnable
3.5 Real-World Interlude: jstack Error “Unable to open socket file”
The first time I ran jstack, I hit a classic trap:
[root@cainiao ~]# jstack 9501
9501: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
The error text is highly misleading. The process is obviously alive (1424% in top), so both "not responding" and "VM not loaded" are false. The real cause: user identity mismatch.
jstack connects to the target JVM via the socket file /tmp/.java_pid9501; the JVM's Attach Listener uses SO_PEERCRED to get the connecting process's uid/gid, which must match the JVM's own, and root is not exempt (HotSpot's implementation has no uid==0 special case). And this error message is a generic fallback that conflates "credential mismatch" with "process truly unresponsive."
The controlled experiment is right in this case: a practice process started by root, jstack as root works; a business process started by admin, jstack as root is always rejected—same jstack, same machine, difference only in process owner.
Solution (by priority):
# preferred: run under the same user (credential matches, completely harmless)
sudo -u admin /opt/java/bin/jstack 9501
# alternative: -F uses Serviceability Agent's ptrace mode, bypasses socket check
# cost: briefly suspends the target process (a tiny STW), and output lacks lock info
jstack -F 9501
General rule: when attach-class tools (jstack/jmap/arthas) report "Unable to open socket file," check the process owner first before assuming the process is unresponsive. Investigation order:
ps -o pid,user,comm -p 9501 # look at the USER column
3.6 Anti-Jitter Reinforcement: Three-Part Snapshot Intersection
jstack is an instant snapshot; it captures the stack at the moment of the shot. If the hot thread isn't stable (the top TID keeps switching), a single snapshot can miss it. A common production technique:
for i in 1 2 3; do
sudo -u admin /opt/java/bin/jstack 9501 >> /tmp/stack_9501_all.log
sleep 5
done
# the call stack that appears in all three snapshots is the real hotspot (intersection-of-samples idea)
This is also the standard "preserve the scene" move before restart (see Chapter 5). A flame graph (perf periodic sampling aggregation) is an automated, upgraded version of "take multiple samples and intersect them."
4. The True Culprit: Business Hotspot or GC Storm
4.1 First Look at the Stack: What's the Thread Called?
The grep from Chapter 3 hit Gang worker (Parallel CMS Threads)—these are concurrent GC worker threads of the CMS collector, not business code. This is the most frequently missed root-cause category in CPU troubleshooting: investigators are used to hunting hotspots in business stacks, and seeing a GC thread gives them pause.
Here's the branching judgment between the two types:
| Top Thread Shape | Conclusion | Next Step |
|---|---|---|
| Business thread name (e.g., http-nio-8080-exec-x) RUNNABLE | Business hotspot | Look at stack top method; combine with flame graph |
| Gang worker / GC Thread / VM Thread | GC storm | Switch to jstat + GC logs, the path this post takes |
| Many WAITING + high sy | Blocking buildup | Check thread pool / connection pool; vmstat cs to confirm |
4.2 Three-Level Evidence Chain for GC Activity
From "suspicion" to "conclusion," evidence is ranked by confidence:
Level one (circumstantial): the three-number contradiction from 2.4—instant %CPU high + low load average + few running tasks = pulsed consumption. Low confidence: scheduled tasks have the same shape.
Level two (direct evidence): top -Hp shows the top thread is Gang worker—the only place in the top ecosystem where you can directly distinguish GC from business. High confidence: identity confirmed.
Level three (quantitative ironclad): let numbers speak:
sudo -u admin /opt/java/bin/jstat -gcutil 9501 1000 5
# sample every 1 second, 5 times
Example output:
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 62.31 95.32 87.51 96.42 94.71 38421 1521.388 2143 642.851 2164.239
0.00 61.08 97.85 88.02 96.42 94.71 38453 1522.502 2148 644.212 2166.714
...
Reading the key fields:
| Field | Meaning | This Case |
|---|---|---|
| E | Eden usage | 95%+ and repeatedly hitting the ceiling = intense object allocation |
| O | Old-gen usage | 88% and rising each second = approaching CMS trigger line |
| YGC/YGCT | Young-gen GC count / time | YGC growing wildly within a sampling interval = high allocation pressure |
| FGC/FGCT | Full GC count / time | FGC increases between two samples = Full GC happening frequently |
| GCT | Total GC time | Rule of thumb: GCT ÷ process uptime > 5% deserves attention, > 20% is basically a GC storm |
4.3 STW: Why a GC Storm Slows Down APIs
High GC consumption has another victim: the API. STW (Stop The World) : when GC reaches certain critical steps (like marking root objects, which require the "world to stand still" to be done safely), all application threads must pause. Threads aren't brutally interrupted; they stop voluntarily at safepoints.
A single Full GC pause of 1–3 seconds means all in-flight requests stall for those 1–3 seconds—this is the full explanation for the p99 long tail from Chapter 1: avg stays flat (most requests unaffected) + p99 spikes (a few requests happen to hit the pause window) = periodic STW. The foreshadowing from the monitoring side now closes the loop.
The CMS trap (a high-frequency point in both interviews and real ops): CMS claims to be a "concurrent" collector, but:
- The initial mark and remark phases are still STW
- After old-gen fragmentation causes a concurrent-mode failure, it can degrade to Serial Old's single-threaded Full GC—pause jumps from tens of milliseconds to seconds, and both CPU and RT collapse at once
To determine the pause source, look at the timestamp density of GC logs, not the collector's name:
# find Full GC timestamp density (in this case, log path is set by the -Xloggc startup flag)
grep "Full GC" /home/admin/logs/gc.log | tail -20
# adjacent timestamps seconds apart = high-frequency Full GC confirmed
4.4 The Final Picture of This Case
Three levels of evidence converge, and the truth is complete:
- Process burns CPU in pulses (instant 14 cores / load average 3)—shape of CMS concurrent cycle
- Top of
top -Hpis all Gang worker—GC thread confirmed - jstat shows old-gen at 88% and continuously approaching the trigger line, FGC growing frequently—quantitative ironclad
Conclusion: CPU 100% and slow APIs are two faces of the same root cause—a GC storm caused by insufficient old-gen space. There is no infinite loop in business code; the real culprit is memory allocation pressure and CMS's concurrent-failure degradation. Subsequent optimization directions: investigate the source of old-gen growth (large objects, memory leak, unbounded cache growth), adjust heap configuration—but that goes beyond this post's "investigation and localization" scope and belongs in a future part on root-cause governance.
5. Stop the Bleeding, Recovery, and Retrospective
5.1 Bleeding Decision: Two Paths—Impact Expanding vs Impact Controllable
Localization and bleeding control can run in parallel, but bleeding control always comes before localization. Two situations:
Impact is expanding (error rate rising, users feel it):
- Pull traffic or roll back first—if you just released 30 minutes ago, the first choice is always rollback, don't hunt root cause online
- If rollback isn't possible: restart is the last stopgap, but you must have "preserved the scene" before restarting
Impact is controllable (CPU high but business not trembling):
- Investigate online, don't rush to restart
- Once root cause is found, fix in small steps, gray-verify
5.2 Preserve the Scene Before Restart: Three Snapshot Dumps
This is the most painful lesson in countless post-incident reviews: after restart, the evidence is gone. Before restarting, you must keep three things:
# 1. Thread stack triple snapshot (5s apart, compare whether the stack is stuck at the same point)
for i in 1 2 3; do
sudo -u admin /opt/java/bin/jstack 9501 >> /tmp/stack_before_restart.log
sleep 5
done
# 2. GC snapshot and logs
sudo -u admin /opt/java/bin/jstat -gcutil 9501 1000 3 > /tmp/gc_before_restart.txt
cp /home/admin/logs/gc.log /tmp/gc_before_restart.log
# 3. Heap dump (required for memory-related issues; file is large, confirm disk space)
sudo -u admin /opt/java/bin/jmap -dump:live,format=b,file=/tmp/heap_before_restart.hprof 9501
These three correspond to three possible root-cause directions: thread stack (CPU / deadlock), GC data (memory pressure), heap dump (object analysis). Better to keep too much than miss one of the three.
5.3 Verify Recovery: Let Metrics Speak
After the fix is applied, verify using the same set of metrics:
| Verification | Command / Source | Return to normal criterion |
|---|---|---|
| CPU | top |
Process %CPU falls back to historical baseline |
| API RT | Monitoring dashboard | avg and p99 both fall; long tail disappears |
| GC rhythm | jstat -gcutil pid 1000 10 |
FGC no longer grows frequently; GCT ratio < 5% |
| System load | uptime |
load returns to a stable 1/5/15-minute trend |
5.4 Retrospective: Five Questions
Conclusions on the incident ticket must carry evidence; the review centers on five questions:
- Why did it happen? (root-cause chain: old-gen growth source → allocation pressure → CMS degradation)
- Why wasn't it caught? (are monitoring thresholds reasonable? is GC included in alerts?)
- Why didn't the alert fire earlier? (do you have a p99 alert, or only avg?)
- Can the bleeding action be turned into a standard script? (is the traffic-pull plan / rollback switch one-click?)
- How to prevent recurrence? (three remediation lines: code level + config level + monitoring level)
6. Tool Quick-Reference
6.1 Monitoring Side (don't touch the machine)
| Goal | Tool / Action |
|---|---|
| Confirm blast radius | APM dashboard: overlay instance CPU curves |
| Shape of API slowness | RT avg vs p99 (long tail = STW suspicion) |
| Cluster commonality | Change logs, QPS curves, cache hit rates, slow SQL list |
6.2 Machine Side (system view)
| Goal | Command | Key Criterion |
|---|---|---|
| System overview | top |
us/sy/wa/hi/si/st, load trend, Tasks states |
| Process sort (scriptable) | ps -eo pid,pcpu,args --sort=-pcpu | head |
Not limited by terminal line count |
| Verify switching storm | vmstat 1 |
cs (context switches per second) |
| Single-process thread view | top -Hp <pid> |
Top %CPU thread name + TIME+ growth |
| Thread listing | ps -T -p <pid> -o tid,pcpu,comm |
Thread-level CPU usage |
6.3 JVM Side (inside the process)
| Goal | Command | Note |
|---|---|---|
| Thread stack snapshot | jstack <pid> > file.log |
Dump to disk then grep; cross-user add sudo -u |
| Reverse-lookup hot thread | printf '%x\n' <tid> + grep -A 30 'nid=0x<hex>' file.log |
Use nid=0x prefix for exact match |
| GC live rhythm | jstat -gcutil <pid> 1000 5 |
Watch FGC growth and GCT ratio |
| Thread state distribution | grep 'java.lang.Thread.State' file.log | sort | uniq -c |
How many running, how many waiting |
| Heap dump | jmap -dump:live,format=b,file=x.hprof <pid> |
For memory direction; file is large |
| Flame graph | arthas profiler / perf record | Automated upgrade of multi-frame jstack sampling |
6.4 High-Frequency Error Reference
| Error | Real cause | Solution |
|---|---|---|
Unable to open socket file |
User identity doesn't match process owner (root attaches to admin process) | sudo -u <owner> jstack <pid> |
-F option can be used... |
Same as above (fallback message misleading) | First check ps -o user -p <pid> then decide |
7. Closing: What This Exercise Really Tests
Back to the three-layer panorama at the start. This post's investigation path looks like a single case, but it's actually the closure of a three-layer methodology:
- Monitoring side: overlay APM dashboard curves to determine blast radius (single-machine / cluster three-way fork) + p99 percentile to preselect direction—horizontal narrowing, know which machine to go to first
- Machine side: read top's eight fields to determine causality (us/sy distribution + thread state + contradictory numbers)—determine direction, where the CPU went
- Four-link localization: process → thread → hex → jstack code line—vertical localization, which line is burning
There is one discipline running through everything:
Conclusions must be backed by evidence. Every "suspicion" must be confirmed by evidence at the next level—circumstantial gives direction, direct evidence confirms identity, quantitative ironclad decides with numbers. Conversely, error messages can lie to you ("not responding" actually meant permissions), sampled values can fool you (%CPU is an instant difference), and a single field can mislead you (us/sy aren't causality). So each layer stays in its lane, every step seeks validation—that's the dividing line between "expert level" and "someone who knows how to run commands."
All commands in this post were verified in a Linux + HotSpot JVM 8 environment. The "order service" case data comes from a real production investigation, anonymized.