A 2 AM CPU 100% Alert That Ended at a GC Storm: A Three-Layer Java Diagnosis Runbook
Java teams still running CMS (common on many JDK 8 deployments) can't afford to guess whether a CPU spike is business code or collector behavior; this runbook turns a 2 AM panic into a repeatable checklist. It also exposes the jstack socket-file permission mismatch that wastes hours, and shows why p99 spikes plus a low load average are a fingerprint of pulse GC rather than a busy loop.
The post starts with a six-step incident framework and then drills into three diagnostic layers. Monitoring side overlays instance CPU curves to decide single-machine vs cluster-wide, and reads avg vs p99 to preselect whether the cause is a GC pause or chronic slowdown. Machine side reads top's load, us/sy/wa, and thread states to establish causality, catching the telltale contradiction of instant 14-core CPU with low load average and only one running task.
The JVM layer runs the four-command chain: `top` to find the process, `top -Hp` to find the hot thread, `printf '%x'` to convert the decimal TID to hex, then `jstack` to reverse-lookup the stack. The stack reveals Gang worker threads of the CMS collector, and `jstat -gcutil` confirms it with old-gen at 88% and FGC counts rising between samples. The article also covers the jstack "Unable to open socket file" permission trap, three-shot thread dumps to filter sampling noise, and the pre-restart evidence protocol—thread stacks, GC logs, and heap dump—so a restart doesn't erase the root cause.
The article's evidence hierarchy—circumstantial (pulsed CPU), direct (thread name), quantitative (jstat)—is what separates a structured investigation from command layer-caking; each hypothesis is tied to a concrete next command, which is what actually shortens mean time to resolution.
The jstack socket error is a warning that operational tooling's error messages are written for the common case, not for privilege mismatches; the same binary on the same machine with a different uid produces a message that says the process is dead when it's simply a permission check.
The 'three numbers in contradiction' technique—instant %CPU, load average, and running-task count—is a cheap way to spot pulse workloads like periodic GC cycles from a single top snapshot; it deserves to be an alert rather than a manual read.
CMS's 'concurrent' branding obscures real stop-the-world windows; the article's insistence on checking Full GC log timestamp density rather than the collector's name is the practical takeaway for anyone tuning or reviewing GC behavior.
The post's advice to preserve three evidence types (thread stack, GC data, heap dump) before a restart institutionalizes the lesson that a restart without evidence is a roll of the dice in incident review.