Foreword

Hang detection here uses Tencent's open source library matrix. This article explains the principle.

RunLoop

The iOS application message loop is based on the RunLoop mechanism:

The official internal logic:

1. Notify the Observer that the RunLoop has entered.
2. Notify the Observer that it is about to process Timers.
3. Notify the Observer that it is about to process non-port-based input sources (about to process Source0).
4. Process the ready non-port-based input sources (process Source0).
5. If a port-based input source is ready and waiting, process it immediately and go to step 9 (process Source1).
6. Notify the Observer that the thread is about to sleep.
7. Put the thread to sleep until one of the following occurs:
   • An event arrives on a port-based input source (Source1)
   • A Timer fires
   • The thread is woken manually from outside
   • The RunLoop's set timeout expires
8. Notify the Observer that the thread has just been woken (events not processed yet).
9. Process pending events:
   • If it is a Timer event, handle the Timer and restart the loop; go to step 2.
   • If an input source is triggered, handle that event (the docs say deliver the event).
   • If the RunLoop was woken manually and has not timed out, restart the loop; go to step 2.
10. When this RunLoop run ends, notify the Observer that the RunLoop has exited.

The main thread's RunLoop organizes all event handling on the main thread, including user interaction, timers registered on the main RunLoop, and tasks submitted to the main queue. If any segment of work takes too long, work that later needs the main thread can be delayed, and the user may perceive a hang.

So the idea for hang detection is direct: observe when the main RunLoop starts working and when it rests; if that interval exceeds a threshold, consider it a hang.

The RunLoop has 6 states observable through an Observer:

• kCFRunLoopEntry: entered the RunLoop, about to begin the event processing loop
• kCFRunLoopBeforeTimers: about to process Timers
• kCFRunLoopBeforeSources: about to process input sources
• kCFRunLoopBeforeWaiting: about to sleep, waiting for an event or timer to fire
• kCFRunLoopAfterWaiting: just woke from sleep, has not yet processed the event for this wake
• kCFRunLoopExit: RunLoop exits

Note that Entry and Exit correspond to entering and exiting one RunLoop run invocation; they are not triggered once per processed event.

For RunLoop principles, see iOS Concept Battle Road (1): RunLoop.

Matrix's implementation

How is a hang determined?

Matrix's core basis for judging main-thread hang is: the RunLoop has started processing work but has not returned to the waiting state, and this continuous busy time exceeds the threshold.

1. The RunLoop wakes up and enters AfterWaiting.

Matrix records the current time and marks the main thread as 'busy'. BeforeTimers and BeforeSources also set this marker, but if already busy, the time is not restarted.

// myRunLoopBeginCallback handling
switch (activity) {
    case kCFRunLoopAfterWaiting:
    case kCFRunLoopBeforeTimers:
    case kCFRunLoopBeforeSources:
        // Only record the time when busy starts
        if (!g_bRun) {
            gettimeofday(&g_tvRun, NULL);
        }

        g_bRun = YES;
        break;

    default:
        break;
}

1. The main thread starts executing event-handling code.

This is where various events are handled: Source0, Source1, timers, and so on. If some callback does time-consuming computation or keeps waiting on a lock, this phase becomes expensive and cannot finish quickly, so the RunLoop cannot reach BeforeWaiting.

1. If it does reach BeforeWaiting, Matrix clears the busy marker.

The busy segment has ended; the normal sleep that follows does not participate in hang determination.

// core handling in myRunLoopEndCallback
if (activity == kCFRunLoopBeforeWaiting) {
    gettimeofday(&g_tvRun, NULL);
    g_bRun = NO;
}

So when checking, Matrix mainly looks at two conditions:

The busy marker is still set, and 'current time - busy start time' exceeds the threshold.

If these are satisfied, the main thread is considered hung.

BOOL isBusy = g_bRun;
struct timeval start = g_tvRun;

struct timeval now;
gettimeofday(&now, NULL);

unsigned long long duration =
    [WCBlockMonitorMgr diffTime:&start endTime:&now];

// simplified core condition; time validity and special-case checks are omitted
if (isBusy && duration > g_RunLoopTimeOut) {
    return EDumpType_MainThreadBlock;
}

For example, record t0 when waking; if the business callback never ends, the busy marker stays YES. During a check, if now - t0 exceeds the threshold, it is a hang. If the thread entered normal sleep long ago, the marker is NO, and no matter how large the time difference is, this judgment is not triggered.

Matrix's default threshold is 2 seconds.

Collecting reports

The Observer callback runs on the main thread and is responsible for recording busy state and time, but the hang-detection code itself cannot run on the main thread. If the main thread stays in some business code for a long time or is blocked waiting on a lock, it cannot continue processing subsequent events, and monitoring callbacks that depend on the main thread will also be delayed.

So Matrix uses a separate monitor thread to check these states. In Matrix this thread is called m_monitorThread, and its entry point is threadProc. It is responsible for timeout checks, the optional early-sampling switch, and filtering and report handling after a hang is detected.

Knowing that a hang happened is not enough; you also need to know why it happened.

There are two paths:

• Early sampling disabled (default)
• Early sampling enabled

The sampling here mainly captures the main thread's call stack.

Early sampling disabled

By default, the monitor thread checks the main thread state once per second. If the main thread is marked busy and the duration exceeds 2 seconds, it enters the hang handling flow.

At that point the monitor thread grabs the main thread's call stack and decides whether this report needs to be recorded. A hang can last a long time, and not every stack capture needs to be written to a report. If it keeps grabbing the same stack, there is no need to keep writing duplicate reports. The flow looks like this:

Captured stack	Generate report?	Wait until next check
A	Generate, save A	1 s
A	Same, skip	2 s
A	Same, skip	3 s
A	Same, skip	5 s
A	Same, skip	8 s
B	Changed, generate, save B	reset to 1 s
C	Changed again, generate, save C	reset to 1 s
C	Same, skip	2 s
C	Same, skip	3 s
A	Different from saved C, generate, save A	reset to 1 s

As you can see, if the stack stays the same, the next capture interval gets longer and longer. This is an annealing algorithm.

Annealing algorithm

The so-called 'annealing' means: if consecutive checks find that the main thread is still hung and the captured stack has not changed, skip duplicate reports and gradually extend the wait before the next check.

If the program is stuck and the stack is the same, there is no need to generate a report every time. Generating a report requires formatting the full thread call stack, memory state, context information, and writing it to disk. That operation itself consumes resources, and doing it while the system is already hung only makes things worse; repeated reports are also meaningless.

How does the backoff work exactly?

Matrix saves a 'previous interval' and a 'current interval', both initially set to the base check interval, which by default is 1 second.

Each time the same stack is found:

newInterval = previousInterval + currentInterval
previousInterval = currentInterval
currentInterval = newInterval

So when the same stack keeps appearing, the wait times become:

1 s -> 2 s -> 3 s -> 5 s -> 8 s -> 13 s...

At the same time, the current check skips report generation.

Check time	Check result	This handling	Wait until next check
0 s	first capture stack A	save comparison stack, generate report	1 s
1 s	still timeout, still stack A	skip duplicate report	2 s
3 s	still timeout, still stack A	skip duplicate report	3 s
6 s	still timeout, still stack A	skip duplicate report	5 s
11 s	still timeout, still stack A	skip duplicate report	8 s

When the next check obtains a valid but different stack, even if the main thread is still over the timeout, the base interval is restored and the new stack is saved as the comparison basis. That ends the current annealing strategy.

Early sampling enabled

Early sampling means capturing call stacks in advance. Once enabled, it captures a call stack every 50ms, no matter what, busy or idle, it samples anyway.

The captured stacks go into a queue. The queue capacity is calculated as:

check interval / sampling interval = 1000ms / 50ms = 20

Later stacks overwrite earlier ones.

When a hang is confirmed, Matrix picks the most suitable complete call stack from the recorded stack queue. Matrix chooses the sample whose stack-top address has the largest consecutive repeat count; on a tie, the newer sample is chosen. If several consecutive samples land at the same position, that position deserves priority.

The process is:

1. Each time a stack is captured, first look at its stack-top address.

For example, two samples:

previous: [address A, address X, address Y]
current:  [address A, address M, address N]

The comparison is on the first address A. Even if the lower call chains differ, this step still considers the stack-top the same.

2. Record the consecutive repeat count.

current top == previous top -> current count = previous count + 1
current top != previous top -> current count = 0

Only the stack-top address is used to accumulate the consecutive repeat count; the whole call chain does not need to be the same.

3. Take the sample with the largest count from the cache.

Assume the cache has the following samples; A and B are different stack-top addresses:

sampling order: 1  2  3  4  5  6  7  8  9
stack-top:      A  A  A  B  A  B  B  B  A
repeat count:   0  1  2  0  0  0  1  2  0

The 3rd and 8th samples both have count 2, tied as the maximum. Matrix searches from the newest sample, so it selects the complete call stack captured at the 8th sample and sends it to the report.

So even though A appears 5 times and B appears 4 times, the final choice is B, because the rule uses consecutive repeat count, not total occurrence count.

The final report

Earlier I kept emphasizing the main thread, but the actual report does not contain only the main thread's call stack. When generating a report, Matrix captures the call stacks of all threads in the current process and writes them all into the report.

Early sampling disabled:

Main-thread timeout detected
    ↓
Immediately capture main-thread stack, used for filtering and annealing
    ↓
Pass filtering
    ↓
Call dumpFileWithType, enter KSCrash report flow
    ↓
Capture scene stacks of main thread and other threads
    ↓
Write report:
main-thread scene stack + other threads scene stacks

Early sampling enabled:

Before timeout, capture main-thread stacks at sampling interval and put them in queue
    ↓
Main-thread timeout detected
    ↓
Take [latest main-thread stack] from queue, used for filtering and annealing
    ↓
Pass filtering
    ↓
Select [main-thread representative stack] from queue
    ↓
Call dumpFileWithType, enter KSCrash report flow
    ↓
Capture scene stacks of main thread and other threads
    ↓
Write report:
main-thread representative stack + main-thread scene stack + other threads scene stacks

Enabling early sampling adds one more main-thread representative stack. When writing the report, the current call stacks of all threads in the process are captured. The main thread is the primary focus, and other threads' stacks are auxiliary information.

Also, when early sampling is disabled, the main-thread stack captured immediately during the timeout phase is used only for filtering. After it passes filtering and enters the report-writing flow, the main-thread stack and other threads' stacks are captured again at that point; that is what actually appears in the final report.

When early sampling is enabled, the main-thread stack that was being captured continuously is used to pick a representative stack after filtering, then the report flow captures the main thread's current stack plus other threads' stacks. That means two main-thread stacks end up in the report.

A real report

{
  "report": {
    "id": "D4D3E0A4-0DCB-4B34-99D4-D78851E43E27",
    "timestamp": 1792729529,
    "type": "standard",
    "process_name": "HTAPMKitDemo"
  },

  "crash": {
    "threads": [
      {
        "backtrace": {
          "contents": [
            {
              "instruction_addr": 9622005768,
              "symbol_name": "__semwait_signal"
            },
            {
              "instruction_addr": 6875891756,
              "symbol_name": "nanosleep"
            },
            {
              "instruction_addr": 6634837256,
              "symbol_name": "+[NSThread sleepForTimeInterval:]"
            },
            {
              "instruction_addr": 4324661340,
              "symbol_name": "static HTHangValidationModel.blockMainThreadForFourSeconds()",
              "source_file": "HTHangValidation.swift",
              "source_line": 379
            },
            {
              "instruction_addr": 4324651328,
              "symbol_name": "closure #1 in HTHangValidationModel.triggerForegroundHang()",
              "source_file": "HTHangValidation.swift",
              "source_line": 240
            },
            {
              "instruction_addr": 6646869856,
              "symbol_name": "swift::runJobInEstablishedExecutorContext(swift::Job*)"
            }
          ]
        }
      }
      // other 11 threads omitted
    ]
  },

  "htapm": {
    "module": "hang",
    "detector": "matrix",
    "block_duration_us": 2105855,
    "runloop_threshold_us": 2000000,

    "collection": {
      "scene_threads": {
        "enumerated_thread_count": 12,
        "items": [
          {
            "thread_identifier": 3120216,
            "is_main_thread": true,
            "status": "captured",
            "frame_count": 6,
            "report_array_index": 0
          }
          // other threads' metadata omitted
        ]
      },
      "main_thread": {
        "state": "not_requested",
        "reason": "disabled"
      }
    }
  }
}

This is a real report collected after extracting Matrix's logic into the Hang module of an APM system. Some content has been omitted and only the key information is kept, and it has been symbolized for readability.

• module: "hang": the hang module
• runloop_threshold_us: 2000000: the hang threshold is 2 seconds
• block_duration_us: 2105855: about 2.106 seconds had already passed when the detection happened

The report includes scene information for 12 threads. The main thread is the focus; in htapm.collection.scene_threads.items:

is_main_thread = true
report_array_index = 0

This means the main thread in this report is at crash.threads[0]. report_array_index is the mapping between the two locations.

Also, collection.main_thread.reason is disabled, which means early sampling was not enabled this time. Therefore, the analysis here uses the main-thread scene stack captured when the report was generated.

__semwait_signal
nanosleep
+[NSThread sleepForTimeInterval:]
HTHangValidationModel.blockMainThreadForFourSeconds()
HTHangValidationModel.triggerForegroundHang()

The main thread was stopped in a wait operation, and this wait was caused by sleepForTimeInterval:. Going further back, it reaches the business method blockMainThreadForFourSeconds().

The corresponding source is:

@inline(never)
private static func blockMainThreadForFourSeconds() {
    Thread.sleep(forTimeInterval: 4)
}

Continuing down, it is the closure in triggerForegroundHang(). So the call path for this hang is:

closure in triggerForegroundHang
    ↓
blockMainThreadForFourSeconds()
    ↓
Thread.sleep(forTimeInterval: 4)
    ↓
main thread waits synchronously and cannot continue processing events
    ↓
exceeds 2-second threshold, triggers hang report

So this report ultimately points to the Demo executing a synchronous sleep on the main thread.

Other approaches

CADisplayLink

Our local debugging tool has an FPS detection feature, implemented with CADisplayLink.

CADisplayLink is directly tied to the screen refresh rate. The hardware sends a frame synchronization (VSync) signal, and CADisplayLink listens for that signal to trigger a callback, so its rhythm matches the screen refresh.

Then you can count callbacks; for example, how many callbacks arrive in one second. On a 60Hz screen, if only 35 callbacks arrive in one second, severe frame drops can be inferred.

The problem is that to judge main-thread hang, it needs to be registered on the main thread's RunLoop. If it is registered on the main thread, then when the main thread is stuck, the callback is delayed too.

Suppose the main thread executes an image decode that occupies it for 3 seconds:

CADisplayLink normal callback
        ↓
main thread starts decodeImage
        ↓
runs for 3 seconds
CADisplayLink callbacks cannot run during this period
        ↓
decodeImage returns
        ↓
CADisplayLink callback fires again and sees a long interval

According to the main RunLoop execution mechanism, even if the logic 'capture stack when more than xxx seconds have passed' is written into the CADisplayLink callback, that logic itself has no chance to run during this period.

This causes a problem: by the time it discovers the anomaly, the call stack that caused the hang may already be gone. After those 3 seconds, when it captures the main-thread stack, it captures the scene of the callback currently running, and may not see the previous time-consuming call chain.

So the limitation is that it can discover after recovery that a callback was delayed, but it cannot reliably locate the scene at the time of the hang.

Additionally, FPS below 60 per second does not necessarily mean a hang. CADisplayLink's callback frequency is also affected by configuration, device refresh rate, low-battery and other system policies. And even if the callback is normal, it cannot prove that every frame was displayed in time; a frame still has to pass through rendering and GPU stages. Apple explicitly distinguishes the latency of the application submission stage from the latency of the rendering stage. Counting only callbacks cannot cover the entire display pipeline.

So CADisplayLink is generally useful as an auxiliary metric, for example as a small component during development and debugging, but it is not suitable as APM hang detection.

Sub-thread Ping

Sub-thread Ping means a child thread periodically sends a very light task to the main thread and asks the main thread to reply.

When the child thread sends the task, it starts timing. When the main thread executes the task, it replies to the child thread. If the child thread does not receive the reply after the threshold time, it judges this response as timed out and captures the main thread's stack.

The probe task has to wait for the main thread to execute it, but the timeout judgment and triggering the stack capture do not need to wait.

This approach has limitations:

• Different timing starting point

Ping measures 'how long the probe task waited from submission to execution'. If the main thread has already been busy for 800ms before Ping is sent, Ping does not count the previous 800ms.

• The wait time may be accumulated from multiple tasks

If many tasks are queued before the Ping task, the Ping task may also be delayed. Therefore, a timeout cannot directly prove that one function took too long; it can only prove that the probe task was not handled in time.

All things considered, the RunLoop approach is more suitable.

Wrap-up

Hang monitoring is mainly answering two questions:

• Has the main thread been unable to respond for a long time
• Why was it unable to respond at that time

The approach described here, the one used in Matrix, uses a RunLoop Observer to record the main thread's busy state and start time, and uses an independent monitor thread to check for timeout, separating anomaly discovery from main-thread execution.

After a timeout is found, you also need to think about how to preserve the scene and how to handle duplicate reports. Matrix obtains the main-thread stack immediately, then does duplicate filtering and applies the annealing strategy. If early sampling is enabled, it records main-thread call stacks into a queue and eventually selects one representative main-thread stack.

After entering the report collection flow, it records the scene of the main thread and other threads; enabling early sampling adds one extra main-thread representative stack.

It also gives a real report showing how to interpret a hang report and confirm the final cause of the hang. It also compares CADisplayLink and sub-thread Ping approaches.

Nothing much more to say.