Backend

From Wire to Userland: The Full Lifecycle of a Packet Inside Linux

By 那咋乎吧 · Aug 29, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Understanding the full receive path — from interrupt affinity and NAPI quotas to Netfilter hook ordering and the TCP fast-path conditions — is what separates diagnosing a throughput regression from guessing. Every hook point is a linked-list traversal with a cost, and the fast-path bypass is fragile: one out-of-order segment collapses it.

Summary

DMA writes incoming packets into pre-allocated ring buffers shared between the NIC and kernel. Modern multi-queue NICs use MSI-X to deliver per-queue interrupts to specific CPUs, avoiding the interrupt storm that would paralyze a 10G NIC at line rate. The hard interrupt handler does almost nothing: it masks the queue interrupt and schedules NAPI, deferring real work to the NET_RX_SOFTIRQ softirq.

Inside the softirq, the poll function allocates sk_buffs from per-CPU caches, attempts GRO merging for consecutive same-flow TCP segments, and hands the packet up through the protocol stack. Netfilter's PRE_ROUTING and LOCAL_IN hooks run iptables rules and conntrack tracking; route cache turns microsecond lookups into nanosecond hits for subsequent packets in the same flow.

At the transport layer, established TCP sockets take a fast path that skips all out-of-order and retransmission logic when the sequence number matches exactly. The data is queued directly onto the socket's receive queue, and tcp_data_ready wakes every process blocked on read or epoll in one shot.

Takeaways

— A 10G NIC at line rate generates 15 million packets per second; per-packet interrupts would stall the CPU, so NAPI batches processing in softirq context with packet-count and time budgets.

— MSI-X lets each RX queue target a dedicated CPU via its own interrupt vector, keeping packet processing local to one core and avoiding cross-CPU contention.

— The hard interrupt handler only masks the queue interrupt and schedules NAPI; all actual packet processing happens later in NET_RX_SOFTIRQ or ksoftirqd.

— GRO merges consecutive same-flow TCP segments into a single sk_buff before the protocol stack runs, collapsing multiple stack traversals into one.

— tcpdump captures packets at the ptype_all hook, which fires before iptables PREROUTING — so it sees packets that iptables later drops.

— Route cache turns full routing-table lookups (microseconds) into cache hits (nanoseconds) for packets belonging to an already-seen flow.

— The TCP fast path in tcp_rcv_established triggers only when the TCP header has no options and the sequence number matches exactly; one out-of-order segment forces the slow path with full retransmission and reordering logic.

— tcp_data_ready wakes every process waiting on the socket — epoll, blocking read, or otherwise — in a single call after data is queued.

Conclusions

The entire receive path is a chain of budgeted, deferrable work units: DMA is asynchronous, the hard irq does almost nothing, NAPI polls under a time/packet budget, and even softirq backlog spills to ksoftirqd. The design assumes overload is normal and builds in back-pressure at every stage.

Netfilter hook ordering explains why eBPF and XDP are placed before the stack: every iptables rule is a linked-list traversal inside a hook, and PRE_ROUTING runs on all inbound packets regardless of final destination. Moving drop decisions earlier avoids wasting CPU on packets that will be discarded anyway.

The TCP fast path is a binary cliff, not a gradient. A single missing or out-of-order segment collapses the optimized path into the full slow path with SACK processing and retransmission logic, which is why packet loss matters far more than latency for throughput.

Per-CPU data structures appear at every layer — softnet_data poll_list, sk_buff allocation caches, NAPI instances pinned to queues pinned to CPUs. The architecture is fundamentally a partitioned pipeline, and breaking CPU affinity (e.g., by misconfiguring IRQ smp_affinity) forces cache misses and lock contention across the whole path.

Concepts & terms

MSI-X

A PCIe interrupt mechanism where the device sends a memory-write message directly to the CPU's Local APIC instead of asserting a physical interrupt pin. Each MSI-X table entry can target a different CPU with its own interrupt vector, enabling per-queue interrupt steering on multi-queue NICs.

NAPI (New API)

Linux's receive-side interrupt mitigation framework. Instead of one interrupt per packet, the hard irq masks the queue interrupt and schedules a polling instance; the softirq then batch-processes packets from the RX ring under a configurable budget, re-enabling the interrupt only when the ring is drained.

GRO (Generic Receive Offload)

A software receive-side optimization that merges consecutive TCP segments belonging to the same flow into a single larger sk_buff before the protocol stack processes them, reducing per-packet stack traversal overhead.

sk_buff

The core data structure representing a network packet in the Linux kernel. Protocol layers do not copy data; they move head/data/tail pointers and header offsets within the structure, making layer transitions nearly zero-copy.

Netfilter hooks

Five interception points in the Linux network stack (PRE_ROUTING, LOCAL_IN, FORWARD, LOCAL_OUT, POST_ROUTING) where kernel modules like iptables and conntrack register callback chains. Each hook point is a linked-list traversal; more rules mean more CPU cost per packet.

TCP fast path (tcp_rcv_established)

An optimized receive code path taken when a TCP segment arrives in-order with no header options and the exact expected sequence number. It skips out-of-order queue management, SACK processing, and retransmission logic, queuing data directly to the socket and waking waiters immediately.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗