The Message Protocol That Runs a Local LLM Inside a Browser Worker

Running Large Models in the Browser (Part 4): Finally, Someone Has Clearly Explained the Message Design for Worker Streaming Inference

This article is original. It is based on the real project webgpu-deepseek (WebGPU + Transformers.js running DeepSeek-R1-Distill-Qwen-1.5B locally in the browser).

Part 4 of a 5-part series · Previous: "Is navigator.gpu Existing Enough? The Two-Layer Trap of WebGPU Detection" · Next Preview: "Encapsulating the useLLM Hook: Wrapping Worker Inference into a React Hook"

First, look at a conversation. You send a message in the browser's chat box, and a few seconds later, the AI "spits out" a reply word by word—exactly like the ChatGPT web version.

But what happens behind this conversation? How does the message get from the React component to the Web Worker? How do tokens stream back from the model? After you click the interrupt button, why does the model actually stop?

This article completely dismantles the entire message-driven inference engine for you to see. After reading it, you can write your own browser-side large model inference messaging system.

sequenceDiagram
    participant R as React Main Thread
    participant W as Web Worker
    participant T as Tokenizer
    participant M as ONNX Model (WebGPU)

    R->>W: postMessage({ type: &#34;generate&#34;, data: messages })
    W->>W: stopping_criteria.reset()
    W->>T: apply_chat_template(messages)
    T-->>W: input_ids + attention_mask
    W->>M: model.generate({...inputs, streamer, stopping_criteria})
    loop Each token
        M-->>W: token_callback (Calculate TPS)
        W->>W: callback_function
        W->>R: postMessage({ status: &#34;update&#34;, output, tps, state })
    end
    M-->>W: { past_key_values, sequences }
    W->>W: Cache past_key_values
    W->>R: postMessage({ status: &#34;complete&#34;, output: decoded })

The diagram above is the map for the entire article. Below, we will dismantle each link in the diagram one by one.

First Hurdle: How Messages Get from React to the Worker

Worker Instantiation—The Base of the Entire Inference Engine

All inference runs inside a Web Worker. Why? Because model inference is a CPU/GPU-intensive operation—if it ran on the main thread, any single model.generate() call would freeze the entire page, making scrolling impossible and buttons unclickable, and the browser might even pop up a "Page Unresponsive" warning.

// App.tsx — Worker is created only once, using useRef to manage the reference
const worker = useRef(null);

useEffect(() => {
  if (!worker.current) {
    // 🔑 Key: Under Vite, use new URL() to import Worker, supporting ESM modules
    worker.current = new Worker(
      new URL("./worker.js", import.meta.url),
      { type: "module" }  // ⚠️ Without this, importing third-party libraries in Worker will error
    );
    worker.current.postMessage({ type: "check" }); // Step 1: Check if WebGPU is available
  }
  // ...listen for messages
}, []);

Why useRef instead of useState?

A Worker instance is a live thread reference, not a UI state. Calling new Worker() on every render causes memory leaks (old Workers won't be GC'd), and all previous model download progress is lost. Using useRef ensures only one Worker instance exists throughout the component's lifecycle—this is the same idea as the singleton pattern discussed in the first article, but managing a Worker instead of a Pipeline.

Message Protocol Design—6 Message Types

Communication between the Worker and the main thread happens via postMessage. The message protocol is the nervous system of the entire inference engine; if the message types are well-designed, everything else falls into place:

// worker.js — Message routing (the entry point of the entire inference engine)
self.addEventListener("message", async (e) => {
  const { type, data } = e.data;

  switch (type) {
    case "check":   check();    break;   // WebGPU capability detection
    case "load":    load();     break;   // Download model + warm-up
    case "generate":           // Start generation
      stopping_criteria.reset();         // 🔑 Reset interrupt signal (might have been interrupted last time)
      generate(data);          break;
    case "interrupt":
      stopping_criteria.interrupt();     // 🔑 Set interrupt flag; the generation loop checks this and stops
      break;
    case "reset":
      past_key_values_cache = null;       // 🔑 Clear KV cache, start a brand new conversation
      stopping_criteria.reset();
      break;
  }
});

Direction	type	Meaning	Carried Data
React → Worker	`check`	Check if WebGPU is available	None
React → Worker	`load`	Start downloading the model	None
React → Worker	`generate`	Initiate an inference	`data`: messages array
React → Worker	`interrupt`	Interrupt current generation	None
React → Worker	`reset`	Reset conversation context	None
Worker → React	`loading`	Model download progress	`data`: progress text
Worker → React	`initiate`	Single file download starts	`file`, `progress`, `total`
Worker → React	`progress`	Single file download progress	`file`, `progress`, `total`
Worker → React	`done`	Single file download complete	`file`
Worker → React	`ready`	Model is ready	None
Worker → React	`update`	Streaming generation, new token arrived	`output`, `tps`, `numTokens`, `state`
Worker → React	`complete`	Generation finished	`output`: full decoded text
Worker → React	`error`	Error inside Worker	`data`: error message

The message types evolved from 5 to 13, not because of "design-first," but because of "problem-driven"—every time you discover "the main thread doesn't know about this state," you add a message type. The best way to design a message protocol is not to draw UML upfront, but to let the code run and add what's missing.

Second Hurdle: generate()—The Core of the Inference Pipeline

The message arrives at the Worker and is routed to generate(data). This function is the heart of the entire inference engine—it turns the messages array into tokens, lets the model generate, and pushes the results back to the main thread word by word via callbacks.

// worker.js — Inference pipeline (core)
async function generate(messages) {
  // Step 1: Get the singleton tokenizer and model
  const [tokenizer, model] = await TextGenerationPipeline.getInstance();

  // Step 2: messages array → token ids the model can consume
  const inputs = tokenizer.apply_chat_template(messages, {
    add_generation_prompt: true,  // 🔑 Automatically append <|im_start|>assistant\n, letting the model continue writing the answer
    return_dict: true,            // Return {input_ids, attention_mask} instead of a plain array
  });

  // Step 3: Parse DeepSeek's  thinking tags to distinguish "thinking" from "answering"
  const [START_THINKING_TOKEN_ID, END_THINKING_TOKEN_ID] = tokenizer.encode(
    " thinking response",
    { add_special_tokens: false }
  );

  let state = "thinking";  // Initial state: thinking
  let startTime;
  let numTokens = 0;
  let tps;  // tokens per second
  // ...(callback functions in the next section)
}

Why use `apply_chat_template` instead of manually concatenating strings?

// ❌ Common mistake beginners make:
const prompt = `User: ${messages[0].content}\nAssistant:`;
const inputs = tokenizer(prompt);

// ✅ Correct approach:
const inputs = tokenizer.apply_chat_template(messages, {
  add_generation_prompt: true,
  return_dict: true,
});

DeepSeek-R1 was trained using a specific chat template format (<|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n). If you manually concatenate strings, the format won't match the training data, and the model's output quality will severely degrade—the "signal" it receives is completely different from training. apply_chat_template ensures the format strictly matches the training. Never manually concatenate prompt strings for an LLM; let the tokenizer handle it.

Third Hurdle: TextStreamer—How Tokens "Stream" Back

This is the most elegant part of the entire inference engine. model.generate() doesn't return all results at once—that would make the user wait too long. Instead, it triggers a callback every time a token is generated.

// worker.js — Setting up streaming callbacks
const token_callback_function = (tokens) => {
  // 🔑 Use performance.now() for high-precision timing (100x more precise than Date.now())
  startTime ??= performance.now();

  if (numTokens++ > 0) {
    // 🔑 TPS = number of tokens / elapsed time (seconds) × 1000
    tps = (numTokens / (performance.now() - startTime)) * 1000;
  }

  // ⚠️ Detect DeepSeek's end-of-thinking marker
  // tokens[0] is the token the model just generated
  if (tokens[0] == END_THINKING_TOKEN_ID) {
    state = "answering";  // Switch from "thinking mode" to "answering mode"
  }
};

const callback_function = (output) => {
  // 🔑 After each token is generated, immediately push it back to the main thread via postMessage
  self.postMessage({
    status: "update",
    output,        // Current accumulated text (grows each time)
    tps,           // Real-time generation speed
    numTokens,     // Total tokens generated so far
    state,         // "thinking" | "answering"
  });
};

const streamer = new TextStreamer(tokenizer, {
  skip_prompt: true,           // Don't repeat the prompt in the output
  skip_special_tokens: true,   // Filter out special tokens like <|im_end|>
  callback_function,           // 🔑 Triggered once per token
  token_callback_function,     // 🔑 Lower-level than callback_function: receives raw token id
});

The difference between the two callbacks is important:

Callback	Trigger Timing	Data Received	Use Case
`token_callback_function`	After model predicts a token (before decoding)	Raw token id (integer)	Performance stats (TPS), state switching
`callback_function`	After token is decoded to text	Accumulated text string	Push to UI for display

The order of token ID first, then text is crucial. If you need performance monitoring (like TPS, detecting special tokens), do it in token_callback—getting the token ID there is zero-cost; if you wait until it's decoded to text to parse, you're doing an extra string operation for nothing.

How are streaming callbacks invoked?

The entire flow is as follows:

Inside model.generate() loop:
  for (let i = 0; i < max_new_tokens; i++) {
    1. Model predicts next token → gets token_id
    2. streamer.token_callback([token_id])  ← Calculate TPS, detect  response
    3. tokenizer.decode(token_id) → text fragment
    4. streamer.callback(accumulated text)          ← postMessage pushes back to main thread
    5. Check stopping_criteria → Interrupted?
  }

TextStreamer is a built-in streaming output tool in Transformers.js. You don't need to manually write the loop—just pass in the callback functions, and it handles the rest (decoding, concatenation, skipping special tokens) for you.

Fourth Hurdle: KV-Cache—Why the Second Conversation is Faster Than the First

You might notice: within the same conversation round, the second reply is faster than the first. This isn't an illusion; it's past_key_values_cache at work.

// worker.js — KV-Cache caching
let past_key_values_cache = null;  // 🔑 Module-level variable, reused across generate() calls

async function generate(messages) {
  // ...
  const { past_key_values, sequences } = await model.generate({
    ...inputs,
    // past_key_values: past_key_values_cache,  // Currently commented out in this version
    max_new_tokens: 2048,
    streamer,
    stopping_criteria,
    return_dict_in_generate: true,  // 🔑 Must be set to true to get past_key_values
  });

  past_key_values_cache = past_key_values;  // 🔑 Cache this round's KV computation results for next reuse
  // ...
}

What is KV-Cache?

Every time a Transformer model generates a new token, it must perform Attention computation on all historical tokens (multiplying the Key and Value matrices for each token). Without KV-Cache:

Generating 1st token: Compute Attention 1 time
Generating 2nd token: Recompute Attention for the first 2 tokens
Generating 3rd token: Recompute Attention for the first 3 tokens
...
Generating Nth token: Recompute Attention for the first N tokens

Total computation is O(N²). With KV-Cache: each round only computes Attention for the new token, reading historical tokens' Key/Value directly from the cache. Total computation drops to O(N).

❌ Without KV-Cache:
  Round 1 conversation: Compute 1+2+3+...+2048 Attentions ≈ 2 million times
  Round 2 conversation: Recompute from scratch → another 2 million times

✅ With KV-Cache:
  Round 1 conversation: Same as above
  Round 2 conversation: Only compute Attention for new tokens → a few thousand times

This is why "continuing a conversation" is faster than "starting a new conversation." It's not that the model got smarter; it's that the previous computation results weren't thrown away.

Why does the reset message need to clear the KV-Cache?

case "reset":
  past_key_values_cache = null;  // 🔑 Start a brand new conversation, discard all previous context
  stopping_criteria.reset();
  break;

If not cleared, the model will carry over all historical tokens' Key/Value from the previous conversation round—the result is:

The new conversation's answers are contaminated with the previous round's context
The cache grows larger, making inference slower and slower
VRAM/memory usage increases indefinitely

Fifth Hurdle: Interruption—After Clicking the Stop Button, Why Does the Model Actually Stop?

Generation is a synchronous for loop (although WebGPU computation is asynchronous), so you can't directly "kill" it. But you can check a flag at each step of the loop:

// worker.js
const stopping_criteria = new InterruptableStoppingCriteria();

// User clicked the stop button:
case "interrupt":
  stopping_criteria.interrupt();  // Sets internal _interrupted to true
  break;

// Inside model.generate():
// After generating each token, check stopping_criteria
// If _interrupted === true → stop immediately, don't generate the next token

stateDiagram-v2
    [*] --> Idle: reset
    Idle --> Generating: generate
    Generating --> Idle: complete
    Generating --> Interrupted: interrupt
    Interrupted --> Idle: reset
    note right of Interrupted: Keep already generated tokens

InterruptableStoppingCriteria is essentially a cooperative interruption—it doesn't forcefully kill the thread (which you can't do in a Worker anyway), but checks before generating each next token. It's like running a marathon where someone at every water station holds up a sign asking "Want to stop?"—you can stop when you see the sign, but you don't need to check between every single step.

Design of keys after interruption: Tokens already generated before interrupt() is called are not lost—they have already been pushed back to the main thread for display via callback_function. Interruption simply stops generating new tokens.

Sixth Hurdle: Think/Answer State Machine—DeepSeek's "Inner Monologue"

DeepSeek-R1 is a reasoning model. Before giving its final answer, it first performs "inner reasoning" inside thinking tags:

 thinking
The equation x² - 3x + 2 = 0, I need to use the quadratic formula...
Discriminant = 9 - 8 = 1 > 0, so there are two real roots
x = (3 ± 1) / 2 = 2 or 1
 response

The solutions to the equation x² - 3x + 2 = 0 are **x = 1** and **x = 2**.

To allow the UI to distinguish between the "thinking process" and the "final answer" (for example, showing the thinking process in a collapsible area), a simple state machine is implemented in the worker:

// worker.js — Think/Answer state machine
const [START_THINKING_TOKEN_ID, END_THINKING_TOKEN_ID] = tokenizer.encode(
  " thinking response",
  { add_special_tokens: false }
);

let state = "thinking";  // Two states: 'thinking' | 'answering'

const token_callback_function = (tokens) => {
  if (tokens[0] == END_THINKING_TOKEN_ID) {
    state = "answering";  // 🔑 Detected  response token, switch state
  }
};

const callback_function = (output) => {
  self.postMessage({
    status: "update",
    output,
    tps,
    numTokens,
    state,  // 🔑 UI uses this field to determine if current text is "thinking" or "answering"
  });
};

This state machine has only 2 states and 1 transition condition, but it's very reliable. Because token ID matching is exact—there's no possibility of regex matching errors. In the model's generated token stream, the response token is always that specific ID.

The UI side can render differently based on the state field (e.g., gray italic text for thinking, normal style for answering), but this part will be expanded in the next article (React Hook encapsulation).

Why Put Inference in a Web Worker? It's Not Just "So the UI Doesn't Freeze"

On Main Thread	In Web Worker
`model.generate()` blocks DOM rendering	Main thread stays idle, user can scroll, click
Browser may show "Page Unresponsive"	Completely seamless
Cannot utilize multi-core CPU	Worker runs on an independent thread
Cannot gracefully clean up when component unmounts	`worker.terminate()` is clean and neat
Large file downloads block user interaction	Downloads in Worker don't affect UI

Someone asks, "Can React's useTransition replace a Worker?" The answer is no. useTransition only marks state updates as low priority, but the JS computation itself still runs on the main thread—when it hits a synchronous blocking operation like model.generate(), it will still freeze. A Worker is true multithreading; computation and UI rendering don't interfere with each other.

Summary: 5 Design Decisions of the Inference Engine

Looking back at this panoramic diagram:

graph LR
    subgraph Main Thread
        A[React UI] -->|1. postMessage| B[Message Router]
    end
    subgraph Worker
        B -->|2. generate| C[&#34;Tokenization<br/>apply_chat_template&#34;]
        C -->|3. token ids| D[&#34;Model Inference<br/>model.generate&#34;]
        D -->|4. Each token| E[&#34;TextStreamer<br/>Dual Callbacks&#34;]
        E -->|5. postMessage| F[UI Update]
        H[&#34;past_key_values<br/>KV-Cache&#34;] -.->|Next reuse| D
        I[&#34;Interruptable<br/>StoppingCriteria&#34;] -.->|Check interrupt| D
    end

Each design decision corresponds to a problem:

#	Decision	Problem Solved
1	Put inference in Worker thread	Main thread doesn't block, page doesn't freeze
2	`apply_chat_template`	Prompt format matches training, model output quality doesn't degrade
3	TextStreamer dual callbacks	Each token pushed to UI in real-time, user sees "typewriter effect"
4	past_key_values_cache	Multi-turn conversations don't recompute Attention, speed doubles from second turn onward
5	InterruptableStoppingCriteria	User can interrupt generation anytime, no wasted computation

Golden Quote: The message design of an inference engine isn't about drawing UML diagrams—it's about adding a postMessage at every moment the "main thread doesn't know."