Vue.js · Frontend · JavaScript

Vue 3.4 Ditches the Token Array for a Streaming State-Machine Parser

By 单线程_01 · Aug 28, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Streaming parse-and-emit removes the memory pressure of a full token array on large templates and cuts one pass from the compilation pipeline. For anyone maintaining a Vue SFC-heavy codebase or building tooling that touches the compiler, the single-pass architecture simplifies reasoning about parse performance and memory profiles.

Summary

Before Vue 3.4, template compilation ran a Scanner to build a full token array, then a Parser consumed it to produce the AST — a two-pass design that forced long templates to hold every token in memory. The new Tokenizer is a character-by-character state machine that recognizes text, tags, interpolations, and attributes inline, firing callbacks that construct AST nodes immediately. No intermediate token array exists.

The change is visible in `baseParse`, which now instantiates a `Tokenizer` with an element stack and callbacks like `onText`. As the state machine walks the input, hitting `<` triggers a text-segment callback and a state transition into tag parsing; ordinary characters pass through with zero overhead. After the loop, a `cleanup` step flushes any remaining text fragment, and `condenseWhitespace` normalizes whitespace — deleting pure-whitespace nodes between elements or at boundaries, compressing internal runs to a single space, and inside `<pre>` only unifying Windows line endings.

Position tracking (`loc`) uses a pre-collected array of newline indices. For arrays longer than 100 entries, a binary search maps a character offset to a line:column pair; shorter arrays use a reverse linear scan. This is what powers precise error messages in Vue’s template compiler.

Takeaways

— Vue 3.4’s Tokenizer is a single-pass state machine that emits AST nodes through callbacks during the character scan, never building a complete token array.

— The default state is `State.Text`; only `<`, `&` (outside the browser), and interpolation delimiters trigger state transitions — all other characters are treated as plain text with no processing overhead.

— A `cleanup` method after the main loop flushes the final text segment, and `condenseWhitespace` then deletes boundary/separator pure-whitespace nodes and compresses internal whitespace runs to a single space.

— Position information (`loc`) is computed by recording every newline index during the scan, then using binary search (for >100 newlines) or reverse linear scan to convert a character offset into a line and column.

— Inside `<pre>` tags, whitespace is never compressed — only `\r\n` is normalized to `\n` for cross-platform consistency.

Conclusions

The old Scanner→Parser pipeline was a textbook two-pass compiler frontend; Vue’s move to a callback-driven state machine mirrors how many production parsers (e.g., SAX-style XML parsers) avoid building intermediate representations to stay memory-bounded.

Recording newline indices and deferring line:column calculation to `getPos` is a deliberate trade-off: it adds a small per-character branch in the hot loop but keeps position lookups O(log n) and avoids storing per-character metadata.

The whitespace condensing rules are surprisingly nuanced — pure-whitespace nodes between elements are deleted only if they contain a newline, otherwise collapsed to a single space. This preserves intentional inline spacing while stripping formatting whitespace, a heuristic that has caused subtle rendering differences across frameworks.

Concepts & terms

Tokenizer (Vue 3.4+)

A character-by-character state machine that scans a template string and synchronously invokes callbacks to build AST nodes, replacing the older Scanner→Parser two-pass design.

condenseWhitespace

A post-parse normalization step that deletes pure-whitespace text nodes at template boundaries or between block-level elements (when they contain newlines), collapses inter-element whitespace without newlines to a single space, and compresses internal whitespace runs in mixed-content text nodes.

loc (SourceLocation)

A position object on each AST node containing start/end line, column, character offset, and the raw source substring. Computed by mapping a character index through a pre-collected array of newline positions using binary or linear search.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗