Vue 3.4 Ditches the Token Array for a Streaming State-Machine Parser

Vue3.4+ Template AST Generation Principles

Comparison of old and new processes

• Before Vue3.4: Template string → Scanner generates a complete Token array → Parser loops to consume Tokens → Build AST. Drawback: All Tokens must be stored in an array first, consuming extra memory for long templates.

• Vue3.4+ new version: Template string → Tokenizer scans character-by-character with a state machine → Identifies units such as text / tags / interpolations / attributes → Synchronously calls back to instantly create AST nodes, no longer generating a complete Token array.

Core source code (original code retained, supplemented with comments and type semantics)

// compile.ts
import { baseParse } from './parser'
import type { CodegenResult, CompilerOptions, RootNode } from './types'

export function baseCompile(
  source: string | RootNode,
  options: CompilerOptions = {},
): CodegenResult {
  // ... other compilation pre-logic omitted
  const ast = isString(source) ? baseParse(source, resolvedOptions) : source
  // ... transform, generate subsequent logic omitted
}

// parser.ts
import Tokenizer from './tokenizer'
import { NodeTypes, RootNode, ElementNode } from './types'
import { getLoc, createRoot } from './utils'

let currentRoot: RootNode | null = null
let currentInput = ''
const stack: ElementNode[] = []

type Callbacks = {
  onText: (start: number, end: number) => void
  // also includes onTagOpen / onTagClose / onInterpolation / onAttribute etc. callbacks, here only onText is shown as an example
}

export function baseParse(input: string, options?: ParserOptions): RootNode {
  currentInput = input
  const root = (currentRoot = createRoot([], input))
  // Instantiate the character scanner, passing in the element stack and node generation callbacks
  const tokenizer = new Tokenizer(stack, {
    onText(start: number, end: number) {
      const parent = stack[0] || currentRoot!
      const content = currentInput.slice(start, end)
      parent.children.push({
        type: NodeTypes.TEXT,
        content,
        loc: getLoc(start, end),
      })
    },
  })
  tokenizer.parse(currentInput)
  currentRoot = null
  return root
}

// tokenizer.ts
import { CharCodes, State } from './constants'

export default class Tokenizer {
  private buffer = ''
  private index = 0
  private sectionStart = 0
  private state: State = State.Text

  constructor(
    private readonly stack: ElementNode[],
    private readonly cbs: Callbacks,
  ) {}

  public parse(input: string): void {
    this.buffer = input
    this.index = 0
    this.sectionStart = 0
    this.state = State.Text
    // Traverse the template string character by character
    while (this.index < this.buffer.length) {
      const c = this.buffer.charCodeAt(this.index)
      // Dispatch processing logic based on current state
      switch (this.state) {
        case State.Text:
          this.stateText(c)
          break
        // Other state branches such as BeforeTagName / InTagName / InAttr etc. omitted
      }
      this.index++
    }
  }

  private stateText(c: number): void {
    // Encountering the < symbol: truncate the current text interval, trigger onText callback, switch to tag parsing state
    if (c === CharCodes.Lt) {
      if (this.index > this.sectionStart) {
        this.cbs.onText(this.sectionStart, this.index)
      }
      this.state = State.BeforeTagName
      this.sectionStart = this.index
    }
    // Other branch judgments for interpolation {{, newlines etc. in text state omitted
  }
}

Example execution flow: source = "123<div></div>"

1. baseCompile: Determines source is a string, calls baseParse

2. baseParse

   • Initializes the root node, assigns global currentRoot
   • Creates a Tokenizer instance, passing in the element stack and onText callback (used to generate TEXT nodes)
   • Executes tokenizer.parse(source) to start character-by-character scanning

3. Tokenizer.parse

   Begins traversing characters one by one:

   • Reads 1, 2, 3 sequentially, in State.Text state
   • Reads <, triggers text truncation: calls onText(0,3), generates a TEXT node from "123", mounts it to the root node's children
   • State switches to State.BeforeTagName, continues parsing the div opening tag, pushes the element node onto the stack
   • Subsequently recognizes the closing </div>, pops the element from the stack

4. After scanning all characters, baseParse clears the global currentRoot, returns the completed Root AST

When source="123<div></div>", the execution logic is as follows:

1. Tokenizer: Creates the tokenizer object via new Tokenizer, registers callback functions
2. baseCompile: Calls baseParse passing in the source string
3. baseParse: Assigns the source string to the global variable currentInput, creates the root node root via the createRoot function and assigns it to the global variable currentRoot, calls the instantiated tokenizer.parse, and finally returns the root node root
4. tokenizer.parse: Scans the template character by character, calls callbacks based on state, for example the onText callback mounts the TEXT node to the root node

Understanding Tokenizer Source Code by Scenario

Based on Vue source code __tests__/parse.spec.ts plain text scenario test cases, a complete breakdown of the Tokenizer's parsing logic for ordinary text, loc position information generation rules, and newline/excess whitespace handling principles, retaining core source code throughout, organized by the logic of "execution flow + core principles + case evidence", clearly restoring the underlying mechanism of Vue template text parsing.

Scenario One: Text Characters Only (source="\n some text")

Test scenario: The template source contains only newlines, spaces, and ordinary text, with no tags or interpolation syntax — the most basic template parsing scenario.

describe('Text', () => {
    test('simple text', () => {
      const ast = baseParse('\n   some text')
      const text = ast.children[0] as TextNode

      expect(text).toStrictEqual({
        type: NodeTypes.TEXT,
        content: ' some text',
        loc: {
          start: { offset: 1, line: 1, column: 0 },
          end: { offset: 13, line: 2, column: 13 },
          source: '\n   some text',
        },
      })
})

Core questions:

• How does baseParse parse plain text characters segment by segment and generate text AST nodes?
• How are the loc line/column numbers, offsets, and source intervals in AST nodes precisely calculated?
• How are newline characters and leading/trailing/consecutive excess whitespace in templates preprocessed and compressed?

Overall Execution Chain: baseParse Entry Flow

All template parsing starts from the baseParse function. Core flow: Create root AST node → Tokenizer tokenizes character by character → Whitespace normalization processing → Backfill root node information and return AST.

// baseParse core source code
let currentRoot;
export function baseParse(input: string, options?: ParserOptions): RootNode {    
  // parser.ts lines 1073~1076    
  const root = (currentRoot = createRoot([], input)) // Bind global root node variable    
  tokenizer.parse(currentInput) // Start tokenizer to parse template character by character    
  root.loc = getLoc(0, input.length) // Bind global position information to root node    
  root.children = condenseWhitespace(root.children) // Normalize whitespace characters    
  currentRoot = null // Clear global temporary variable    
  return root    
}

Step-by-step execution logic

1. Create root node: Call createRoot to generate the template root AST node, initializing basic properties;
2. Tokenize and parse: Execute tokenizer.parse, traversing the template string character by character, identifying syntax such as text, tags, interpolations;
3. Whitespace processing: Compress and clean up redundant newlines and spaces via condenseWhitespace;
4. Wrap up and return: Bind position information to the root node, clear global variables, output the complete AST.

createRoot root node creation source code: The root node is the top-level node of the entire AST, storing global information such as the template's original source code, child nodes, and compilation auxiliary variables.

// ast.ts lines 603~622
export function createRoot(    
  children: TemplateChildNode[],    
  source = '',    
): RootNode {    
  return {    
    type: NodeTypes.ROOT,    
    source,    
    children,    
    helpers: new Set(),    
    components: [],    
    directives: [],    
    hoists: [],    
    imports: [],    
    cached: [],    
    temps: 0,    
    codegenNode: undefined,    
    loc: locStub,    
  }    
}

Tokenizer Text Tokenization Core Logic

The Tokenizer's default initial state is State.Text (text state). The vast majority of ordinary characters go through text parsing logic; only special trigger characters switch the parsing state.

Tokenization main loop source code: Traverses the template character by character, separately captures newline character positions for subsequent loc line/column calculation, and by default enters the text parsing branch.

export default class Tokenizer {    
  // tokenizer.ts lines 944~955    
  public parse(input: string): void {    
    this.buffer = input    
    while (this.index < this.buffer.length) {    
      const c = this.buffer.charCodeAt(this.index)    
      // Separately identify newline characters, record positions, used for loc line/column parsing (excluding entity character scenarios)    
      if (c === CharCodes.NewLine && this.state !== State.InEntity) {    
          this.newlines.push(this.index)    
      }    
      // Dispatch processing logic based on current parsing state    
      switch (this.state) {    
          case State.Text: {    
              this.stateText(c)    
              break    
          }    
          // Other state branches omitted (tags, interpolations, attributes, etc.)    
      }    
      this.index++    
    }    
    this.cleanup() // Wrap up remaining text fragments    
    this.finish()    
  }    
}

Text state handling stateText: Only switches state for special characters; ordinary text characters are directly retained without additional processing. There are only 3 types of special characters that trigger state switching: tag left angle bracket, entity character, interpolation start character. All other characters are determined to be ordinary text.

// tokenizer.ts lines 334~348
private stateText(c: number): void {    
  // Encounter tag start character <, end current text parsing, switch to tag parsing state    
  if (c === CharCodes.Lt) {    
    if (this.index > this.sectionStart) {    
        this.cbs.ontext(this.sectionStart, this.index)    
    }    
    this.state = State.BeforeTagName    
    this.sectionStart = this.index    
  } 
  // Outside browser environment, encounter entity character &, enter entity parsing state
  else if (!__BROWSER__ && c === CharCodes.Amp) {    
    this.startEntity()    
  } 
  // In non-v-pre environment, encounter interpolation start character, enter interpolation parsing state
  else if (!this.inVPre && c === this.delimiterOpen[0]) {    
    this.state = State.InterpolationOpen    
    this.delimiterIndex = 0    
    this.stateInterpolationOpen(c)    
  }    
  // Ordinary text characters: no logic, continue traversing    
}

Text fragment wrap-up cleanup: After the tokenization main loop ends, cleanup is called to handle the last unfinished text fragment, triggering the ontext callback to generate a text node.

// tokenizer.ts lines 1098~1117
private cleanup() {    
  if (this.sectionStart !== this.index) {    
    // In text state, CDATA state, wrap up text fragment
    if (    
      this.state === State.Text ||    
      (this.state === State.InRCDATA && this.sequenceIndex === 0)    
    ) {    
      this.cbs.ontext(this.sectionStart, this.index)    
      this.sectionStart = this.index    
    } 
    // Attribute value state wrap-up logic (omitted)
    else if (    
      this.state === State.InAttrValueDq ||    
      this.state === State.InAttrValueSq ||    
      this.state === State.InAttrValueNq    
    ) {    
      this.cbs.onattribdata(this.sectionStart, this.index)    
      this.sectionStart = this.index    
    }    
  }    
}

ontext callback: Generates text AST nodes. The tokenizer binds the ontext callback during initialization, receiving the start and end indices of the text fragment, ultimately generating/merging text nodes and mounting them to the root node's children array.

// parser.ts lines 100~105
const tokenizer = new Tokenizer(stack, {    
  ontext(start, end) {    
    onText(getSlice(start, end), start, end)    
  },    
})

// parser.ts lines 594~614
function onText(content: string, start: number, end: number) {    
  // When there are no nested nodes, text is mounted to the root node
  const parent = stack[0] || currentRoot    
  const lastNode = parent.children[parent.children.length - 1]    
  // Consecutive text nodes: merge content, update end position
  if (lastNode && lastNode.type === NodeTypes.TEXT) {    
    lastNode.content += content    
    setLocEnd(lastNode.loc, end)    
  } 
  // Brand new text node: create node, generate loc position information and push into child node array
  else {    
    parent.children.push({    
      type: NodeTypes.TEXT,    
      content,    
      loc: getLoc(start, end),    
    })    
  }    
}

loc Position Information Generation Mechanism

loc is the core of Vue template compilation error precise positioning, containing: start/end line and column numbers, character offset, and corresponding source code fragment, calculated through the two-layer methods getLoc + getPos. getLoc uniformly encapsulates position information

// parser.ts lines 916~924
function getLoc(start: number, end?: number): SourceLocation {    
  return {    
    start: tokenizer.getPos(start), // Start position (line, column, offset)    
    end: end == null ? end : tokenizer.getPos(end), // End position    
    source: end == null ? end : getSlice(start, end), // Corresponding source code fragment    
  }    
}

// Extract the source string for the corresponding index interval
function getSlice(start: number, end: number) {    
  return currentInput.slice(start, end)    
}

getPos index-to-line/column core algorithm: Through the pre-stored this.newlines (array of all newline character indices), converts a one-dimensional string offset index into a two-dimensional editor line/column position.

Optimization strategy: When the newline character array length exceeds 100, binary search is used; otherwise, reverse traversal is used, ensuring parsing performance.

// tokenizer.ts lines 296~328
public getPos(index: number): Position {
    // Default value: if there is no preceding newline, it is line 1, column defaults to index+1
    let line = 1
    let column = index + 1
    const length = this.newlines.length // this.newlines array of newline character positions
    let j = -1 // -1 means there is no newline before index

    // If the newline array is relatively long, use binary search for optimization
    if (length > 100) {
      let l = -1
      let r = length
      while (l + 1 < r) {
        const m = (l + r) >>> 1 // Unsigned right shift, equivalent to Math.floor((l+r)/2)
        // If current newline position < index → target is on the right, update left boundary
        this.newlines[m] < index ? (l = m) : (r = m)
      }
      j = l // After loop ends, l is the last subscript satisfying newlines[l] < index
    } else {
      // Array is very short, directly traverse in reverse, find the first newline less than index
      for (let i = length - 1; i >= 0; i--) {
        if (index > this.newlines[i]) {
          j = i
          break
        }
      }
    }

    // Found a preceding newline character
    if (j >= 0) {
      line = j + 2 // newlines[j] is the newline at the end of line j+1, so current is line j+2
      column = index - this.newlines[j] // Distance of current index from the previous newline (column starts from 1? Note this!)
    }

    return {
      column,
      line,
      offset: index,
    }
}

Algorithm case walkthrough

Test template: source = a

bc

def, formatted equivalent: a\nbc\ndef, newline character index array: this.newlines = [1,4]

• index=0 (character a): No preceding newline, j=-1 → line=1, column=1, offset=0
• index=1 (newline character \n): No newline less than 1, j=-1 → line=1, column=2, offset=1 (newline character belongs to the end of the previous line)
• index=2 (character b): Preceding nearest newline index 1, j=0 → line=2, column=1, offset=2
• index=4 (second \n): Preceding nearest newline index 1, j=0 → line=2, column=3, offset=4
• index=5 (character d): Preceding nearest newline index 4, j=1 → line=3, column=1, offset=5

Whitespace Character Normalization Processing (Newlines/Excess Spaces)

After tokenization is complete, baseParse finally executes condenseWhitespace, performing unified normalization processing on newlines, spaces, and tabs in text nodes — this is the core of Vue template whitespace elegant rendering.

Core configuration rules

• Default mode: whitespace: 'condense' (compression mode)
• Special mode: whitespace: 'preserve' (preserve all whitespace, no compression logic executed)
• Inside <pre> tags: Only unify newline characters \r\n to \n, all whitespace preserved as-is

Whitespace compression complete source code

// parser.ts lines 837~883
function condenseWhitespace(nodes: TemplateChildNode[]): TemplateChildNode[] {    
  const shouldCondense = currentOptions.whitespace !== 'preserve'    
  let removedWhitespace = false    
  for (let i = 0; i < nodes.length; i++) {    
    const node = nodes[i]    
    // Only process text nodes; element and comment nodes are skipped directly
    if (node.type === NodeTypes.TEXT) {    
      // Not inside pre tag, execute whitespace compression logic
      if (!inPre) {    
        // Scenario 1: Text node is [pure whitespace] (only spaces/newlines/tabs)
        if (isAllWhitespace(node.content)) {    
          const prev = nodes[i - 1] && nodes[i - 1].type    
          const next = nodes[i + 1] && nodes[i + 1].type    
          // Directly delete pure whitespace nodes if the following conditions are met:
          // 1. Leading/trailing whitespace nodes  2. Whitespace between comments or between comment and element  3. Whitespace between elements containing newlines
          if (    
            !prev ||    
            !next ||    
            (shouldCondense &&    
              ((prev === NodeTypes.COMMENT &&    
                (next === NodeTypes.COMMENT || next === NodeTypes.ELEMENT)) ||    
                (prev === NodeTypes.ELEMENT &&    
                  (next === NodeTypes.COMMENT ||    
                    (next === NodeTypes.ELEMENT &&    
                      hasNewlineChar(node.content))))))    
          ) {    
            removedWhitespace = true    
            nodes[i] = null as any    
          } 
          // Whitespace between elements without newlines: preserve a single space
          else {    
            node.content = ' '    
          }    
        } 
        // Scenario 2: Non-pure-whitespace text, compress internal consecutive whitespace to a single space
        else if (shouldCondense) {    
          node.content = condense(node.content)    
        }    
      } 
      // Inside pre tag: only unify newline characters, do not compress whitespace
      else {    
        node.content = node.content.replace(windowsNewlineRE, '\n')    
      }    
    }    
  }    
  // Filter out whitespace nodes that were set to null
  return removedWhitespace ? nodes.filter(Boolean) : nodes    
}

Whitespace handling core rules summary

1. Pure whitespace text nodes (only spaces/newlines/tabs)

   • Template leading/trailing whitespace, whitespace between comments and elements, whitespace between elements containing newlines: directly deleted

   • Whitespace between elements without newlines: preserve 1 space

2. Text nodes containing valid content: All internal consecutive spaces, newlines, tabs: uniformly compressed to a single space

3. Inside <pre> tags

   • No whitespace compression executed, original formatting preserved

   • Only Windows newlines \r\n are uniformly converted to \n, unifying cross-platform rendering consistency

Full Text Process Summary

1. Tokenization phase: Tokenizer traverses characters in default text state, only switches state for special syntax, all ordinary text is retained, and raw text nodes are generated after completion;
2. Position calculation: Through the newline character index array + binary/traversal algorithm, converts one-dimensional character offsets into two-dimensional line/column positions, precisely generating loc information;
3. Whitespace optimization: Through condenseWhitespace, cleans up redundant newlines, compresses consecutive spaces, balancing rendering aesthetics with browser and SSR consistency.