Two Core Foundations of LLMs: Tokenization and Embedding Semantic Vectorization

Tokenization and text vectorization are the foundation of all large model applications and are key knowledge points that distinguish ordinary users from developers. Below is a complete explanation combining theory and locally runnable code.

1. Tokenization: The Smallest Unit for Model Computation and Billing

Basic Token Pricing Rules

A token is the smallest unit for processing text in a large model and is also the statistical standard for API billing. The splitting logic is not limited to words or Chinese characters:

• English text: Approximately 3 English characters equal 1 token
• Chinese text: Approximately 2 Chinese characters equal 1 token
• Cost reference: The cost for calling a million tokens is only a few yuan, making personal learning and debugging very low-cost

Core Reasons Why Text Must Be Tokenized

1. Neural networks can only recognize numbers, vectors, and matrices, and cannot directly read text. Text needs to be converted into a sequence of numerical IDs;
2. The generation logic of large models is based on the preceding text sequence to predict the next most probable subword. Tokenization is a prerequisite step in the generation process;
3. Mainstream GPT series models use BPE (Byte Pair Encoding) subword splitting rules, adapting to various text scenarios including mixed Chinese and English, rare vocabulary, and long words.

Local Demo Code for Token Encoding and Decoding

Use the open-source tiktoken tokenization library to implement text encoding and decoding restoration. The code can be run directly locally:

javascript

import { getEncoding } from 'js-tiktoken';
// GPT general-purpose tokenization dictionary
const tokenEncoder = getEncoding('cl100k_base');
const inputText = "Hello tiktoken，大模型分词原理讲解";

// Convert text to an array of token numerical IDs
const tokenList = tokenEncoder.encode(inputText);
console.log('Token ID array:', tokenList);
console.log('Total token count:', tokenList.length);

// Restore the original text from token numerical IDs
const recoverText = tokenEncoder.decode(tokenList);
console.log('Restored text:', recoverText);

Billing logic explanation: The user's input content is split into input tokens, and the model's returned content is split into output tokens. The sum of the two is the total number of tokens billed for this call.

2. Embedding Semantic Vectorization: Enabling Models to Understand Text Semantics

Complete Text Processing Pipeline

Original text → Tokenization splitting → Embedding conversion to high-dimensional vectors → Transformer model computation → Output token sequence → Decoding into readable text

Core Functions of Embedding

1. Converts natural language into 1024-dimensional floating-point vectors, with numerical values ranging from [-1, 1];
2. The vector carries the deep semantics of the text. Texts with similar semantics are closer in distance within the high-dimensional vector space;
3. Quantifies the similarity between two pieces of text using cosine similarity. This is the underlying core of knowledge base retrieval, document matching, and RAG systems.

Complete Code for Vector Generation + Cosine Similarity (Using Alibaba Cloud Bailian API)

Alibaba Cloud Bailian provides an embedding interface compatible with the OpenAI standard. Domestic accounts can apply and use it directly. The code includes a security reminder for the API key:

javascript

import { getEncoding } from 'js-tiktoken';
import OpenAI from 'openai';
import dotenv from 'dotenv';
dotenv.config();

// Initialize Alibaba Cloud Bailian client
// Important security reminder: It is forbidden to upload or publish real API keys in plain text. Store them uniformly in local environment variables to avoid asset loss.
const llmClient = new OpenAI({
  apiKey: process.env.DASHSCOPE_API_KEY,
  baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1'
});

/**
 * Generate a 1024-dimensional semantic vector for text
 * @param {string} text Input text
 * @returns {number[]} Semantic vector array
 */
async function createTextEmbedding(text) {
  const result = await llmClient.embeddings.create({
    model: 'text-embedding-v4',
    input: text,
    dimensions: 1024
  });
  return result.data[0].embedding;
}

/**
 * Calculate the cosine similarity of two vectors. The closer the value is to 1, the higher the semantic similarity.
 * @param {number[]} vec1 Vector 1
 * @param {number[]} vec2 Vector 2
 * @returns {number} Similarity value [-1, 1]
 */
function calculateCosSimilarity(vec1, vec2) {
  let dotProduct = 0;
  let vec1Length = 0;
  let vec2Length = 0;
  for (let index = 0; index < vec1.length; index++) {
    dotProduct += vec1[index] * vec2[index];
    vec1Length += Math.pow(vec1[index], 2);
    vec2Length += Math.pow(vec2[index], 2);
  }
  return dotProduct / (Math.sqrt(vec1Length) * Math.sqrt(vec2Length));
}

// Local test for semantic similarity
async function runSimilarDemo() {
  const textA = "Karpathy详解LLM底层BPE分词与Token机制";
  const textB = "卡帕西分享大模型子词分词Tokenization原理";
  const textC = "今日多云，适合外出骑行踏青";

  const vecA = await createTextEmbedding(textA);
  const vecB = await createTextEmbedding(textB);
  const vecC = await createTextEmbedding(textC);

  console.log('Similarity of similar professional texts:', calculateCosSimilarity(vecA, vecB));
  console.log('Similarity of unrelated texts:', calculateCosSimilarity(vecA, vecC));
}
runSimilarDemo();

Summary of Local Debugging Practice (Original Personal Experience)

After debugging this code locally multiple times, I have summarized two practical development conclusions:

1. Vectors generated from long Chinese texts are more stable, while the similarity calculation for short sentences fluctuates more;
2. 1024-dimensional vectors provide sufficient performance for similarity calculations and are suitable for small local knowledge bases. For massive document retrieval involving hundreds of thousands or millions of documents, a professional vector database is needed to optimize query efficiency.

Operational effect explanation: The similarity of two highly semantically overlapping professional texts will approach 1 infinitely; the similarity of unrelated texts will approach 0. This logic is widely used in intelligent Q&A, document plagiarism checking, and knowledge base retrieval scenarios.