Tokenization and Embeddings: The Two Primitives Every LLM Application Sits On
Tokenization and embeddings are not academic details — they determine API costs, retrieval quality, and whether a RAG pipeline returns relevant documents or noise. Misunderstanding either leads to bloated bills or broken search.
Every LLM call starts with tokenization: text is broken into subword IDs using BPE, with roughly 3 English characters or 2 Chinese characters per token. These IDs are the only input neural networks can process, and the sum of input and output tokens determines API cost. The open-source tiktoken library encodes and decodes this locally with a single function call.
After tokenization, embedding models convert text into 1024-dimensional vectors where semantically similar sentences sit close together. Cosine similarity between these vectors powers retrieval-augmented generation, document matching, and plagiarism detection. Alibaba Cloud's Bailian API provides an OpenAI-compatible embedding endpoint that returns these vectors directly.
Practical debugging reveals that longer Chinese texts produce more stable vectors than short phrases, and 1024 dimensions handle small knowledge bases fine. At hundreds of thousands of documents, a dedicated vector database becomes necessary to keep queries fast.
Tokenization and embedding are the two gates every LLM application passes through, yet many developers treat them as black boxes until a billing surprise or retrieval failure forces them to look.
The observation that short Chinese texts produce less stable embeddings is a concrete, testable claim that most embedding documentation glosses over — it implies minimum text-length thresholds matter for production RAG quality.
Using Alibaba Cloud's OpenAI-compatible embedding endpoint shows how the API surface has standardized across Western and Chinese providers, making the choice of vendor largely a matter of latency, cost, and data locality rather than integration effort.