Building RAG from Scratch: Embedding Technology

Preface

The previous article introduced how to split markdown files into chunks; refer to Building RAG from Scratch: Chunking Strategies. Next, each chunk is passed to an embedding model, encoded into a set of high-dimensional floating-point numbers, i.e., a vector, and then written into the vector store together with the original text.

Series articles:

1. Building RAG from Scratch: Full Retrieval Pipeline, Giving the LLM Your Own Knowledge Base
2. Building RAG from Scratch: Document Parsing Techniques
3. Building RAG from Scratch: Chunking Strategies

What is Embedding

Embedding maps text, images, audio, video and other information into dense vectors in a high-dimensional space. A vector is an array of floating-point numbers that serves as coordinates: semantically similar objects have close coordinates in that space, while dissimilar ones end up far apart. For example, two passages about pulmonary gas exchange (one about the concept, one about influencing factors) have similar vector directions, while a passage about financial statements falls far away.

Vectors are numeric arrays; embedding is the process of turning information into numeric arrays. To measure how close two vectors are, cosine similarity is commonly used: it compares the angle; the more consistent the direction, the closer the value is to 1, independent of text length. Encoding the customer question as a vector and finding the nearest records in the vector store is vector retrieval. Also note that vectors generated by different embedding models are usually not in the same comparable space; even if dimensions match, they cannot be mixed. When switching models, you typically need to re-encode existing documents and ensure queries and documents use the same vector model.

Embedding Models: From Word Vectors to Sentence Vectors

Around 2013, word vectors were popular: word2vec, GloVe, fastText, and the like. Each word maps to a vector, and semantically similar words sit next to each other in space, e.g., "doctor" and "nurse". The problem is obvious: a word has only one vector, so polysemous words cannot be distinguished. "Apple" gets the same vector whether it refers to the fruit or the company.

Later, as transformers matured, models like BERT and RoBERTa pushed word vectors into a contextual stage. The same word gets different vectors in different sentences, mitigating polysemy. But these models natively produce a vector per token: a sentence of dozens of words yields dozens of vectors. To compute similarity between sentences, you need an extra pooling step, which works poorly.

For retrieval, sentence embeddings are more common: compress an entire piece of text into one vector. Query and document are each encoded separately, then the two vectors are compared by distance. This structure is called a bi-encoder. Document vectors can be computed offline and stored in the database in advance; at query time only the query vector is computed once, so even millions of documents are searchable. As for the more accurate but more expensive approach of passing both sides through the model together, that is left to the reranking discussion in Part 5.

Choosing an Embedding Model

There are many embedding models on the market. When selecting, you usually consider: domain applicability, retrieval accuracy, supported languages, text chunk length, model size, and retrieval efficiency. Beyond technical factors, two more business considerations apply:

• Price: the capability gap among mainstream domestic providers is small; when model performance is similar, unit price and discounts become the deciding factor.
• Provider fit: if the platform already runs entirely on one cloud (e.g., Tencent Cloud), using the same provider for embedding reduces operational cost.

How FastRAG Integrates Embedding

FastRAG centralizes vectorization in one implementation and uses two switches to adapt to different providers.

Query and Document Vectors Are Separate

The domain layer does not depend on any provider's SDK.

Code: domain/interfaces/embedding.go

type IEmbedding interface {
    // EmbedDocs batch encodes documents. Return order corresponds to input.
    EmbedDocs(ctx context.Context, texts []string) ([][]float32, error)

    // EmbedQuery encodes a query string.
    //
    // Must be separate from EmbedDocs: most models (bge-m3 / E5 / GTE) require
    // a prefix for queries (e.g., "query: "), while the doc side does not.
    // Mixing them significantly hurts scores.
    EmbedQuery(ctx context.Context, text string) ([]float32, error)

    Dim() int
    Model() string
}

Request and Response: One Implementation, Two Protocols

The same Embedding implementation uses a protocol switch to interface with two protocols. The three differences (building the request body, sending the query, parsing the response) are each collected in three methods:

Code: infrastructure/serviceimpl/embedding.go

func (e *Embedding) buildBody(texts []string, textType string) any {
    if e.protocol == config.ProtocolDashScope {
        return map[string]any{
            "model": e.model,
            "input": map[string]any{"texts": texts},
            "parameters": map[string]any{
                "text_type": textType,
            },
        }
    }
    return map[string]any{"model": e.model, "input": texts}
}

func (e *Embedding) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
    vecs, err := e.embed(ctx, []string{e.queryPrefix + text}, textTypeQuery)
    if err != nil {
        return nil, err
    }
    if len(vecs) != 1 {
        return nil, apperrors.NewSysError(apperrors.CodeEmbeddingFail,
            fmt.Sprintf("embedding returned %d vectors, expected 1", len(vecs)))
    }
    return vecs[0], nil
}

func (e *Embedding) parseVectors(parsed *embedResponse, n int) [][]float32 {
    out := make([][]float32, n)

    if e.protocol == config.ProtocolDashScope {
        if parsed.Output != nil {
            for _, d := range parsed.Output.Embeddings {
                if d.TextIndex >= 0 && d.TextIndex < n {
                    out[d.TextIndex] = d.Embedding
                }
            }
        }
        return out
    }

    for _, d := range parsed.Data {
        if d.Index >= 0 && d.Index < n {
            out[d.Index] = d.Embedding
        }
    }
    return out
}

Batched Encoding

Batched encoding has one prerequisite: the array positions must not change. Chunks with empty body are filtered out.

Code: infrastructure/serviceimpl/embedding.go

func (e *Embedding) EmbedDocs(ctx context.Context, texts []string) ([][]float32, error) {
    if len(texts) == 0 {
        return nil, nil
    }

    // First pick out the indices that actually need computing; send non-empty ones
    // in original order batched, then scatter back into out by index.
    wanted := make([]int, 0, len(texts))
    for i, t := range texts {
        if strings.TrimSpace(t) != "" {
            wanted = append(wanted, i)
        }
    }

    out := make([][]float32, len(texts))
    for start := 0; start < len(wanted); start += e.batchSize {
        end := start + e.batchSize
        if end > len(wanted) {
            end = len(wanted)
        }

        batch := make([]string, 0, end-start)
        for _, i := range wanted[start:end] {
            batch = append(batch, texts[i])
        }

        vecs, err := e.embed(ctx, batch, textTypeDocument)
        if err != nil {
            return nil, err
        }
        for j, i := range wanted[start:end] {
            out[i] = vecs[j]
        }
    }

    // The remaining unfilled indices are empty texts; leave nil — when serialized
    // to null, ES treats them as absent fields.
    return out, nil
}

Skipped empty texts leave nil in the result rather than a zero vector: ES cosine does not accept a zero-length vector, and filling one in would cause the whole batch write to return 400. Also, a chunk without a title shouldn't have a title vector.

Calling, Validation, and Retry

The single call embedOnce handles sending the request, parsing the response, and validating each item; embed wraps it with retry:

Code: infrastructure/serviceimpl/embedding.go

func (e *Embedding) embed(ctx context.Context, texts []string, textType string) ([][]float32, error) {
    var vecs [][]float32
    err := retry.Do(
        func() error {
            var err error
            vecs, err = e.embedOnce(ctx, texts, textType)
            return err
        },
        retry.Context(ctx), // abort immediately when ctx is canceled; no further attempts
        retry.Attempts(2),  // initial + one retry, same as rerank
        retry.Delay(defaultEmbedRetryDelay),
        retry.DelayType(retry.FixedDelay), // fixed-interval backoff, not default exponential + jitter
        retry.LastErrorOnly(true),         // pass through only the last error, no layered wrapping
    )
    if err != nil {
        return nil, err
    }
    return vecs, nil
}

func (e *Embedding) embedOnce(ctx context.Context, texts []string, textType string) ([][]float32, error) {
    payload, err := json.Marshal(e.buildBody(texts, textType))
    if err != nil {
        return nil, apperrors.ErrEmbeddingFailed.Wrap(err)
    }

    body, status, err := postJSON(ctx, e.client, e.url, e.apiKey,
        payload, maxEmbedResponseBytes, apperrors.ErrEmbeddingFailed)
    if err != nil {
        return nil, err
    }

    var parsed embedResponse
    if err := json.Unmarshal(body, &parsed); err != nil {
        return nil, apperrors.NewSysError(apperrors.CodeEmbeddingFail, fmt.Sprintf(
            "embedding response is not valid JSON, HTTP %d: %s", status, truncateBody(body, 512)))
    }
    if err := parsed.apiError.check(status, apperrors.CodeEmbeddingFail, "embedding", body); err != nil {
        return nil, err
    }

    out := e.parseVectors(&parsed, len(texts))

    for i, v := range out {
        if len(v) == 0 {
            return nil, apperrors.NewSysError(apperrors.CodeEmbeddingFail,
                fmt.Sprintf("embedding item %d missing", i))
        }
        // Dimension mismatch must be caught here: if it slips through to ES,
        // you get a dense_vector mapping error, far from the real cause.
        if e.dim > 0 && len(v) != e.dim {
            return nil, apperrors.NewSysError(apperrors.CodeEmbeddingFail, fmt.Sprintf(
                "embedding dimension is %d, config says %d; they must match", len(v), e.dim))
        }
    }
    return out, nil
}

If the config says 1024 dimensions but the actual vector is 768, this gives a clear error here, which is easier to debug than waiting for an ES mapping error. But matching dimensions doesn't mean the spaces are the same: if the query uses model B and the document uses model A, the length still passes the check, yet the distance is meaningless, so when switching models you have to re-embed the entire batch of documents and queries together.

embed wraps a retry around it: initial plus one, fixed 100 ms interval, stops on context cancellation. This layer exists because a medium-sized document involves dozens of batches; a fluke on any one batch would fail the whole import. But this is per-batch retry, not checkpointed resume for the entire import; if it fails midway, you start over.

Storing to the Database: Title and Content Dual Paths

A single chunk actually computes vectors twice: once for the title, once for the content.

Code: application/service/ingest/ingest.go

func (s *Service) embedChunks(
    ctx context.Context, e interfaces.IEmbedding, chunks entity.Chunks,
) (titleVecs, contentVecs [][]float32, err error) {
    titleVecs, err = e.EmbedDocs(ctx, chunks.Titles()) // title vectorization
    if err != nil {
        return nil, nil, apperrors.ErrEmbeddingFailed.Wrap(err)
    }

    contentVecs, err = e.EmbedDocs(ctx, factory.EmbeddingTexts(chunks)) // content vectorization
    if err != nil {
        return nil, nil, apperrors.ErrEmbeddingFailed.Wrap(err)
    }

    if len(titleVecs) != len(chunks) || len(contentVecs) != len(chunks) {
        return nil, nil, apperrors.NewSysError(apperrors.CodeEmbeddingFail, fmt.Sprintf(
            "vector count mismatch: title %d / content %d, chunks %d",
            len(titleVecs), len(contentVecs), len(chunks)))
    }

    return titleVecs, contentVecs, nil
}

Title and content are computed and stored separately: the title is short and low-noise, used for reranking; the content is long and information-dense, used for recall. The content path takes the Content field; the chunker already prepends the heading chain to the beginning, so the section context rides along.

Finally, they are assembled with the chunks by position into the object to be written:

Code: domain/factory/vector_doc.go

func BuildVectorDocs(
    kb *entity.Knowledge,
    docID uint64,
    chunks entity.Chunks,
    titleVecs, contentVecs [][]float32,
    now time.Time,
) ([]repository.VectorDoc, error) {
    if len(titleVecs) != len(chunks) || len(contentVecs) != len(chunks) {
        return nil, apperrors.NewSysError(apperrors.CodeInternal, fmt.Sprintf(
            "vector count does not match chunk count: title %d / content %d / chunks %d",
            len(titleVecs), len(contentVecs), len(chunks)))
    }

    ts := now.UnixMilli()
    docs := make([]repository.VectorDoc, len(chunks))
    for i, c := range chunks {
        docs[i] = repository.VectorDoc{
            ChunkID:     ChunkID(docID, c.Order, c.Content),
            Tenant:     kb.Tenant,
            KBID:        kb.ID,
            DocID:       docID,
            Order:       c.Order,
            Title:       c.Title,
            Content:     c.Content,
            HeadingPath: c.HeadingPath,
            TitleVec:    titleVecs[i],
            ContentVec:  contentVecs[i],
            CreateTs:    ts,
        }
    }
    return docs, nil
}

Summary

Embedding turns text into a set of high-dimensional floating-point numbers, letting semantically similar text move closer in space so that retrieval can work by distance.

Full code: PycMono/FastRAG