跪拜 Guibai
← Back to the summary

Behind a Product RAG: SurrealDB, Embeddings, and an AI Tool in SvelteKit

Product RAG Knowledge Base and AI Consultant: Design and Implementation

Author: Swot

This document is the normative design and implementation specification for a product knowledge base (RAG) plus an AI product consultant, intended for later integration and secondary development.

Core code:

  • Data access (server): src/lib/server/products.js
  • Data access (client adapter): src/lib/products.js
  • Write entry point: src/routes/api/products/+server.js
  • Vector service: src/lib/server/embed.js (server-only, see §4.1)
  • Database layer: src/lib/server/db.ts
  • AI tools and chat: src/routes/api/test-vercel-api-chat/+server.js
  • Self-check / backfill scripts: scripts/check-product-search.mjs, scripts/backfill-product-embeddings.mjs

Design Structure Diagram

The following diagram shows the system structure of this "product RAG knowledge base + AI consultant": who calls whom, how data flows, where vectors come from. Three access paths all converge on the single implementation at $lib/server/products.js.

mermaid-diagram-2026-11-19-114918.png


1. Overview

Products are independent records in the SurrealDB product table. The whole chain is divided into three layers, with boundaries as follows:

                        ┌──────────────────────────────────────────────────────────┐
      浏览器 / 编辑器    │  $lib/products.js ──HTTP──> /api/products ──>             │
      (客户端)         │        (Bearer 鉴权,写操作记审计日志)                     │
                        └──────────────────────────────────────────────────────────┘
                                                            │
                                              $lib/server/products.js ──> SurrealDB
                                                            ▲
                        ┌───────────────────────────────────┴───────────────────────┐
      服务端            │  ① 页面 load / SSG entries                                 │
      (同进程直接 import) │     +page.server.ts / +layout.server.js                │
      无 HTTP、无鉴权     │     └─ import $lib/server/products.js ── getAllProducts()  │
      无序列化           │  ② AI 工具 getProducts                                    │
                        │     chat 端点的 streamText tools 内                        │
                        │     └─ import $lib/server/products.js ── searchProducts()  │
                        └───────────────────────────────────────────────────────────┘

Differences among the three access paths:

Path Form Caller → Callee Auth / Serialization
Browser / editor Cross-process HTTP fetch $lib/products.js in browser → server /api/products Bearer auth, JSON serialization
Page load / SSG entries Same-process function call +page.server.ts / +layout.server.js load functions → getAllProducts() in $lib/server/products.js None
AI tool getProducts Same-process function call execute function of the tool in test-vercel-api-chat/+server.js → searchProducts() in $lib/server/products.js None

"Same-process function call" means a normal in-memory function call within the same Node.js process (i.e., the SvelteKit server process):

These two paths are direct imports between server-side modules, not another HTTP round trip; there is no network round trip, no authentication, no serialization, and you get in-memory objects. They share the same $lib/server/products.js as the browser path, so read/write logic exists in one implementation only, no "one SQL for pages, another for API" fork.

Database access is unified by safeQuery() in src/lib/server/db.ts. Modules under $lib/server/ are not included in the browser bundle, so credentials and SDKs do not leak.


2. Data Model

The product table stores medical products that the company's R&D center "Product Center" displays externally; one record is one product. Products are of two types: research (in development) and approved (approved), the latter carrying a registration number (regNumber). Each record contains product title, intro, detailed description, core advantages, applicable scenarios (scope), structure, usage scenarios, and other textual information, plus the rich-text body of the detail page (content) and images. The AI product consultant queries this table when answering "what products exist", "what product should I use for a symptom", or "what does a specific registration number correspond to".

2.1 Table and Schema

The product table is SCHEMALESS. SurrealDB's SCHEMALESS means:

So this project does not write DEFINE TABLE; the entire table has only one field constraint:

DEFINE FIELD embedding ON product TYPE option<array<float>>;

array<float> does not lock the dimension; both 1024 and 1536 dimensions work; only HNSW index DIMENSION locks the dimension, and this project does not create HNSW (see §4.2).

2.2 Field Layers

Layer Fields Notes
Business writable WRITABLE_FIELDS title, type, subtitle, summary, description, content, image, imageAlt, coreAdvantages, coreTech, regNumber, scope, structure, scenes, advantages, bullets, sort Only these fields accept external writes; undefined is skipped, empty strings / empty arrays are preserved as "explicit clearing"
Internally maintained embedding, embeddingModel, embeddingVersion, createdAt, updatedAt Generated by the server, not accepted for external writes
List projection BASE_FIELDS Business fields minus content, plus id and accounting fields (see explanation below) Default projection of getAllProducts(), does not include content or embedding
Detail projection BASE_FIELDS + content withContent: true, used only by the detail page and editor
Model-visible MODEL_FIELDS BASE_FIELDS minus imageAlt / sort / embeddingModel / embeddingVersion / createdAt / updatedAt Derived from BASE_FIELDS; adding a business field only needs to change one place

What are "accounting fields": They refer to metadata maintained by the system for internal management, not product business content—specifically embeddingModel, embeddingVersion, createdAt, updatedAt (and sort used for ordering). They record "which model, which version, when was this vector computed", used only for operations troubleshooting and version management; they do not help answer user questions, so MODEL_FIELDS removes them and does not feed them to the large language model.

type is either research (in development) or approved (approved); regNumber is the registration number for exact matching. Lists are always ordered by ORDER BY sort ASC, id ASC; when old data lacks sort, it naturally degrades to ascending id order.

Externally returned data never includes embedding: a 1024-dimensional float array should not go to the frontend, nor into the LLM context.

2.3 ID Rules

Why URLs don't contain random strings: SurrealDB record ids can be generated with type::rand() as ULID/uuid, which would appear as a long random string in URLs. This project deliberately manually assigns sequential integer ids (product:1, product:2, ...), so detail page URLs look like /rd-center/product/1, clean and predictable.

Why numeric ids must stay number type: In SurrealDB, type::thing('product', 1) produces a numeric id product:1, while type::thing('product', '1') produces a string id product:⟨1⟩—these are two different records in SurrealDB. Existing records (1..6) were created with numeric ids. If normalizeId() normalizes "1" to a string, subsequent queries with type::thing('product', '1') will not match the existing product:1, causing "can't find old records" or even "accidentally create duplicate records". Therefore normalizeId() only returns Number for pure numeric strings, keeping id types stable.


3. Read/Write Paths

3.1 Read Points

Location Call Purpose
src/routes/[...slug]/+page.server.ts getAllProducts() Inject templateData.products into the R&D center list page
src/routes/rd-center/product/[id]/+layout.server.js getAllProducts({ withContent: true }) Detail page content and prev/next navigation
src/routes/rd-center/product/[id]/+page.server.ts getAllProducts() SSG entries, only queries DB when BUILD_MODE=ssg; returns [] early in SSR builds to avoid build failure when database is unreachable
src/lib/components/templates/RDCenterTemplate.svelte dynamic import('$lib/products') Re-fetch full data when entering edit mode
src/routes/rd-center/product/[id]/+page.svelte dynamic import('$lib/products') Refresh after save on detail page
src/routes/api/test-vercel-api-chat/+server.js searchProducts() getProducts tool for AI consultant

Server-side load imports $lib/server/products directly; the browser side can only import $lib/products (wrapping /api/products requests, and also exporting two shared constants: MODEL_FIELDS as the field whitelist fed to the model, and MIN_SEARCH_SCORE as the minimum cosine similarity threshold for semantic search—below it, treated as "no match", see §4.2). RDCenterTemplate keeps the product list (templateData.products) and page configuration data (page.metadata.rdcenter) separate.

3.2 Write Entry Point /api/products/+server.js

Method Semantics
GET List (?type=, ?withContent=true) or single (?id=, includes content)
POST Save a single record, MERGE semantics, untouched fields remain unchanged
PUT Full overwrite of a list: array order is sort; matched records use MERGE, negative or unmatched ids get new numeric ids and are created, records missing from the list are DELETEd, then all vectors are recomputed uniformly
DELETE Delete a single record
PATCH Update only sort, without touching text, without recomputing vectors (sorting does not affect semantic content)

All /api/* are uniformly validated by src/hooks.server.js for Authorization: Bearer, with the only whitelist being /api/admin/login; write operations log audit entries with target: 'product'.

Where MERGE appears (annotated):

  1. Single-save upsertProduct():
await queryRows(`UPSERT type::thing('product', $id) MERGE $fields`, { id, fields });
  1. In full-overwrite saveProductList(), matched existing records use MERGE:
await queryRows(`UPDATE type::thing('product', $id) MERGE $fields`, {
    id,
    fields: { ...fields, embeddingVersion: 0, updatedAt: now }
});

But new records (negative id / newly added in the list) use CONTENT $fields (entire record write, equivalent to CREATE), because the target id is guaranteed not to exist:

await queryRows(`UPSERT type::thing('product', $id) CONTENT $fields`, { ... });

Why writes must go through the server: upsertProduct() needs to call the SiliconFlow embedding recomputation after saving the text, and SILICONFLOW_API_KEY can only exist on the server (browser has no process.env). If the browser connected directly to the database to save, the text would be stored but the vector would forever stay at embeddingVersion = 0, and semantic search would break.

Why MERGE instead of CONTENT: It does not accidentally delete unsubmitted fields, nor overwrite existing embedding. But when the text changes, the old vector becomes invalid, so on update embeddingVersion is explicitly reset to 0, leaving it to ensureProductEmbeddings() to recompute.


4. Vectors and Retrieval

4.1 Generation and Backfill (Embedding Service)

The vector service is in src/lib/server/embed.js, using plain fetch to call SiliconFlow's OpenAI-compatible interface (no new SDK dependency): model BAAI/bge-m3, dimension 1024, EMBED_VERSION = 1, key from SILICONFLOW_API_KEY.

productToEmbedText() only concatenates title / subtitle / summary / description / coreAdvantages / scope. type / regNumber are exact-filter fields and do not participate in semantics; content is an HTML long text, used only for detail page rendering, not for vectors nor fed to the model.

Write order is "save text first, compute vector later":

  1. UPSERT … MERGE saves the text.
  2. Call embed() based on the merged full text (old record + new fields).
  3. UPDATE writes back embedding / embeddingModel / embeddingVersion.

Vector generation failure does not block text saving; it only sets embeddingVersion to 0. ensureProductEmbeddings() scans for records with empty vectors or stale versions for idempotent repair; when switching embedding models, just bump EMBED_VERSION and recompute everything, no business code changes.

4.2 Hybrid Search searchProducts()

After a higher-priority branch is hit, it does not fall through:

Priority Condition Behavior
1 regNumber passed WHERE regNumber = … exact match, no vector; the number is unique, so type is ignored
2 query passed vector::similarity::cosine exact cosine sort, ORDER BY score DESC, LIMIT clamped to 1~20; can combine type filter; discard results with score < 0.45
3 No query Return full list, can filter by type

Basis for MIN_SEARCH_SCORE = 0.45: cosine ranking can always produce a top-k, but measured unrelated questions (asking about weather, registration count) score at most 0.37~0.40, while relevant long-tail queries ("what to use for periodontal surgery") reach 0.52. Raising the threshold would kill long-tail, lowering it would let unrelated questions get "answer-like" products. Returning an empty list is a legitimate result, not an error.

Currently no HNSW: with about 100 products, brute-force cosine is milliseconds; HNSW would lock the dimension, introduce approximate recall loss, and add tuning cost. Upgrade when data reaches tens of thousands or measurements show it is the bottleneck.

4.3 AI Tool Side

The chat endpoint uses AI SDK's streamText() (model @ai-sdk/deepseek deepseek-flash) + tool() to define the getProducts tool, with stopWhen: isStepCount(5) allowing multi-step calls.

The getProducts tool's toModelProduct() trims fields according to MODEL_FIELDS, omitting undefined / null / empty strings / empty arrays to save tokens, and the semantic branch additionally keeps score to 3 decimal places. The tool description and system prompt both state "returning an empty list means no matching product" and "must not present general science as company products", and require that when count = 0 or the top score is too low, the user is explicitly told there is no corresponding product; when the tool throws, it returns { count: 0, products: [], error }, not blocking the conversation.


5. Key Design Decisions

Decision Reason
Products must be independent records, not nested arrays in another table SurrealDB's indexes (including HNSW) operate on records; array elements cannot be returned as independent results; nested structures cannot be incrementally edited, paginated, or efficiently filtered by type / regNumber. Future "product document long text" will also need an independent vector space
Writes go through /api/products, not browser direct DB connection Keys exist only on the server
MERGE + explicit embeddingVersion = 0 Protects unsubmitted fields and existing vectors, while ensuring old vectors are invalidated when text changes
No external return of embedding; content only in detail projection Saves bandwidth and tokens, avoids vectors entering frontend / LLM context
Structured fields not in vectors, content not in vectors nor fed to model type / regNumber are more reliably matched with WHERE exact match; HTML long text has high token cost and limited help for Q&A
Semantic search threshold + empty-result convention Without a threshold, "what's the weather today" would be treated as a product answer
Sorting uses PATCH and does not recompute vectors Sorting does not change semantic content; recomputation is pure waste
Embedding service is server-only (in $lib/server/) Reads process.env keys, calls external API; compile-time forbids client import
.env loaded via process.loadEnvFile() in hooks.server.js SvelteKit only feeds .env into $env/*; server-side modules reading process.env would not get SILICONFLOW_API_KEY, manifesting as the tool reporting "product data retrieval failed"

6. Operations and Verification

6.1 Environment Variables

Variable Purpose
SILICONFLOW_API_KEY Generate / query vectors; only in .env or platform environment, do not commit (single-person private repo is fine as long as the repo is not exposed)
SURREAL_URL / SURREAL_USER / SURREAL_PASS / SURREAL_NS / SURREAL_DB Read by src/lib/server/db.ts at runtime; missing any one errors on first connection; no defaults (do not give defaults: when copying projects and changing params, forgetting to change them can corrupt the previous project's database)
DEEPSEEK_API_KEY AI conversation model
SMTP_* Lead / notification emails

⚠️ Environment variables have "two sources of truth", easy to contaminate each other (tested conclusion):

  1. .env file: loaded via process.loadEnvFile() in hooks.server.js; scripts load with --env-file-if-exists=.env.
  2. ecosystem.config.cjs (PM2 config) env block: in production, PM2 directly injects process environment variables, and it also hardcodes a set of SURREAL_*, SILICONFLOW_API_KEY, DEEPSEEK_API_KEY, SMTP_*.

The critical pit is precedence: Node's process.loadEnvFile() does not override already-existing environment variables (tested on Node v22). This means:

Conclusion / rule: keep only one source of truth per environment. Local development uses .env; production uses ecosystem.config.cjs (or platform environment variables), and do not leave a different .env file on the server. When changing config, first confirm "where do this process's environment variables come from".

6.2 Common Commands

package.json registers an npm script for product retrieval self-check:

"check:products": "node --env-file-if-exists=.env scripts/check-product-search.mjs"
# Product hybrid retrieval self-check (key, count, vector version, recall scores, field leakage)
pnpm check:products
pnpm check:products -- 拔牙后创面怎么护理 预防龋齿
pnpm check:products -- --limit=5

# Idempotent vector backfill (when switching models or stale records appear)
node --env-file=.env scripts/backfill-product-embeddings.mjs

--env-file-if-exists=.env allows environments without a .env (relying only on platform environment variables) to run; if keys are missing, the script reports an error and exits with code 1. Full text of both scripts is in the appendix.

6.3 Verification SQL

SELECT count() FROM product GROUP ALL;
SELECT id, array::len(embedding) AS dim, embeddingModel, embeddingVersion FROM product ORDER BY id;
SELECT count() AS stale FROM product WHERE embedding IS NONE OR embeddingVersion != 1 GROUP ALL;
SELECT id, (content IS NOT NONE) AS has_content FROM product ORDER BY id;

stale should be 0, dim should be 1024. Frontend regression checks three places: /rd-center two Tab product counts and order, /rd-center/product/1 content, and BUILD_MODE=ssg npm run build generated product/1..N.html.

6.4 Version Constraints (Pitfalls Encountered)


Appendix A: scripts/check-product-search.mjs

/**
 * Product hybrid retrieval self-check
 *
 * Purpose: debug "AI consultant can't answer product questions" — one-shot confirm
 *   1) SILICONFLOW_API_KEY exists in the running environment
 *   2) Product count and vectors are ready in SurrealDB
 *   3) Semantic retrieval recalls correct products (including scores and margins)
 *   4) Returned fields do not leak content / embedding
 *
 * Run:
 *   pnpm check:products
 *   pnpm check:products -- 拔牙后创面怎么护理 预防龋齿
 *   pnpm check:products -- --limit=5
 */

import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { createServer } from 'vite';

const projectRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');

/** Default queries: cover semantic retrieval / type filter / reg number exact matching branches */
const DEFAULT_QUERIES = ['拔牙后创面怎么护理', '预防龋齿', '3D打印软组织修复材料'];
const DEFAULT_REG_NUMBER = '浙械注准20252171626';

const args = process.argv.slice(2);
let limit = 3;
/** @type {string[]} */
const queries = [];
for (const arg of args) {
	if (arg === '--') continue;
	const matched = arg.match(/^--limit=(\d+)$/);
	if (matched) limit = Math.min(Math.max(Number(matched[1]), 1), 20);
	else queries.push(arg);
}
const finalQueries = queries.length > 0 ? queries : DEFAULT_QUERIES;

const pad = (text, width) => String(text).padEnd(width);

/** Accumulated failures, decide exit code at end */
const failures = [];

/* ------------------------------ 1. Environment check ------------------------------ */

console.log('== Environment check ==');
const apiKey = process.env.SILICONFLOW_API_KEY;
if (apiKey) {
	console.log(`SILICONFLOW_API_KEY: ✓ (${apiKey.length} characters)`);
} else {
	failures.push('Missing SILICONFLOW_API_KEY (.env not loaded or not configured)');
	console.log('SILICONFLOW_API_KEY: ✗ not set');
}

/* --------------------------- 2~4. Go through the real retrieval layer --------------------------- */

const server = await createServer({
	root: projectRoot,
	configFile: path.join(projectRoot, 'vite.config.ts'),
	logLevel: 'error',
	server: { middlewareMode: true }
});

try {
	const { searchProducts, getAllProducts, MIN_SEARCH_SCORE } = await server.ssrLoadModule('/src/lib/server/products.js');
	const { EMBED_VERSION } = await server.ssrLoadModule('/src/lib/server/embed.js');

	console.log('\n== Data check ==');
	const all = await getAllProducts();
	const stale = all.filter((p) => p.embeddingVersion !== EMBED_VERSION).length;
	console.log(`Product count: ${all.length}`);
	console.log(`Vectors ready: ${all.length - stale}/${all.length} (stale=${stale}, expected version ${EMBED_VERSION})`);
	console.log(`Semantic threshold: score >= ${MIN_SEARCH_SCORE} (below means no match)`);
	if (all.length === 0) failures.push('product table has no data');
	if (stale > 0) failures.push(`${stale} product vectors need backfill, run scripts/backfill-product-embeddings.mjs first`);

	console.log(`\n== Semantic retrieval (limit=${limit}) ==`);
	for (const query of finalQueries) {
		const rows = await searchProducts({ query, limit });
		console.log(`\nquery: ${query}`);
		if (rows.length === 0) {
			console.log(`  (no match: highest similarity below ${MIN_SEARCH_SCORE}, expected if the question is product-unrelated)`);
			continue;
		}
		rows.forEach((row, index) => {
			const score = typeof row.score === 'number' ? row.score.toFixed(4) : '   -  ';
			console.log(`  ${index + 1}. ${score}  ${pad(row.title, 24)} ${pad(row.type, 10)} ${row.regNumber ?? ''}`);
			if (index === 0 && rows.length > 1) {
				console.log(`     margin: ${(row.score - rows[1].score).toFixed(4)}`);
			}
		});
	}

	console.log('\n== Exact filter branches ==');
	const approved = await searchProducts({ type: 'approved' });
	const research = await searchProducts({ type: 'research' });
	console.log(`type=approved: ${approved.length} items -> ${approved.map((p) => p.title).join(' / ') || '(empty)'}`);
	console.log(`type=research: ${research.length} items -> ${research.map((p) => p.title).join(' / ') || '(empty)'}`);

	const byRegNumber = await searchProducts({ regNumber: DEFAULT_REG_NUMBER });
	console.log(`regNumber=${DEFAULT_REG_NUMBER}: ${byRegNumber.length} items -> ${byRegNumber[0]?.title ?? '(empty)'}`);
	if (byRegNumber.length !== 1) failures.push(`reg number exact match abnormal, returned ${byRegNumber.length} items`);

	console.log('\n== Field leakage check ==');
	const forbidden = ['content', 'embedding'];
	const leaked = [
		...new Set(
			[...all, ...approved, ...research, ...byRegNumber].flatMap((row) =>
				forbidden.filter((key) => Object.prototype.hasOwnProperty.call(row, key))
			)
		)
	];
	if (leaked.length > 0) {
		failures.push(`return value leaked fields: ${leaked.join(', ')}`);
		console.log(`✗ leaked: ${leaked.join(', ')}`);
	} else {
		console.log('✓ no content / embedding');
	}
} catch (error) {
	failures.push(`retrieval process threw: ${error?.message ?? error}`);
	console.error('\n✗ execution failed:', error);
} finally {
	await server.close();
}

console.log('\n== Conclusion ==');
if (failures.length === 0) {
	console.log('✓ all passed');
	process.exit(0);
}
for (const failure of failures) console.log(`✗ ${failure}`);
process.exit(1);

Appendix B: scripts/backfill-product-embeddings.mjs

/**
 * Backfill embeddings for the product table (idempotent)
 *
 * Run:
 *   node --env-file=.env scripts/backfill-product-embeddings.mjs
 *
 * Logic:
 *   Scans records with empty embedding or embeddingVersion not matching the current version,
 *   re-calls the embedding service and writes back. In future, switching models only requires changing EMBED_VERSION.
 *
 * Note: This script connects directly to the database, hardcoding root:root and yinlife / site,
 *   only SURREAL_URL can be overridden by environment; verify before running in non-local environments.
 */

import { Surreal } from 'surrealdb';
import { embed, productToEmbedText, EMBED_MODEL, EMBED_VERSION } from '../src/lib/server/embed.js';

const SURREAL_URL = process.env.SURREAL_URL || 'http://localhost:8000';
const SURREAL_NS = 'yinlife';
const SURREAL_DB = 'site';

const db = new Surreal();
await db.connect(SURREAL_URL);
await db.signin({ username: 'root', password: 'root' });
await db.use({ namespace: SURREAL_NS, database: SURREAL_DB });

const [rows] = await db.query(
	'SELECT * FROM product WHERE embedding IS NONE OR embeddingVersion != $v ORDER BY id',
	{ v: EMBED_VERSION }
);

console.log(`[backfill] products to fill: ${rows.length}`);

for (const p of rows) {
	const text = productToEmbedText(p);
	const [vec] = await embed([text]);

	if (!vec || vec.length === 0) {
		console.warn(`[backfill] ${p.id} vector empty, skip`);
		continue;
	}

	await db.query(
		`UPDATE $id SET
			embedding = $vec,
			embeddingModel = $model,
			embeddingVersion = $version,
			updatedAt = time::now()`,
		{ id: p.id, vec, model: EMBED_MODEL, version: EMBED_VERSION }
	);

	console.log(`[backfill] ✓ ${p.id} ${p.title} (${vec.length} dims)`);
}

const [check] = await db.query(
	'SELECT id, title, array::len(embedding) AS dim, embeddingModel, embeddingVersion FROM product ORDER BY id'
);
console.table(check);

await db.close();
Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

Swot

This content only presents the high-level design approach.