AI Programming · OpenAI

A Three-Step Upload Pipeline That Eliminates Dirty Data in RAG Apps

By 第一行代码HW · Aug 4, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

RAG demos are easy; RAG that doesn't leave dirty state after a failed upload is harder. The three-step preview-then-commit pattern and the dual-ID strategy are directly portable to any Spring AI project that ingests user documents, and the pitfall catalog saves hours of debugging against Alibaba's OpenAI-compatible endpoints.

Summary

Uploading documents into a RAG pipeline usually means a single endpoint that parses, chunks, and vectorizes everything in one shot. Any failure leaves half-written database rows and orphaned files. This system splits the process into three distinct steps: a parse preview that uses a temporary file and touches no database, a chunk preview that lets users adjust chunk size and overlap on sliders and see the result in real time, and a final atomic save wrapped in a Spring @Transactional that persists the file, slices, and Redis vectors together or not at all. The design eliminates the most common source of dirty state in RAG ingestion.

The stack pairs MySQL for metadata with Redis for vectors, using a dual-ID scheme where auto-increment keys handle table relationships and UUIDs serve as external identifiers and Redis document IDs. Multi-knowledge-base isolation is enforced through metadata tags on every vector, with FilterExpression queries scoping retrieval to a single knowledge base. A custom SSE protocol delivers reference sources as a JSON preamble before the streaming answer, and the frontend uses a three-state renderer that shows plain text during streaming to avoid broken Markdown syntax, then runs marked.parse() only after the stream completes.

A dedicated recall-test interface exposes the retrieval black box directly: it returns the top-K chunks with similarity scores and source filenames, color-coded by relevance. The author catalogs three production pitfalls—null chunks in Bailian's streaming responses, a "choices is not set" error from content moderation or rate limiting, and vector dimension mismatches when embedding models are swapped without rebuilding the Redis index.

Takeaways

— Apache Tika parses PDF, Word, Excel, PPT, and TXT through a single Spring AI document reader.

— A three-step upload flow—parse preview, chunk preview, atomic save—keeps the database clean by never persisting anything until the user confirms the chunking result.

— Auto-increment primary keys handle internal table relationships; UUIDs serve as external identifiers and Redis vector document IDs, preventing data-scale leakage and deletion mismatches.

— Multi-knowledge-base isolation works by tagging every vector with knowledgeBaseId, fileId, and sliceId metadata, then filtering retrieval with FilterExpression.

— A custom SSE protocol prepends a __SOURCES__: JSON frame before the streaming answer so the frontend receives reference chunks and the generated text through a single channel.

— Streaming Markdown must be rendered as plain text until the stream completes; running marked.parse() on half-finished syntax produces garbled output.

— A standalone recall-test endpoint returns top-K chunks with similarity scores and source filenames, making retrieval quality directly inspectable.

— Alibaba Bailian's streaming responses can contain null chunks (heartbeat or end frames) that require triple-null guards to avoid NullPointerException.

— A "choices is not set" error from the OpenAI Java client usually means Bailian's content moderation or rate limiting intercepted the request.

— Swapping embedding models without dropping and rebuilding the Redis index causes a vector dimension mismatch; the model version must be pinned in configuration.

Conclusions

Separating document ingestion into preview and commit phases is an underappreciated pattern. Most RAG tutorials skip it, but in any multi-tenant system where users upload their own files, a single-step ingest endpoint guarantees support tickets about corrupted state.

The dual-ID design solves a real operational headache: auto-increment IDs in Redis leak data volume, and deleting by auto-increment ID after a database resync is fragile. UUIDs as the external contract decouple the vector store from the relational primary key.

Tethering the LLM with a system prompt that forbids fabrication is table stakes, but the real leverage is in the independent recall-test interface. Without it, you cannot tell whether a good answer came from good retrieval or from the model's own knowledge.

The three-state Markdown renderer is a small detail that separates a janky streaming experience from a polished one. Most streaming chat UIs either flash broken syntax or wait until the end to render, losing the real-time feel.

Concepts & terms

RAG (Retrieval-Augmented Generation)

A pattern where a user query first retrieves relevant document chunks from a vector database, then injects those chunks as context into the LLM prompt so the model answers from provided sources rather than its training data.

TokenTextSplitter

A Spring AI splitter that divides text based on token count using a tokenizer, respecting semantic boundaries better than character-based splitting.

SSE (Server-Sent Events)

A unidirectional HTTP streaming protocol where the server pushes data to the client over a single long-lived connection. Used here with a custom frame format to deliver reference sources and streaming text through one channel.

RedisVectorStore

Spring AI's abstraction over Redis RediSearch that stores document embeddings as vectors and supports metadata filtering via TAG fields for scoped similarity searches.

FilterExpression

A Spring AI API for building metadata filter conditions (e.g., eq, and, or) that restrict vector similarity searches to documents matching specific tag values.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗