Under One Yuan: Regex, SQLite, and an LLM Turn 1,500 AI Chats into a Personal Knowledge Base
Most users let hundreds of AI conversations rot in history logs. This pipeline demonstrates that a small deterministic front-end (regex, SQLite hashing) can make LLM processing affordable enough for an individual, and that a context-aware classification prompt beats a generic tagger. The same pattern applies to anyone holding exported chat JSON from any LLM.
A 60MB conversations.json export becomes a searchable knowledge base through three layers: local regex and keywords filter out about 70% of casual chat, a SQLite-backed hybrid fingerprint skips unchanged conversations so the API is never re-billed, and a DeepSeek call then classifies and scores the remaining sessions into categories. The classification prompt embeds the user's identity and focus areas, which separates hard technical troubleshooting from emotional venting far better than a generic tagger.
A separate deep_scan.py handles marathon conversations of over 1,000 messages, extracting all user utterances to pull out late-stage breakthroughs that a first-pass look would miss. Pitfalls along the way include a Windows file lock from an open Excel sheet, CTF sessions mis-scored because security keywords were missing from the initial list, and a dedup logic that had to balance missing new messages against re-sending old ones. The final outputs are an Excel scoring sheet and distilled Markdown notes ready for Obsidian.
Skipping vector search in favor of regex filtering plus LLM scoring is a deliberate trade-off: retrieval gives fragments, but the stated goal is distillation into structured troubleshooting notes, so the evaluator approach fits the output.
The hybrid fingerprint is a pragmatic incremental-processing trick. Hashing session ID, message count, and last timestamp detects changes without re-reading message bodies, balancing cost and completeness.
Embedding the user's persona into the classification prompt is a low-cost accuracy booster. It shows why generic classifiers fail: they lack the context to distinguish a Docker error from a rant.
The decision to do coarse filtering locally before any API call shifts most of the cost burden to deterministic code, making large-scale personal document processing viable on a hobbyist budget.