Under One Yuan: Regex, SQLite, and an LLM Turn 1,500 AI Chats into a Personal Knowledge Base

I Spent Less Than One Yuan to Turn a Year of 1,500 Conversations into a Personal Knowledge Base Using Python

1. Background: Do you have this pain point?

As a heavy AI user, over the past year I've accumulated more than 1,500 conversations across various AI platforms (especially DeepSeek).

My mind remembers a lot, but more is scattered in hundreds of cyber corners. I'm a first-year information security student, every day setting up environments, hitting errors, doing CTF, and occasionally falling into self-reflection.

Early on, when I got good answers, I'd copy-paste them obsessively. Later I found that too draining, so I just left them in history. Until one day I wanted to retrieve the troubleshooting steps for a "Kali Linux source error" from six months ago. I searched through history and was baffled by hundreds of titles.

I suddenly realized: most of these 1,500 conversations are junk, a tiny few are gold. If I don't sort them, they'll forever be cyber garbage occupying server space.

How do you filter through 1,500 messy JSON files to find genuinely valuable technical troubleshooting and cognitive breakthrough moments, and distill them into a searchable personal knowledge base? This post shares my complete hands-on pitfalls.

2. Overall Design (Architecture)

If I just wrote a script to store content, that would at best be "backup". But the core of this pipeline is "cleaning and distillation".

The whole system works like this:

A crucial design choice: why didn't I use a vector database for semantic search?

Many developers' first reaction: slice conversations, embed them, store in a vector DB, then search for relevant ones later.

But I rejected that. Because my goal is not "retrieval" but "filtering high-value conversations and distilling them into notes."

Vector search gives you "fragments", but I wanted a systematic troubleshooting guide (symptom -> root cause -> steps). So I chose: rule-based coarse filtering (local regex) + LLM scoring/classification + manual review calibration. Treat the LLM as an "evaluator", not a "searcher". (It's worth noting that coarse filtering saves money, and the rare missed high-value items are left to a later "long-conversation deep scan" as a safety net.)

3. Core Highlights (Technical Dimensionality Reduction)

Turning a 60MB JSON into a knowledge base, the fear is "blindly calling APIs". Feeding everything to the model at once risks exceeding context limits and is extremely costly. Three design choices hit the pain points:

1. Extreme cost savings: local regex + SQLite fingerprint dedup

Local coarse filter: I wrote a set of regexes and keywords (like CTF, Linux, config, error) to filter out 70% of chat and junk locally. Only the rest goes to the API.
Fingerprint dedup: Every time the script runs, it first checks SQLite. If the conversation hasn't changed, it skips it. This way, even if you run the script ten times a day, you won't spend a cent extra.

2. Making the model "know you": highly customized prompts

Why do generic classifiers fail? Because they lack context. In my classification and scoring prompt, I explicitly wrote:

"I am a first-year information security student who often does CTF, deals with a lot of environment errors and code configurations, and occasionally explores psychological cognition."

With just that sentence, the model knows that a "Docker error" is rigorous hands-on practice, not casual chat; a "self-review" is a valuable psychological breakthrough. Letting the AI play the role of a senior who knows you pushes classification accuracy through the roof.

3. "Dimensionality reduction" for long conversations

DeepSeek's exported JSON has an annoying structure: fragments. It contains not only user questions (REQUEST) and AI answers (RESPONSE), but also a ton of long thinking processes (THINK). If you extract blindly, tokens explode. I wrote a dedicated extractor that automatically filters out THINK and keeps only clean Q&A.

Even more, for epic conversations like "I'm done" with over 1,000 messages, looking only at the first few lines might misjudge it as pure emotional venting (scored 1). I wrote a deep_scan.py that forces extraction of all your utterances in that conversation, stitches them into one long text, and feeds it to the LLM, which then pulls out the later "cognitive awakening" and golden quotes from tens of thousands of words.

4. Pitfall Log (Real Development Has Its Messes)

How can a personal project avoid pitfalls? Three moments impressed me most:

1. Classic Windows file lock

Mid-run, the terminal spewed red text: PermissionError: [Errno 13] Permission denied: 'classified.csv'. After much debugging, I found that I had opened the table in Excel to preview intermediate results. On Windows, when a file is in use, Python can't write to it. I added try...except, and now if it detects you're holding the file, it gently prompts: "Please close your Excel first."

2. When CTF gold was scored 1: a rescue plan

On the first full run, I found that "CISCN2019Web11 Detailed" and "BUUCTF Web Problem 1" were all scored as "Other/1". Why? The initial keyword list didn't include security-related terms, so they were missed. I didn't stupidly reset all 1,500 entries and rerun. Instead, I wrote a script that selected only conversations whose titles contained CTF/security/injection, rolled back their SQLite status, and re-ran locally. Modifying the keyword list + partial re-run fixed all misclassifications for a few cents.

3. The dedup dilemma: what counts as a "new conversation"?

This was the most mentally taxing part:

If you dedup only by "session ID": you ask a new question in the same conversation today, the script thinks it already ran, and you miss newly produced knowledge.
If you use "whole text hash": you ask one more question, the whole hash changes, and it feeds the previous dozens of messages to the model again, wasting money.

To solve this, I introduced a hybrid fingerprint: SHA-256(session ID + message count + last message timestamp). If the ID hasn't changed, and the message count and last timestamp haven't changed, the conversation hasn't moved, skip it. If two more messages appear, the fingerprint changes and triggers re-scoring. This saves money and never misses new thinking.

5. Personal Use Guide (Pipeline Run Checklist)

If you want to run this pipeline, here's my 8-step checklist:

Data acquisition: Export the full conversations.json through official means (or packet capture).
Structure parsing: Write a small script to probe the JSON hierarchy (mapping -> fragments) and understand how the data is nested.
Coarse filter & incremental: Configure local keywords and use SQLite to store fingerprints so each run processes only new or modified conversations.
Classification & scoring: Call the DeepSeek API. Don't be stingy with prompt length: the more detail you provide about your identity, pain points, and focus areas, the more accurate the classification and scoring.
Manual calibration & recovery: After the run, export review.csv, scan it in Excel. For AI misjudgments (like treasure scored 1), modify the taxonomy.yaml keyword list, locally reset, and re-run.
Deep extraction (optional): For very long conversations (like psychology reviews or long-term project discussions), run them through deep_scan separately to force full-text reading and pull out later awakening points.
Knowledge base construction: Filter the "computer" category's pitfall/hands-on sessions, then have the AI distill them into structured Markdown troubleshooting notes.
Daily review: Open the final Excel, add data bars, sort by score descending, and your "highlight moments" are clear at a glance.

6. Results

1. Excel classification and scoring table

2. Auto-generated Markdown knowledge base

7. Open Source and "Teach to Fish"

I plan to clean up this code and open-source it on GitHub, repo name AI-Chat-Archiver. (Will release in a few days.)

The script took me a few evenings; the core code is only a few hundred lines. But rather than hand you a packaged tool, I want to share an engineering mindset: treat the LLM as a data processing engine.

Because everyone's JSON structure differs, and everyone's focus area is totally different. If you don't want my code, or your exported structure differs, I've prepared a "teach to fish" custom guide. Just paste the following text, along with the first few lines of your JSON file (desensitized), to any code-capable AI (like DeepSeek or Claude), and it will tailor a set for you:

I am a [your major/identity, e.g., information security student], I exported a conversations.json of size XX MB.
Its top-level structure is a list, each conversation has id, title, and mapping. The fragments in mapping contain REQUEST (user), RESPONSE (AI), THINK (thinking process).
I need you to write a Python script:
1. Extract all user questions (REQUEST) and AI answers (RESPONSE), discard THINK.
2. Use keywords to filter conversations containing '[your core focus words, e.g., CTF/error/Linux/psychology]'.
3. Write a generic utils module that calls the LLM API (with SQLite fingerprint dedup to avoid duplicate costs).
4. Generate config.yaml and txt templates in prompts/ for me.
Please directly give me the complete code structure.

Conclusion

In this era of AI everywhere, free content generation is no longer rare. What's truly scarce is your own real struggle-and-error experience, and the ability to structure it.

I spent less than 1 yuan to turn a year's 1,500+ conversations into hundreds of searchable troubleshooting guides and cognitive reviews in Obsidian. This feeling of alchemizing "cyber junk" into a "personal external brain" is really satisfying.

I hope this post gives you some inspiration to go dig through your own history; maybe there's a gold mine hiding inside.