Pytest Caught 11 Hidden Bugs in an AI Long-Term Memory Module That Manual Testing Missed

Replacing Manual Verification of AI Long-Term Memory Tests with Pytest: 11 Hidden Bugs Found in 2 Days

At 1:37 AM, a colleague posted a screenshot in the group chat: the AI assistant had mistaken a user's allergy history from three months ago as a new symptom today and suggested "stop the medication immediately." A chill ran down my spine—this was the most serious incident since our long-term memory module went live.

During the post-mortem, the root cause was a boundary bug in the memory deduplication logic, but what scared me more was: this bug had gone through three rounds of manual testing and was missed every time. It wasn't that the testing was inadequate; the human brain simply cannot keep up with the combinatorial explosion of states. That night, I decided that memory consistency testing had to be fully automated. Two days later, a test suite built with pytest uncovered 11 issues in one go, 3 of which were P0-level. This article covers the pitfalls I encountered and the complete code.

Problem Breakdown: Why Manual Testing Can't Handle AI Long-Term Memory?

Our scenario is straightforward: a user converses with an AI, and the AI needs to remember facts the user has stated (preferences, health information, schedules, etc.) and accurately reference them in subsequent responses. The long-term memory module exposes four core operations: Add, Query, Update, Delete, plus automatic expiration and capacity limits.

Consistency risk points look like this:

Writing the same key twice: does the second write overwrite or append?
Updating a non-existent key: does it fail silently or throw an exception?
When capacity is full and the oldest memory is deleted, but there happen to be multiple "oldest" entries (same timestamp), which one gets deleted?
When an expired memory is queried, is it filtered out, or does it also take valid memories with it?

In the past, we ran regression tests with Excel test cases: manually start a minimal environment, insert print statements in the code to check return values. One round took 30 minutes, left your eyes strained, and only covered the main paths. The biggest trap of manual testing isn't the slowness; it's that after refactoring, you simply don't want to test again. We needed an automated test suite that could perform a complete regression in under 5 seconds, making the memory module as reliable as a database.

Solution Design: Pytest + In-Memory Implementation + Parameterization Hell

The technical choice was unequivocally pytest. The reason for not using unittest: parameterization is too verbose to write, and fixture lifecycle control isn't granular enough. Our core strategy: push the system under test to the purest in-memory implementation, thoroughly test logical consistency first, then talk about integration.

The system under test is a class MemoryStore with the following interface:

class MemoryStore:
    def add(self, user_id, key, value, ttl=None): ...
    def get(self, user_id, key): ...
    def update(self, user_id, key, value): ...
    def delete(self, user_id, key): ...
    def get_all(self, user_id): ...  # Returns in reverse chronological order

The basic storage structure is a defaultdict containing an OrderedDict, maintaining insertion order, combined with timestamps to implement LRU eviction and TTL. During testing, this class is instantiated directly, without touching any database, isolating network and IO noise.

Three pillars of test organization:

fixtures provide clean store instances and various preset states.
Parameterization covers all critical paths and boundary conditions, including exceptional inputs.
Indirect parameterization works with fixtures to precisely control preset data.

No mocking was used. The memory module's logic is essentially purely functional; mocking would actually obscure the real state transitions.

Core Implementation: From the Most Basic to the Most Error-Prone Tests

1. Basic CRUD Consistency

This code verifies the most fundamental create, read, update, and delete operations, while also testing the overwrite behavior when the same key is added repeatedly.

import pytest
from datetime import datetime, timedelta
from collections import defaultdict, OrderedDict
import time

# ---- Implementation under test (simplified version, placed here to ensure runnability) ----
class MemoryStore:
    def __init__(self, capacity=100):
        self.capacity = capacity
        self._store = defaultdict(OrderedDict)  # user_id -> OrderedDict(key -> (value, expire_at))
    
    def add(self, user_id, key, value, ttl=None):
        expire_at = datetime.now() + timedelta(seconds=ttl) if ttl else None
        if key in self._store[user_id]:
            # Duplicate key: move to end (update access time), overwrite value
            self._store[user_id].move_to_end(key)
        elif len(self._store[user_id]) >= self.capacity:
            # Evict the oldest
            self._store[user_id].popitem(last=False)
        self._store[user_id][key] = (value, expire_at)
    
    def get(self, user_id, key):
        entry = self._store[user_id].get(key)
        if entry is None:
            return None
        value, expire_at = entry
        if expire_at and datetime.now() > expire_at:
            del self._store[user_id][key]  # Lazy deletion
            return None
        self._store[user_id].move_to_end(key)  # Update access time, affects LRU
        return value
    
    def update(self, user_id, key, value):
        entry = self._store[user_id].get(key)
        if entry is None:
            raise KeyError(f"Key '{key}' not found for user {user_id}")
        _, expire_at = entry
        self._store[user_id][key] = (value, expire_at)
    
    def delete(self, user_id, key):
        self._store[user_id].pop(key, None)
    
    def get_all(self, user_id):
        now = datetime.now()
        result = []
        # Iterate in reverse, lazily deleting expired data
        for key in reversed(list(self._store[user_id].keys())):
            value, expire_at = self._store[user_id][key]
            if expire_at and now > expire_at:
                del self._store[user_id][key]
                continue
            result.append((key, value))
        return result


# ---- Test Code ----
@pytest.fixture
def store():
    """Each test case gets a brand new store to avoid state pollution"""
    return MemoryStore(capacity=5)

def test_basic_crud(store):
    # Add
    store.add("u1", "color", "blue")
    assert store.get("u1", "color") == "blue"
    
    # Update
    store.update("u1", "color", "red")
    assert store.get("u1", "color") == "red"
    
    # Delete
    store.delete("u1", "color")
    assert store.get("u1", "color") is None

def test_duplicate_key_should_overwrite(store):
    store.add("u1", "key", "v1")
    store.add("u1", "key", "v2")
    assert store.get("u1", "key") == "v2"
    # Duplicate add should not increase entry count, should still be 1
    assert len(store.get_all("u1")) == 1

2. LRU Eviction Logic When Capacity Is Full

This is the most error-prone part: during eviction, is the "oldest added" or the "oldest accessed" entry evicted? Our get operation calls move_to_end(key), meaning accessed memories should not be evicted. The following test directly caught a P0 bug in our production environment—the initial version of get did not update the access position, causing information the user had just asked about to be pushed out the next second.

def test_lru_eviction_most_recently_used_survives(store):
    """When capacity is full, the least recently used entry is evicted, the most recently used is retained"""
    # Fill the store with capacity 5
    for i in range(5):
        store.add("u1", f"k{i}", f"v{i}")
    # Access k0, making it the most recently used
    store.get("u1", "k0")
    # Insert a new key, triggering eviction
    store.add("u1", "new", "new_value")
    
    all_keys = [k for k, v in store.get_all("u1")]
    assert "k0" in all_keys           # k0 should survive because it was accessed
    assert "new" in all_keys
    assert "k1" not in all_keys       # k1 is the oldest entry never accessed after insertion, should be evicted

3. TTL Expiration and Boundary Timing

TTL expiration seems simple, but the behavior at the exact moment of expiration is very prone to ambiguity. We use datetime.now() > expire_at, so a query at the exact moment of expiration should return None. This test case uses time.sleep to precisely control the time window, while also verifying that lazy deletion truly removes the underlying data, not just returns None.

def test_ttl_expiry(store):
    store.add("u1", "temp", "data", ttl=1)  # Expires after 1 second
    assert store.get("u1", "temp") == "data"
    time.sleep(1.1)  # Wait for expiration
    assert store.get("u1", "temp") is None
    # After lazy deletion, no residue should remain, get_all should also not return this entry
    assert len(store.get_all("u1")) == 0

def test_ttl_edge_expiry_exactly_at_boundary(store):
    """Test behavior exactly at the expiration boundary: inserting a key with the same name immediately after expiration should be treated as a brand new entry"""
    store.add("u1", "k", "v", ttl=0)  # Expires immediately
    time.sleep(0.1)  # Ensure time has passed
    # Add a key with the same name after expiration
    store.add("u1", "k", "v2")
    assert store.get("u1", "k") == "v2"

Pitfall Log: Pytest Details the Official Docs Don't Tell You

Pitfall 1: Wrong fixture scope, tests "poison" each other

Initially, I set the store fixture's scope to scope="session", thinking I could save time by reusing the object. As a result, one test modified the store's data, and all subsequent test cases that used it exploded—not in the same test case, but randomly in different places. The symptoms were bizarre, and it took two hours of investigation to realize it was state leakage. Lesson: Unless the system under test is stateless, the fixture scope must be function, giving each test its own instance. Parameterized tests sharing the same fixture are also safe because pytest re-runs the fixture for each parameter combination.

Pitfall 2: List objects in parameterization modified by multiple tests

We had a parameterized test to test eviction behavior under different capacities:

@pytest.mark.parametrize("capacity,keys_to_add,expected_evicted", [
    (3, ["a","b","c","d"], "a"),
])

Initially, keys_to_add was directly a list literal (mutable object). One test case accidentally appended to it, which affected subsequent parameterized combinations—Python's default argument sharing problem. The solution is simple: use tuples instead of lists, or use ids to explicitly mark immutable data.

Pitfall 3: Time-related tests fail intermittently in CI

sleep(1.1) in the CI pipeline sometimes actually sleeps longer than 1.1 seconds due to high machine load, causing test instability. The ideal solution of injecting a controllable "virtual clock" in the background was too cumbersome. Our compromise: set TTL to 2 seconds, sleep(2.2), and add a small tolerance window to datetime.now. A better approach is to use freezegun to freeze time, but that introduces an extra dependency; the current 2-second loose window is sufficient.

Effectiveness Verification

Metric	Manual Testing	Pytest Automation
Full Regression Time	~30 minutes	5.2 seconds
Test Cases Covered	12	47 (including boundary/exception)
Hidden Bugs Found	-	11 (3 P0)
Developer Refactoring Confidence	Too scared to touch	Make changes, run it, see the truth in 5 seconds

The most noticeable change: the team started proactively adding new features to the memory module because the test suite acts as a safety net.

Ready-to-Use Code

Copy the MemoryStore class and tests above into test_memory.py, and run it with a single command:

pytest -v test_memory.py

If you are using poetry, remember to add pytest to the dev group.

#Python #AIEngineering #pytest #TestAutomation #LongTermMemory

About the Author

A hands-on developer who constantly moves between backend and AI engineering, firmly believing that "anything that can be automated should never be manual." Writes code, and also writes articles to help peers avoid pitfalls.

GitHub: https://github.com/baofugege
Sponsor: https://github.com/sponsors/baofugege — If this article saved you a day of debugging, buy me a coffee
Services: Python Backend Performance Optimization / Tool Customization / Technical Consulting, contact Telegram @baofugege