Frontend · CSS

Pytest Caught 11 Hidden Bugs in an AI Long-Term Memory Module That Manual Testing Missed

By 用户05954017446 · Jul 29, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

AI features that rely on persistent user memory—health info, preferences, schedules—carry real safety risk when consistency bugs slip through. A fast, deterministic test harness turns a module nobody wants to touch into one the team confidently extends, and the same pytest patterns apply to any stateful in-memory logic with LRU and TTL semantics.

Summary

An AI assistant's long-term memory module misread a three-month-old allergy as a current symptom, triggering a dangerous medical recommendation. The root cause was a boundary bug in deduplication logic that three rounds of manual testing had missed. Manual regression took 30 minutes and covered only 12 main-path cases; developers avoided re-running it after refactoring, so bugs accumulated.

A pytest suite with 47 parameterized tests now exercises the full CRUD surface, LRU eviction, TTL expiration, and edge cases like duplicate timestamps and exact-expiry-boundary behavior. The suite runs in 5.2 seconds and uses pure in-memory instances with no mocking, isolating logical consistency from IO noise.

Key pitfalls included fixture scope leaking state between tests when set to `session`, mutable default arguments in parameterized lists corrupting subsequent test runs, and time-sleep flakiness in CI pipelines. The fix for CI instability was a looser TTL window rather than introducing a time-freezing library.

Takeaways

— A deduplication boundary bug caused an AI assistant to treat a three-month-old allergy record as a current symptom, prompting a dangerous medication instruction.

— Three rounds of manual testing missed the bug because human testers cannot track the combinatorial state space of add, update, delete, TTL expiry, and capacity eviction.

— A 47-case pytest suite now completes a full regression in 5.2 seconds, compared to 30 minutes for the previous 12-case manual process.

— The suite found 11 hidden bugs total, three of them P0 severity.

— Using `scope="session"` on a fixture leaked mutable state across tests, causing random, hard-to-diagnose failures.

— Parameterized test lists mutated by one case corrupted subsequent cases due to Python's mutable-default sharing; tuples fixed it.

— Time-sleep-based TTL tests flaked in CI under load; a 2-second TTL with a 2.2-second sleep and a tolerance window stabilized them without adding `freezegun`.

— Mocking was deliberately avoided because the memory store's logic is purely functional; mocks would have hidden real state-transition bugs.

Conclusions

Manual testing's real failure mode is not slowness but avoidance: after refactoring, developers skip re-running tedious manual suites, so bugs quietly accumulate.

The LRU eviction bug—failing to update access position on `get`—is a classic example of a semantic error invisible to happy-path tests but catastrophic in production when recently accessed data gets evicted.

Pytest's fixture-per-function default is the correct safety boundary for stateful tests; optimizing for speed with session-scoped fixtures creates non-deterministic cross-test contamination that wastes more time than it saves.

Time-dependent tests without a virtual clock are inherently flaky in CI; the pragmatic tradeoff of a wider tolerance window is often cheaper than pulling in a library like `freezegun` for a small suite.

Concepts & terms

LRU eviction

Least Recently Used eviction: when a fixed-capacity store is full, the entry that was accessed least recently is removed to make room. Correctness depends on updating the access timestamp on both writes and reads.

TTL (Time-To-Live)

A duration after which a stored entry is considered expired. Lazy deletion checks and removes expired entries only when they are accessed, rather than running a background cleanup process.

Lazy deletion

A strategy where expired data is removed only at the moment of access (get or get_all), rather than by a periodic background job. This avoids scheduling overhead but requires careful handling so expired entries don't leak into results.

Fixture scope in pytest

Controls how often a fixture is re-created. `function` scope (default) creates a fresh instance per test; `session` scope creates one instance for the entire test run, which can leak state between tests if the object is mutable.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗