A Contract Review Bot That Diagnoses Its Own Regex Failures
Contract review tools that silently miss pages or misattribute clauses create liability, not efficiency. This build shows that a self-diagnosing pipeline—one that inspects its own rule coverage and flags incomplete OCR—can catch those failures before a human does, making agent-assisted legal review safer to deploy in procurement and compliance workflows.
A procurement contract review pipeline built on WorkBuddy chains together General Accurate OCR, Table Recognition V3, and a Document Extraction Agent to handle TXT, DOCX, text-layer PDFs, and scanned images. The system detects format, runs the appropriate OCR path, extracts 20 predefined contract elements, and applies a seven-category risk rule engine to produce an HTML report with traceable evidence snippets.
During testing across five fictional contracts, the assistant initially missed most risks on a plain-text sample. It then inspected its own `risk_rules.py`, diagnosed that the regex patterns were overfit to a single test contract, and rewrote the rules to cover real-world Chinese contract phrasing. The fix lifted risk detection from 3 to 11 items and element extraction from 12/20 to 17/20.
A six-page scanned PDF exposed a different failure mode: the main script OCR'd only page one, producing a clean false negative. The agent supplemented a page-by-page processing script, pulling 2,497 characters across all pages and surfacing seven risks that the single-page run had completely missed. Directional bugs also surfaced—jurisdiction and price-adjustment clauses were occasionally attributed to the wrong party, a weakness the rule templates still carry.
The self-diagnosis and self-repair behavior on the TXT sample is more instructive than a clean first run: it demonstrates that a rule engine's real weakness is coverage drift, and that back-testing against real phrasing is the only way to surface it.
False negatives from incomplete page processing are more dangerous than false positives in a contract review context—a missed risk carries real financial exposure, and the six-page scan failure shows how easily a naive single-page assumption can produce them.
The directional bugs (jurisdiction and price-adjustment assigned to the wrong party) suggest that regex-based rule engines lack the semantic role-labeling needed to reliably distinguish Party A from Party B, which is a hard requirement for any legal-review tool.
Packaging three discrete API skills into one orchestrated skill turns a manual multi-step chore into a reusable asset; the value is less in the individual OCR calls and more in the workflow automation that removes the human as the integration layer.