跪拜 Guibai
← All articles
AI Programming · CI/CD

One Team's AI Code Review Pipeline: How to Let AI Review AI and Actually Trust It

By 工程小白菜 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

This is a concrete blueprint for turning AI code review from a one-off suggestion box into a self-improving, measured part of the delivery pipeline, showing how a small team can operationalize AI review with commodity tools and without continuous human babysitting.

Summary

The system uses Alibaba's open-source Open Code Review (OCR) as its engine and a cheap internal model, betting that rule quality matters more than model horsepower. Rules are split into a team-level baseline stored in a shared git repo and per-project rules in each repo's root; both are merged and versioned, with path matching, three severity levels, and exceptions to reduce false positives.

A shared CI template runs the review job in parallel with the build, tolerating failure and using smart diff baselines (MR, push, or new-branch fetch). Results go three ways: GitLab comments anchored to exact lines, Feishu cards color-coded by severity with per-author splitting, and real-time ingestion into a metrics store.

A reception layer identifies AI comments by format, dual-channels ingestion (real-time + reconciliation), and infers status from developer replies (fixed, false positive) to feed a dashboard. The false-positive rate becomes the input for rule optimization. Finally, the coding AI itself consumes the findings: it fixes redlines locally, replies to CI comments with fixed/obsolete/false-positive justifications, and those replies flow back into the dashboard. In one test, 14 of 17 comments were fixed, 2 obsolete, 1 false positive.

Takeaways
— The system uses Alibaba's open-source Open Code Review (OCR) as the review engine to reuse validated rules and avoid building diff parsing from scratch.
— A cheaper model tier was selected for review; the team found that finding quality bottlenecks on rules, not model intelligence.
— Rules are structured JSON with path matching, three levels (background, redline, team_standard), and a merge_system_rule flag that stacks team rules on top of built-in ones.
— CI runs the review job in parallel with build with allow_failure, using shared template and variables for frontend/backend differences.
— Diff baselines adapt to context: MR diff base, previous commit for push, or fetch target branch for new branches.
— Results distribute to three exits: GitLab commit comments anchored to lines, Feishu cards color-coded by severity, and real-time ingestion into a metrics service.
— AI comments are identified by body format, not by account, making collection stable across token changes.
— Developer replies like "👍 Fixed" or "🤡 False Positive" are parsed to infer finding status (adopted, falsePositive, pending).
— The false-positive rate from the dashboard becomes a direct input for rule improvements.
— The coding AI applies local redline gates, replies to CI comments with fixed/obsolete/false-positive explanations, and escalates uncertain cases to humans.
— In a real run, 17 CI comments were processed: 14 fixed, 2 obsolete, 1 false positive, all replies AI-generated and human-confirmed.
Conclusions

Treating review rules as versioned, mergeable artifacts alongside code turns AI review into a controllable, testable system rather than black-box prompt magic.

The false-positive loop is the strongest part: users vote with their feet, and the metric directly drives rule changes, which is a pragmatic way to operationalize AI quality.

Handing the review conclusions back to the code-writing AI closes a loop that most teams stop at distribution; this is what moves humans from reviewing every comment to surfacing exceptions.

The reliance on Feishu as a notification channel is China-specific; Western equivalents would need Slack or Teams, but the pattern is identical.

Concepts & terms
Open Code Review (OCR)
An open-source AI code review tool from Alibaba that uses LLMs to analyze diffs against structured rules, outputting findings as JSON that can be embedded into CI pipelines.
Redline rule
A hard rule that the review system treats as mandatory to fix; violations block merging or are intercepted locally, unlike background info or team standards which are advisory.
merge_system_rule
A flag in OCR rule schemas that controls whether custom rules stack on top of the engine's built-in system rules (merge) or replace them entirely; the article uses it to layer team rules over general ones.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗