One Team's AI Code Review Pipeline: How to Let AI Review AI and Actually Trust It
This is the sequel to "From Copilot to WorkBuddy: Several Practices for AI Self-Learning, Self-Acceptance, and Cross-Project Work." The previous article planted a hook: AI writes code far faster than humans can review it, and our solution was to embed "LLM semantic review" into CI. This article breaks down the implementation of this AI code review system—how rules are managed, how CI runs, how results are distributed, and how review quality is quantified.
Overview
The entire system has four layers, each with a single responsibility:
Division of labor in one sentence: CI handles "reviewing," Feishu handles "calling out," the dashboard handles "calculating," and the AI companion handles "fixing."
1. Selection: Don't reinvent the engine, prioritize cost-effective models
For the review engine we directly used Alibaba's open-source tool Open Code Review (OCR for short), for three reasons:
- Quality reuse: Directly inherit Alibaba's validated high-quality rules, avoiding the need to figure out prompts on our own.
- Avoid reinventing the wheel: Skip tedious low-level development like handling code context and diff parsing.
- Zero maintenance overhead: Stay aligned with large-model iterations without dedicated staff.
The model connects to the company's internal model gateway, selecting a cheaper tier—this kind of code review task doesn't need a flagship model, cost-effectiveness first. Practice has proven this judgment correct: the bottleneck in review finding quality lies more in rules than in model intelligence.
2. Rule Splitting: Team Baseline + Project Customization
OCR's review rules are structured JSON, versioned in git like code:
{
"meta": { "version": "1.2.0" },
"rules": [
{
"path": "src/request/**",
"level": "background",
"rule": "【Exemption】src/request is the unified interface layer; import axios is normal usage there and does not apply to the following centralized request redline.",
"merge_system_rule": true
},
{
"path": "**/*.{js,ts,tsx}",
"level": "redline",
"rule": "【Redline】All backend requests must be centralized in the src/request directory; pages and components must not directly create axios instances.",
"merge_system_rule": true
}
]
}
Several key designs in this schema:
- path matching: Rules can be precise to directories and file types, e.g., Props type standards only for component directories.
- three levels:
backgroundgives background instructions to the AI (including exceptions—telling it which cases are not violations; crucial for reducing false positives),redlineis a must-fix red line,team_standardis team standards. - merge_system_rule: Controls whether to merge with OCR's built-in system rules. Team rules stack on top of general rules, not replace them.
What we did is split them into two layers: team baseline rules are maintained in a separate git repository, fetched and merged at CI runtime; if the fetch fails, fall back to system defaults. Project custom rules live in each repository's root .opencodereview/rule.json to add red lines specific to the project. Linting takes effect on push, and both the CI side and the local review gate share the same rule file.
3. CI Job: Shared Template + Variabilized Differences
The review job is condensed into one shared CI template, referenced by both frontend and backend via include: job definitions, review logic, severity thresholds, and comment format are maintained in one place; differences (scan exclusion lists, suggested code block language, runner, stage) are extracted as CI variables, and the user overrides only a few lines of shell configuration.
ai_review_job:
image: node:20
only:
- develop
- release
variables:
GIT_DEPTH: "0" # Full history, needed for diff baseline
allow_failure: true # Review runs in parallel with build; failure does not block deployment
script:
- npm install -g <ocr-cli>
# 1. Configure model gateway (variables in GitLab group-level CI Variables; project zero-config)
# 2. Fetch team rules + merge project rules
# 3. ocr review --from <baseline> --to $CI_COMMIT_SHA --format json
# 4. Result distribution: GitLab comment + Feishu card + real-time dashboard ingestion
Diff baseline is chosen from three scenarios: MR mode uses the MR diff base; ordinary push uses the previous commit; the first commit on a new branch has no parent, so it automatically fetches the target branch as the baseline; if nothing can be obtained, it degrades to a single-commit scan.
allow_failure ensures seamless embedding: The review job runs in parallel with build; if it fails or times out, deployment is unaffected, and developers' usage is identical to before integration.
Exclusion lists are variabilized by tech stack: Frontend excludes dist/, *.less; backend excludes target/, *.class; compressed artifacts and style files never enter the LLM.
4. Result Distribution: One Result, Three Exits
After OCR outputs JSON, a Node script distributes it in three directions:
GitLab commit comments. Each finding gets its own comment, anchored to a specific code line using the official API's path / line / line_type: "new" parameters—open the commit page and see exactly which line has the issue. The comment body carries a fixed format (🤖 AI Review Suggestion (severity)). After the comment is created, we capture the noteId—this ID is the key for precise positioning in the later collection layer.
Feishu message cards. After the comment is done, a private message is pushed to the reviewed person. Each card explains three things clearly:
- How serious it is: The card title is colored by the highest severity level in this batch—high→red, medium→yellow, low→blue; a glance tells whether to act immediately
- Where to look: Each issue has a severity dot; clicking the file line number jumps directly to the corresponding GitLab comment; the "Review Statistics Dashboard" button at the bottom jumps to the dashboard for the current branch, and "View All Comments" returns to the commit page
- Whose responsibility: The card includes project/branch columns and trigger info; in mixed multi-author scans, findings are split by author, so each person only receives comments related to their changed files
Replies to the card ("False Positive / Fixed") are automatically categorized into dashboard statistics.
Real-time ingestion. The same result is pushed with a token to the collection service's ingestion endpoint. After unified severity thresholds, it lands in the database—review records are visible in seconds, no waiting for offline collection.
5. Reception Layer: Making Review Quality Quantifiable
Distribution is just the start. Comments scatter across hundreds of commits, nobody can keep up, and the question we need to answer is: Is this AI review actually accurate? Do people accept it?
So we built a collection service and dashboard:
- AI comment identification is based on body format, not account. The token used by CI and the account used by humans may change, but the comment body format is ours—identifying by format is inherently stable, and comments from both frontend and backend systems can be collected uniformly
- Dual-channel ingestion: CI pushes in real-time to the collection service (comments visible immediately), plus a scheduled full reconciliation job (fills in replies, corrects status, catches anything the real-time channel missed). Real-time handles "speed," reconciliation handles "accuracy"
- Reply status inference: When developers reply under an AI comment with "👍 Fixed" or "🤡 False Positive," the collection layer uses regex to infer the final status of that finding: adopted / falsePositive / pending. False positives are marked by users voting with their feet, not determined by us
- Dashboard: Trend chart (how many review findings per day, severity distribution), by-project/by-person detail tables, false-positive rate statistics
An unexpected payoff came from this layer: the false-positive rate becomes a direct input for rule optimization. If a rule's false-positive count stays high → the dashboard shows it at a glance → modify the rule → the false-positive rate drops in the next version, all backed by data.
6. Closure Layer: Review Conclusions Return to the Code-Writing AI
Up to this point, "found" issues still require humans to fix. But our daily development is AI-written (as in the previous article), so the consumer of review conclusions should also be the AI itself—in the athlete/referee separation, the "referee" must pass the ball back to the "athlete."
Redline issues are blocked locally. After the AI companion modifies code, it proactively runs a local review gate—a set of redline regexes that are instant and deterministic; if hit, they block; as a safety net, it always runs. It fixes and re-runs one by one until the redlines are zero.
CI comments are processed and replied to one by one. Replies follow a fixed format under the GitLab comment:
- Fixed: describe the fix, attach the fix commit, the comment can be traced back to code
- Obsolete: The issue was already implemented or no longer exists in later commits, with justification
- False Positive: Give a reason—e.g., the parameter structure is defined by the backend OpenAPI, integration testing on the test environment already passed
- When uncertain, escalate to a human; do not answer boldly
In one real test run: 17 CI comments, 14 fixed, 2 obsolete, 1 false positive; all replies were generated by AI and submitted, a human only did the final confirmation.
Replies automatically flow back to the dashboard. The collection layer classifies replies by keywords and infers status. AI reviews AI's code, AI replies to AI's comments, humans only make the final call at critical points.
Final Thoughts
Let me string together the whole chain: After AI modifies code, it first passes the local redline gate; push only after they are cleared. CI does semantic review, comments anchored to lines, cards pushed to people, results ingested in real-time. Conclusions return to the companion, which replies one by one per the agreed format. Replies are collected, classified, and flow back to the dashboard; the false-positive rate in turn optimizes rules—the next round of review starts from more accurate rules.
There's an even more practical benefit: when AI both writes code and replies to comments, the human's role shifts from "processing each review comment" to "watching data and making key decisions." The previous article concluded that "the bottleneck of AI efficiency is precipitation, acceptance, and scaling." This one makes up the other half of the acceptance line—acceptance is not just AI self-checking; there must be an independent referee, and the referee itself must be measured.
Questions are welcome in the comments.