Alibaba Open-Sourced Its Internal Code Review AI: Open Code Review

Alibaba open-sourced Open Code Review (OCR) — an AI-powered code review CLI that served as the company's internal review assistant for two years, reportedly used by tens of thousands of developers and credited with catching millions of defects before its public release. It's now on GitHub with 29.6k stars, an Apache-2.0 license, and a specific, checkable claim: it beats general-purpose coding agents like Claude Code at review precision, using a fraction of the tokens.

That claim comes with an important asterisk worth understanding before you adopt it: the benchmark backing it up was built entirely by Alibaba. That doesn't make the numbers false — but it does mean this is a case where reading past the headline claim to the actual methodology matters more than usual.

Update (October 9, 2026): What developers are saying after the Hacker News launch

The repository has since climbed to roughly 44.8k stars and 3.2k forks, and a Hacker News thread (284 points, 73 comments at the time of writing) gave the first batch of independent, hands-on reports. These are individual commenters' accounts, not verified results.

• Independent benchmark run, with the opposite shape of Alibaba's numbers. User eranation ran OCR on 10 of the 50 PRs in a third-party code-review benchmark and reported recall around 74% but precision around 12%, which would drag F1 to roughly 20% and near the bottom of that leaderboard if it held across all 50. That is the reverse trade-off from the precision-over-recall positioning in the README, and a ten-PR sample is small.
• A compatibility bug. User sfortis reported that OCR failed with GPT-5.x models because a hardcoded max_tokens parameter is rejected in favor of max_completion_tokens. They also said a plain review skill running on a Codex subscription worked fine and consumed subscription usage instead of API tokens.
• The "why not just ask my agent?" question. User faangguyindia asked what OCR adds over asking Codex in another tab. The maintainer, lizhengfeng101, replied elsewhere in the thread that skills run as sub-agents are a good approach but can be hard to debug, evaluate and tune, and that OCR adds deterministic file filtering, bundling and rule matching around the agent.
• Rule files are in Chinese. Several commenters, including hrpnk and nutifafa, noted the built-in review rules are written in Chinese. User pramodbiligiri published a machine-translated copy of the rules in English.
• Alternatives people already use. Commenters mentioned paid reviewers such as CodeRabbit, a hand-built multi-model "meta-review" skill, and complaints about changing bot pricing, which shows how crowded the category is.

The takeaway matches our original caution: the architecture is interesting and free to try, but the precision claim comes from Alibaba's own benchmark, and the first outside measurement we have seen points the other way. Run it on your own pull requests before trusting either set of numbers. For a comparable engineered-context approach, see code-review-graph.

TL;DR

The actual architecture: deterministic engineering + agent, not just a prompt

OCR's core design argument is specific and worth taking seriously on its own terms, independent of the benchmark dispute: general-purpose agents like Claude Code with review skills tend to fail at code review in three particular ways — incomplete coverage on large changesets (the agent "cuts corners" and skips files), position drift (reported issues don't match the actual line), and unstable quality (natural-language-driven instructions are hard to debug and fluctuate with minor prompt changes).

OCR's fix is to split the review pipeline into two parts, each handling what it's actually good at:

Deterministic engineering handles the steps that must not go wrong — precise file selection (which files need review, which get filtered out), smart file bundling (grouping related files like message_en.properties and message_zh.properties into one review unit, each running as an isolated sub-agent for stability at scale), fine-grained rule matching (via a template engine rather than natural-language rule guidance), and separate positioning/reflection modules that specifically check comment location and content accuracy after the fact.

The agent handles dynamic decisions — retrieving relevant context, using a toolset the team says was "distilled from deep analysis of tool-call traces in large-scale production data" (call frequency, per-tool repetition rates, and how new tools affected the overall call chain) rather than a generic agent toolkit.

That's a genuinely different design philosophy from "give a capable general agent good review instructions," and it maps onto a broader pattern this blog has covered before: code-review-graph made a similar bet — that structural, engineered context delivery beats asking a general agent to re-derive that structure from scratch every time — applied there via a Tree-sitter code graph rather than OCR's deterministic file-bundling and rule-matching pipeline.

The benchmark numbers, and the honest caveat

AACR-Bench, the evaluation Alibaba built to support these claims, is a real, substantial effort: 50 popular open-source repositories, 200 real pull requests, 10 programming languages, and 1,505 annotated ground-truth issues cross-validated by more than 80 senior engineers. The headline results, comparing OCR against Claude Code on the same underlying model:

Read those numbers carefully rather than just the "significantly higher" framing in Alibaba's own marketing copy. 33.9% precision means roughly two out of every three issues OCR reports are still false positives — a real improvement over Claude Code's reported 7.2% (which implies over 9 in 10 flagged issues are noise), but still a substantial rate a team would need to budget review triage time around, not a "solved" precision problem. Claude Code's higher recall (28.9% vs 20.0%) reflects OCR's deliberate design trade-off toward precision over noise — it's finding fewer of the real issues, on purpose, in exchange for fewer false alarms.

Why "Alibaba built the benchmark it wins on" matters

This is the single most important caveat to weigh before treating these numbers as settled fact. As one critical analysis of the benchmark put it directly: Alibaba built both the review tool and the dataset used to evaluate it, which creates a structural conflict of interest that 80+ external engineer annotators cross-validating ground truth doesn't fully resolve — the annotators verified what counts as a real defect in the dataset, but they didn't independently design the comparison methodology, choose which competing tools and configurations to benchmark against, or run the evaluation itself. An instrument that agrees with its own maker's design goals isn't automatically wrong, but it isn't independent validation either.

The same critique raises a fair, checkable point about scope: OCR's built-in ruleset targets specific defect categories (NPE, thread-safety, XSS, SQL injection), and the benchmark's value would be strengthened significantly by comparison against dedicated static analysis tools and linters that already catch many of these same categories deterministically — a comparison the current benchmark doesn't include. Until an independent party reproduces these numbers on a benchmark they control, or Alibaba's own numbers get replicated by outside teams running the public AACR-Bench dataset themselves, the responsible read is: real, substantial engineering, promising early numbers, unverified by anyone outside Alibaba.

What people are actually asking before adopting this

• "Do I need to give up my existing coding agent?" No — delegation mode lets Claude Code, Codex, Cursor, or OpenCode perform the actual review using their own LLM, while OCR just handles file selection and rule resolution. This is likely the lower-risk way to trial OCR's engineering approach without a new API key or provider relationship.
• "Does this replace my linters and static analysis tools?" No, and it shouldn't be positioned that way — OCR's ruleset overlaps with categories linters already catch deterministically (NPE, SQL injection patterns). It's a complement to existing static analysis, not a replacement, and the critique above suggests that direct comparison hasn't been published yet.
• "What does ocr scan do that ocr review doesn't?" ocr review works from Git diffs (staged/unstaged changes, branch ranges, specific commits); ocr scan reviews whole files without needing git history — useful for auditing an unfamiliar codebase or a directory with no meaningful diff to review.
• "Is this open-washing an internal tool for marketing, or a real contribution?" Two years of internal production use at Alibaba's scale, a genuinely different architectural approach (deterministic pipeline + agent hybrid) from most review tools, and 178 contributors plus 127 releases since going public are real signals of substance — independent of whether the specific benchmark numbers hold up to outside scrutiny.

The bigger pattern: engineered pipelines vs. general agents for narrow tasks

OCR is one more data point in a trend worth naming explicitly: for well-defined, repeated tasks like code review, purpose-built pipelines that constrain an LLM's role to specific, engineered steps are increasingly outperforming general-purpose agents on cost and consistency, even when the general agent is more capable in the open-ended sense. This is the same underlying logic behind TypeSafe AI's Jev trading text generation for structured decisions, and behind code-review-graph trading a general agent's repo re-reading for a pre-built structural graph. The pattern across all three: identify exactly which parts of a task benefit from a general model's judgment, and engineer everything else — file selection, context retrieval, output structure — so the model's freedom is spent only where it adds value, not on re-deriving structure that a deterministic system could hand it for free.

For a team evaluating whether to build custom review tooling versus buying an off-the-shelf general agent's review skill, that's the real strategic question OCR's existence raises: not "is this specific tool better," but "does our review workflow have enough volume and enough repeated structure to justify engineering around an LLM, rather than just pointing a capable general agent at the diff and accepting its default trade-offs."

FAQ

What is Open Code Review (OCR)? Alibaba's internal AI code review CLI, used for two years before open-sourcing, combining deterministic file/rule engineering with an LLM agent for line-level review comments.

How does OCR compare to using Claude Code for review? Alibaba's own benchmark shows 33.9% precision and 25.1% F1 for OCR vs. 7.2% precision and 11.6% F1 for Claude Code on the same model, using ~1/9th the tokens — though Claude Code had higher recall.

Is the AACR-Bench comparison trustworthy? It's a real, substantial benchmark, but Alibaba built both the tool and the benchmark — a genuine conflict-of-interest limitation that hasn't been independently reproduced yet.

What does "33.9% precision" actually mean in practice? Roughly two-thirds of OCR's reported issues are still false positives, even in its best-case benchmark — better than Claude Code's reported rate, but not a solved problem.

What is "delegation mode"? A mode where your existing coding agent performs the review with its own LLM, while OCR handles file selection and rule matching — no separate API key required.

Do I need my own LLM API key to use OCR? Yes in default mode (OpenAI- or Anthropic-compatible provider); no in delegation mode.

Related reading

• code-review-graph: stop AI coding agents from re-reading your whole repo
• What are Agent Skills? A complete guide
• What is MCP? Model Context Protocol architecture guide
• AI benchmarks: a complete guide
• GPT-5.6 Terra and CheatingBench: the real score, fact-checked
• TypeSafe AI's Jev: a "System One Model" for structured decisions
• Official: alibaba/open-code-review on GitHub · AACR-Bench on Hugging Face

Figures in this piece reflect Alibaba's own published AACR-Bench results and GitHub repository stats as of September 16, 2026. Independent reproduction of the benchmark claims has not been published; verify against your own codebase before adopting for production review workflows.