**Building a Multimodel Review System for Agent-Generated Code**
When AI agents accelerate development, code review becomes the bottleneck. What happens when your reviewer is also an AI? This post details a proven system for reviewing agent-written code, catching hallucinations, and scaling reviews alongside agent throughput—all while preserving a human-in-the-loop.
—
### The Problem with Agent Reviews
Two major issues arise when agents review their own work:
1. **Hallucinations are inevitable**. Even well-designed agents invent plausible but nonexistent filenames, parameters, or edge cases—delivered with full confidence.
2. **Throughput breaks human review loops**. A single engineer running multiple parallel agent sessions can produce thousands of lines of code faster than a human can read it.
The result? A bottleneck where human reviewers either approve bad code or become overwhelmed.
—
### The Solution: A Cross-Provider Automated Review System
The answer isn’t more human reviewers—it’s a smarter, automated review pass that happens before humans ever look. The system described here uses a **separate AI model from a different provider** to review pull requests (PRs) automatically, catching issues the original agent may have introduced.
**Key design principles:**
– **Decentralized review.** A model from a different provider reviews code written by another model to catch blind spots.
– **Stateful feedback.** Every PR maintains a single, evolving review table that tracks findings, severity, and resolution status.
– **Machine-readable verdicts.** A final `` tag lets CI pipelines enforce gating logic.
– **Writer-first workflow.** The human reviewer acts as editor-in-chief, resolving disagreements and setting standards rather than line-editing code.
—
### How the Pipeline Works
1. **Trigger and concurrency**
– Every PR to `dev` triggers the review pipeline.
– In-flight reviews are canceled on new pushes to avoid wasted work.
2. **Context collection**
– The system fetches issue comments, review summaries, and inline comments from three GitHub APIs.
– These are merged chronologically and passed to the reviewing model for full context.
3. **Model execution**
– A dedicated GitHub Action runs OpenAI’s Codex agent on the PR diff.
– The model reviews only the changes introduced in the PR, compares them against the base commit, and reports issues.
4. **Review output**
– Findings are posted as a single, continuously updated PR comment.
– The commit status turns red or green based on unresolved high-severity issues.
5. **Author agent loop**
– Author agents (Claude Code skills) handle triage: fixing issues, declining with reasoning, or deferring.
– A “reply first, push after” rule ensures the reviewer sees responses before re-evaluating.
—
### Key Components and Configuration
– **Two GitHub workflows**: One orchestrates the review (`pr.yml`), the other encapsulates the Codex action as a reusable composite.
– **Minimal but safe permissions**: The action can read code, post comments, and set statuses—but it cannot merge or push.
– **Model pinning**: The reviewing model is explicitly pinned (e.g., `gpt-5.3-codex`) to control cost and behavior.
– **Effort setting**: `medium` effort balances thoroughness with speed and cost.
– **Retry logic**: Two attempts with a short delay handle transient API errors without masking real problems.
—
### Real-World Example
In a recent PR fixing a capacity grid overflow:
– **Finding 1** (High): Truncation beyond batch cap—fixed with adaptive batching.
– **Finding 2** (Medium): Move cap to config—declined with a reasoned explanation.
The review table captured both decisions. One fix, one informed decline—both preserved in history. The PR turned green, and no human had to debug the comment thread.
—
### Common Pitfalls and Lessons Learned
1. **Pinning models matters**. Default model changes can silently increase costs or break compatibility.
2. **Separate review jobs are risky**. Output can be lost if GitHub blocks secrets in job logs. Keeping everything in one job avoids this.
3. **Feature branches are mandatory**. Direct pushes to main bypass the entire review pipeline.
4. **Treat PR text as untrusted**. Prompt injection is possible; least-privilege tokens and sandboxing mitigate risk.
5. **Don’t chase every false positive**. The goal is merge-ready PRs, not perfect reviews—humans remain the final arbiters.
—
### FAQ
**Q: Why use a different model for review instead of the author’s model?**
A: Author and reviewer models share the same blind spots. A different provider fails differently, catching hallucinations the original model confidently invents.
**Q: Can this work with any LLM provider?**
A: Yes. The pattern is provider-agnostic. The key is having a separate model with its own training biases review code written by another.
**Q: Does this eliminate the need for human reviewers?**
A: No. Humans remain essential for architectural decisions, ethical judgments, and resolving model disagreements. The system removes grunt work, not oversight.
**Q: How are PR comments handled across multiple review rounds?**
A. A single feedback summary table is updated in place. Resolved items are marked ✅, declined items include reasoning, and new issues trigger new review cycles.
**Q: Is this safe for open-source repositories?**
A. Reviewing agent-written code is safest in private, trusted contributor environments. Fork PRs should not be reviewed using secrets or internal runners.
—
### Conclusion
The future of code review isn’t humans reading every diff or models writing unchecked code—it’s **humans curating model reviews**. By combining multiple AI providers, enforcing accountability, and structuring feedback, teams can scale agent productivity without sacrificing quality.
The goal isn’t to remove humans from the loop, but to elevate them. Instead of proofreading outputs, engineers define standards, adjudicate disputes, and own the merge gate—while models do the heavy lifting of checking, catching, and correcting.
If your agents are writing faster than you can review, it’s time to build a reviewer that works at their speed.
—
**Further reading:**
– Follow for AI engineering insights on [LinkedIn]
– Read more on [Towards Data Science] and [Medium]
– Explore the full code examples in the accompanying repository
*Read the previous posts in this series for details on prompt contracts, Claude Code skills, and CI hardening techniques.*
