**The Hidden Failure Mode of AI Debugging: Why Tests Can Pass While Code Breaks**
Debugging is often presented as a clear frontier for large language models and coding agents: fix tricky production bugs in open-source libraries by reading code and tests. In practice, the real-world performance of these systems is shaped less by raw problem-solving skill and more by the nature of the bug itself, especially whether the correct fix depends only on information that lives in the repository.
I recently ran a set of blind, scored experiments on three recently fixed bugs from popular JavaScript/TypeScript libraries: `ky`, `immer`, and `decimal.js`. Across 28 runs using multiple model tiers, workflows, and review checkpoints, a clear pattern emerged. Bugs whose solutions were discoverable directly from code consistently succeeded. Bugs that required reasoning about undocumented contracts, implicit usage patterns, or the real-world behavior of callers consistently failed—and often failed in dangerous ways. These failures passed existing tests, looked plausible, and slipped through human review.
What follows is a detailed look at those experiments, what worked, what did not, and what it means for teams building or deploying coding agents.
### The Bugs I Tested
I selected three real, production-critical bugs fixed in July 2026. Each came with its merged fix and regression tests, which I held out as hidden evaluators. The bugs span “easy”, “hard”, and “very hard” on paper, and the results upend simple intuitions about difficulty.
– **ky #867**: An HTTP client where setting `retry` as a number on a base client and then extending it with an object silently dropped the numeric limit. The correct fix was narrow: apply the numeric conversion only at the root option.
– **immer #1255**: With the array-methods plugin enabled, calling `reverse()` or `sort()` on a draft array could mutate the original caller’s state. The fix required understanding how Immer tracks array positions and re-drafting displaced elements.
– **decimal.js #260**: The `asin()` function lost precision near x = 1 due to catastrophic cancellation in `1 – x²`. The correct fix was an algebraic rewrite to `(1 – x)(1 + x)`, not increased internal precision.
### My Testing Methodology
Each run started from a clean checkout of the repository at the pre-fix commit. An AI agent (or pipeline of agents) received only the bug report, the repository, and the existing visible tests. Hidden regression tests, taken from the actual fix, were applied after the run to determine success or failure.
I used three model tiers (Claude Haiku 4.5, Sonnet 5, Opus 4.8), three workflows (naive single agent, structured “gstack investigate” pipeline, and a parallel diagnose–implement–review pipeline), and 28 total runs. Agents could call visible tests as often as they wanted, but never the hidden ones.
### Results: The Easy Bug Broke Everything
The most surprising finding was that the “easy” ky bug defeated every model and workflow. Across 12 runs, all 12 produced a fix that passed the visible 84-test retry suite but corrupted user payloads when `retry` intersected with user data. The hidden tests consistently caught these runs.
The root cause was a mismatch between surface-level understanding and implicit API contracts: the deep-merge logic that applies numeric retry defaults at any nesting level incorrectly overrode user-supplied `retry` fields. The correct fix constrained the normalization to the top-level option only.
In contrast, both the Immer and decimal.js bugs were solved in all 16 attempts. For Immer, agents consistently identified the positional assumption in the proxy implementation and proposed either wrapping displaced elements or removing the assumption entirely. For decimal.js, most runs rediscovered the algebraic reformulation, while a minority used adaptive precision. None relied on “just increasing precision,” even though that approach looked tempting and passed all visible tests.
### Why the Model and Workflow Weren’t the Bottleneck
Contrary to expectation, model strength did not predict success. All three Claude tiers failed on ky, and the two harder bugs were solved equally well by weaker and stronger models. Increasing structural rigor—adding explicit investigation steps or a multi-agent pipeline with review—also did not eliminate failures. In one reviewed run, the reviewer correctly identified user-data corruption yet still approved the change, demonstrating that process alone cannot fix flawed judgments.
The consistent failure on ky, despite varying models and methods, points to a deeper issue: correctness depended on information outside the code. The bug report and the repository did not state that user payloads could contain keys like `retry`. Discovering this unstated contract requires reasoning about real usage, not just code invariants.
### The Key Insight: Sort Problems by Information, Not Apparent Difficulty
These results suggest a better way to think about AI debugging tasks. The most important question is not “Is this bug technically hard?” but “Is everything needed to produce the correct fix visible in the code and the ticket?”
– **Hard-but-discoverable problems**, such as the Immer proxy invariant or the decimal.js algebraic rewrite, were reliably solved. AI can excel here.
– **Easy-looking problems that depend on unstated contracts**, such as ky’s retry merging, remain brittle. Models and processes alike can miss them, especially when tests do not exercise the intersection of features.
This distinction also highlights a gap in typical evaluation practice. Passing unit tests is necessary but insufficient. A fix can be “green” while still being wrong for users, particularly when those users rely on undocumented or unforeseen interactions.
### Two Places to Improve
First, improve the bug report. Every unstated contract you can surface—calling conventions, data shape expectations, lifecycle rules—gives the agent information it cannot infer from code alone. One extra sentence in a ticket can be more valuable than a model upgrade.
Second, harden the review gate. If an AI reviewer flags potential side effects or data corruption, that finding should block the merge automatically. In the one run that identified the ky payload risk, the workflow ignored it. A process that depends on human judgment here is unreliable; a rule that stops merges on flagged risks is robust.
### Limitations
These findings are based on a small sample of one bug per library and a primarily JavaScript/TypeScript context. Difficulty ratings were not independently validated, and some apparent “easy” fixes may have been latent in model training data. That said, the consistency of failure on ky, across all models and workflows, strengthens the core lesson.
### Conclusion
AI debugging is not inherently unreliable. Across 16 runs, two hard bugs were solved consistently and correctly. The problem is not the models or the methods alone; it is the mismatch between what can be inferred from code and what depends on external usage patterns. When contracts are explicit and information is complete, modern coding agents perform strongly. When they are not, even the best models and processes can pass flawed fixes as correct.
For teams, the implication is clear: invest in making contracts visible, and enforce hard gates on changes that touch data-critical logic. AI is a powerful debugger, but it works best when the problem definition contains everything a human reviewer would want to know.
Thank you for reading
