# How a Verification Layer Made AI Coding Agents Remember What Still Matters
## The Core Problem
AI coding agents work beautifully in short sessions. Give them a few instructions, let them write some code, and the job gets done. But long-running projects tell a different story.
After dozens of turns, fundamental constraints begin to quietly disappear. Nobody deletes them. The context window is not full. They are still sitting somewhere in the chat log. But the agent stops checking for them because each new request feels unrelated to the old rule.
Imagine telling an agent on day one to never expose internal database IDs in API responses. Sixty messages later, you ask it to build a new authentication flow. That new request says nothing about IDs. Without a strong reason to look backward, the agent skips the check and ships an endpoint leaking the exact data you tried to protect.
This is not hypothetical. It is the exact failure mode tested and measured below.
## What This Article Proves
I built a complete, working pipeline in pure Python and ran real benchmark experiments. Every number shown here comes from actual test runs, not simulations. No external APIs, no embedding models, no vector databases. The entire system runs on the Python standard library alone.
The headline finding is striking. A basic search setup recovered only 57% of the requirements a coding agent actually needed. That is enough to pass four out of eight tasks, but not enough to be reliable. Adding a verification layer that checks whether retrieved rules are still valid pushed recovery to 100% and task success from 4 out of 8 to 8 out of 8.
The baseline, which had no access to historical decisions at all, scored zero.
## Retrieval Versus Verification Versus Intent Continuity
These three terms get mixed up frequently, so it helps to draw a sharp line between them.
**Retrieval** asks the question: what information from the past might be relevant to this current request? Standard keyword search, vector similarity, and basic semantic search all operate here. They find records that share words, concepts, or numerical closeness with the current task.
**Verification** asks a different question: is that retrieved information still accurate? A rule from three weeks ago may have been superseded by a newer decision. Two records might contradict each other but apply to entirely different environments. Verification catches these distinctions before anything reaches the agent.
**Intent continuity** combines both steps into a single coherent process. It carries an old requirement into a new task without the user repeating it, while dropping any rule that has been overridden by something more recent.
Most current approaches to agent memory stop at retrieval. They find something that looks related and assume it is still correct. That assumption is where the failures start.
## The Pipeline Architecture
The system processes raw chat history through five stages, all written in pure Python with no external dependencies.
**Stage 1: Extraction.** A lightweight parser scans each message for sentences that look like requirements. It flags sentences containing trigger phrases such as “must,” “never,” “required,” “should,” “prefer,” and similar language. It then identifies the system component the requirement targets and extracts specific values when they are present. In testing against seventy interactions, this extractor achieved perfect recall, catching every planted requirement. Precision was deliberately kept at 80% because three ordinary sentences tripped the filter by using trigger words in non-rule contexts, such as mentioning a dentist appointment.
**Stage 2: Candidate Retrieval.** Once requirements are extracted and structured, the system decides which past records to check before running verification. Two signals drive this decision. First, does the record share the same system component as the current task? Second, does it belong to a linked component defined in a domain schema? That schema maps relationships once, such as authentication work affecting security and API behavior, or API work requiring testing and security reviews. The same schema applies uniformly to every task without modification.
**Stage 3: Verification.** This is the stage that actually handles intent continuity. For each candidate rule, two checks run automatically. Has a newer rule replaced it? Does it apply to the current task context? The system uses a simple but powerful rule for supersession. If two records share the same component, scope, and target key but carry different values, the newer one replaces the older one. Rules targeting different scopes, such as production versus prototype, coexist because they apply to separate environments. This automated distinction matters enormously. Treating them as conflicting would break a core use case.
**Stage 4: Compilation and Deterministic Agent.** Surviving records get flattened into a clean key-value context and fed into a template that mimics a coding agent. The simulated agent starts with fixed defaults, updates itself only with fields it actually receives, and has zero ability to infer rules it was never handed. Every result you see below comes entirely from what the search strategy recovered. There is no language model introducing randomness.
**Stage 5: Grading.** A single grading function checks every run against the exact same ground-truth field list. No approach receives special treatment or a different rubric. The required fields for each task are locked in before any search strategy runs.
## The Experiment Setup
A synthetic project history was constructed instead of using real chat logs. The goal was to create a ground truth that could be fully verified rather than relying on a dataset where determining what the agent should have known becomes a subjective judgment.
The dataset contains seventy chronologically ordered interactions. Twelve genuine planted requirements are buried within them. Three trap sentences are designed to trip up keyword extractors without being actual rules. Fifty-five lines of ordinary noise fill out the dataset, including standup reminders, pull request comments, and casual conversation.
The twelve requirements span different components including authentication, database, security, UI, API, performance, testing, and deployment. Some are constraints, some are decisions, and some are preferences. Four of them form supersession pairs where a later record explicitly replaces an earlier one.
Eight tasks were created that do not restate any of their dependencies. Each task describes a new piece of work and leaves it to the retrieval system to find the historical constraints that still apply.
## Task-by-Task Results
The numbers reveal a clear pattern. The baseline with no history failed every single task, averaging 14 violations per run. Standard lexical retrieval passed 4 out of 8 tasks but still left 7 violations and 17 irrelevant records in its wake. Intent-aware retrieval passed all 8 tasks with zero violations and only 10 irrelevant records, most of which were genuinely valid context that simply was not part of that specific task’s checklist.
On task T1, implementing a new authentication flow, the baseline missed all three hidden constraints and produced three violations. Standard retrieval found the correct authentication method but missed backward compatibility and ID exposure rules because those sentences share no words with the task prompt. Intent-aware retrieval recovered four requirements, including the OAuth2 migration, the ID hiding rule, the backward compatibility constraint, and a rate-limiting rule that happened to be valid historical context for the task but not part of its graded checklist.
On task T3, setting up production database configuration, standard retrieval pulled in both the correct production decision and the stale prototype decision. Without a verification layer, the prototype decision won the field simply because it appeared later in the assembly process. Intent-aware verification explicitly flagged the prototype record as applying to a different scope and kept only the production decision.
## The Bug I Almost Shipped
The experiment went through a version that was subtly rigged, and catching it required deliberate sabotage of my own design.
An earlier iteration of the domain schema allowed per-task component relationship hints. Task T1 was individually told in advance that it also depended on the API and security components. Task T4 received the same treatment for its own specific dependencies. Those two hand-picked declarations happened to be exactly the components those tasks’ correct answers needed, and nothing more.
That is not a discovery mechanism. That is an answer key disguised as a retrieval rule. It directly undercut the entire premise of the project, which is supposed to work without being told where to look.
I caught it the only way that actually works: I deleted the per-task hint and reran the experiment with nothing replacing it. The score dropped from 8 out of 8 to 6 out of 8. The two failing tasks were precisely the two that had been individually hand-fed their answers.
The fix was replacing the per-task hints with a single general schema authored once and applied uniformly to every task, including the six that never needed extra help. This change increased irrelevant records retrieved from 4 to 10 and tokens supplied from 161 to 199. The result stayed at 8 out of 8, but this time it was earned.
This experience underscored an important lesson about synthetic benchmarks. When you build your own test data and already know the answers, it is the easiest thing in the world to unconsciously rig the experiment. Delete the part you suspect is doing too much heavy lifting and see what breaks. It is the only sanity check that actually worked here.
## Honest Design Decisions
Several trade-offs were made intentionally, and they are worth stating clearly.
The extraction dictionaries and component relationships are hand-authored for this specific domain rather than learned from data. This is a controlled demonstration of the verification mechanism, not a general-purpose extraction system that could be pointed at an arbitrary codebase immediately.
Retrieval uses plain lexical word overlap instead of an embedding model or vector database. This keeps retrieval quality from becoming a confounding variable. If a specific embedding model were used, a skeptical reader could reasonably argue the entire comparison depended on which model happened to be selected. The verification layer downstream does not need to change if a different retrieval method is plugged in, making it a clean swap point.
Eight tasks and twelve requirements make up a demonstration rather than a statistically powered study. The scope is sized to be fully inspectable and reproducible, not to generalize with confidence to arbitrary production codebases.
The deterministic agent template exists specifically to isolate what each retrieval strategy recovers. Replacing it with an actual coding model would test a different hypothesis: whether the model correctly uses the recovered context, rather than just whether the context was successfully found.
The grading rubric only tests fields explicitly declared in each task’s ground truth. That is a narrow rubric by design, not a general measure of code quality.
## Frequently Asked Questions
**Why use pure Python with no external dependencies?**
Two reasons. First, convenience. Anyone should be able to clone the repository and run the experiment in under a second with zero setup friction. Second, control. If the pipeline relied on an embedding model, the benchmark results would become tangled with how good or bad that specific model happened to be. The verification logic does the heavy lifting, and it needed to stand entirely on its own.
**What happens if a rule targets a different scope but the same component?**
The verification step keeps both records active. For example, a production database decision and a prototype database decision can coexist because they apply to separate environments. Treating them as conflicting would break a core use case.
**Does intent continuity compress context?**
No, not necessarily. Intent-aware retrieval actually uses more tokens than standard keyword retrieval, roughly 28 percent more in these experiments. The core claim is about correctness, not compression. The system trades tokens for accuracy by recovering requirements that standard retrieval misses entirely.
**What would happen with a real embedding model for retrieval?**
The verification layer would remain unchanged. The retrieval step is a clean swap point. Swapping in embeddings would likely change recall numbers for standard retrieval, but the underlying argument about verification remains the same. The verification step is what does the interesting work.
**Why not just use a longer context window?**
Longer context windows do not solve the problem. Research has shown that models miss details stuck in the middle of long prompts even within their stated limits. And even with total recall, a model still has to connect an old rule about database IDs to a new login task. Memory failure is not the real problem. Deciding what still matters is.
**Is this ready for production use as-is?**
No. The rule-based extractor works because the dataset is controlled. A production version would need a genuinely robust extraction front end, likely a small classifier rather than a simple trigger-phrase list. The system is a demonstration of the verification principle, not a drop-in production solution.
**What about cross-session persistence?**
Everything in the current implementation runs completely in-process. A lightweight persistent store sharing the same record interface would allow intent continuity to survive across restarts and is a natural next step for production use.
## Conclusion
The gap between retrieving related history and retrieving requirements an agent can currently trust is the central finding of this work. Retrieval gets you what is related. Verification gets you what is valid. Intent continuity gets you what still matters, right now, for the task in front of you.
Most systems optimize the first step and skip the other two. That is why they keep shipping code that quietly breaks a decision someone made weeks ago. The verification layer is not optional extras. It is the difference between an agent that remembers everything and an agent that remembers what is actually still correct.
The results are clear and reproducible. Zero external dependencies. Real benchmark numbers from actual runs. And an honest accounting of the mistakes made along the way, including the version of the experiment that almost shipped with rigged results.
Building systems that carry old requirements into new tasks without repeating them is a solvable problem. The solution does not require bigger models, longer context windows, or more sophisticated embedding search. It requires a verification step that asks the simple but powerful question: is this rule still valid?
Thank you for reading
