# Loop Engineering: From One-Shot Prompts to Autonomous Improvement Loops
Most people still use AI like a 2015 search box: you type, you read, you type again. Loop engineering replaces that manual back-and-forth with a self-executing cycle: the model plans, acts, checks its own result, and repeats. When paired with an objective metric, a loop can become the bottleneck limiter—meaning you are the constraint, not the compute.
This guide explains loop engineering using two verified artifacts: Andrej Karpathy’s `autoresearch` repository and the paper *Bilevel Autoresearch*. The framing follows a write-up by @0xCodila.
—
## What Is Loop Engineering?
A prompt is a single instruction; you decide the next step. A loop is a goal the model pursues until it arrives. The model plans, acts, checks its own output, and repeats. You define the objective once, and the loop handles iteration. Crucially, a loop only earns its cost when the work is measurable.
—
## The Three Parts That Make a Loop Work
Every reliable loop has three components:
1. **A verifier** that grades each attempt (a test, a metric, or a build).
2. **State** that records what was tried, what failed, and what remains.
3. **A stop condition** that prevents runaway cost—either when the goal is met or after N attempts.
Without these, the agent simply agrees with itself.
—
## The Karpathy Loop: Inside `autoresearch`
On March 7, 2026, Andrej Karpathy released `autoresearch`, an MIT-licensed repository that implements the Karpathy Loop. It ships with about 630 lines of code and has garnered tens of thousands of GitHub stars.
The design is deliberately small but strict:
– The agent edits only `train.py`, which holds the model, optimizer, and training loop.
– It cannot touch evaluation utilities in `prepare.py`, preventing test manipulation.
– A human writes `program.md`, the instructions the agent must follow.
Each cycle:
1. Reads the code.
2. Proposes a change.
3. Trains for five minutes.
4. Keeps or rolls back based on `val_bpb` (validation bits per byte), where lower is better.
With roughly 12 experiments per hour, runs overnight can yield measurable gains. In one case, the loop found a missing scalar multiplier in a `QK-Norm` implementation and reduced GPT-2 training time by 11% (from 2.02 to 1.80 hours).
—
## Prompt vs Loop vs Bilevel Loop
| Aspect | One-shot Prompt | Karpathy Loop (`autoresearch`) | Bilevel Autoresearch |
|————————-|———————————-|—————————————|——————————–|
| You define | Each step | The goal, once | The goal, once |
| Who iterates | You | Inner agent | Inner + outer agent |
| Verifier | You, manually | `prepare.py` (`val_bpb`) | Same metric, two levels |
| State | Chat only | Experiment log | Log plus injected code |
| Human role | Engineer | Author of `program.md` | Author of `program.md` |
| Reported result | Varies | 700 runs → 20 fixes, 11% speedup | 5x larger val_bpb drop |
—
## The Five Building Blocks
Modern AI engineering teams assemble working loops from five reusable pieces:
1. **Automation** that fires the loop on a schedule, event, or trigger.
2. **A skill** that stores project knowledge in a markdown file, read on every run.
3. **Sub-agents** that split the writer from the reviewer.
4. **Connectors** that let the loop act inside real tools (e.g., issue trackers, Slack).
5. **A verifier** that rejects bad work (ships in tools like Claude Code and Codex).
—
## Bilevel Autoresearch: A Loop on Top of the Loop
If autoresearch is research, can you autoresearch autoresearch? The paper *Bilevel Autoresearch: Meta-Autoresearching Itself* says yes.
– The **inner loop** matches Karpathy’s original: propose, train, evaluate, keep or discard.
– The **outer loop** watches the inner loop’s code and traces, identifies where the search is stalling, writes new Python mechanisms, injects them at runtime, and reruns the inner loop.
In benchmarks, the outer loop reduced `val_bpb` an additional 5x compared to a single loop (–0.045 vs –0.009), using the same LLM. The key insight: the inner loop kept returning to stale priors. The outer loop forced exploration.
—
## Use Cases With Examples
These ideas transfer beyond pretraining:
– **Model work**: search hyperparameters until `val_bpb` drops.
– **Software**: refactor until tests, types, and the build pass.
– **Content**: rewrite until every rubric score clears a threshold.
– **Data**: tune a pipeline until schema checks hold.
The common trait: an automatic gate that can fail the work.
—
## Try It Yourself: A Loop in One Prompt
Paste this into any capable model and watch it self-correct:
“`text
You will work in a loop until the task meets the bar.
TASK:
[describe exactly what you want produced]
SUCCESS CRITERIA (be strict):
– [criterion 1]
– [criterion 2]
– [criterion 3]
LOOP PROTOCOL, repeat every turn:
1. PLAN – state the single next step.
2. DO – produce or improve the work.
3. VERIFY – score the result 1-10 on each criterion. Be honest.
4. DECIDE – if every criterion is 8+, print FINAL and stop.
Otherwise print ITERATING and fix the weakest point first.
RULES:
– Never call it done until every criterion is 8 or higher.
– Each pass must fix the weakest score from the last VERIFY.
– Do not ask questions. Make a sensible assumption and continue.
Begin.
“`
Underneath, the control flow is simple:
“`python
current = baseline
best = evaluate(current) # verifier
for step in range(MAX_STEPS): # stop condition 1
candidate = propose_change(current) # agent edits
score = train_and_eval(candidate) # train, then verify
if score < best: # keep only real improvements
current, best = candidate, score # commit
if best <= TARGET: # stop condition 2
break
```Both versions are limited: you are still the trigger, and closing the tab erases state. Adding automation, a state file, and a real gate turns this into an autonomous loop.---See It RunThe interactive demo below animates one full loop: propose, train, verify, then keep or roll back. Adjust the target and step limit, and watch `val_bpb` fall until the stop condition fires.
