# How AI Agents Learn From Real User Behavior: A New Platform for Continual Improvement
## The Problem Every Agent Team Knows Too Well
Building an AI agent that works well in a demo is one thing. Making it reliably handle complex, multi-step workflows inside a real product at scale is another challenge entirely. Even agents built on powerful foundation models often struggle to carry out end-to-end tasks consistently enough for production use.
Most teams respond by writing hundreds or even thousands of hand-crafted test cases, then tuning prompts, fixing harnesses, and adding special rules until the agent passes enough evaluations to ship. It’s slow, manual work — and it still misses something important.
The people using the product every day already know exactly how these tasks should be completed. Their daily actions contain the answer. But most agent stacks have no way to capture that knowledge and turn it into learning for the agent.
## Why the Agent’s View Is Only Half the Picture
Today’s agent observability tools are impressive. They record every prompt, every tool call, and every model response. They show teams where an agent made a mistake, which is incredibly valuable for debugging.
But they don’t show what the correct behavior looks like in practice. That information lives on the product side — in the clicks users make, the buttons they press, the workflows they follow, and the workarounds they quietly adopt when something doesn’t work as expected.
Product analytics tools like session replays and funnel tracking capture this user behavior, but that data is designed for humans reading dashboards, not for machines learning a task. The two sides of a transaction — what the agent does and what the user does — almost never end up on the same timeline, let alone in the same learning pipeline.
## Full-Stack Trajectories: Capturing Both Sides at Once
A new approach solves this by capturing what happens on both sides of a task in a single, unified record called a **full-stack trajectory**.
This works through a protocol that sits between an agent and the user-facing application. Because the protocol handles communication in both directions, it naturally sees everything: the agent’s reasoning, tool calls, and state changes on one side, and the user’s screen, clicks, and server requests on the other. If the user messages the agent during a task, that conversation is also captured and placed in the correct order on the same timeline.
The result is a complete replay of a real work session — not just what the AI did, but what the human did to get the job done. This is the raw material that makes machine learning from real work possible.
## The Four-Stage Learning Pipeline
Turning raw trajectories into usable agent capabilities happens through a structured four-stage process:
### Stage 1: Capture and Organize
Every session streams into a central intelligence layer and is sorted into containers. Each container groups together sessions related to a specific kind of work — for example, handling billing questions, onboarding new customers, or closing out a project sprint. A routing rule determines which container a session belongs to, and sessions can be reorganized later as needs change. Each container learns independently, so patterns stay focused and relevant.
### Stage 2: Surface Patterns Automatically
Once a container accumulates enough sessions, an automated analysis runs on a schedule or can be triggered manually. It examines the frozen snapshot of all sessions and surfaces recurring patterns: common requests, repeated corrections the agent needed, and friction points where users quietly adapted or worked around a problem.
User friction is the strongest signal in this process. When thousands of sessions show the same workaround, it tells the system exactly what the agent should learn next. Every pattern surfaced includes citations pointing back to the specific sessions that produced it, so teams can always verify the evidence.
### Stage 3: Human Review Before Anything Ships
No pattern becomes an agent capability automatically. When the evidence supports a reusable lesson, the system proposes it as a structured skill — a plain file that defines a step-by-step procedure. A team member reviews each proposal, either through a visual interface or a command-line tool, and approves or rejects it. Published skills are versioned, so teams can track changes and roll back if needed.
### Stage 4: Delivery and Export
Approved skills reach the agent without requiring a redeployment. Teams that prioritize stability can pin the agent to a specific version of a skill. For agents or tools outside the main platform, skills can be downloaded as standalone files. The raw trajectory data itself can also be exported for use in other systems.
In addition to skills, the same pipeline generates three other valuable outputs: production-based evaluation test cases built from real user tasks, datasets suitable for fine-tuning smaller and more cost-effective models, and complete raw session records for any observability platform.
## A Real-World Example
Imagine a project management platform where engineering teams run sprints. The platform has a one-click button that moves every unfinished task into the next sprint. An agent told to “close out the sprint” simply clicks that button and moves everything forward.
But experienced team leads know that’s not the full job. After the bulk move, they check for issues that were already carried over once and send them back to the backlog. They unassign them. They look for blocked tasks and leave comments tagging whoever owns the blocker. These follow-up steps are never documented as a formal process, but every sprint close in practice follows this pattern.
A system that captures full-stack trajectories would see this hidden routine across thousands of sprints, surface it as a pattern, and propose a complete skill that covers both the initial move and the follow-up work. Once published, the agent — and any external agent connected through an integration layer — would follow the full routine without being told each step. As user habits evolve, new sessions keep the skill updated.
## The Problems This Approach Solves
For leaders running agent programs at scale, this method addresses several persistent headaches:
– **Struggling to pass evaluations**: The most time-consuming part of shipping an agent is getting it to handle the workflows that matter in production. Learning directly from how real users complete those tasks gives the agent the steps it was missing. The evaluation suite then grows organically from real edge cases, not just hypothetical ones.
– **Context overload and tool sprawl**: Each agent mistake often leads to another rule in the system prompt or another tool added to the agent. Over time, this bloat makes behavior unpredictable. Keeping learned behavior in a separate skill catalog, loaded only when needed, keeps the agent lean and focused.
– **Rising costs**: More tools and longer prompts mean more compute per task. Datasets built from real trajectories let teams move high-volume workflows to smaller, cheaper open-source models. Workflow-level containers also show exactly where an agent struggles, so improvements can be targeted.
– **Fragmented tooling across teams**: Large organizations use different agent frameworks across web apps, mobile apps, Slack, and Microsoft Teams. Because the underlying protocol is broadly supported, the learning platform works regardless of which framework or model sits underneath.
– **Uncontrolled agent adoption**: When teams adopt agent tools on their own without centralized oversight, it becomes difficult to track what each agent has learned and why. Every skill goes through human review, traces back to its source sessions, and stays versioned and reversible. The entire platform can also be run inside a company’s own infrastructure, keeping all data within its network.
## Deployment and Privacy
The platform is designed to be fully self-hosted. Teams can deploy it on their own Kubernetes cluster using standard packaging tools, or on cloud infrastructure like AWS, with their own databases and identity providers. All trajectory data, skills, and learned patterns stay inside the company’s network or on-premises environment. There is also a managed cloud option for teams that want the fastest path to getting started.
## FAQ
**What is a full-stack trajectory?**
A full-stack trajectory is a complete, time-ordered record of a real work session that captures both what an AI agent does (tool calls, reasoning, responses) and what the user does in the product (clicks, screen changes, server requests). Both sides are captured on a single timeline, making it possible to see the full picture of how a task was actually completed.
**What kinds of outputs does the system produce?**
The system generates four outputs from real user sessions: structured skills that agents can load for specific tasks, production-based evaluation cases built from actual workflows, fine-tuning datasets for training smaller and more efficient models, and raw session data that can be exported to any observability tool.
**Does this replace existing evaluation and observability tools?**
No. The goal is to complement existing stacks. Teams keep their current evaluation platforms and observability tools, and the system feeds them the data those tools have been missing — real-world execution patterns drawn from actual user behavior.
**How is human oversight maintained?**
Every proposed skill must be reviewed and approved by a team member before it reaches any agent. The review happens through a visual interface or command-line tool, and each skill links back to the sessions that generated it. Skills are also versioned and can be retired at any time.
**What does “self-hosted by design” mean in practice?**
It means the entire platform can run on a company’s own infrastructure, using their own Kubernetes clusters, cloud accounts, databases, and identity providers. User activity data and agent traces never need to leave the company’s network.
**Which agent frameworks and models does this work with?**
Because it is built on an open protocol that is supported by major providers including Google, Microsoft, Anthropic, Amazon, and Oracle, it works with virtually any agent framework and any underlying model. Supported frameworks include LangChain, Mastra, Pydantic AI, Google ADK, and others.
**Can teams use historical data, or does it only work with new sessions?**
Teams that already have agents in production can import existing conversation history rather than starting from zero. This allows them to begin surfacing patterns from data they already have.
## Conclusion
The biggest gap in current AI agent development isn’t a lack of powerful models — it’s a lack of learning from real work. Full-stack trajectory capture bridges that gap by putting agent behavior and user behavior on the same timeline, then turning that combined data into actionable skills, evaluations, and training material through a structured, human-reviewed pipeline.
For teams that are already shipping agents but struggling with consistency, rising costs, and the slow grind of hand-writing test cases, this approach offers a way to let the millions of real user sessions they already collect do the heavy lifting. The result is agents that improve continuously, grounded not in synthetic test cases but in the actual patterns of how work gets done.
Thank you for reading
