**Building Agent Systems That Actually Work: Principles From a $100M+ Client Churn Case Study**
Many teams building agent systems put most of their effort into engineering the agent harness, while giving far less attention to the product, workflow, and human decisions that determine how it operates inside a business. When those decisions aren’t thought through carefully, sophisticated agent systems get deployed yet change very little about how work actually gets done.
Based on real-world experience, I’ve distilled five principles for building agent systems that succeed inside a business, earn people’s trust, and improve over time. These principles are illustrated through an agent system I built for a $100M+ managed IT provider to tackle one of its most pressing problems: client churn.
—
### 1. Tailor the Product Shape to the Business Workflow
The common instinct is to put an agent system behind a chat interface. In one previous engagement, a team tried exactly that—an enterprise data agent that let employees ask natural-language questions. But chat interfaces are often the wrong product shape for enterprise workflows for two key reasons:
– Business work usually starts because something happened, not because a person remembers to ask about it. Early churn detection, for example, must begin when new evidence arrives and run continuously in the background.
– Making employee adoption a prerequisite for the business outcome is fragile. The agent system should run without requiring human direction and fit seamlessly into existing behavior.
In this case, churn risk showed up as recurring service issues spread over weeks or months. Connecting those dots manually across technicians, account managers, and dashboards was prohibitively manual at scale. The solution was an event-driven agent system, not a chat-based question-and-answer tool. It owns the detection work itself, watches for new evidence, carries client history forward, and starts workflows without waiting for a prompt.
—
### 2. Give Each Workflow Step to the Simplest Mechanism That Handles It Well
The most common mistake is handing an agent the entire workflow—comparing values, routing results, and making commercial judgments in one opaque LLM run. Stronger production systems assign each step to the simplest mechanism that handles it well, keeping the overall workflow accurate, understandable, and inexpensive to run at scale.
The workflow in this case study looks like this:
– A once-off preparation builds a compact picture of each client’s recent issues.
– A scheduled weekly run compares new tickets with that picture and investigates recurring patterns.
– Only findings that meet both technical and commercial criteria escalate to a human.
Key roles in the workflow:
– **A summarizer agent** compresses long ticket threads into concise issue summaries to cut context size and noise.
– **Semantic similarity grouping**, handled by embeddings and clustering (not an agent), efficiently groups related tickets.
– **Deterministic code** enforces exact recurrence thresholds.
– **An investigation agent** makes two separate judgments: whether tickets represent one continuing issue, and whether that pattern warrants an account manager’s attention.
– **A human** makes the final decision to act, because the right response depends on client context outside the technical evidence.
Each step is given to the simplest mechanism that handles it well.
—
### 3. Redesign the Human’s Job at the Agent Handoff
Adding an agent system changes what people do. One of the most important product decisions is where the system stops and what it hands over to humans.
The best handoff turns a 30-minute reconstruction of what happened into a 1-minute decision. The system does the searching, analysis, and evidence assembly, then stops where human authority still matters. In this case, the account manager still owns the client relationship, so the system emails a consolidated, fully investigated risk alert—no need for the manager to dig through tickets.
But this handoff only works if the manager can trust the system’s findings.
—
### 4. Design the Agent Handoff for Human Verification and Feedback
Trust in agent systems starts at product design, not at the last step. A background system can repeat bad judgments across many cases before anyone notices, so verification has to shape what the system produces, not be an afterthought.
Key design choices included:
– **Making the agent’s work easy to verify** by including reviewable evidence as an intentional output.
– **Starting with how humans already check findings**—account managers confirm recurring issues by seeing frequency, ticket summaries, and source records.
– **Using progressive disclosure** so managers see the finding and most useful evidence immediately, with full details available via links.
– **Collecting feedback at each decision point** rather than a single thumbs-up on the final alert. This makes it clear where the system grouped tickets correctly and where a confirmed pattern actually matters commercially.
The easier feedback is to give during normal work, the more likely people are to provide it. Breaking findings into smaller units also makes annotation less expensive and produces more useful signals for improvement.
—
### 5. Build a Self-Improvement Loop From Human Feedback
After collecting feedback, the next step is turning it into improvements. I recommend building this loop by hand at first. Have domain experts review feedback, decide whether it should change the system, and annotate what the change should look like. These annotations turn team judgment into examples that can later be used to automate parts of the loop.
Automating too early is risky. A generic coding agent doesn’t know the company’s domain, taste, or priorities. By first refining the loop manually, teams can later use coding agents to group feedback, find similar agent runs, and propose updates—scaling human judgment rather than replacing it.
—
### FAQ
**Q: Why not use a chat-based agent for churn detection?**
A: Chat interfaces wait for someone to ask the right question at the right moment. Churn detection must run continuously in the background, starting when evidence arrives, not when someone remembers to ask. Event-driven systems that own the workflow are more effective.
**Q: What does “simplest mechanism that handles it well” mean in practice?**
A: It means using deterministic code for exact rules, embeddings and clustering for semantic similarity, agents for messy evidence and investigation, and humans for final decisions with real commercial consequences. Mixing tools this way keeps the system accurate, understandable, and affordable at scale.
**Q: How does the system group related tickets without using an agent?**
A: It uses embeddings to represent issue summaries numerically and clustering to group them. This is efficient, repeatable, and cheap compared to using an agent for every possible pair.
**Q: Why is human verification designed into the handoff from the start?**
A: Trust comes from traceability and verifiability. Designing the handoff to make verification easy—by showing evidence, explaining reasoning, and separating judgments—ensures people can trust and use the system.
**Q: How should feedback be used to improve the system?**
A: Start with small batches of manually reviewed feedback, annotate what should change and how, then use those examples to gradually automate improvements. Coding agents help scale the process, but human judgment should lead.
—
### Conclusion
In this case study, the agent system turned a prohibitively manual client churn detection workflow into a feasible, ongoing operation at significantly lower costs. But the deeper value is more durable: agent systems capture and compound a company’s operating judgment. That record—what matters, what evidence people trust, and when to act—becomes an asset that grows more valuable over time.
While much attention goes to new model releases, the real competitive advantage belongs to companies that systematize their people’s decisions. The longer such a system runs, the more it becomes its own, uniquely valuable operating capability.
I hope you found this journey into building practical agent systems as insightful as I found it writing it. See you in the next one—sheila
