# CloudWatch Omni: A Comprehensive Guide to Observability for AI Agents and Applications
## Introduction
Modern AI systems are fundamentally different from traditional software. When a user interacts with an agent powered by large language models, the system doesn’t simply return a response — it makes a series of complex decisions, selecting tools, composing prompts, and chaining multiple sub-calls together. Understanding and monitoring these behaviors has become one of the most pressing challenges for engineering teams deploying agentic AI at scale.
Amazon CloudWatch Omni is a next-generation observability platform designed to address these challenges head-on. It provides a unified, app-centric experience for monitoring both traditional applications and AI workloads, leveraging the power of artificial intelligence itself to deliver deeper insights, automated evaluations, and seamless integration into the tools engineers already use.
## The Observability Gap in AI Agent Systems
Traditional monitoring tools focus on system-level metrics such as latency, error rates, and throughput. While these measurements are valuable for conventional applications, they fall short when it comes to AI agents. An agent’s behavior is inherently non-deterministic — a small change in a prompt can significantly degrade response quality even when all standard system metrics appear normal.
Engineering teams often find themselves spending hours manually sifting through logs scattered across multiple systems, struggling to identify what changed and why a particular agent output became unreliable. On top of that, many existing solutions force teams into an uncomfortable choice: either rely on siloed monitoring built specifically for generative AI, or cobble together fragmented tools that require constant switching between a coding environment and a browser-based dashboard.
CloudWatch Omni was built to bridge this gap. It captures every trace generated by an AI agent and includes a rich library of built-in evaluators that assess dimensions such as correctness, coherence, retrieval quality, and tool selection accuracy. This means teams can compare different prompt versions side by side, construct test datasets from real production traffic, run experiments across various configurations, and automatically detect regressions as they emerge.
## Two Surfaces for Development and Operations
One of the defining architectural decisions behind CloudWatch Omni is its dual-surface approach. The platform delivers observability through two complementary interfaces, each tailored to a different stage of the software lifecycle.
### The IDE Experience
Developors work primarily inside their integrated development environments, and CloudWatch Omni brings observability directly into that workflow. A native extension is available for Visual Studio Code and Kiro, allowing developers to view agent traces in real time as they run their applications. The extension includes a built-in playground where agents can be tested and evaluated with a single click. This eliminates the friction of context-switching between the IDE and an external monitoring dashboard.
### The Standalone Web Experience
Operations teams and stakeholders who need to monitor agents in production can access a dedicated web experience that is entirely separate from the AWS Management Console. This browser-based interface supports single sign-on authentication, so teams can investigate issues, review dashboards, and collaborate without needing full console access. The data is shared seamlessly — a trace that a developer debugs during a coding session is the same trace that an operations engineer investigates later.
## Connecting Local Development to the Cloud
CloudWatch Omni includes a feature called Cloud Login, which bridges the local development environment and AWS cloud infrastructure. Once connected, telemetry data flows from the IDE to Amazon CloudWatch for persistent storage, team sharing, and production monitoring. However, this connection is entirely optional. Developers can use the full suite of CloudWatch Omni capabilities locally during the development phase, then establish the cloud connection when the time comes to monitor agents in production.
## Getting Started with CloudWatch Omni
There are two primary paths for onboarding with CloudWatch Omni. The first is through the IDE extension, available for both VS Code and Kiro from their respective marketplaces. The second path goes directly through the cloud experience, where users can begin sending telemetry data to CloudWatch without installing any IDE extension at all.
### Setting Up the Extension
Once the extension is installed, a dedicated CloudWatch Omni icon appears in the Activity Bar of the IDE. From there, users can choose to load a pre-configured sample project that includes an agent implementation and example datasets, or they can create a brand-new agent from scratch. The setup process is guided step by step, including instructions for adding OpenTelemetry instrumentation. Users can also leverage the interactive chat interface to define the agent’s purpose, select a model provider, and configure available tools.
### Running and Tracing Your First Agent
After the development environment is configured, the local dev server can be started. When a question is sent to the agent, selecting the “View Trace” option reveals a detailed breakdown of exactly how the agent processed the request. This is where CloudWatch Omni distinguishes itself from a typical chatbot interface — every decision the agent made is captured and visualized in a structured format.
The platform also integrates with AI-powered code assistants such as Claude Code and Codex, enabling these assistants to configure the development server, install dependencies, and set up instrumentation automatically. This reduces the time from installation to a fully traced agent session to just a few minutes.
## The Trace Explorer: Understanding Agent Behavior
At the heart of CloudWatch Omni is the Trace Explorer, which provides a detailed, hierarchical timeline of every step an agent takes during execution. This includes large language model calls, tool invocations, reasoning steps, and any intermediate actions the agent performs.
Each span in the timeline can be drilled into to inspect detailed inputs, outputs, token consumption, and latency metrics. This level of granularity is critical for understanding agent behavior, because agents make multiple decisions per single invocation — and a problem at any one of those decision points can cascade into incorrect or unexpected outputs.
### Compare Mode and AI-Powered Analysis
The Trace Explorer includes a Compare mode that places two traces side by side, making it straightforward to see how different prompts or configurations affect agent behavior. This is invaluable for debugging regressions and understanding the impact of prompt changes.
Additionally, the Ask Assistant feature leverages an AI agent to analyze traces and surface patterns or anomalies. Users can ask natural-language questions such as “Why did the agent call this tool twice?” and receive contextual insights drawn directly from the trace data.
## Evaluation: Turning Observability into Actionable Insights
Observability without evaluation is incomplete. Traditional metrics like response time and error count cannot tell an engineering team whether an agent’s output was helpful, factually accurate, or logically coherent. CloudWatch Omni addresses this by embedding evaluation directly into the observability workflow.
### Built-In Evaluators
The platform includes 17 pre-built evaluators covering key quality dimensions such as coherence, helpfulness, faithfulness, and routing correctness. Users can select traces from the Trace Explorer, choose the relevant evaluators, and run assessments to receive both per-example scores and aggregate metrics — all without needing to build or maintain a custom evaluation framework.
### The Playground and Experiments
The Playground feature allows teams to test different system prompts and model configurations side by side in real time. This enables rapid iteration and comparison before committing any changes to production.
The Experiments view takes this further by allowing users to run the same dataset against multiple agent variants simultaneously. Evaluation scores, latency, and token usage are compared side by side, providing a clear basis for selecting the best-performing configuration.
### Prompt Management
CloudWatch Omni includes a Prompt Management capability that enables teams to version and track prompt configurations over time. When a new prompt version underperforms, teams can quickly roll back to a previous configuration, reducing the risk of deploying changes that degrade agent quality.
## Session Explorer and Agent Topology
Beyond individual traces, CloudWatch Omni provides tools for understanding agent behavior at a broader scale.
The **Session Explorer** allows teams to review full conversation histories, making it easier to understand how agents handle multi-turn interactions and maintain context across extended dialogues.
The **Agent Topology** view visualizes the architecture of the entire agent system, including sub-agents, tools, and their interconnections. Users can drill into any node in the topology to inspect performance metrics and identify potential bottlenecks in the agent’s workflow.
## The Cloud Experience
CloudWatch Omni also offers a standalone web experience that can be accessed from any browser without requiring an IDE. This is designed for teams that need collaborative access to observability data, application monitoring, analytics, and AI-powered investigations from anywhere in the organization. The web interface brings all of the platform’s capabilities to a familiar browser-based environment.
## Supported Frameworks and Open Standards
CloudWatch Omni is designed to work with the agent frameworks that teams are already using in production. This includes LangChain, LangGraph, CrewAI, the OpenAI SDK, Strands, the Vercel AI SDK, and many others, across both Python and TypeScript environments. The platform also provides native observability for agents built with Amazon Bedrock AgentCore, integrating with Bedrock AgentCore’s evaluation capabilities directly within the Omni workflow.
Instrumentation is built on open standards, including OpenInference and ADOT (AWS Distributed Tracing OpenTelemetry), ensuring compatibility regardless of where agents run — whether on AWS Lambda, ECS, EKS, or other cloud environments. For evaluation, the platform integrates with third-party evaluators such as AutoEval and DeepEval, alongside built-in datasets, a playground, and batch experimentation capabilities. No re-platforming or framework migration is required.
## Pricing and Availability
Amazon CloudWatch Omni is now generally available. The IDE extension is free to use, and an AWS account is not required to get started. Users only need AWS credentials for Amazon Bedrock models or API keys for other providers such as OpenAI or Anthropic.
## Frequently Asked Questions
**Q: Can I use CloudWatch Omni without an AWS account?**
A: Yes. The IDE extension can be used entirely locally without any cloud connection. You only need AWS credentials if you plan to use Amazon Bedrock models or send telemetry data to the cloud for persistent storage and team collaboration.
**Q: Which IDEs are supported by the CloudWatch Omni extension?**
A: The extension is currently available for Visual Studio Code and Kiro. Support for additional IDEs may be added in the future.
**Q: Can CloudWatch Omni monitor agents built with frameworks other than the ones mentioned?**
A: CloudWatch Omni supports a wide range of popular agent frameworks, and its instrumentation is built on open standards (OpenInference and ADOT), which means it can work with many custom or less common frameworks as well.
**Q: What makes the built-in evaluators different from custom evaluation frameworks?**
A: The built-in evaluators cover 17 quality dimensions out of the box, including coherence, helpfulness, faithfulness, and routing correctness. This eliminates the need to design, implement, and maintain a custom evaluation pipeline from scratch.
**Q: How does Compare mode help with debugging?**
A: Compare mode places two traces side by side, highlighting differences in agent behavior, tool usage, and outputs. This makes it easy to identify exactly how a change in prompts, configuration, or model version affected the agent’s performance.
**Q: Is the cloud experience separate from the AWS Management Console?**
A: Yes. CloudWatch Omni’s web experience is a standalone interface accessible through a browser with SSO authentication. It does not require access to the AWS Management Console.
**Q: How does Prompt Management help with production stability?**
A: Prompt Management allows teams to version, track, and roll back prompt configurations over time. This ensures that if a new prompt version introduces unexpected behavior, teams can quickly revert to a previous working version.
## Conclusion
CloudWatch Omni represents a significant evolution in how engineering teams observe and improve AI agent systems. By combining comprehensive tracing, built-in evaluation, real-time experimentation, and seamless IDE integration into a single platform, it addresses the unique observability challenges that agentic AI introduces. Whether you are a developer debugging a single agent interaction locally or an operations engineer monitoring a fleet of agents in production, CloudWatch Omni provides the tools needed to understand, evaluate, and optimize AI-driven workflows. Its support for open standards and existing agent frameworks ensures that teams can adopt it without disrupting their current architecture.
Thank you for reading
