# How a Major Engineering Organization Replaced Its Legacy Metrics Pipeline with OpenTelemetry Without Dropping a Beat
## The Legacy Problem No One Wanted to Fix
For years, one engineering organization relied on a homegrown, open-source StatsD implementation to power its entire metrics infrastructure. The system ran as a sidecar on every host and an aggregation tier at the backend, ingesting data from roughly 100,000 hosts spread across 14 geographic regions. It operated at a 99.95% service-level objective with minimal latency, and it worked so well that almost nobody thought about it. In infrastructure, that quiet reliability is usually a sign that things are running exactly as they should.
But reliability has a shelf life.
Over the course of a few years, the observability community began standardizing heavily on OpenTelemetry (OTel). New instrumentation libraries, exporters, and collector components were being shipped at a rapid pace — all designed around the OTel data model. The legacy StatsD system, by contrast, was fundamentally limited: it was UDP-only, it had no support for traces or logs, and it couldn’t ingest the growing volume of OTel-native telemetry that was now being generated by services across the organization.
The team faced a dilemma. The obvious approach — rip out the old pipeline, ask every engineering team to re-instrument their services using the OTel SDK, and flip a switch — was essentially impossible. It would require a multi-year, organization-wide effort on a pipeline that couldn’t tolerate an outage, with a real risk of losing the exact metrics that alerts depend on.
So the team asked a sharper question: how do you replace the entire engine of a mission-critical metrics pipeline without causing widespread disruption?
## The Core Insight: Preserve the Contract, Rebuild the Engine
The breakthrough came from reframing the problem. A metrics pipeline is fundamentally a contract between two sides. On one side is the service owner: “Send StatsD over UDP to this address, and your metrics will show up in the backend.” On the other side is everything that happens between the packet arriving and the data reaching long-term storage.
Service teams care deeply about the first part — the interface they interact with every day. They care very little about the middleware. So the team decided to keep the interface completely intact and rebuild everything behind it. This transformed what would have been an organization-wide migration into a platform-team migration.
With that decision locked in, the team structured the rebuild around four distinct stages, each running its own purpose-built OpenTelemetry Collector distribution. This modular approach meant that work on one stage could proceed independently without affecting the others. Additionally, the collection layer was configured to speak both StatsD and OTLP simultaneously, meaning service teams didn’t need to swap their existing StatsD client libraries for the OTel SDK before the migration could begin.
The team also benefited from prior experience: their tracing team had already been running the OTel Collector in production at scale for years, so the question of whether it was production-ready had already been answered affirmatively.
## Stage-by-Stage: How the Migration Was Executed
### Collection
The first step was replacing the gostatsd sidecar on every host with an OTel Collector distribution. Crucially, the application side remained completely unchanged — services continued sending StatsD over UDP exactly as before. On day one, no engineering team noticed any difference.
The hidden win here was operational consolidation. Previously, many hosts were running two sidecars: one for StatsD metrics and another for tracing. By folding metrics collection into the existing tracing sidecar and retiring the dedicated StatsD sidecar, the team saved approximately 3.9% CPU on average per service across their most resource-intensive microservices. At fleet scale, this translated to roughly a 30% reduction in sidecar compute cost. At the same time, the new collection layer gained the ability to receive and forward OTel metrics natively via OTLP.
### Ingest
Aggregation is a stateful process — every datapoint belonging to a given time series must hit the same aggregator, which means traditional load-balancing strategies don’t work. For years, an in-house proxy had handled routing by hashing the combination of service name and environment to a specific shard. However, the team discovered that their service-to-metric load distribution was highly non-uniform and followed a long-tail pattern. The shards assigned to the largest services became overloaded hot spots, while smaller shards sat idle.
The fix came from adopting a smarter routing strategy: hashing by streamID — the unique identity of an individual time series — rather than by service name. This meant that even a single large service would have its time series spread evenly across the entire pool of aggregators, while any given series would always land on the same shard. After implementing this change, per-shard CPU utilization transformed from a handful of tall spikes with idle replicas into a flat, even distribution. Even load meant tighter autoscaling bands, meaningful off-peak scale-down, and an end to hot-shard incidents.
### Aggregation
This stage is where the numbers become affordable. The pipeline was taking in approximately 4.8 billion datapoints per minute and reducing that to roughly 220 million — a 96% reduction. Most of the metrics used delta temporality, and no upstream component had been aggregating those deltas in the way that end users expected. To solve this, the team built a custom delta aggregation processor and open-sourced it.
The results were striking. The aggregation tier, processing the same volume of traffic, now runs on approximately half the CPU it previously required. The gains came from three sources: the aggregators no longer needed to parse the legacy StatsD protocol, the load was evenly distributed across shards, and the team inherited performance tuning from the broader OTel Collector community.
### Forward
The final stage — forwarding processed metrics to downstream backends — was the simplest to migrate. A previously bespoke internal forwarder was replaced with a stateless OTel Collector distribution (called metrics-gateway), built on top of upstream community exporters. It supports fan-out to multiple backends, including commercial observability platforms and cloud storage, without requiring any custom backend integrations. Features like retries, queuing, and backpressure come from the community by default. Adding a new destination is now a configuration change, not a new project.
### Lambda
Serverless workloads present a unique challenge — you can’t run a sidecar on a Lambda function. The team addressed this by building an OTel Lambda extension that replaced the gostatsd sidecar for serverless functions. It maintained the same StatsD address, the same environment variables, and required zero code changes from the functions it served. This completed the collection layer, ensuring that metrics from serverless services flowed into the pipeline seamlessly.
## Operational Lessons Learned
The team shared four principles that guided them through a migration of this magnitude:
**Start with the right early adopters.** Identify teams that have the most to gain and are willing to iterate with the platform team. Beginning with development and staging workloads, and prioritizing the services that felt the most pain, provided fast, actionable signal from people who were forgiving while rough edges were smoothed out.
**Profile continuously in production.** The real cost and behavior of any component — whether built in-house or drawn from the open-source community — only reveals itself under genuine production load. Small benchmarks and tests were insufficient; continuous profiling in the live environment was what actually pointed the team toward optimization opportunities.
**Match existing operational workflows.** A migration this large runs for months or even years, and for most of that duration, both the old and new systems must operate side by side. Keeping the operational overhead of running two systems as low as possible means carrying the same operational primitives across both and maintaining parity so that no one needs to learn a second way of doing things.
**Use progressive rollouts.** Begin in lower environments and with less critical services, then gradually increase exposure — 1%, 10%, 50%, 100%. The goal is to discover problems where they’re cheap to fix, not on the most critical production path.
## The Bigger Picture and What Comes Next
Running the OTel Collector at every stage of the pipeline gives the team a single codebase and a single operational model. Extending the pipeline means writing a new component rather than standing up a new service. This architecture unblocks OTel-based instrumentation without sacrificing the aggregation and cost controls that make the organization’s scale financially sustainable. It also allows wasteful datapoints to be dropped at ingest — the cheapest possible place to do so.
The financial case is compelling. The legacy gostatsd aggregators and the in-house routing proxy (nomad) together accounted for roughly 38% of CPU requests in the metrics clusters. Nomad alone consumed about 13% of total resources. Removing them frees up significant infrastructure capacity and represents real cost savings — and it closes the gap between the current state and a fully OpenTelemetry end-to-end pipeline.
Looking ahead, the team has unblocked their internal users to begin migrating their metrics instrumentation to OpenTelemetry. The next phase involves shifting the instrumentation itself off vendor-specific and in-house client libraries — such as Datadog’s DogStatsD and legacy StatsD implementations — and onto the OTel SDK directly. The team also plans to explore deeper into the OTel ecosystem to tackle additional large-scale observability challenges, contributing improvements back to the community as their usage and scale grow.
## Frequently Asked Questions
**Why was replacing gostatsd necessary if it was working well?**
The legacy system was reliable but fundamentally limited. It was UDP-only, couldn’t handle traces or logs, and lacked support for the OTel data model that the broader observability ecosystem had standardized on. As more services began emitting OTel-native telemetry, maintaining a parallel system became unsustainable.
**How did the team avoid disrupting service teams during the migration?**
By keeping the interface unchanged. Service teams continued sending StatsD over UDP to the same address. The collection layer handled both the old StatsD protocol and the new OTLP protocol simultaneously, so no code changes were required on the application side during the initial phase.
**What is streamID-based routing and why does it matter?**
StreamID is the unique identity of an individual time series. Instead of routing metrics by service name (which caused hot shards when large services dominated), hashing by streamID spreads a single service’s time series evenly across all available aggregators while still guaranteeing that any given series always lands on the same shard.
**How much compute was saved by retiring the legacy sidecar?**
The team saved approximately 3.9% CPU on average per service across their most expensive microservices, which translates to roughly a 30% reduction in sidecar cost at fleet scale.
**What is the delta aggregation processor and why was it needed?**
Most metrics in the pipeline use delta temporality, meaning they represent changes since the last measurement. The legacy system couldn’t aggregate deltas the way end users expected, so the team built a custom processor to handle this correctly and open-sourced it for the community.
**How did serverless functions fit into the migration?**
Serverless functions can’t run traditional sidecars. The team built an OTel Lambda extension that maintained the same StatsD address and environment variables, requiring no code changes from the functions while seamlessly forwarding their metrics into the pipeline.
**What are the next steps for the team?**
The immediate next step is moving instrumentation off vendor-specific and legacy client libraries onto the OTel SDK. The team also plans to contribute more heavily to the OpenTelemetry ecosystem as they encounter and solve large-scale observability challenges at their scale.
## Conclusion
Migrating a metrics pipeline at scale is rarely about the technology — it’s about minimizing risk while maximizing incremental value. This organization’s approach of preserving the developer-facing interface, rebuilding the backend in modular stages, and leveraging the OpenTelemetry Collector as a unified platform provides a compelling blueprint for any team facing a similar challenge. The result is a pipeline that is cheaper to run, easier to extend, and aligned with the broader observability ecosystem — all achieved without a single service interruption.
Thank you for reading
