# ZGateway: How a Proxy Layer Revolutionized Access to One of the World’s Largest Key-Value Stores
## Introduction
At massive scale, even the most robust distributed databases face architectural challenges that go beyond raw storage and retrieval speed. When millions of client applications each need to communicate directly with thousands of database shards distributed across hundreds of thousands of hosts, the resulting connection topology can become a systemic vulnerability. A engineering team at a major technology company recently introduced ZGateway, a proxy tier positioned between client applications and ZippyDB — a high-throughput key-value store that supports product metadata, counters, and configuration across billions of operations every second.
What began as a targeted fix for connection sprawl evolved into a full-featured infrastructure layer capable of batching requests, enforcing admission control, caching hot data, and managing failover across regions. This article explores the problem that motivated ZGateway, how it works, what capabilities it introduced, and what lessons can be drawn from its deployment.
—
## The Connection Crisis: Why a Proxy Became Necessary
In a direct-access architecture, every client establishes and maintains connections to every database host it needs. For a single client touching tens of thousands of shards, this means tens of thousands of Transport Layer Security (TLS) connections. Both the client and the database host bear the cost of each idle connection — consuming memory, CPU cycles, and file descriptors on both ends of the link.
The problem compounded with every new client cohort. As the number of clients grew, inbound connection counts on each database host ballooned proportionally. Reconnection storms, sometimes triggered by routing bugs, led to cascading failures. In one documented incident, a misconfiguration caused every client to open a separate connection per shard, driving the entire fleet into a reboot loop as file descriptors were exhausted and out-of-memory conditions triggered across hosts.
Client-side fixes were considered impractical because the client fleet was maintained by hundreds of different engineering teams, each with their own release cycles and constraints. A centralized intermediary was the only viable path forward.
—
## What Is ZGateway?
ZGateway is a stateless proxy tier that sits between client applications and the ZServer database fleet. It is deployed as regional tiers discovered through a service mesh, and it operates in two modes: a pure proxy mode for straightforward request forwarding and a read-through cache mode that serves hot reads from local in-process memory.
The engine powering ZGateway is the same thick C++ client library used by ZippyDB itself, meaning ZGateway is essentially a managed instance of the ZippyDB client. This design choice keeps replica selection logic, shard resolution, and TLS termination consistent with the existing client stack while centralizing connection management at the proxy layer.
When a client sends a request, it flows over a sticky connection to a regional ZGateway host. The gateway terminates TLS, authorizes the request against access control lists scoped to the use case, applies per-tenant admission control and traffic shaping, resolves the target shard, checks a local cache (on caching tiers), batches the request alongside other in-flight work targeting the same shard, and forwards it to the correct replicas. Responses are then demultiplexed back to the originating client, with per-use-case metrics, distributed traces, and quota usage recorded throughout the process.
—
## The Math Behind the Architecture
The engineering team modeled the fleet as a balls-and-bins problem: with B shards and H hosts, the probability that any given host receives a connection from a single client follows a well-known probabilistic distribution. Using mock figures based on Meta’s deployment — 20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, and 50,000 shards per client — per-host connection counts collapsed by approximately 97 to 98 percent. Total persistent connections across the fleet dropped roughly 19-fold.
The deeper architectural win is in scaling behavior. Under direct access, fan-in grows linearly with the number of clients. With ZGateway, fan-in reduces to approximately the number of regions multiplied by the shard density per host, making it independent of both the client fleet size and the proxy fleet size. This decoupling is what allows the system to absorb continuous growth in client applications without proportionally increasing the load on database hosts.
ZGateway currently handles more than one billion operations per second and carries approximately 40 percent of ZippyDB traffic, with projections to pass 60 percent in the near future. The computational overhead for an average use case sits at about 6 percent.
—
## Capabilities That Emerged from the Proxy Layer
Beyond solving the connection problem, ZGateway became the home for several production-grade capabilities that would have been difficult or impossible to deploy at the client level.
### Safe Migration and Configuration Rollouts
Configuration flags scoped per service and shard prefix enable gradual percentage ramps, region-level filters, and global kill switches. This allows teams to roll out changes to a subset of traffic, observe behavior, and instantly revert if necessary — all without coordinating across the hundreds of teams that own client implementations.
### Discriminant Load Shedding (DLS)
DLS maps incoming requests to per-tenant buckets split by priority level, with each bucket drained on a round-robin basis. When a single tenant begins flooding the system, only its own bucket fills and overflows. In a controlled stress test conducted at over 90 percent CPU utilization across roughly 1,350 tenant buckets, only six noisy neighbors had their requests shed. The remaining tenants experienced zero rejections, and overall goodput held steady at 97 to 98 percent. The DLS machinery itself consumed approximately 8 percent of CPU.
### Read Caching
Cache tiers serve frequently accessed reads directly from in-process memory. On a cache miss, a per-key fill lock ensures that only one request fetches the data from the backend while other concurrent requests wait. Cache freshness is maintained through change-data-capture events under a bounded-staleness contract, ensuring that cached data does not diverge significantly from the source of truth.
### Adaptive Load Balancing
ZGateway tiers mix hosts ranging from 26 cores to 126 cores. A control-plane balancer continuously adjusts each host’s ServiceRouter weight in the opposite direction of its recent CPU utilization, ensuring that heavier hosts receive proportionally less traffic while lighter hosts absorb the excess.
### Cross-Region Resilience
Global routing, mega-regions, and ring-based topologies allow a saturated regional tier to fail over to healthy capacity in nearby regions. This provides geographic redundancy without requiring client-side awareness of regional topology changes.
### Transaction Support
Client-side bookkeeping that was previously scattered across application logic has been consolidated within the gateway. The team reduced transaction management to nine distinct phases and rolled it out to 100 percent of transaction traffic with no reliability regressions.
—
## Lessons and Implications
ZGateway demonstrates that a well-designed proxy layer can transform the scalability characteristics of a distributed database system. The key insight is that fan-in — the number of connections any single database host must sustain — can be decoupled from the growth of the client population by introducing a bounded intermediary tier.
The cross-client batching mechanism is particularly notable because it eliminates hot-key stampedes without requiring application-level coordination. When multiple unrelated clients request the same key within the same shard simultaneously, the gateway coalesces those requests into a single backend fetch, serving all callers from one result.
It is also worth noting that ZGateway is not a product that can be simply deployed outside its originating environment. The real value lies in the architectural patterns and engineering principles it embodies: centralized connection management, tenant-isolated load shedding, in-process caching with coherence guarantees, and adaptive traffic shaping. These patterns can inform proxy designs in other large-scale systems regardless of the specific technology stack.
—
## Frequently Asked Questions
**What problem does ZGateway solve?**
ZGateway solves the connection sprawl problem that arises when millions of client hosts each maintain tens of thousands of direct connections to database shards. It collapses these connections into a bounded pool of proxy-to-database links, dramatically reducing resource consumption and eliminating failure modes caused by file descriptor exhaustion.
**How much overhead does ZGateway introduce?**
For an average use case, ZGateway adds approximately 6 percent computational overhead while handling over one billion operations per second.
**What is Discriminant Load Shedding and how does it work?**
Discriminant Load Shedding is a mechanism that isolates traffic by tenant into dedicated priority-based buckets. Each bucket drains independently on a round-robin schedule. When a tenant floods the system, only its own bucket overflows and sheds requests, leaving all other tenants unaffected.
**Can ZGateway be used as a standalone product?**
ZGateway is tightly integrated with the ZippyDB client library and Meta’s internal service mesh infrastructure. However, the architectural patterns — connection pooling, tenant-aware load shedding, cross-client batching, and adaptive load balancing — can be adapted as design principles for similar proxy layers in other systems.
**How does the read-through cache ensure data freshness?**
The cache tier uses change-data-capture events to stay synchronized with the database. A bounded-staleness contract ensures that cached data does not fall too far behind the source of truth, and per-key fill locks prevent redundant backend fetches on cache misses.
**What role does the service mesh play in ZGateway’s architecture?**
Meta’s service mesh, called ServiceRouter, is responsible for discovering and routing traffic to regional ZGateway tiers. It also manages TLS termination within the stack and enables the control-plane balancer to adjust host weights dynamically.
—
## Conclusion
ZGateway represents a significant evolution in how large-scale key-value stores can be accessed at hyperscale. By inserting a stateless proxy tier between clients and databases, the engineering team resolved a critical connection management problem, unlocked powerful new capabilities like tenant-isolated load shedding and cross-client request batching, and created a foundation for safe, gradual configuration changes across a fleet maintained by hundreds of teams.
The project illustrates that sometimes the most impactful infrastructure improvements come not from optimizing the database itself, but from rethinking how clients interact with it. As distributed systems continue to grow in scale and complexity, proxy layers like ZGateway offer a proven blueprint for managing connection density, isolating failures, and maintaining high throughput under adverse conditions.
Thank you for reading
