**Building a Multi-Cluster MongoDB Deployment on Kubernetes for High Availability**
Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build a **multi-cluster MongoDB deployment on Kubernetes** that can withstand those failures. We will use the Percona Operator for MongoDB, an open-source Apache 2.0 Licensed Kubernetes Operator, as our example. For relational workloads, similar patterns are available through projects like Vitess and CloudNativePG.
If you are new to multi-cluster MongoDB on Kubernetes, Ivan Groenewold’s post is a great place to start. It walks you through a complete multi-cluster setup you can follow hands-on.
—
### **Single Point of Failure: The Risk of a Single-Cluster Setup**
Standard Kubernetes excels at self-healing within a single cluster, automatically recovering from crashed Pods or failed Nodes. However, it has no built-in mechanism to handle failures at the cluster level itself: *a regional outage, a corrupted control plane, or a network partition can take down your entire database with no automatic recovery path.*
Distributing MongoDB nodes across separate Kubernetes clusters addresses this gap and enables three core scenarios:
– **Disaster Recovery (DR)**: If an entire region goes down, nodes in a secondary cluster already hold a full copy of the data and enough votes to elect a new Primary and resume writes automatically.
– **Live Migrations**: Running nodes across two clusters lets you shift application traffic gradually between environments, for example, during a cloud provider migration, without taking the database offline. *(Note that coordinating a clean cutover still requires careful application-level planning around connection strings and write consistency.)*
– **Maintenance without Stopping**: You can drain and upgrade one cluster entirely while the database continues accepting writes through the nodes in the remaining cluster, with no scheduled downtime.
**Architecture Overview**
To manage the complexity of multi-cluster deployments, the architecture divides clusters into two roles, ensuring that Kubernetes-level operations (handled by the Operator) and database-level operations (handled by MongoDB) do not interfere with each other. All nodes, regardless of which cluster they run in, **belong to a single MongoDB replica set**, which is what enables cross-cluster voting and leader elections.
#### **1. Defining Site Roles**
Each cluster is assigned to one role to prevent conflicts between independent Kubernetes control planes:
– **Main Site**: The primary cluster, fully managed by the Operator. It holds the Primary node and handles application writes. The Operator here is responsible for provisioning TLS certificates, user credentials, and replica set configuration.
– **Replica Site**: The Operator runs with *unmanaged* mode to *true*, which means it does not generate certificates or user credentials, and does not attempt to initialize a new replica set. Instead, the user needs to copy **TLS secrets and credentials from the Main Site**, allowing the Replica Site nodes to authenticate and join the existing replica set. This prevents the problem where two independent operators attempt to control the same database simultaneously.
#### **2. Connecting Clusters with the MCS API**
Even though each Kubernetes cluster runs an independent control plane, the MongoDB replica set spans across all of them. The Kubernetes Multi-Cluster Services API (MCS API) makes this possible. When `multiCluster.enabled: true` is set, the Operator creates `ServiceExport` and `ServiceImport` resources so nodes in different clusters can discover and communicate with each other via a shared DNS zone (`svc.clusterset.local`).
> **Note** that the MCS API is not included in a standard Kubernetes installation and requires a separate implementation such as Submariner, Cilium ClusterMesh, or a managed provider solution like GKE’s native MCS.
To enable multi-cluster service discovery with the Percona Operator for MongoDB, set `multiCluster.enabled: true` in your `cr.yaml`:
“`yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
name: main-cluster
spec:
crVersion: 1.23.0
image: perconalab/percona-server-mongodb-operator:main-mongod8.0
imagePullPolicy: Always
updateStrategy: SmartUpdate
multiCluster:
enabled: true # <--- Enable multi-cluster service discovery
DNSSuffix: svc.clusterset.local # <--- The Global DNS standard
```After applying the configuration, verify that the Operator created the `ServiceExport` resources. Note that it takes approximately five minutes for resources to sync across the fleet. This is how it looks:```bash
kubectl get serviceexport
NAME AGE
main-cluster-rs0 6m
main-cluster-rs0-0 6m
main-cluster-rs0-1 5m
```The **Main Site** (Cluster A) runs the Primary node and is fully managed by the Operator. The **Replica Site** (Cluster B) holds Secondary nodes in `unmanaged` mode. `ServiceExport` and `ServiceImport` resources bridge the two clusters via a shared `svc.clusterset.local` DNS zone, enabling cross-cluster replica set membership.---### **How to Design for High Availability**Before choosing how to distribute clusters, it helps to understand how MongoDB itself decides which node is in charge.A MongoDB replica set requires a strict majority of its voting members to elect a Primary and accept writes. With four voting members spread equally across two locations, neither side can reach a majority if the network between them is severed, and both sides stop accepting writes, even if every server is healthy.The solution is a fifth member, placed in a third location. With 5 total votes, one side reaches 3 votes; a majority, and elects a new Primary. This is the **2+2+1 pattern**.**Image 2: The following image is in a Normal State with no Network Partition***(Image referenced in original post)***Image 3: After Network Partition***(Image referenced in original post)*Have an odd number of MongoDB voting members, placed in different locations, so that if one location or network link fails, one side can still have enough votes to keep/elect a Primary.With the Percona Operator for MongoDB, the 2+2+1 pattern is configured directly in `cr.yaml`. Start by defining the total number of data-bearing replica set members across both sites:```yaml
replsets:
- name: rs0
size: 2 # 2 nodes in Main Site + 2 nodes in Replica Site
expose:
enabled: true
type: ClusterIP
```…Then add the extra member as a separate entry under the same `externalNodes` section:```yaml
externalNodes:
- host: third-site-rs0-4.psmdb.svc.clusterset.local
votes: 1
priority: 1
```---### **Deployment Workflow**The following steps deploy the 2+2+1 pattern across two Kubernetes clusters using the Main and Replica site roles described above.1. **Connect the Networks**: Set up cross-cluster network connectivity using Cilium ClusterMesh, Submariner, or your cloud provider’s native solution (such as GKE Fleet). This is a prerequisite; the MCS API requires an implementation to be installed before the Operator can create functional `ServiceExport` and `ServiceImport` resources.2. **Deploy the Main Site**: Deploy the Operator and apply your `cr.yaml` on the primary cluster. Ensure replica set pods are exposed with `type: ClusterIP`, this is required by the **MCS API** since nodes communicate internally across clusters. See the Main Site configuration guide.3. **Copy the Secrets**: The Main and Replica sites must share the same TLS certificates and user credentials to communicate. Run `kubectl get secrets` on the Main Site to confirm the exact names in your deployment.4. **Deploy the Replica Sites**: Start the remote clusters, but remember to set their operators to **“unmanaged” mode**. This tells the nodes to join the existing database instead of trying to create a new one.```yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
name: replica-cluster
spec:
# ----------------------------------------------------
# This tells the Operator NOT to start a new database,
# but to join the existing one to prevent split-brain.
# ----------------------------------------------------
unmanaged: true
multiCluster:
enabled: true
DNSSuffix: svc.clusterset.local
```5. **Finalize the Replica Set**: Update `cr-main.yaml` on the Main Site to add the Replica Site nodes as `externalNodes`. Two nodes are added as voting members with lower priority, and the third as a non-voting member, this prevents split-brain if the Replica Site loses connectivity.```yaml
replsets:
- name: rs0
size: 2
externalNodes:
- host: replica-cluster-rs0-0.psmdb.svc.clusterset.local
votes: 1
priority: 1
- host: replica-cluster-rs0-1.psmdb.svc.clusterset.local
votes: 0
priority: 0
```Then apply: `kubectl apply -f deploy/cr-main.yaml.`Repeat this step on the Replica Site, adding the Main Site nodes as `externalNodes` in `cr-replica.yaml`. Please check the interconnect guide if you want to see a full configuration.```yaml
replsets:
- name: rs0
size: 2
externalNodes:
- host: main-cluster-rs0-0.psmdb.svc.clusterset.local
votes: 1
priority: 1
- host: main-cluster-rs0-1.psmdb.svc.clusterset.local
votes: 0
priority: 0
```Once these steps are complete, the replica set spans both clusters and is ready for the failover behavior described in the next section.---### **Failover Behavior: The Election Process**MongoDB’s replica set protocol includes built-in failure detection and automatic leader election. If the Primary node goes offline, whether a single pod crashes or an entire region is down, the remaining nodes automatically elect a new Primary without any operator or human intervention.Let’s look back at our 2+2+1 setup. To stay online, the database needs a strict majority (3 out of 5 votes). If Cluster A completely fails, the two nodes in Cluster B plus the node at Cluster C give us exactly the 3 votes we need to choose a new Primary.Under the default configuration, the median time before a cluster elects a new primary does not typically exceed 12 seconds; this includes the time to detect the failure and complete the election. In a multi-region setup, cross-cluster network latency may extend this window, so your application connection logic should include **retryable writes** to handle the brief period during which no Primary is available. The `electionTimeoutMillis` setting can be tuned if your latency requirements demand faster detection.---### **Architecture After Failover**After a regional failure, your database is still working, but it enters a “degraded” state. This means it is functioning, but with fewer resources than normal.- **The New Primary**: A node in Cluster B is promoted to be the new leader and starts accepting new data.
- **Less Redundancy**: Your database is now running with the exact minimum number of nodes needed for a majority. In our 2+2+1 setup, you are surviving on just those 3 remaining members.
- **Reduced Fault Tolerance**: Even though your application survived the crash of Cluster A, your safety margin is completely gone. *If you lose just one more node in Cluster B or the node at Cluster C, the database will lose its majority and lock into “read-only” mode.***Image 4: Failover in a multi-cluster MongoDB deployment using the Kubernetes MCS API***(Image referenced in original post)*The replica set detected the failure, held an election, and promoted a new Primary, automatically, without human intervention. That is the 2+2+1 pattern working as designed. In a follow-up post, we will put this architecture under deliberate pressure with chaos engineering experiments to measure exactly how it holds under real failure conditions.This blog post was reviewed by **Chetan Shivashankar**, Technical Lead, Kubernetes at Percona.Here are some resources you can review to learn more about the topic.---## **Frequently Asked Questions (FAQ)**### **1. What is the 2+2+1 pattern in MongoDB multi-cluster deployments?**
The 2+2+1 pattern refers to distributing MongoDB voting members across three locations: two nodes in the main site, two nodes in a replica site, and one additional node in a third location. This configuration ensures that in the event of a network partition or regional failure, the remaining nodes can still form a majority (3 out of 5 votes) and elect a new Primary, maintaining high availability.### **2. What is the role of the Percona Operator for MongoDB in multi-cluster setups?**
The Percona Operator for MongoDB automates the deployment, management, and scaling of MongoDB clusters on Kubernetes. In multi-cluster deployments, it enables multi-cluster service discovery via the MCS API, manages TLS certificates and user credentials, and allows nodes in different clusters to join a single replica set while preventing split-brain scenarios through unmanaged mode on replica sites.### **3. What is unmanaged mode in the Percona Operator?**
Unmanaged mode is a configuration where the Operator on a replica site does not create new resources such as certificates, users, or a new replica set. Instead, nodes on that cluster join an existing replica set managed by the Operator on the main site. Unmanaged mode is essential for multi-cluster setups to prevent multiple operators from attempting to control the same database.### **4. What is the MCS API, and why is it required for multi-cluster MongoDB?**
The Kubernetes Multi-Cluster Services (MCS) API enables service discovery and connectivity across multiple Kubernetes clusters. For multi-cluster MongoDB deployments, the Operator uses the MCS API to create `ServiceExport` and `ServiceImport` resources, allowing MongoDB nodes in different clusters to discover and communicate with each other via a shared DNS zone (e.g., `svc.clusterset.local`).### **5. How does failover work in a multi-cluster MongoDB deployment?**
MongoDB’s replica set protocol automatically detects failures and elects a new Primary. In a 2+2+1 configuration, if the entire main site fails, the two nodes in the replica site plus the one node in the third location can still form a majority and elect a new Primary. This failover happens automatically without human intervention, typically within seconds.### **6. What happens after a failover in a degraded state?**
After a failover, the database continues operating with fewer nodes—specifically, the minimum number needed for a majority (3 nodes in a 2+2+1 setup). While the database remains functional, its fault tolerance is reduced. If another node fails, the database may lose its majority and enter read-only mode.### **7. What are the prerequisites for deploying a multi-cluster MongoDB setup?**
Key prerequisites include:
- Kubernetes clusters with network connectivity between them
- Implementation of the MCS API (e.g., via Submariner, Cilium ClusterMesh, or cloud provider solutions)
- Proper configuration of the Percona Operator with multi-cluster enabled
- Sharing of TLS certificates and user credentials between the main and replica sites
- Careful planning of the 2+2+1 voting member distribution---## **Conclusion**Building a multi-cluster MongoDB deployment on Kubernetes is a powerful way to achieve high availability and resilience against regional failures. By leveraging the Percona Operator for MongoDB and the Kubernetes Multi-Cluster Services API, teams can design architectures that ensure automatic failover, minimize downtime, and provide flexible options for disaster recovery, live migrations, and maintenance without interruption.Key takeaways include:
- Avoid single points of failure by distributing MongoDB nodes across multiple clusters and regions
- Use the 2+2+1 voting pattern to maintain majority during network partitions
- Carefully manage cluster roles and secrets to prevent split-brain scenarios
- Rely on MongoDB’s built-in election protocols for automatic failover
- Prepare for degraded states and plan for recoveryThis approach represents a best practice for running stateful databases at scale in Kubernetes environments, enabling organizations to meet strict uptime and reliability requirements. As Kubernetes and multi-cluster technologies continue to evolve, these patterns will become increasingly vital for modern cloud-native applications.
