**Title:** Mastering Multi‑Region Kubernetes with Linkerd: Federation, Flat Mirroring, and Gateway Mirroring
Modern applications increasingly span multiple regions and clusters to meet latency, resiliency, and data‑locality goals. But as soon as you move beyond a single cluster, you face the same set of hard questions: *How do I make a service available everywhere? How do I route traffic in the event of a region outage? And how do I let a client pick a specific cluster when it needs to?*
Linkerd’s multicluster extension provides a unified way to connect services across clusters using three complementary patterns—federated, flat (pod‑to‑pod), and gateway mirroring. In this post, we’ll wire all three across three GKE clusters, deploy demo services using each mode, verify behavior, and test automatic failover when an entire cluster disappears.
—
### Linkerd Multicluster Modes: Gateway, Flat, and Federated
Linkerd multicluster supports three modes, and on a real platform you’ll likely use all three. The mode for a given service is selected via a label, so the same clusters can host federated, flat, and gateway‑mirrored services side by side.
| Mode | Label | What happens | Network requirement |
|——|——-|————–|———————|
| **Hierarchical (gateway)** | `mirror.linkerd.io/exported=true` | Service mirrored as `
| **Flat (pod‑to‑pod)** | `mirror.linkerd.io/exported=remote-discovery` | Service mirrored as `
| **Federated** | `mirror.linkerd.io/federated=member` | All same‑name services unioned into `
The distinction that matters operationally is simple:
– **Flat and federated modes** require pod‑to‑pod reachability across clusters. On GCP, VPC‑native GKE clusters on peered VPCs provide this for free.
– **Gateway mode** works on any network, as long as the gateway IP is reachable.
Most platform teams end up with a mix: core workloads federated for automatic failover, specialized services mirrored flat for region‑aware routing, and a few gateway‑mirrored services when flat connectivity isn’t available.
—
### Multi‑Region Architecture: GKE Cluster Setup
Our demo uses three GKE clusters across three regions (west, east, north), fully linked with a full‑mesh of six directional links. Three demo services illustrate each mode:
– **frontend** – Federated, deployed in all three clusters. A single federated service load‑balances across all nine pods (3 replicas × 3 clusters). When a cluster disappears, traffic automatically shifts to the remaining pods.
– **api** – Flat‑mirrored in `west` and `east`. The `north` cluster consumes them as `api-west` and `api-east`, allowing clients to pick a specific backend (for example, to keep requests in‑region).
– **analytics** – Gateway‑mirrored, running only in `east`, exposed through a Linkerd gateway so consumers in `west` and `north` can reach it without flat network connectivity.
The full diagram shows six directional links and the per‑cluster Linkerd control plane, with a gateway in `east` to support hierarchical mirroring.
—
### Deployment Prerequisites: GKE, Linkerd, and CLI Tools
Before you begin, ensure you have:
– A GCP account (free‑tier credits cover this; small node pools keep costs low)
– `gcloud` CLI, authenticated
– `kubectl` v1.28+
– `step` CLI (for certificate generation)
– `helm` v3
– ~30 minutes for the full setup
The provided infrastructure script enables the required APIs, so a brand‑new project works out of the box.
—
### Step 0: Configure
Clone the companion repository, copy the example environment file, and set at minimum your `GCP_PROJECT`. Everything else ships with sensible defaults for three regions, one zone per region, and small node pools to control cost.
“`bash
git clone
cd blog-linkerd-federation
cp env.example .env
# Edit .env and set GCP_PROJECT
source .env
“`
Every script reads from this file and fails loudly on missing variables (`set -euo pipefail`), so the defaults in `env.example` include the full set of VPC and cluster names.
—
### Step 1: Provision Three GKE Clusters with VPC Peering
Run the infrastructure script. This (~10–15 minutes) enables APIs, creates three non‑overlapping VPCs, establishes full‑mesh peering with custom route export/import, and creates three GKE Standard clusters:
“`bash
./scripts/01-infra.sh
“`
**Important notes:**
– Use non‑overlapping pod and service CIDRs. Overlapping ranges break peering silently.
– We pin `–node-locations` to a single zone so `CLUSTER_NODE_COUNT=1` equals one node per cluster, keeping costs down.
– The script sets kubectl contexts to `west`, `east`, and `north`.
—
### Step 2: Install Linkerd with a Shared Trust Anchor
Install Linkerd into all three clusters using a shared root CA and per‑cluster issuer certificates. This enables cross‑cluster mTLS:
“`bash
./scripts/02-linkerd-install.sh
“`
The script generates a shared root CA (`root.crt`) and per‑cluster issuers. A shared root is a production best practice: if one issuer is compromised, you can rotate it without affecting the others.
—
### Step 3: Install Multicluster Components and Create a Full‑Mesh Link Topology
Set up the multicluster components and six directional links (every cluster to every other):
“`bash
./scripts/03-multicluster-setup.sh
“`
Each cluster gets a set of controllers per link it consumes. For example, `west` and `north` each run an extra controller for the gateway link to `analytics` in `east`.
Then generate six links:
– Flat links for federated and flat‑mirrored services.
– A separate gateway‑aware link (`east-gw`) for hierarchical mirroring.
—
### Step 4: Deploy the Demo Services
Deploy the demo workloads and label them according to the mode:
“`bash
./scripts/04-deploy-app.sh
“`
**frontend (federated)**
“`bash
for ctx in west east north; do
kubectl –context $ctx -n mc-demo label svc/frontend mirror.linkerd.io/federated=member
done
“`
**api (flat‑mirrored)**
“`bash
kubectl –context west -n mc-demo label svc/api mirror.linkerd.io/exported=remote-discovery
kubectl –context east -n mc-demo label svc/api mirror.linkerd.io/exported=remote-discovery
“`
**analytics (gateway‑mirrored)**
“`bash
kubectl –context east -n mc-demo label svc/analytics mirror.linkerd.io/exported=true
“`
—
### Step 5: Verify All Three Modes
Generate traffic and confirm expected behavior:
“`bash
./scripts/05-verify.sh
“`
Expected output:
– **federated**: `[federated] frontend from east`, `[federated] frontend from west`, `[federated] frontend from north`
– **flat‑west**: `[flat-west] api from west`
– **flat‑east**: `[flat-east] api from east`
– **gateway**: `[gateway] analytics from east`
You can also inspect endpoints:
– Federated: endpoints across all three clusters
– Flat mirror: endpoints are remote pod IPs
– Gateway mirror: endpoint is the gateway IP on a non‑standard port
—
### Step 6: The Chaos Test – Kill a Cluster
Simulate a full cluster failure:
“`bash
./scripts/06-chaos-test.sh
“`
Results:
– **Federated (`frontend-federated`)**: Traffic redistributes immediately with zero errors.
– **Flat‑mirrored (`api-east`)**: Returns 503—expected, since the client explicitly targeted east. Failover must be handled by the client or an additional abstraction (e.g., TrafficSplit).
– **Gateway‑mirrored (`analytics-east-gw`)**: Returns 502—the gateway in `east` is down too.
Bring `east` back up and scale deployments to restore service. The federated service rebalances automatically; mirrored services recover as endpoints return.
**Lesson:** Federation gives automatic failover with no client changes. Mirroring gives explicit control, but the client must handle unavailability.
—
### Step 7: Teardown
When you’re done:
“`bash
./scripts/99-teardown.sh
“`
This removes clusters, VPCs, peerings, and firewall rules, stopping ongoing charges.
—
### Selecting Your Multicluster Architecture Strategy
Use this decision framework:
| Question | → Mode |
|———|——–|
| Should the client be cluster‑agnostic? | Federated |
| Does the client need to pick a specific cluster? | Flat mirror |
| Is there no flat network between clusters? | Gateway mirror |
| Do you need automatic failover with no app changes? | Federated |
| Do you need traffic splitting with explicit weights? | Flat mirror + TrafficSplit |
| Is the service a singleton (only in one cluster)? | Mirror (flat or gateway) |
You can mix modes freely across the same mesh; the service label decides behavior independently for each service.
—
### Gotchas and Configuration Lessons
– **VPC peering route exchange**: You must pass both `–export-custom-routes` and `–import-custom-routes` on both sides.
– **Regional clusters**: A regional cluster with `–num-nodes 1` actually creates three nodes. Pin `–node-locations` to one zone to control cost.
– **Non‑overlapping CIDRs**: Always set explicit pod and service CIDRs; GKE defaults overlap.
– **Controller count**: One service‑mirror controller per link per consumer. Miss one, and mirroring silently doesn’t happen.
– **Federated service naming**: The federated suffix is fixed (`-federated`). Plan your client targets accordingly.
– **Gateway + flat can’t share one link**: A single Link resource is either gateway or flat. Use two links with different names to get both.
—
### Key Takeaways: Mastering Multi‑Region Linkerd Deployments
The official docs show each mode in isolation, but real platforms need all three simultaneously. Federation handles “run‑of‑the‑mill” multi‑region failover with no app changes. Flat mirrors give you explicit, cluster‑aware routing for data locality. Gateway mirrors bring hierarchical reach when a flat network isn’t available.
Building this is mostly wiring. Once the trust anchor is shared and links are in place, adding a service to the federation is a single `kubectl label`, and removing a cluster is as simple as letting it go down—the mesh adjusts on its own.
For teams running across regions, that’s a significant operational win: services run everywhere, traffic finds healthy copies, and you pick the multicluster mode per service based on its actual needs.
—
**References**
– [Linkerd Federated Services](https://linkerd.io/2/tasks/federated/)
– [Linkerd Multicluster Reference](https://linkerd.io/2/tasks/multicluster/)
– [Linkerd Multicluster Communication](https://linkerd.io/2/tasks/multicluster-communication/)
– [Installing Multicluster Components](https://linkerd.io/2/tasks/install-mc/)
– [Linkerd 2.17 Announcement – Federated Services](https://linkerd.io/267/)
– [GKE VPC‑Native Clusters](https://cloud.google.com/kubernetes-engine/docs/howalias/alias-pods-services)
—
