**Automating Kubernetes Control Plane Upgrades: A Real-World GitOps Journey**
Upgrading a Kubernetes control plane has traditionally been a nerve-wracking, manual process. The typical approach involved SSH-ing into each node, executing upgrade commands, and closely monitoring etcd quorum — all while hoping for the best. However, with the increasing pace of vulnerabilities and the demand for higher reliability, the industry is moving toward automated, reliable, and secure upgrade mechanisms. In this post, we explore how to build a trusted, fully automated, GitOps-driven Kubernetes control plane upgrade pipeline using CNCF-native tools.
—
### **The Roots of the Platform**
Everything starts with a solid foundation. The platform in this case was bootstrapped using OpenTofu (a CNCF-incubated tool) to deploy a highly available K3s control plane with Cilium CNI. Provisioned as code from the start, this cluster serves as the root of a growing platform — meant to host more tooling, more workloads, and more clusters over time.
The goal wasn’t just to add features, but to establish a genuinely simple and reliable upgrade process that could be trusted — even when unattended. Using CNCF tools minimized vendor lock-in and built upon foundational knowledge like that of the “Golden Kubestronaut.”
—
### **The Setup: Immutable Infrastructure with Kairos**
The control plane runs on three nodes using K3s in HA mode, managed by the Kairos “Hadron” Linux distribution. What makes this powerful is Kairos’ immutable approach: upgrades are performed by writing a new OS image to an inactive partition and rebooting into it. Rollback is as simple as booting the previous partition.
This method offers strong supply-chain guarantees — but only if the upgrade is executed correctly. A prior version of the pipeline suffered from a concurrency setting mistake (`concurrency: 0` interpreted as “all at once” instead of “one at a time”), which caused all nodes to reboot simultaneously. While etcd survived, it was pure luck.
The fix — `concurrency: 1` — became the foundation for the entire workflow.
—
### **The Full Pipeline: Six Tools, One Mission**
The pipeline is composed of six specialized tools, each responsible for a single task:
1. **Gitea** – Hosts Git repositories and runs CI via Gitea Actions, storing all manifests, policies, and upgrade specs.
2. **Renovate** – Monitors the Quay.io image stream for new versions and opens PRs that update image tags and corresponding custom resource names.
3. **Kyverno** – Acts as a gatekeeper through a ClusterPolicy, rejecting any upgrade CR that doesn’t reference the approved image.
4. **Cosign** – Verifies image signatures using OIDC identities from GitHub Actions, ensuring authenticity and integrity.
5. **ArgoCD** – Implements GitOps by detecting merged PRs, syncing the cluster state automatically without manual `kubectl apply`.
6. **kairos-operator** – The actual upgrade executor. It watches for `NodeOpUpgrade` resources, cordons one node, applies the new image, reboots, waits for rejoining, and proceeds sequentially.
These tools together create a secure, automated, and observable upgrade process.
—
### **A Real Upgrade: July 7th to v0.4.0**
Here’s what a real upgrade looked like:
| Time | Event |
|——|——-|
| 11:40:20 | Baseline: 3 nodes ready, kernel 7.0.10-hadron |
| 11:42:00 | PR #9 merged after Renovate bump |
| 11:42:30 | ArgoCD detects drift |
| 11:42:37 | ArgoCD syncs: old CR deleted, new CR applied |
| 11:42:43 | Image pull starts (324 MB) |
| 11:43:09 | Image pulled in 25.5 seconds |
| 11:43:11 | First node cordoned |
| 11:43:33 | Node upgraded and rebooted |
| ~11:47 | First node ready with kernel 7.1.0-hadron |
| ~11:49 | Second and third nodes upgraded sequentially |
| 11:51:36 | All nodes ready — upgrade complete |
The entire upgrade took **11 minutes** with:
– Zero human intervention
– No broken etcd quorum
– No workload disruption
The change that started it all? A simple two-line diff updating the image and CR name.
—
### **Why the Name Must Change**
The `NodeOpUpgrade` custom resource is designed as one-shot — once completed, Kubernetes never reprocesses it. Simply patching the image on an existing CR doesn’t work because the operator sees it as already finished. That’s why the PR must also update `metadata.name` — forcing ArgoCD to delete the old CR and create a new one, which correctly triggers the operator.
—
### **An Unexpected Bug and a Humble Fix**
During the first real upgrade, a subtle issue appeared. Renovate’s custom regex manager used a non-existent field — `extractVersionTemplate` — which prevented proper version conversion.
As a result:
– The image updated correctly,
– But the CR name stayed the same,
– So the operator ignored it.
The pipeline appeared broken, but it was actually silently failing. The solution was switching to `currentValueTemplate`, which correctly converts “v0-3-0” into “v0.3.0” and reconstructs the proper CR name format.
This incident highlights a critical lesson: **automation can be wrong in perfectly convincing ways**. That’s why human review still matters — not to press buttons, but to catch what the automation misses.
—
### **What’s Next?**
The upgrade pipeline is now being extended:
– The **gateway cluster** — running Netnode on a single-node Kairos setup — is being integrated into the same GitOps loop.
– A **dry-run stage** is in development, using `kairos-agent upgrade –recovery` to test images before any PR merges. This will catch failures early, before they touch production nodes.
—
### **FAQ**
**Q: Why use Kairos for upgrades?**
A: Kairos provides immutable OS updates with A/B partitions, enabling fast rollbacks and verified supply-chain updates via cosign signatures.
**Q: What does concurrency: 1 do?**
A: It ensures only one node is upgraded at a time, preventing simultaneous outages and protecting cluster availability.
**Q: Why doesn’t patching the CR work?**
A: `NodeOpUpgrade` is a one-shot resource. Once completed, the operator ignores changes to existing CRs. A new CR with a new name is required.
**Q: How does Renovate know which image to use?**
A: Renovate watches Quay.io tags and opens PRs to update both the image tag and the CR name automatically.
**Q: What role does Kyverno play?**
A: Kyverno enforces policy at admission time, blocking any upgrade CR that doesn’t reference the approved image.
**Q: Why is human review still necessary?**
A: Automation can succeed while doing the wrong thing — like updating the image but not the CR name. Humans catch these subtle failures.
—
### **Conclusion**
Automated, GitOps-driven Kubernetes control plane upgrades are not only possible — they’re essential for modern, secure, and reliable clusters. By leveraging CNCF tools like Kairos, ArgoCD, Kyverno, Cosign, and Renovate, teams can build upgrade pipelines that are fast, consistent, and observable.
This journey shows that while automation can handle the execution, human oversight remains vital for catching the unexpected. As platforms evolve, the combination of immutable infrastructure, strong supply-chain verification, and thoughtful automation creates a upgrade process you can truly trust — without losing sleep over it.
