# Building a Fully Open-Source Secrets Backend on Kubernetes with OpenBao and CloudNativePG
## Why Open-Source Secrets Management Matters
Every organization running workloads on Kubernetes needs a secrets backend — but choosing one that is self-healing, transparent, and free of proprietary lock-in is easier said than done. Many enterprises end up tethered to a single cloud provider’s key management service, which limits portability, raises costs at scale, and introduces a single point of failure outside the team’s direct control.
The combination of **OpenBao** and **CloudNativePG** addresses all of these concerns. OpenBao, maintained under the Linux Foundation as the open-source successor to HashiCorp Vault, provides a mature secrets engine with encryption, key rotation, and access policies. CloudNativePG, a Cloud Native Computing Foundation Sandbox project currently under evaluation for Incubation, delivers a production-grade PostgreSQL engine as a Kubernetes-native operator — complete with synchronous replication, automatic failover, and certificate-based authentication.
When OpenBao’s PostgreSQL storage backend is pointed at a CloudNativePG-managed cluster, the result is a fully open-source, self-contained secrets infrastructure where every byte of sensitive data lives inside a database that heals itself, replicates without data loss, and authenticates every connection without ever relying on a password sitting in a config file.
—
## How the Architecture Works
At a high level, the stack has four defining characteristics:
1. **OpenBao’s PostgreSQL storage backend** turns any PostgreSQL cluster into an encrypted key-value store. With high availability enabled via a dedicated lock table, OpenBao maintains quorum across its replicas without requiring local disk persistence on any node.
2. **A three-instance CloudNativePG cluster** runs the database that stores OpenBao’s data. The cluster uses quorum-based synchronous replication (`method: any, number: 1`), which means any single standby replica satisfies the durability requirement, and failover to a new primary happens with zero data loss.
3. **Workload isolation** through Kubernetes node selectors, tolerations, and required zonal pod anti-affinity ensures that the PostgreSQL instances run on dedicated infrastructure nodes separate from general-purpose workloads, following the operator’s scheduling guidance for production resilience.
4. **Passwordless mutual TLS (mTLS)** authentication is enforced at two levels: the CloudNativePG `DatabaseRole` custom resource issues client certificates for both the schema owner and the application role OpenBao connects as, and explicit `pg_hba.conf` rules mandate certificate presentation rather than allowing password-based fallback.
The result is a stack where no secret password exists anywhere in the configuration — not in environment variables, not in Helm values, not in init scripts.
—
## Setting Up a Local Test Environment
To experiment with this architecture locally, the fastest path is the CloudNativePG Playground repository, which is pre-configured with the operator and a ready-to-use Kind cluster. The playground provisions a six-node Kind cluster: one control plane node, one infrastructure workload node, one application workload node, and three worker nodes carrying a Postgres-specific taint.
This taint is significant because it is what the `Cluster` manifest’s tolerations target, and it also means OpenBao itself has only two general-purpose nodes available for scheduling. This becomes important later when pod anti-affinity is introduced.
**Prerequisites:** Docker, Kind, Helm, and `kubectl`.
The setup workflow is straightforward:
“`bash
# Clone the repository and provision a single local cluster
git clone
cd cnpg-playground
./scripts/setup.sh openbao
# Deploy the operator, cert-manager, the Barman Cloud plugin,
# and a ClusterImageCatalog — but skip the demo databases
REQUIREMENTS_ONLY=true ./demo/setup.sh
“`
The `REQUIREMENTS_ONLY` flag is useful because it avoids deploying sample databases that are irrelevant to this recipe, letting you start clean with only the infrastructure components needed for the secrets backend.
—
## Step 1: Deploy the Database Cluster, Roles, and Schema
The first real step is creating the CloudNativePG `Cluster` resource. A key design choice in the manifests is the use of an `imageCatalogRef` instead of hard-pinning a PostgreSQL image tag. By pointing at the `postgresql-minimal-trixie` catalog, the operator resolves to the latest minimal PostgreSQL 18 image in that catalog automatically. This means the same manifest continues to pick up patch-level updates without any manual edit to the `Cluster` resource.
The cluster declaration includes several critical pieces:
– **Synchronous replication** with `method: any` and `number: 1`, paired with the default `dataDurability: required` setting for zero-data-loss guarantees.
– **Workload isolation** via node selectors, tolerations, and required zonal pod anti-affinity.
– **Two `pg_hba` rules** that enforce certificate authentication for the roles OpenBao will use, removing any possibility of password-based connections.
Two `DatabaseRole` resources follow:
– **`role-openbao`** — the schema owner, used once to run DDL (data definition language) statements.
– **`role-openbao-rw`** — the restricted runtime role that OpenBao itself connects as. This role has only the DML (data manipulation language) privileges needed for day-to-day operations.
Both roles receive a `clientCertificate` block within the `DatabaseRole` spec. This is deliberate: a one-shot DDL job is no more entitled to a stored password than the application role is. Certificates eliminate that risk entirely.
Once the cluster reconciles, the operator creates two client certificate secrets following a predictable naming convention (`
An important security detail: every manifest that mounts a client certificate secret sets `defaultMode: 0640` on the volume. Kubernetes mounts `Secret` volumes at `0644` by default, which the PostgreSQL client library (`libpq`) refuses outright. The library rejects any private key file that is group- or world-readable, whether owned by root or by the connecting user. Since the mounted files remain root-owned and only their group matches the pod’s `fsGroup`, `0640` is the setting that satisfies the library while maintaining appropriate access control.
—
## Step 2: Initialize the Schema and Grant Table Privileges
`DatabaseRole` does not yet manage table-level grants — the `permissions` stanza that would let a `Database` object express `GRANT` and `REVOKE` declaratively is still an open proposal in the project’s roadmap. Until that feature lands, a one-time Kubernetes Job running DDL as the schema owner is the correct approach to create OpenBao’s tables and grant the restricted role the privileges it needs.
OpenBao’s PostgreSQL storage backend expects exactly two tables when high availability is enabled:
– **`openbao_kv_store`** — stores the key-value pairs with columns for `parent_path`, `path`, `key`, and `value`, with a primary key on `(path, key)`.
– **`openbao_ha_locks`** — holds the HA lock records that coordinate failover between OpenBao replicas.
Getting the schema wrong is an easy pitfall. If the `key` column or the primary key definition is incorrect, OpenBao will silently attempt to create the tables itself on first connection using its own DDL. That path only succeeds if the connecting role has `CREATE` privileges — which the restricted `openbao-rw` role deliberately does not have. Pre-creating both tables under the owner role and setting `skip_create_table` on the OpenBao side keeps that DDL entirely off the restricted runtime role.
The initialization Job also addresses a gap that PostgreSQL leaves open by default: every database grants `CONNECT` to `PUBLIC`, and the `public` schema grants `USAGE` to `PUBLIC`. This means any role that can log into the cluster can connect to the `openbao` database and inspect its contents unless explicitly told otherwise. The schema-init Job revokes these public grants and grants back only what `openbao-rw` actually needs.
The Job runs eight SQL statements and produces eight confirmations — both tables created, the DML grant issued, and three `REVOKE/GRANT` pairs that lock down the database and schema to the intended role.
—
## Step 3: Configure and Deploy OpenBao via Helm
With the database and schema ready, OpenBao is deployed using the official Helm chart. The configuration has several nuances worth calling out:
**No local persistence.** OpenBao’s `dataStorage` is explicitly disabled. The entire point of this stack is that OpenBao carries no local state — all secrets live inside the CloudNativePG cluster. Without this setting, the chart would create a 10Gi persistent volume claim per pod that sits unused.
**Certificate mounting.** The `role-openbao-rw-client-cert` secret and the cluster’s CA certificate are mounted using the `server.volumes` and `server.volumeMounts` fields, which pass straight through to the Pod spec. The chart’s `server.extraVolumes` field uses a different, simplified schema (`type/name/path`) that does not accept raw Kubernetes Secret volumes, and there is no `extraVolumeMounts` field for the server StatefulSet. Using the direct volume fields avoids this mismatch entirely.
**Connection string.** OpenBao’s PostgreSQL storage configuration points at the restricted `openbao-rw` role with `sslmode=verify-full` and references the certificate files at the mounted paths. The `skip_create_table` parameter is set explicitly to prevent the runtime role from attempting DDL it cannot execute.
**Pod anti-affinity.** OpenBao’s Helm chart defaults to a `requiredDuringSchedulingIgnoredDuringExecution` rule keyed on `kubernetes.io/hostname`, ensuring no two replicas land on the same node. In the playground environment specifically, with Postgres nodes tainted and unavailable, only two general-purpose nodes remain — one short of what three replicas need. A toleration for the control-plane taint is added to give OpenBao a third landing spot, which is acceptable on a single-developer Kind cluster but should never be carried into production.
“`bash
helm repo add openbao
helm repo update
helm install openbao openbao/openbao
–namespace openbao
-f openbao-values.yaml
“`
—
## Step 4: Verification and Initialization
After deployment, the OpenBao pods come up in order due to the StatefulSet’s default `OrderedReady` policy. `openbao-0` must be fully initialized and unsealed before `openbao-1` is even created, and the same sequence repeats for each subsequent pod.
**Initialization** happens on `openbao-0` using the `bao operator init` command, which generates unseal keys and a root token. These must be stored securely.
**Unsealing** must happen per pod, per key, in order — `openbao-0`, then `openbao-1`, then `openbao-2`. Each pod holds an independent Shamir quorum in memory, so unsealing one pod has no effect on the others. Until all three keys are submitted to a pod, readiness probes report `Unseal Progress: 0/3` and emit `Warning Unhealthy` events. These are expected and resolve automatically once the pod receives its keys.
When a pod is fully unsealed, its status shows `Sealed: false` with `Storage Type: postgresql` — confirming that OpenBao is writing its keyring, root key material, seal configuration, and all bootstrap state directly into the CloudNativePG cluster.
**Verification** involves logging in with the root token, enabling KV v2 secrets engine, writing a test secret, and reading it back. A direct query of the `openbao_kv_store` table then shows the test payload stored as an encrypted `BYTEA` blob — never in plaintext, even to someone with direct PostgreSQL access.
—
## Operational Considerations
### Certificate Renewal
CloudNativePG’s client certificates carry a 90-day validity period and are renewed automatically approximately one week before expiry. The operator replaces the contents of the `role-openbao-rw-client-cert` secret in place without requiring any manifest changes.
However, OpenBao does not pick up renewed certificates automatically. The PostgreSQL storage backend opens its connection pool once at process startup and never re-reads the certificate files afterward. A renewed certificate only takes effect after a rolling restart of the OpenBao pods. This is a property of the storage plugin’s connection lifecycle, not something the CloudNativePG operator can control — the operator’s job ends at keeping the secret current.
The practical implication: schedule periodic rolling restarts of OpenBao within the 83-day window (the period between automatic renewal and expiry) to ensure the new certificate is picked up before the old one expires.
### Backups and Disaster Recovery
This recipe deploys a single-cluster configuration. Production workloads need additional layers:
– **Automated PostgreSQL backups** can be configured using the Barman Cloud Plugin (deployed as a CNPG plugin) with `Backup` and `ScheduledBackup` resources, enabling continuous WAL archiving and point-in-time recovery against object stores like AWS S3, Google Cloud Storage, or Azure Blob Storage.
– **Disaster recovery** across regions can leverage CloudNativePG’s distributed topology: an asynchronous replica cluster in a second Kubernetes cluster can be promoted to primary if the original cluster is lost entirely, meeting real RTO and RPO targets.
—
## FAQ
**Q: Why use OpenBao instead of HashiCorp Vault directly?**
A: OpenBao is the Linux Foundation’s open-source fork of HashiCorp Vault, created to ensure the project remains fully open-source and vendor-neutral under a foundation governance model. It maintains API compatibility with Vault while removing any dependency on a single commercial entity’s roadmap or licensing decisions.
**Q: Can this architecture work with an existing PostgreSQL cluster that isn’t managed by CloudNativePG?**
A: OpenBao’s PostgreSQL storage backend works with any PostgreSQL-compatible database. However, the self-healing, synchronous replication, and certificate-authentication features described here come from CloudNativePG. Using a manually managed PostgreSQL instance would require you to handle replication, failover, certificate lifecycle, and scheduling isolation yourself.
**Q: What happens if one of the three OpenBao pods goes down?**
A: Because OpenBao uses Shamir-based quorum sealing with three shares and a threshold of three, all three pods must be unsealed for the cluster to be operational. If one pod is lost, the remaining two still hold their shares but the cluster becomes read-only until the third pod is recovered and unsealed. Kubernetes will automatically recreate the pod, and you would need to unseal it with the same three keys.
**Q: Is there a way to avoid manually unsealing each OpenBao pod?**
A: Yes — in production environments, you can integrate OpenBao with an auto-unseal mechanism using a cloud KMS or a sealed-secrets approach. This recipe focuses on the open-source stack without external dependencies, but auto-unseal is a standard OpenBao feature documented in the official Helm chart and operator guides.
**Q: Why does the schema owner role (`role-openbao`) need a client certificate if it’s only used once?**
A: The principle is that no credential lying around is worth trusting, even for a one-time operation. A client certificate issued by CloudNativePG’s `DatabaseRole` is automatically rotated on a 90-day cycle, and its private key never needs to be stored in a config file or environment variable. If the schema owner used a password instead, that password would persist indefinitely in the manifest or secret store — a far larger attack surface.
**Q: Can I use a different PostgreSQL version or image?**
A: The `ClusterImageCatalog` approach means you can switch between supported PostgreSQL versions by changing the catalog reference. The current recipe uses PostgreSQL 18 minimal builds, but any PostgreSQL version available in a configured catalog will work with both CloudNativePG and OpenBao’s PostgreSQL storage backend.
**Q: What is the role of cert-manager in this stack?**
A: cert-manager is deployed as part of the playground setup to handle the issuance and lifecycle management of TLS certificates used for internal cluster communication (e.g., between OpenBao replicas and the database). The client certificates for application roles are managed separately by CloudNativePG’s `DatabaseRole` CRD.
—
## Conclusion
This architecture demonstrates how two CNCF projects — Kubernetes, long graduated, and CloudNativePG, currently in the Sandbox stage — can combine with OpenBao to form a complete, fully open-source secrets management stack with no cloud database dependencies and no vendor lock-in.
The key takeaways are:
– **Synchronous replication** in CloudNativePG delivers zero-data-loss failover without any third-party storage.
– **Certificate-based authentication** via `DatabaseRole` turns the absence of a password into an enforced policy, not an assumption.
– **Pod anti-affinity and node isolation** keep the database’s failure domains separate from general-purpose workloads.
– **No local persistence** on the OpenBao side means the entire secrets infrastructure is stateless at the application layer, with all data living inside a self-healing PostgreSQL cluster.
The open-source ecosystem has matured to the point where enterprises can run production-grade secrets management entirely on their own infrastructure, backed by community-governed projects with transparent governance and no licensing surprises.
—
Thank you for reading
