Registry mirror authentication with Kubernetes secrets and techniques

Half I: Structure and Implementation

In manufacturing Kubernetes clusters, pulling container pictures from personal registries occurs 1000’s of instances per day. Kubernetes distributions from main cloud distributors present credential suppliers for his or her respective registries like AWS ECR, Google GCR, and Azure ACR. Nevertheless, the issue emerges when it is advisable to authenticate to personal registry mirrors or pull-through caches. That is significantly widespread in air-gapped environments the place organizations run their very own mirror registries to scale back egress prices, enhance efficiency, or meet compliance necessities.

Conventional approaches to reflect authentication require node-level configuration and direct entry to nodes. This implies credentials have to be configured globally on the node stage, shared throughout all namespaces. This breaks tenant isolation and violates the precept of least privilege.

The CRI-O venture offers a credential supplier that solves this drawback by enabling authentication for registry mirrors utilizing commonplace Kubernetes Secrets and techniques. This strategy maintains safety boundaries whereas preserving the efficiency advantages of registry mirrors.

Stipulations

The issue with node-level credentials

Conventional container registry authentication in Kubernetes has a elementary limitation when working with personal registry mirrors. The kubelet itself has no information of mirror configuration. Mirrors are configured on the container runtime stage by means of recordsdata like /and so on/containers/registries.conf. There are at the moment no Kubernetes enhancement proposals to vary this structure, so mirror configuration stays exterior the Kubernetes API.

This creates a technical drawback: whereas you need to use namespace-scoped secrets and techniques with imagePullSecrets for pulling from supply registries, this doesn’t work while you wish to use personal mirrors or pull-through caches. Mirrors require node-level configuration, forcing you to make use of world credentials. This results in three vital points:

1. Safety isolation is damaged. When credentials are configured globally on the node stage, they change into accessible throughout all namespaces. A compromised pod in namespace A can doubtlessly entry credentials supposed for namespace B. This violates primary safety boundaries.

2. Operational complexity will increase. Cluster directors should handle credentials centrally, creating bottlenecks. Groups lose autonomy and should coordinate with platform groups for each credential replace.

3. Compliance issues come up. In regulated environments, credential sharing throughout venture boundaries violates safety insurance policies. Audit trails change into murky when credentials aren’t scoped to particular namespaces.

That is significantly problematic in multi-tenant platforms the place completely different groups want remoted entry to their respective personal registries. The false selection between utilizing mirrors for efficiency and sustaining safety isolation just isn’t acceptable in manufacturing environments.

The answer: Kubelet credential supplier plugins

Kubernetes launched the kubelet picture credential supplier plugin API in model 1.20 and graduated it to secure in model 1.26. This API permits exterior executables to offer registry credentials dynamically, changing the necessity for static node-level configuration.

The plugin mannequin works by means of commonplace enter and output (stdin/stdout). When the kubelet wants to drag a picture, it invokes the plugin binary, passing a CredentialProviderRequest by way of stdin. The plugin returns a CredentialProviderResponse by way of stdout containing authentication credentials or, within the case of the CRI-O credential supplier, alerts that authentication is dealt with by means of various means.

Kubelet configuration necessities

Two kubelet flags allow credential supplier plugins:

• --image-credential-provider-config: Absolute path to the configuration file.
• --image-credential-provider-bin-dir: Listing containing plugin binaries referenced by their title.

The configuration file specifies which plugins deal with which picture patterns. Beginning with Kubernetes 1.33, the KubeletServiceAccountTokenForCredentialProviders characteristic gate permits the kubelet to move service account tokens to credential supplier plugins. That is required for the CRI-O credential supplier to operate, because it wants the token to extract the namespace and authenticate to the Kubernetes API utilizing the pod’s personal identification.

A whole configuration seems to be like this:

apiVersion: kubelet.config.k8s.io/v1
form: CredentialProviderConfig
suppliers:
  - title: crio-credential-provider
    matchImages:
      - docker.io
      - quay.io
    defaultCacheDuration: "1s"
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    tokenAttributes:
      serviceAccountTokenAudience: 
      cacheType: "Token"
      requireServiceAccount: false

The tokenAttributes part specifies how service account tokens are dealt with:

• serviceAccountTokenAudience: Specifies who the token is meant for (sometimes the Kubernetes API server URL). The API server validates this matches earlier than accepting the token.
• cacheType: Determines caching scope (Token for per-token caching or ServiceAccount for per-service-account caching)
• requireServiceAccount: Whether or not the plugin requires the pod to have a service account. The kubelet passes the token from both the pod’s explicitly specified service account or the default service account for the namespace.

This configuration permits namespace-scoped credential retrieval with out requiring node-level permissions.

How the CRI-O Credential Supplier Works

The CRI-O credential supplier implements a workflow that extracts namespace info from service account tokens, discovers registry mirrors, retrieves secrets and techniques from that namespace, and generates short-lived authentication recordsdata consumed by CRI-O.

Simplified workflow

1. Kubelet invokes the credential supplier with a picture title and repair account token
2. The credential supplier:
   • Parses the JSON Internet Token (JWT) to extract the namespace
   • Discovers configured mirrors from native configuration
   • Queries the Kubernetes API for all registry secrets and techniques within the namespace
   • Matches secrets and techniques towards the mirrors and supply picture
   • Generates a short-lived auth file for CRI-O
   • Returns an empty success response to kubelet
3. CRI-O makes use of the auth file for mirror authentication and cleans it up after the picture pull

Full workflow with element interactions

The diagram illustrates the whole interplay between parts. When a picture pull request arrives, the kubelet invokes the credential supplier as an exterior executable, passing each the picture reference and the pod’s service account token. The credential supplier acts as a bridge between Kubernetes (the place secrets and techniques are saved) and CRI-O (the place mirrors are configured).

Token parsing and discovery

The supplier parses the JWT token to extract the namespace from the kubernetes.io declare with out validating the token signature (validation occurs when the token is used to name the Kubernetes API). Utilizing this namespace info, it queries the Kubernetes API for all dockerconfigjson sort secrets and techniques and reads the mirror configuration from the container runtime’s /and so on/containers/registries.conf file.

Secret retrieval and credential matching

The credential supplier makes use of the pod’s service account token to authenticate to the Kubernetes API and retrieve secrets and techniques from the pod’s namespace. This requires RBAC permissions for the service account to checklist and get secrets and techniques, plus cluster-level permissions for nodes to make use of credential suppliers.

It decodes every dockerconfigjson secret from base64, extracts the registry credentials, and normalizes registry URLs by stripping http:// and https:// prefixes. It then makes use of prefix matching to establish which credentials apply: a mirror URL like quay.io matches secrets and techniques for each quay.io and quay.io/myorg. A number of secrets and techniques in the identical namespace are all merged collectively.

The supplier additionally reads the worldwide kubelet auth file at /var/lib/kubelet/config.json if it exists and merges it with the namespace secrets and techniques, with namespace credentials taking priority over world ones.

Auth file technology and cleanup

The auth file is written to /and so on/crio/auth/–.json the place the SHA256 hash of the picture reference ensures a novel filename that forestalls collisions between concurrent pulls of various pictures. CRI-O discovers this file by means of the naming conference, makes use of it to authenticate towards the mirror registries in the course of the picture pull, and cleans it up after the pull completes. The supplier returns an empty success response to the kubelet, signaling that authentication is dealt with.

This structure maintains separation of issues: Kubernetes manages secrets and techniques and identification, the container runtime manages mirror configuration, and the credential supplier bridges the hole between them.

The implementation consists of a number of efficiency optimizations: buffer pooling to scale back rubbish assortment strain, streaming JSON parsing to keep away from studying all enter into reminiscence, pre-allocated information constructions to reduce allocations, and early loop exits when credentials are discovered. These optimizations matter as a result of the credential supplier executes for each picture pull that matches the configured matchImages patterns. In a busy cluster pulling from configured registries, this could imply tons of or 1000’s of invocations per hour.

An actual world instance

Let’s stroll by means of a concrete instance with actual configuration recordsdata. The credential supplier binary may be downloaded from the releases or constructed from the repository and put in on all nodes within the listing specified by –image-credential-provider-bin-dir.

Configure kubelet

Configure the kubelet to make use of the credential supplier by creating

/and so on/kubernetes/credential-providers/config.yaml:
apiVersion: kubelet.config.k8s.io/v1
form: CredentialProviderConfig
suppliers:
  - title: crio-credential-provider
    matchImages:
      - docker.io
    defaultCacheDuration: "1s"
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    tokenAttributes:
      serviceAccountTokenAudience: 
      cacheType: "Token"
      requireServiceAccount: false

Notice: The defaultCacheDuration and cacheType fields are required by the API however don’t at the moment have an effect on conduct for the reason that CRI-O credential supplier doesn’t assist credential caching. Auth recordsdata are generated contemporary for every picture pull.

Add the next kubelet flags to reference this configuration:

--image-credential-provider-config=/and so on/kubernetes/credential-providers/config.yaml
--image-credential-provider-bin-dir=/usr/libexec/kubernetes/credential-providers
--feature-gates=KubeletServiceAccountTokenForCredentialProviders=true

Arrange registry mirror

Arrange a personal registry mirror on the node. Begin a neighborhood registry at localhost:5000 with primary authentication:

# Create htpasswd file with credentials (username: myuser, password: mypassword)
mkdir -p /tmp/registry/auth
podman run --rm --entrypoint htpasswd httpd:2 -Bbn myuser mypassword > /tmp/registry/auth/htpasswd

# Begin the registry with authentication
podman run -d -p 5000:5000 
  --name registry 
  -v /tmp/registry/auth:/auth 
  -e REGISTRY_AUTH=htpasswd 
  -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" 
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd 
  docker.io/library/registry:2

Push a take a look at picture to the mirror:

# Log in to the native registry
podman login localhost:5000 -u myuser -p mypassword

# Pull and push a take a look at picture
podman pull docker.io/library/nginx:newest
podman tag docker.io/library/nginx:newest localhost:5000/library/nginx:newest
podman push localhost:5000/library/nginx:newest

Configure mirror in CRI-O

Configure this registry as a mirror for docker.io in /and so on/containers/registries.conf. When CRI-O makes an attempt to drag a picture from docker.io, it should first attempt to pull from the mirror:

unqualified-search-registries = ["docker.io"]

[[registry]]
location = "docker.io"

[[registry.mirror]]
location = "localhost:5000"
insecure = true

Create registry credentials secret

Create a namespace secret containing credentials for authenticating to the mirror registry. The credential supplier will retrieve this secret from the default namespace and use it to generate the auth file:

apiVersion: v1
form: Secret
sort: kubernetes.io/dockerconfigjson
metadata:
  title: my-secret
  namespace: default
information:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwOi8vbG9jYWxob3N0OjUwMDAiOnsidXNlcm5hbWUiOiJteXVzZXIiLCJwYXNzd29yZCI6Im15cGFzc3dvcmQiLCJhdXRoIjoiYlhsMWMyVnlPbTE1Y0dGemMzZHZjbVE9In19fQo=

The base64-encoded .dockerconfigjson information decodes to:

{
  "auths": {
    " {
      "username": "myuser",
      "password": "mypassword",
      "auth": "bXl1c2VyOm15cGFzc3dvcmQ="
    }
  }
}

Configure RBAC

Configure RBAC to permit the pod’s service account to learn secrets and techniques in its namespace. The credential supplier makes use of the pod’s service account token to authenticate to the Kubernetes API and retrieve registry secrets and techniques. With out these permissions, the API will reject the credential supplier’s request to checklist and get secrets and techniques.

Cluster-level RBAC

Create cluster-level RBAC to permit nodes to make use of the credential supplier:

apiVersion: rbac.authorization.k8s.io/v1
form: ClusterRole
metadata:
  title: node-credential-providers
guidelines:
  - apiGroups: [""]
    assets: ["serviceaccounts"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    assets: ["*"]
    verbs: ["request-serviceaccounts-token-audience"]
---
apiVersion: rbac.authorization.k8s.io/v1
form: ClusterRoleBinding
metadata:
  title: node-credential-providers-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  form: ClusterRole
  title: node-credential-providers
topics:
  - apiGroup: rbac.authorization.k8s.io
    form: Person
    title: system:node:your-node-name

The system:node:your-node-name topic should match the precise node identification. It’s good to create a ClusterRoleBinding for every node in your cluster, or add a number of topics to a single binding.

Namespace-level RBAC

Create namespace-level RBAC to permit service accounts to learn secrets and techniques:

apiVersion: rbac.authorization.k8s.io/v1
form: Position
metadata:
  title: secrets-role
  namespace: default
guidelines:
  - apiGroups: [""]
    assets: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
form: RoleBinding
metadata:
  title: secrets-role-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  form: Position
  title: secrets-role
topics:
  - apiGroup: rbac.authorization.k8s.io
    form: Person
    title: system:serviceaccount:default:default

Operating the instance

Create a file named pod.yaml that defines a pod pulling the nginx picture from docker.io:

apiVersion: v1
form: Pod
metadata:
  title: nginx
  namespace: default
spec:
  containers:
    - title: nginx
      picture: docker.io/nginx

Deploy the pod:

kubectl apply -f pod.yaml

When the pod is created and the kubelet makes an attempt to drag the picture, the credential supplier is invoked. The supplier logs (obtainable by way of journald) present the whole circulate:

Operating credential supplier
Studying from stdin
Parsed credential supplier request for picture "docker.io/library/nginx"
Parsing namespace from request
Matching mirrors for registry config: /and so on/containers/registries.conf
Acquired mirror(s) for "docker.io/library/nginx": "localhost:5000"
Getting secrets and techniques from namespace: default
Unable to search out env file "/etc/kubernetes/apiserver-url.env", utilizing default API server host: localhost:6443
Acquired 1 secret(s)
Parsing secret: my-secret
Discovered docker config JSON auth in secret "my-secret" for "
Checking if mirror "localhost:5000" matches registry "localhost:5000"
Utilizing mirror auth "localhost:5000" for registry from secret "localhost:5000"
Wrote auth file to /and so on/crio/auth/default-7e59ad64326bc321517fb6fc6586de5ee149178394d9edfa2a877176cdf6fad5.json with 1 variety of entries
Auth file path: /and so on/crio/auth/default-7e59ad64326bc321517fb6fc6586de5ee149178394d9edfa2a877176cdf6fad5.json

Notice: The warning concerning the lacking /and so on/kubernetes/apiserver-url.env file is predicted. Platforms could use that env file to specify the API server URL, which might be picked up by the credential supplier. In any other case, it falls again to localhost:6443.

CRI-O logs

CRI-O then picks up the auth file and efficiently pulls from the mirror. The CRI-O debug logs present it discovering and utilizing the namespace-scoped auth file:

DEBU[...] Searching for namespaced auth JSON file in /and so on/crio/auth for picture docker.io/nginx
DEBU[...] Utilizing auth file for namespace default: /and so on/crio/auth/default-7e59ad64326bc321517fb6fc6586de5ee149178394d9edfa2a877176cdf6fad5.json
INFO[...] Attempting to entry "localhost:5000/library/nginx:latest"
INFO[...] Pulled picture: docker.io/library/nginx@sha256:5ff65e8820c7fd8398ca8949e7c4191b93ace149f7ff53a2a7965566bd88ad23

Registry logs

The registry logs affirm the authenticated request:

time="..." stage=information msg="authorized request" http.request.host="localhost:5000" http.request.methodology="GET" http.request.uri="/v2/" http.request.useragent="cri-o/1.34.0" auth.person.title="myuser"
time="..." stage=information msg="authorized request" http.request.host="localhost:5000" http.request.methodology="GET" http.request.uri="/v2/library/nginx/manifests/latest" http.request.useragent="cri-o/1.34.0" auth.person.title="myuser"
time="..." stage=information msg="authorized request" http.request.host="localhost:5000" http.request.methodology="GET" http.request.uri="/v2/library/nginx/blobs/sha256:..." http.request.useragent="cri-o/1.34.0" auth.person.title="myuser"

Success! The pod begins and the picture was pulled from the personal mirror utilizing solely namespace-scoped credentials, with no world credentials required.

Safety concerns

The safety mannequin depends on Kubernetes’ current RBAC system mixed with service account token scoping. The basic safety property is that every pod can solely entry secrets and techniques in its personal namespace. That is enforced by means of a number of layers: the service account token is namespace-scoped, the Kubernetes API server enforces RBAC when the credential supplier queries for secrets and techniques, and the auth file is cleaned up by CRI-O after the picture pull completes.

Contemplate a risk situation the place a compromised pod in namespace A makes an attempt to entry credentials from namespace B. The assault fails as a result of the service account token within the pod solely grants entry to namespace A, so the Kubernetes API server rejects makes an attempt to checklist secrets and techniques in namespace B. Even when the attacker might one way or the other invoke the credential supplier instantly, it will solely retrieve namespace A’s secrets and techniques. This maintains protection in depth the place credentials by no means exist in a location accessible to different namespaces.

The auth recordsdata themselves are short-lived. They’re created on-demand when a picture pull begins, utilized by CRI-O in the course of the pull operation, atomically moved to a brief location throughout concurrent pulls to forestall race situations, and deleted after the pull completes. This minimizes the window throughout which credentials exist on disk.

Conclusion

The CRI-O credential supplier permits namespace-scoped authentication for personal registry mirrors, sustaining safety boundaries whereas preserving the efficiency advantages of mirrors. The answer leverages current Kubernetes primitives like service account tokens, secrets and techniques, and RBAC. This makes it a pure match for multi-tenant clusters.

In Half II (coming quickly), we’ll see how platforms like OKD and OpenShift are integrating this functionality natively, simplifying deployment by means of declarative APIs.