GitOps policy-as-code: Securing Kubernetes with Argo CD and Kyverno

A hands-on information to deploying Kyverno with Argo CD and imposing customized insurance policies

As Kubernetes environments develop, GitOps with Argo CD has turn into the usual for declarative, self-healing infrastructure. But with out guardrails in your deployments, misconfigured, insecure, or non-compliant assets can simply make it to manufacturing.

This weblog explores the right way to deploy Kyverno alongside Argo CD, utilizing baseline insurance policies from the official Kyverno Insurance policies Helm chart and demonstrating the right way to add your personal customized insurance policies on prime.

What’s Kyverno?

Kyverno is a CNCF graduated venture that acts as a coverage engine for Kubernetes. It allows you to outline guidelines for what’s and isn’t allowed in your cluster, all written as customary Kubernetes YAML. Kyverno operates on the admission controller degree, which means it intercepts useful resource requests earlier than they hit the cluster and acts on them based mostly in your insurance policies. 

You possibly can consider Kyverno like a checkpoint earlier than manifests are created. They should comply based mostly on a set of insurance policies you need to have set in your cluster.

There are 4 kinds of insurance policies it helps:

• Validate: block or audit assets that don’t meet your guidelines
  • Block: If useful resource doesn’t comply it is not going to be created
  • Audit: If useful resource doesn’t comply It would nonetheless be created however Log that the useful resource doesn’t comply as a result of coverage
• Mutate: As a substitute of rejecting a useful resource, Kyverno modifications it robotically to fulfill your guidelines earlier than it’s created.
• Generate: create outlined assets in coverage robotically
• Cleanup/deleting: take away stale or undesirable assets on a schedule

It covers the complete lifecycle of a useful resource in your cluster.

Why Kyverno with Argo CD?

In the event you’re working Argo CD, you have already got your cluster state dwelling in Git. Kyverno extends that very same construction to coverage enforcement as nicely. Collectively, they permit a totally GitOps-driven method to safety and governance.

Right here’s why the 2 work so nicely collectively:

Coverage as code:

Kyverno insurance policies are outlined utilizing customary Kubernetes YAML, following the identical construction as Kubernetes manifests (`apiVersion`, `type`, and `spec`). This enables insurance policies to be versioned, reviewed, and promoted by means of environments utilizing acquainted Git workflows.

Constant enforcement:

As soon as insurance policies are dedicated to Git, Argo CD robotically syncs them to the cluster. Kyverno enforces these insurance policies at admission time, making certain that each one assets comply together with your guidelines no matter how they’re created.

Protected rollouts with audit and implement modes:

Kyverno insurance policies can run in Audit mode to report violations with out blocking something, or Implement mode to dam assets that don’t meet the coverage. With Argo CD, transferring a coverage from audit to implement is only a Git change.

Configuration and setup: Kyverno with Argo CD

The setup proven on this tutorial relies on the construction used within the ITGix ADP (Software Improvement Platform), created by our staff at ITGix ( It offers a ready-made App-of-Apps sample for managing cluster infrastructure with Argo CD. On the ADP, enabling Kyverno is so simple as setting enabled: true in your platform’s environment-specific configuration, the platform handles the remaining.

That mentioned, you don’t want the ADP to comply with alongside. The setup makes use of a typical App-of-Apps sample that works with any Argo CD set up. The thought is straightforward: one root Argo CD software factors at a listing in your Git repo that incorporates particular person Software manifests for every service. When the basis app syncs, it creates all of the youngster apps, and every youngster app manages its personal Helm chart. The whole lot is pushed from Git with no guide kubectl apply wanted after the preliminary bootstrap.

On the finish the repo construction will appears to be like like this:

global-infra/
  infra-services/  #incorporates Software manifests for Kyverno + insurance policies
    kyverno.yaml
    kyverno-policies.yaml
  kyverno/              # kyverno youngster app factors right here
    Chart.yaml
    values.yaml
  kyverno-policies/     # kyverno-policies youngster app factors right here
    Chart.yaml
    values.yaml
    templates/          # your customized coverage YAMLs stay right here

From right here we assume you have already got Argo CD put in, with a root app watching your infra-services/ listing. All that you must do is drop the next Software manifests into that listing and Argo CD handles the remaining.

Step 1: Add Kyverno as an Argo CD software

This may set up the complete Kyverno bundle into your cluster (the brains of Kyverno), together with:

– All Kyverno controllers (coverage engine, mutate, generate, cleanup, and many others.)

– Customized Useful resource Definitions (CRDs) required by Kyverno

– Default configurations from the Helm chart  

Inside infra-services/, create kyverno.yaml. This defines the Argo CD Software that deploys Kyverno into your cluster. It makes use of an area Helm chart wrapper that pulls within the official Kyverno chart as a dependency. The sync-wave annotation ensures Kyverno is all the time put in earlier than any insurance policies are utilized.

infra-services/kyverno.yaml

apiVersion: argoproj.io/v1alpha1
type: Software
metadata:
  title: kyverno
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
spec:
  venture: default
  supply:
    repoURL: 
    targetRevision: important
    path: kyverno
    helm:
      valueFiles:
        - values.yaml
  vacation spot:
    server: 
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

A few issues price noting right here:

– ServerSideApply=true

  That is wanted for Kyverno as a result of its CRDs are giant and may exceed the annotation measurement restrict utilized by client-side apply.

– IncludeMutationWebhook=true

  This evaluate choice tells Argo CD to consider Kyverno’s mutating webhooks when calculating diffs, stopping fixed out-of-sync warnings on assets that Kyverno mutates.

– ServerSideDiff=true

  This tells Argo CD to calculate diffs utilizing the server-side illustration of assets as a substitute of the native manifest. This works higher with instruments like Kyverno that mutate assets at admission time and helps keep away from noisy or incorrect diffs.

Step 2: Wrap the official Kyverno Helm chart

Helm repository at

Chart: kyverno

kyverno/Chart.yaml

The `kyverno/Chart.yaml` wraps the official chart as a dependency, pulled from the Kyverno

apiVersion: v2
title: kyverno
description: Helm Chart for Kyverno
sort: software
model: 1.0.0
dependencies:
 - title: kyverno
   model: "3.7.1"
   repository: 

kyverno/values.yaml

Word: To make use of the Deleting/Cleanup Controller to robotically clear up assets, add the next to your kyverno/values.yaml file. In the event you don’t plan on utilizing any insurance policies for cleanup/deleting, the default settings from the official chart will work simply advantageous.

kyverno:
 cleanupController:
   rbac:
     clusterRole:
       extraResources:
         - apiGroups:
             - ""
           assets:
             - pods
           verbs:
             - get
             - listing
             - watch
             - delete

Step 3: In ArgoCD UI Affirm Argo software Kyverno is created

After pushing these modifications, Argo CD will deploy kyverno to the cluster:

Word: If kyverno is caught on unsynced ensure you added the annotations within the Software:

argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true

Step 4:  Add Kyverno Insurance policies as an Argo CD Software

This step installs the precise insurance policies you need Kyverno to implement.  

It offers you a set of beginning insurance policies in your cluster, which you’ll customise later utilizing your values.yaml or creating your personal in templates/.

With Kyverno working, create kyverno-policies.yaml in infra-services/ to deploy your insurance policies. It follows the identical sample, pointing on the kyverno-policies/ listing and working at sync-wave `”2″` so it all the time comes after Kyverno itself is prepared:

infra-services/kyverno-policies.yaml

apiVersion: argoproj.io/v1alpha1
type: Software
metadata:
 title: kyverno-policies
 namespace: argocd
 annotations:
   argocd.argoproj.io/sync-wave: "2"
spec:
 venture: default
 supply:
   repoURL: 
   targetRevision: important
   path: kyverno-policies
   helm:
     valueFiles:
       - values.yaml
 vacation spot:
   server: 
   namespace: kyverno
 syncPolicy:
   automated:
     prune: true
     selfHeal: true
   syncOptions:
     - CreateNamespace=true

Step 5: Wrap and configure the Kyverno insurance policies Helm chart

On this step, we outline a Helm wrapper for Kyverno’s official insurance policies chart. This enables us to:

• Set up baseline, production-ready insurance policies maintained by the Kyverno venture
• Configure coverage habits (Audit vs Implement, severity, PSS degree) utilizing values.yaml
• Add our personal customized insurance policies alongside the official ones in a single GitOps workflow

The kyverno-policies youngster software created within the earlier step factors to this listing.

Helm repository at Chart: kyverno-policies

kyverno-policies/Chart.yaml

The kyverno-policies/Chart.yaml file wraps the official kyverno-policies Helm chart as a dependency. The chart is pulled straight from the Kyverno Helm repository.

apiVersion: v2
title: kyverno-policies
description: Helm Chart for Kyverno Insurance policies
sort: software
model: 1.0.0
dependencies:
 - title: kyverno-policies
   model: "3.7.1"
   repository: 

kyverno-policies/values.yaml

And kyverno-policies/values.yaml units the pod safety customary and the default validation mode for these library insurance policies:

kyverno-policies:
 policyType: ValidatingPolicy
 podSecurityStandard: baseline
 validationFailureAction: Audit
 podSecuritySeverity: Medium

– policyType: ValidatingPolicy: Tells the chart to generate validation insurance policies (insurance policies that test assets towards guidelines) quite than mutating ones. These insurance policies will both block or report assets that don’t meet the outlined requirements.

– podSecurityStandard: baseline: Selects which set of Pod Safety Customary insurance policies to put in. Kubernetes defines three ranges: privileged (unrestricted), baseline (prevents recognized privilege escalations), and restricted (closely locked down). baseline is an efficient place to begin because it blocks issues like privileged containers and host networking with out being so strict that it breaks widespread workloads.

– validationFailureAction: Audit: Controls what occurs when a useful resource violates a coverage. Audit means violations are logged and visual in coverage reviews, however the useful resource remains to be allowed to be created. Setting this to Implement would block non-compliant assets from being created solely. Beginning with Audit allows you to see what would break earlier than you begin blocking something.

– podSecuritySeverity: Medium: Units the severity degree reported when a violation is detected. This reveals up in Kyverno’s coverage reviews and can be utilized for filtering and prioritization. Choices are Low, Medium, Excessive, and Crucial.

After pushing these modifications, Argo CD will deploy kyverno-policies to the cluster:

Step 6: Add assist for customized Kyverno insurance policies (non-obligatory)

To this point, we’ve put in:

• Kyverno itself
• A baseline set of official Kyverno insurance policies managed by way of Helm

In lots of real-world eventualities, you’ll additionally need to implement organization-specific guidelines that aren’t lined by the default coverage library. To assist this, we add a listing for customized coverage manifests.

Create the next listing in your repository:

kyverno-policies/templates/

Any Kyverno coverage YAML positioned on this listing can be robotically included when the Helm chart is rendered. As soon as dedicated to Git, Argo CD detects the change and Kyverno enforces the coverage, no guide steps required.

This retains the built-in coverage library and customized insurance policies cleanly separated whereas nonetheless following the identical GitOps workflow.

Step 7: Utilizing customized insurance policies

Good instance of all kinds of Insurance policies will be discovered at Official Kyverno web site:

For example for the weblog here’s a coverage I discovered from the official Kyverno web site, you’ll be able to add to the kyverno-policies/templates/ listing.

Validating coverage: Prohibit utilization of exterior IPs

Sort: Validate | Mode: Audit | Severity: Medium

Description: Setting externalIPs on a Service is a recognized vector for a man-in-the-middle assault (CVE-2020-8554). It permits visitors destined for an arbitrary IP to be intercepted and redirected, which is a severe danger in any multi-tenant cluster. This coverage validates that no Service has the externalIPs subject set, blocking that assault floor solely.

kyverno-policies/templates/clusterpolicy-restrict-external-ips.yaml

apiVersion: kyverno.io/v1
type: ClusterPolicy
metadata:
 title: restrict-external-ips
 annotations:
   insurance policies.kyverno.io/title: Prohibit Exterior IPs
   insurance policies.kyverno.io/class: Greatest Practices
   insurance policies.kyverno.io/minversion: 1.6.0
   insurance policies.kyverno.io/severity: medium
   insurance policies.kyverno.io/topic: Service
   insurance policies.kyverno.io/description: "Service externalIPs can be used for a MITM attack (CVE-2020-8554). Restrict externalIPs or limit to a known set of addresses. See:  This policy validates that the `externalIPs` field is not set on a Service."
spec:
 validationFailureAction: Audit
 background: true
 guidelines:
 - title: check-ips
     match:
       any:
         - assets:
             varieties:
               - Service
     validate:
       message: externalIPs are usually not allowed.
       sample:
         spec:
           X(externalIPs): "null"

Viewing coverage violations

As soon as Kyverno insurance policies are working in your cluster, it turns into straightforward to see when a useful resource doesn’t adjust to them. The precise habits relies on the validationFailureAction configured in your insurance policies. As mentioned earlier, insurance policies can run in Audit mode (report violations) or Implement mode (block assets solely).

Argo CD UI

If a manifest violates a coverage working in Implement mode, Argo CD will fail to sync the useful resource. Within the Argo CD UI, the affected software will seem OutOfSync or Degraded, and the sync error will embody the Kyverno validation message outlined within the coverage.

This makes it instantly seen throughout a GitOps workflow when a change launched in Git violates cluster coverage.

Kyverno coverage reviews

When insurance policies run in Audit mode, the useful resource will nonetheless be created, however the violation is recorded in a Kyverno PolicyReport or ClusterPolicyReport useful resource.

You possibly can view these with:

• kubectl get policyreport -A
• kubectl get clusterpolicyreport

To examine the main points of a violation:

• kubectl describe clusterpolicyreport

These reviews present:

• The useful resource that violated the coverage
• The coverage title
• The rule that failed
• The severity degree
• The message outlined within the coverage

This enables groups to start out with Audit mode to look at how insurance policies have an effect on workloads earlier than switching them to Implement mode to actively block non-compliant assets.