When Kubernetes launched a decade in the past, its promise was clear: make deploying microservices so simple as operating a container. Quick ahead to 2026, and Kubernetes is not “just” for stateless net companies. Within the CNCF annual survey launched in January 2026, 82% of container customers report operating Kubernetes in manufacturing, and 66% of organizations internet hosting generative AI fashions use Kubernetes for some or all inference workloads.
The dialog has basically shifted from stateless net functions to distributed information processing, distributed coaching jobs, LLM inference, and autonomous AI brokers. This isn’t simply evolution, it’s platform convergence pushed by a sensible actuality: operating information processing, mannequin coaching, inference, and brokers on separate infrastructure multiplies operational complexity whereas Kubernetes supplies a unified basis for all of them.
Three eras, one platform
The Kubernetes journey mirrors how software program has developed.
- Microservices period (2015–2020): hardened stateless companies, rollout patterns, and multi-tenant platforms.
- Knowledge + GenAI period (2020–2024): introduced distributed information processing and GPU-heavy coaching/inference into the mainstream.
- Agentic period (2025+): shifts workloads from request/response APIs to long-running reasoning loops.
Every wave builds on the final, making a single platform the place information processing, coaching, inference, and brokers coexist.
Basis: Knowledge processing at scale
Earlier than fashions prepare, information have to be ready. Kubernetes is now the unified platform the place information engineering and machine studying converge, dealing with each steady-state ETL and burst workloads scaling from a whole bunch to hundreds of cores inside minutes. In line with the 2024 Knowledge on Kubernetes neighborhood report, practically half of organizations now run 50% or extra of their information workloads on Kubernetes in manufacturing, with main organizations surpassing 75%.
Apache Spark stays the gold normal for large-scale information processing. The Kubeflow Spark Operator permits declarative Spark administration inside Kubernetes. Organizations run Spark at large scale: hundreds of nodes, 100k+ cores on single clusters, spanning a whole bunch of clusters. Spark preprocesses petabytes of coaching information and triggers downstream coaching jobs, all orchestrated by native Kubernetes primitives.
Orchestration: Connecting the pipeline
With petabytes of ready coaching information and fashions needing retraining on schedules, coordinating multi-step workflows turns into vital. A typical ML pipeline includes Spark preprocessing, distributed coaching throughout hundreds of GPUs, mannequin validation, and mannequin deployment. Working these manually doesn’t scale.
Kubeflow Pipelines supplies transportable ML workflows with experiment monitoring. Argo Workflows permits complicated DAGs spanning Spark jobs, PyTorch coaching, and KServe deployments. The orchestration layer transforms ad-hoc scripts into manufacturing pipelines that set off retraining when information drift is detected
Coaching: Gang scheduling and useful resource coordination
As soon as orchestration triggers a coaching job, distributed coaching’s basic problem emerges: useful resource coordination. Request 120 GPUs however solely 100 accessible? These 100 sit idle, burning cash and blocking work. That is the default state in shared clusters the place a number of groups compete for GPUs.
Gang scheduling grew to become desk stakes. Initiatives like Volcano and Apache Yunikorn pioneered the sample the place multi-node coaching jobs solely begin when all requested sources can be found.
Kueue is rising because the neighborhood normal for batch workload administration on Kubernetes. It brings quota administration, fair-share scheduling, and multi-tenancy controls, fixing the issue of a number of groups competing for restricted GPU sources. JobSet enhances Kueue by offering a local API for managing distributed Job teams with coordinated failure dealing with.
Serving: Inference at scale
After coaching completes, serving predictions to customers requires a basically completely different strategy. Coaching is batch and GPU-saturated. Inference is on-line, latency-sensitive, cost-critical, and should deal with unpredictable visitors.
vLLM and SGLang grew to become requirements for high-throughput LLM serving utilizing PagedAttention and steady batching for Inference workloads on Kubernetes.
KServe supplies a standardized mannequin serving layer with autoscaling, versioning, and visitors splitting. KServe integrates with Knative for scale-to-zero GPU workloads. For multi-host inference serving fashions with 400B+ parameters, LeaderWorkerSet treats pod teams as a single unit.
Agentic workloads: Constructing the agent working system
With inference serving predictions at scale, the latest sample emerges: autonomous brokers. In contrast to single predictions, brokers make chains of LLM calls, preserve dialog state, entry exterior instruments, and run for minutes or hours. They’re long-running reasoning loops needing orchestration, state administration, and safety boundaries.
Are you able to construct and orchestrate AI brokers on Kubernetes? Completely. Frameworks like LangGraph present stateful agent orchestration with sturdy execution. KEDA permits event-driven autoscaling, vital when 100 person requests want 100 agent pods, scaling to zero when idle. StatefulSets present persistent volumes for agent state, whereas vector databases deal with semantic reminiscence.
Safety requires protection in depth. Workload id viaSPIFFE/Spire offers each agent a verifiable id. Sandboxed execution utilizing gVisor or Kata Containers helps isolate untrusted code paths. Coverage enforcement with OPA or Kyverno defines runtime guardrails enforced on the pod admission layer.
Optimizing for the GPU economic system
Throughout all these workloads, GPU availability and price dominate. The bottleneck isn’t CPU or reminiscence, it’s accessing GPUs when wanted and maximizing utilization.
GPU sharing developed. Multi-Occasion GPU (MIG) partitions GPUs into remoted cases. Time-slicing interleaves execution. Multi-Course of Service (MPS) permits concurrent kernels. Dynamic Useful resource Allocation (DRA) in Kubernetes strikes past Machine Plugins, permitting runtime GPU partitioning and reassignment.
On the infrastructure layer, Karpenter (kubernetes-sigs) provisions actual node varieties and aggressively deprovisions idle capability to optimize prices. Container picture acceleration utilizing Seekable OCI (SOCI) scale back startup time for big pictures—particularly related for model-serving containers.
Multi-cluster orchestration and AI conformance
As AI workloads scaled, even optimized single clusters hit limits. Groups now run a whole bunch of clusters for batch processing, distributed coaching, and inference. When a 100-GPU coaching job saturates capability, inference queues again up and information processing stalls.
Multi-cluster scheduling grew to become vital. Options like Armada (CNCF Sandbox) deal with a number of clusters as a single useful resource pool with clever workload distribution, world queue administration, and gang scheduling throughout boundaries.
As Kubernetes turns into the AI substrate, the ecosystem can also be formalizing portability expectations. The CNCF neighborhood has launched work on Kubernetes “AI conformance,” aiming to outline baseline capabilities for operating AI workloads persistently throughout conformant clusters.
What’s subsequent: Improvements pushed by AI scale
AI scale is pushing innovation into areas few anticipated. Management airplane scalability is being reimagined as normal etcd turns into a bottleneck at ultra-scale. Cloud suppliers are innovating past etcd with customized replication programs and in-memory storage. Whereas upstream etcd v3.6.0 delivered 50% reminiscence discount, 100k+ node clusters require rethinking the management airplane datastore.
Unified agent operators are rising to simplify agent deployment with built-in scaling, safety, and lifecycle administration. Multi-cluster workload-aware scheduling is evolving to deal with a whole bunch of clusters as a single clever useful resource cloth the place workloads land based mostly on GPU availability, community topology, and price.
The trail ahead
Platform metrics are altering. Success is more and more tokens-per-second-per-dollar, not pod density. Reliability consists of detecting output drift and degraded mannequin high quality. Observability should hint reasoning loops, instrument calls, and immediate/context paths.
The excellent news: a lot of that is being constructed within the open—throughout CNCF and Kubernetes SIG initiatives—turning Kubernetes into the platform the place AI groups construct end-to-end programs, not simply deploy containers.
—-
Able to discover these patterns? The neighborhood maintains hands-on workshops and reference architectures for Knowledge and AI workloads on Kubernetes. Take a look at the Kubeflow documentation,CNCF panorama guides, and cloud-specific examples to your platform.
