# Sharing GPUs Fairly Across Teams in Kubernetes Without Buying Extra Hardware
## The GPU Sharing Problem
Running machine learning and AI workloads in Kubernetes has always been tricky when it comes to GPUs. The fundamental issue is simple but painful: Kubernetes schedules GPUs as whole, indivisible units. If a team only needs 4 GB of an 80 GB GPU, the entire card gets locked up — and every other team on that cluster has to wait.
Most organizations solve this in one of two ways, and neither is great. You can give each team its own cluster, but that means paying for separate control planes. Or you can throw everyone into a single shared cluster, but isolation becomes weak and resource conflicts are constant.
What if there was a way to give every team its own isolated Kubernetes experience while packing multiple workloads onto a single GPU card?
## A Two-Project Approach
The answer involves pairing two open-source tools that solve different parts of the puzzle at the same time.
The first tool lets you run each team’s Kubernetes control plane as lightweight pods inside a single management cluster. Instead of standing up separate physical or virtual machines for control planes, every tenant gets a fully independent API server, scheduler, and controller manager — all orchestrated from one place.
The second tool lets pods request a slice of a GPU. Rather than handing out whole cards, it allows teams to ask for a specific amount of GPU memory (in megabytes) and a percentage of compute power. Multiple pods can then share a single physical card without stepping on each other.
Together, these two projects create a system where each tenant has complete isolation at the API level, and GPU hardware is shared efficiently down to the fraction.
## How It All Fits Together
At the highest level, everything runs as containers on a single Linux machine — no special hardware required for the test setup. One cluster hosts the management plane, and each tenant gets its own worker container that joins the network.
The tenant’s API server lives inside the management cluster, but the GPU scheduling logic lives on the worker node itself. This means that when a pod requests GPU resources, the API server has to reach through a secure tunnel back to the worker to talk to the scheduling plugin. That round trip is the critical integration point, and getting it right is what makes the whole system work.
Each tenant’s data is kept completely separate — the only shared components are the management cluster itself and the underlying datastore that holds tenant configurations.
## Simulating GPUs Without Real Hardware
One of the most interesting parts of building this kind of system is testing it when you don’t have actual GPUs on hand. A mock device plugin skips the need for real NVIDIA drivers and hardware discovery. Instead, you describe fake GPUs through node annotations, specifying fake memory and compute capacity. The scheduling and quota logic then operates exactly as it would on real cards.
In a sample setup, two tenants were created:
– **Tenant A** was configured with one simulated high-end GPU (80 GB memory, 100 compute cores).
– **Tenant B** was configured with two simulated GPUs (24 GB and 100 cores each).
This made it easy to verify that each tenant could only see and use its own resources.
## Watching GPU Pods Get Scheduled
When a pod requests a fractional GPU, the scheduling path goes through several hops. The pod is created in the tenant’s namespace, the tenant’s API server receives it, and the request gets forwarded to the GPU-aware scheduler running on the tenant’s worker. That scheduler checks whether the requested memory and compute slice are available, and if so, places the pod on the node.
The admission call has to travel from the management cluster back through the tenant worker via a secure tunnel. If that connection breaks, the pod falls back to the default scheduler and GPU fractionality is lost.
## Testing the System
A series of tests verified that the setup worked correctly. Three pods were scheduled to share a single simulated GPU card, each requesting 20 GB of memory and 30% of compute. All three landed on the same node and ran simultaneously, using 60 GB of memory and 90% of the card’s compute capacity combined.
A fourth pod that asked for more than the remaining available resources was correctly rejected and left in a pending state, with the scheduler clearly explaining why — either insufficient memory or insufficient compute, depending on what the pod requested.
Most importantly, cross-tenant isolation was verified. When querying each tenant’s API server, neither could see the other’s nodes, pods, or GPU allocations. Even though both control planes were running side by side in the same management cluster, their data remained completely siloed.
## Common Pitfalls
A few things broke during the build process, each with straightforward fixes. One issue was that the default Docker network setup for the kind cluster mixed IPv6 and IPv4 gateway addresses in a way that confused the load balancer. The fix was to manually extract the IPv4 subnet and configure the address pool explicitly.
Another issue was that GPU pods got stuck waiting for a “node lock” that never released. This happened because the mock device plugin doesn’t automatically complete the allocation process the way a real NVIDIA plugin would. Manually marking the binding as successful on each pod resolved the lock immediately.
## What This Does Not Prove
It’s important to note what this kind of test environment can and cannot demonstrate. The setup proves that the routing and scheduling decisions work correctly, but it does not enforce real runtime memory or compute isolation — that requires actual NVIDIA drivers and hardware. No real CUDA workloads were run, so performance characteristics and noisy-neighbor effects remain untested. The mock plugin also required manual intervention for allocations, whereas real device plugins handle this automatically.
—
## FAQ
**Q: Do I need a physical GPU to try this setup?**
A: No. The mock device plugin simulates GPU capacity through node annotations, letting you build and test the entire scheduling and isolation pipeline without any real GPU hardware.
**Q: How is tenant isolation enforced?**
A: Each tenant gets its own Kubernetes API server, scheduler, and controller manager running as separate pods. Network traffic between the API server and worker nodes travels through an encrypted Konnectivity tunnel, and each tenant’s etcd data is kept separate.
**Q: What happens if two pods request more GPU resources than a card has available?**
A: The GPU-aware scheduler refuses the placement and leaves the pod in a pending state. The scheduling failure message will clearly indicate whether the rejection was due to insufficient memory or insufficient compute.
**Q: Can this approach work on real hardware?**
A: Yes. The same architecture works with real NVIDIA GPUs and the actual HAMi device plugin. The mock plugin is only needed for testing without physical hardware.
**Q: Is there a supported way to add worker nodes to this setup?**
A: The approach demonstrated here uses kindest/node containers joined over the Docker network. The supported production paths include using kubeadm directly on real machines or deploying via Cluster API.
**Q: Do these two projects require custom integration code?**
A: No. Kamaji and HAMi operate at different layers of the stack and work together without any glue code or custom patches.
—
## Conclusion
By combining a multi-tenant control plane solution with a GPU slicing scheduler, teams can share expensive hardware fairly without sacrificing isolation or control. Each team gets its own independent Kubernetes API, and every GPU card becomes a divisible pool of memory and compute that multiple workloads can share.
The most critical integration — the API server reaching back through a secure tunnel to the GPU-aware scheduler on the worker node — works reliably, and the entire platform can be built and validated on a single machine without any GPU hardware at all. Once the wiring is proven with simulated devices, swapping in real GPUs is a straightforward next step.
Thank you for reading
