How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure
Quick Answer
NVIDIA's tutorial demonstrates how to run isolated Kubernetes clusters for multiple teams on a shared GPU infrastructure using KAI Scheduler and vCluster, allowing teams to maintain autonomy without hardware fragmentation.
Quick Take
This setup enables three teams to run their own workloads on a single NVIDIA L40S GPU while ensuring resource isolation and independent control planes.
Key Points
- KAI Scheduler optimizes GPU resource allocation for AI workloads in large-scale environments.
- vCluster provides fully isolated tenant clusters while sharing the same underlying nodes.
- Each team can manage their own CRDs and RBAC without interfering with others.
- The tutorial is designed for a single NVIDIA L40S GPU shared among three teams.
- The setup is scalable to larger clusters with hundreds of GPU nodes.
DeepSignal Analysis
What happened
NVIDIA's tutorial outlines a method for running isolated Kubernetes clusters on shared GPU infrastructure, specifically using KAI Scheduler and vCluster. This approach allows multiple teams to operate independently without the need for separate hardware, facilitating resource management and autonomy.
Key evidence
- The tutorial demonstrates how three teams can share a single NVIDIA L40S GPU while maintaining isolated Kubernetes control planes, ensuring that each team only accesses its own workloads.
- KAI Scheduler is designed for optimizing GPU resource allocation for AI workloads and can manage large-scale GPU clusters, dynamically allocating resources based on workload demands.
- vCluster provisions fully isolated tenant clusters that share underlying nodes and hardware, providing each team with its own API server, CRDs, and RBAC, similar to a dedicated cluster.
Why it matters
This setup addresses the common challenge of resource fragmentation in organizations that require multiple teams to run their workloads. By allowing teams to share GPU resources while maintaining control and isolation, organizations can optimize their infrastructure usage and reduce operational overhead.
📖 Reader Mode
~12 min readRunning a dedicated Kubernetes cluster per team often results in more isolation than an organization requires. While one cluster can be successfully shared across many teams, the coordination costs increase as the number of teams grows. Challenges include conflicting CRD versions, overlapping RBAC, and no clean way to carve GPU capacity into team-level budgets. At a certain scale, teams might start asking for their own clusters just to regain autonomy.
This post provides a pattern that preserves team autonomy without splitting the hardware. This solution involves a single control plane cluster with a GPU pool, GPU sharing with per-team quotas, and isolated Kubernetes control plane per team including an API server, controller, data store, syncer, and scheduler. This can be achieved using two open source tools: KAI Scheduler and vCluster.
Follow along with the tutorial steps and you’ll have three teams running real GPU Kubernetes pods in their own tenant clusters, all sharing a single physical GPU. You’ll also be able to verify that each team only sees their own workloads.
Tutorial prerequisites and notes
To keep the process reproducible for users that have limited resources, this tutorial uses a cluster with one NVIDIA L40S GPU and three teams sharing fractions of it. This makes the moving parts easy to see and try. The process works the same way on a larger cluster with hundreds of GPU nodes and dozens of teams. You can scale the node pool, the queue hierarchy, and the number of tenant clusters.
KAI Scheduler is a robust, efficient, and scalable topology-aware Kubernetes scheduler that was purpose-built for optimizing GPU resource allocation for AI workloads. It’s designed to manage large-scale GPU clusters, including thousands of nodes, and a high throughput of workloads. With KAI Scheduler, you can dynamically allocate GPU resources to workloads. It can run alongside the default kube-scheduler. Any pod with schedulerName: kai-scheduler is handled by KAI Scheduler. Everything else goes through the usual kube-scheduler process.
The vCluster Kubernetes platform provisions fully isolated tenant clusters on your infrastructure or directly on bare metal. Each tenant cluster gets its own API server, custom resource definitions (CRDs), and role-based access control (RBAC), indistinguishable from a dedicated Kubernetes cluster, while sharing the underlying nodes and hardware. The virtualized control plane is invisible to tenants: no shared control plane nodes, no in-cluster agent pods, and no lateral path between environments. That makes vCluster a natural fit for GPU infrastructure where teams need their own clean cluster experience without splitting the hardware.
The vCluster shared-nodes model is used for this tutorial, so teams share the GPU node while each gets its own isolated control plane, the right fit for trusted internal teams. For untrusted tenants needing node-, network-, and storage-level separation, the same pattern extends to vCluster private nodes.
The example in this post uses three teams: NLP Team, Vision Team, and Recommender System Team. The NLP Team wants to install their own CRDs. The Vision Team wants cluster-admin to debug scheduling. The Recommender Team is on a different Kubeflow version. Nobody wants to share a kubectl context and accidentally break one another’s environments.
Using vCluster, each team gets their own isolated control plane, RBAC, namespaces, and CRDs. They can each have cluster-admin access. Underneath, all tenant clusters share the same nodes and GPUs.
Demo environment
This demo runs on an NVIDIA Brev GPU instance on Nebius with:
- One NVIDIA L40S, 40 vCPUs, 160 GiB RAM, 256 GiB disk (48 GB VRAM)
- Ubuntu 24.04.4 LTS
- MicroK8s v1.36.2 – Kubernetes was preconfigured by Brev, including the MicroK8s gpu addon, which pre-installs the NVIDIA GPU Operator into the gpu-operator-resources namespace
- KAI Scheduler v0.16.4
- vCluster CLI 0.35.1
Note: For different setups, the cluster-creation and GPU Operator install steps will differ on GKE/EKS/AKS/vanilla k8s/k3s. Step 3 (KAI Scheduler) onward is identical on any Kubernetes that has the NVIDIA GPU Operator running with Container Device Interface (CDI) enabled.
Step 1: Local tooling
Install standalone kubectl and helm to avoid prefixing every command with microk8s. Then wire up a kubeconfig and pin MicroK8s so snap doesn’t auto-upgrade the control plane during the demo.
sudo snap refresh --hold microk8s sudo snap install kubectl --classic --channel=1.35/stable sudo snap install helm --classic mkdir -p ~/.kube sudo microk8s config > ~/.kube/config sudo chown $USER:$USER ~/.kube/config chmod 600 ~/.kube/config
Next, make sure the required MicroK8s addons are enabled:
microk8s enable dns microk8s enable hostpath-storage # vCluster needs PVCs
Then verify:
kubectl get nodes -o wide kubectl get storageclass
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME brev-8dq0cch1j Ready <none> 57d v1.36.2 10.0.0.20 <none> Ubuntu 24.04.4 LTS 6.11.0-1016-nvidia (amd64) containerd://2.2.3 NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE microk8s-hostpath (default) microk8s.io/hostpath Delete WaitForFirstConsumer false 20m
Step 2: Add the Helm repo
Add the NVIDIA Helm repo:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update
Step 3: Confirm the GPU Operator
kubectl get pods -n gpu-operator-resources
NAME READY STATUS RESTARTS AGE gpu-feature-discovery-wpcjm 1/1 Running 0 8m gpu-operator-57d75775c8-npjzz 1/1 Running 0 9m gpu-operator-node-feature-discovery-... 1/1 Running 0 9m nvidia-container-toolkit-daemonset-ft4hb 1/1 Running 0 8m nvidia-cuda-validator-8vqdv 0/1 Completed 0 8m nvidia-device-plugin-daemonset-l6fj6 1/1 Running 0 8m nvidia-operator-validator-cdnmj 1/1 Running 0 8m
If you’re using an older GPU Operator, upgrade in place to 26.3.x:
helm upgrade gpu-operator nvidia/gpu-operator \ -n gpu-operator-resources \ --version v26.3.3 \ --reset-then-reuse-values
If needed, install NVIDIA GPU Operator directly:
helm install gpu-operator nvidia/gpu-operator \
-n gpu-operator-resources --create-namespace \
--version v26.3.3 \
--set driver.enabled=false \
--set operator.defaultRuntime=containerd \
--set toolkit.env[0].name=CONTAINERD_CONFIG \
--set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd.toml \
--set toolkit.env[1].name=CONTAINERD_SOCKET \
--set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \
--set-string toolkit.env[2].name=CONTAINERD_SET_AS_DEFAULT \
--set-string toolkit.env[2].value=1
Step 4: Install KAI Scheduler
Next, install the KAI Scheduler:
helm upgrade -i kai-scheduler \ oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ -n kai-scheduler --create-namespace \ --version v0.16.4 \ --set "global.gpuSharing=true"
Then verify:
kubectl get pods -n kai-scheduler
NAME READY STATUS RESTARTS AGE admission-57556f949-sp98t 1/1 Running 0 47s binder-66785d8dd9-9frgk 1/1 Running 0 46s kai-operator-6fdf595c4d-292d7 1/1 Running 0 52s kai-scheduler-default-6bb667b767-vq2q4 1/1 Running 0 46s pod-grouper-84dfc7759b-v5qtb 1/1 Running 0 47s podgroup-controller-5878f48dbb-v7wcn 1/1 Running 0 47s queue-controller-7796bb8984-hdn5r 1/1 Running 0 46s
Step 5: Define team queues
KAI Scheduler uses a Queue CRD to model an org → team hierarchy. This step involves creating one parent (ml-org, with a total budget of one GPU) and three child queues. Each is guaranteed 0.33 of the GPU and allowed to increase to the full GPU when others are idle.
Save the following as create-queues.yaml:
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: ml-org
spec:
resources:
gpu: { quota: 1, limit: -1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: team-nlp
spec:
parentQueue: ml-org
priority: 100
resources:
gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: team-vision
spec:
parentQueue: ml-org
priority: 100
resources:
gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: team-recommender
spec:
parentQueue: ml-org
priority: 100
resources:
gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
Here, quota is the guaranteed minimum, limit is the maximum allowed, and overQuotaWeight controls how surplus is split.
Next, apply and list:
kubectl apply -f create-queues.yaml kubectl get queues
queue.scheduling.run.ai/ml-org created queue.scheduling.run.ai/team-nlp created queue.scheduling.run.ai/team-vision created queue.scheduling.run.ai/team-recommender created NAME PRIORITY PARENT CHILDREN DISPLAYNAME default-parent-queue ["default-queue"] default-queue default-parent-queue ml-org ["team-nlp","team-vision","team-recommender"] team-nlp 100 ml-org team-recommender 100 ml-org team-vision 100 ml-org
Note that default-parent-queue and default-queue are created automatically by KAI Scheduler on first install. They are the fallback queue for any pod that doesn’t specify one.
Step 6: Spin up a vCluster per team
Next, install the vCluster CLI:
curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" sudo install -m 755 vcluster /usr/local/bin/vcluster rm vcluster vcluster --version
vcluster version 0.35.1
Define the vCluster config. The critical setting is setOwner: false. The KAI Scheduler pod-grouper walks ownership chains (Job → Pod, Deployment → ReplicaSet → Pod) to auto-group workloads. Disabling vCluster owner rewriting allows KAI Scheduler to see the real hierarchy.
cat > vcluster.yaml <<'EOF'
experimental:
syncSettings:
setOwner: false
sync:
fromHost:
nodes:
enabled: true
selector:
all: true
EOF
Then create one vCluster per team:
vcluster create team-nlp --values vcluster.yaml --connect=false vcluster create team-vision --values vcluster.yaml --connect=false vcluster create team-recommender --values vcluster.yaml --connect=false
Verify that all three vClusters are running:
vcluster list
NAME | NAMESPACE | STATUS | VERSION | CONNECTED | AGE
-------------------+---------------------------+---------+---------+-----------+-------
team-nlp | vcluster-team-nlp | Running | 0.35.1 | | 102s
team-recommender | vcluster-team-recommender | Running | 0.35.1 | | 88s
team-vision | vcluster-team-vision | Running | 0.35.1 | | 93s
Each team sees the real nodes, including the GPU node:
vcluster connect team-nlp -- kubectl get nodes
19:09:22 done vCluster is up and running NAME STATUS ROLES AGE VERSION brev-8dq0cch1j Ready <none> 6m20s v1.36.2
Step 7: Deploy a workload from each team
Each team deploys their GPU workload from their own vCluster. The pod spec is simple, with three fields telling KAI Scheduler what to do:
vcluster connect team-nlp -- kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: nlp-sentiment-model
labels:
kai.scheduler/queue: team-nlp # Which team
annotations:
gpu-fraction: "0.33" # How much GPU
spec:
schedulerName: kai-scheduler. # Use KAI, not default
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: nlp-inference
image: nvidia/cuda:12.4.0-base-ubuntu22.04
command: ["bash", "-c", "nvidia-smi; sleep infinity"]
nodeSelector:
nvidia.com/gpu.present: "true"
EOF
19:13:34 done vCluster is up and running pod/nlp-sentiment-model created
Do the same to deploy from each team, changing the queue name.
Now, verify:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE vcluster-team-nlp nlp-sentiment-model-x-default-x-team-nlp 1/1 Running 0 110s 10.1.171.180 brev-8dq0cch1j vcluster-team-recommender recommender-model-x-default-x-team-recommender 1/1 Running 0 6s 10.1.171.187 brev-8dq0cch1j vcluster-team-vision vision-classifier-model-x-default-x-team-vision 1/1 Running 0 14s 10.1.171.145 brev-8dq0cch1j
Three pods, three different vcluster-team-* namespaces, all running on the same physical node brev-8dq0cch1j.
Each team sees only their own pod from inside their vCluster:
vcluster connect team-nlp -- kubectl get pods -o wide vcluster connect team-vision -- kubectl get pods -o wide vcluster connect team-recommender -- kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nlp-sentiment-model 1/1 Running 0 3m11s 10.1.171.180 brev-8dq0cch1j <none> <none> NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES vision-classifier-model 1/1 Running 0 3m59s 10.1.171.145 brev-8dq0cch1j <none> <none> NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES recommender-model 1/1 Running 0 2m45s 10.1.171.187 brev-8dq0cch1j <none> <none>
Finally, the queue confirms the allocation. You can check all three at the same time:
kubectl describe queue team-nlp | grep -A4 Status kubectl describe queue team-vision | grep -A4 Status kubectl describe queue team-recommender | grep -A4 Status
You will see the same result for all:
Status:
Allocated:
nvidia.com/gpu: 330m
Requested:
nvidia.com/gpu: 330m
Note that KAI Scheduler handles scheduling—which pods land on which GPU and in what proportion. It does not enforce GPU memory isolation at the hardware level when GPU sharing is used. Applications need to respect their memory amount (for example, setting –gpu-memory-utilization in vLLM).
Under the hood, the GPU time-slices between CUDA contexts from each pod at kernel boundaries. For hard memory isolation on supported hardware, NVIDIA Multi-Instance GPU (MIG) provides hardware-level partitioning, which can be scheduled by KAI Scheduler as well.
Get started running isolated tenant Kubernetes clusters
KAI Scheduler decides fairly who gets GPU slices and schedules them as a group using GPU sharing and DRA Driver support, hierarchical queues with guaranteed quotas plus over-quota capabilities, gang scheduling, and topology awareness. Once AI workload scheduling is solved, teams want their own clusters. vCluster gives each team its own isolated Kubernetes control plane without the cost of separate infrastructure.
Together, KAI Scheduler and vCluster provide a dedicated cluster experience for three teams with zero waste on a single GPU. The answer isn’t always more GPUs, but better utilization of infrastructure.
Ready to get started? Check out KAI Scheduler, vCluster, and NVIDIA GPU Operator on GitHub.
Learn more about KAI Scheduler and vCluster integration at KubeCon 2026 North America, November 9-12.
— Originally published at developer.nvidia.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from NVIDIA Developer Blog
See more →
Synthetic Data Generation for Financial AI Research with NVIDIA NeMo
NVIDIA's NeMo pipeline generates 502,536 unique financial news headlines in 82 iterations, addressing data imbalance in financial NLP. The iterative approach uses semantic deduplication and category-weighted sampling to enhance diversity and relevance in generated content.
