Advanced GPU Scheduling in Kubernetes: Beyond the Basics

Basic GPU scheduling in Kubernetes gets you one model per GPU. Advanced scheduling gets you 10x better utilization from the same hardware. This guide covers the techniques that separate efficient GPU clusters from expensive underutilized ones.

The Baseline Problem: One Workload, One GPU

Default GPU scheduling in Kubernetes is all-or-nothing. A pod requests 1 nvidia.com/gpu, gets an entire physical GPU, and holds exclusive access until the pod terminates. For large training jobs that saturate GPU memory and compute, this is fine. For inference workloads that use 5-15% of a GPU's capacity, this wastes 85-95% of very expensive hardware.

Multi-Instance GPU (MIG): Hardware-Level Partitioning

MIG (Multi-Instance GPU) is an NVIDIA feature available on A100, A30, and H100 GPUs. It partitions the physical GPU at the hardware level into isolated instances — each with dedicated compute engines, cache, and memory bandwidth. No resource contention between MIG instances; each performs like a smaller dedicated GPU.

Common MIG profiles for an A100-80GB: 7x 1g.10gb (7 instances, 10GB each), 4x 2g.20gb (4 instances, 20GB each), or 2x 3g.40gb (2 instances, 40GB each). Match the profile to your inference workload memory requirements. The NVIDIA MIG Manager handles profile switching and the device plugin exposes MIG instances as schedulable resources.

GPU Time-Slicing: Broader Model Support

MIG requires specific NVIDIA GPU models. For older hardware (T4, V100, RTX), GPU time-slicing provides shared access via context switching. Multiple containers run on the same GPU, with the GPU switching between them. Unlike MIG, time-slicing does not provide memory isolation — a poorly behaved container can still consume all GPU memory. But it dramatically improves utilization on hardware that doesn't support MIG.

Configure time-slicing in the NVIDIA device plugin ConfigMap. Set replicas: 10 for a GPU to make it appear as 10 schedulable resources to Kubernetes. Each workload requesting 1 nvidia.com/gpu gets time-shared access to the physical device.

Gang Scheduling for Distributed Training

Distributed training jobs (PyTorch DDP, TensorFlow MirroredStrategy, Horovod) require all workers to start simultaneously. Kubernetes' default scheduler provisions pods one at a time — this creates deadlock when a multi-GPU job partially allocates GPU nodes and holds them while waiting for the rest, blocking other jobs from proceeding.

Volcano and Yunikorn both implement gang scheduling for Kubernetes. They hold all pods of a job in a pending queue until all required resources are simultaneously available, then schedule the entire job atomically. This prevents GPU deadlock on large training clusters and improves overall cluster throughput.

Topology-Aware Scheduling: Placing Pods on NVLink-Connected GPUs

Not all GPU-to-GPU communication is equal. NVLink provides 600 GB/s bandwidth between GPUs within the same node. PCIe provides 64 GB/s. Placing distributed training workers on GPUs connected via NVLink versus PCIe can mean 3-5x faster all-reduce communication in multi-GPU training.

The NVIDIA GPU Feature Discovery plugin exposes NVLink topology as node labels. The Topology Manager in Kubernetes (beta, enable with --topology-manager-policy=best-effort or restricted) uses this information to co-locate containers on NUMA/NVLink-aligned GPUs. This is critical for large-scale LLM fine-tuning and Multi-Node training jobs where communication overhead dominates training time.

Dynamic Resource Allocation (DRA): The Future of GPU Scheduling

Dynamic Resource Allocation (DRA) is a Kubernetes feature (beta in 1.29+) that replaces the device plugin model with a more flexible API. DRA lets workloads request structured resource claims — not just count-based limits — including constraints like "give me 4 GPUs all connected via NVLink on the same NUMA node." NVIDIA has published a DRA driver for Kubernetes. DRA is the long-term future of GPU scheduling in Kubernetes; track its graduation to GA in upcoming releases.

Advanced GPU Scheduling with THNKBIG

Getting GPU utilization above 70% on a mixed inference and training cluster requires careful scheduler configuration, topology awareness, and workload profiling. THNKBIG's AI infrastructure team has designed GPU Kubernetes platforms for teams running LLM serving, computer vision training, and multi-tenant AI workloads. Contact us to optimize your GPU cluster.