Gang scheduling waits until the group can run
A distributed training job can waste capacity when some workers start while the rest wait for GPUs. KAI-Scheduler supports gang scheduling: the pod group gets scheduled together or stays waiting. Hierarchical queues then determine whose workloads deserve resources, who can borrow unused capacity, and whose running work can be displaced. That's a sensible combination for a shared AI cluster, provided someone owns the policy decisions. It can operate alongside other Kubernetes schedulers, so adoption can start with selected workloads.
Our checkout contained 1,575 files, roughly 256,611 lines of source, and occupied 12.1 MB. The scope includes GPU sharing, topology-aware placement, and resource reclamation, the process of taking capacity back from running workloads. These features matter when placement and competing teams are already operational problems. On a cluster with one owner and straightforward jobs, another scheduling policy layer can consume attention without solving an urgent problem. Apache-2.0 licensing makes evaluating and modifying the code practical.
What happened when we ran it
Our run installed 759 packages in 109 seconds and successfully built KAI-Scheduler in 370 seconds. We cloned commit 41a22cd on October 8, 2026, into a fresh unprivileged Debian sandbox using golang:1.24-bookworm, with 3 CPUs, 8 GB of RAM, and no secrets. The repository scan found 20 CI workflow files, a Dockerfile, and a tests directory. The successful build establishes that this source revision compiled in that environment. It doesn't establish GPU utilization or scheduling throughput.
The test step failed with exit code 1 after 351 seconds: 189 tests passed and 62 failed out of 251. The log tail lists failures under the end-to-end reclaim, resize, timeaware, and upgrade suites, including TestInPlacePodResize, TestTimeAware, and TestUpgrade. It doesn't contain the underlying exception or assertion text. The tail identifies the failing suites without enough detail to diagnose them. The result is a failed test step, with those named end-to-end suites requiring closer inspection.
Every workload needs a queue and a scheduler name
The quick start requires pods to carry a kai.scheduler/queue label and set schedulerName: kai-scheduler. A fresh deployment creates a default parent queue and leaf queue, but your organizational policy still needs configuring. Submit workloads in a dedicated namespace, not the scheduler's namespace. Kubernetes access and Helm are prerequisites, and the README requires NVIDIA GPU Operator for workloads requesting GPU resources. Installing the chart presumes you're already able to administer that cluster.
The 759-package source install is a contributor cost, separate from the policies you must maintain after deploying the chart. KAI's fairness guide distributes deserved quotas first, then spare capacity according to priority and weight. Queues borrowing beyond their allocation can become reclaim targets. Before rollout, write down which team can interrupt which workload and whether the interrupted job can resume. Queue labels encode those agreements. They can't create agreements your teams haven't made.
Default fractional scheduling does not enforce GPU memory limits
KAI's GPU-sharing guide names NonMemoryEnforced as the default mode. A pod can request a fraction of a GPU or an amount of GPU memory, yet the container may see and use more memory than requested. That is an important distinction for teams sharing hardware between less-trusted tenants. A scheduling reservation describes placement and accounting. The default mode doesn't turn that reservation into a CUDA memory boundary.
Our 3-CPU, 8 GB sandbox run didn't measure GPU-sharing isolation. For runtime enforcement, the documentation points to HAMi-core or NvFractions, each with additional components and configuration. NvFractions needs the GPU-sharing operator, with CDI/NRI configured when enforcement is required. Air-gapped operators must mirror the additional chart and images as well as KAI's own artifacts. If enforced memory limits are a requirement, choose and validate that runtime path before accepting fractional requests from users.
Consolidation can restart work that cannot recover
Open issue 2283 describes consolidation deleting running pods so their controllers can recreate them elsewhere. The reporter says this failed training replicas using restartPolicy: Never and permanently removed bare pods. The requested per-workload consolidation opt-out is attached to an open proposal. Don't assume it's in the released chart. Becoming non-preemptible has a policy cost too: the issue says that workload must fit its queue's quota and loses reclaim eligibility. Teams relying on borrowed capacity should examine that tradeoff.
The 62 failures in our suite don't establish a connection to that consolidation report. A separate open issue 2371 reports components receiving authorization errors after kai-config is deleted and recreated, including through ArgoCD self-heal. That is directly relevant to GitOps recovery, rather than evidence about ordinary placement. Both reports deserve a targeted check on your chosen release. Neither justifies treating the entire scheduler as unusable, but restart-sensitive training and automatic configuration recovery need more proof than a successful compile.
October maintenance supports a trial on a pinned release
KAI-Scheduler's last push was October 9, 2026. GitHub returned 1,561 stars and 226 open issues and PRs, with recent discussion about binder behavior and consolidation. Release v0.18.3 arrived October 6 with a GPU-fractioning chart update. The support policy lists v0.18 as an active LTS line through September 2027. Those dates and current reports show maintenance. They also make release selection consequential: documentation on main can describe behavior beyond the chart you pin.
The 20 workflow files give you upstream checks to inspect, while our 351-second failed test step leaves adoption work unfinished. Evaluate KAI when GPU sharing and hierarchical team quotas solve a problem you can name. Compare Volcano when broader batch scheduling is the priority, or Kueue when deciding which jobs may start is the main need. For KAI, make the acceptance workload include a reclaimed training job, a fractional-memory request, and configuration recovery. Require the chosen release to behave correctly when a worker is interrupted, not merely when capacity is free.