mrkeyoor.com_
Sat 08 Aug 21:02 UTC
Automationevaluationupdated 08 Aug 2026

kueue

Kueue is a Kubernetes controller that decides when batch, AI, and other resource-heavy jobs are allowed to start. It solves the shared-cluster problem where many teams compete for scarce CPUs, memory, GPUs, or devices and ordinary pod scheduling alone cannot enforce fair job-level queues and quotas.

Verdict

Kueue is the best first choice for Kubernetes-native job queueing when you want to keep the normal scheduler and existing job APIs. Its queue model, broad integrations, quota sharing, and production test matrix are unusually strong, but policy design and upgrades deserve platform-level ownership. Adopt it for genuinely contested capacity; do not add it to a lightly used cluster merely because a queue sounds tidy.

Setup3/5Simple controller install, demanding quota and preemption design
Docs5/5Deep concepts, tasks, integrations, metrics, and upgrade guidance
Community5/5Very active Kubernetes SIG project with rapid review and issue traffic
Maturity4/5Production adopters and v1beta2 API, with advanced areas still moving

Who it’s for

  • Platform teams sharing expensive GPU, accelerator, or large CPU pools across several tenants.
  • Kubernetes operators who need job priorities, quotas, borrowing, fair sharing, and preemption at workload level.
  • Organizations running Kubernetes Jobs, Kubeflow training jobs, Ray, JobSet, pod groups, or supported serving workloads.
  • Multi-cluster teams prepared to operate a manager cluster, worker credentials, and Kueue on every participating cluster.

Who it’s NOT for

  • Clusters older than Kubernetes 1.34: the v0.19 README says only 1.34 and newer are officially supported and tested.
  • Teams seeking a replacement for pod placement or node packing: Kueue makes job admission decisions, then the normal Kubernetes scheduler places pods.
  • Users needing a complete workflow or DAG engine: integration with workflow frameworks is explicitly listed only as a long-term aspirational goal.
  • MultiKueue designs where the manager must also be its own worker: the current documentation lists that arrangement as unsupported.
  • Operators expecting minor upgrades to be automatic: v0.19 has mandatory upgrade notes covering manual Helm resource cleanup, changed quota accounting, a new readiness timeout default, and kubeconfig path restrictions.
  • Teams requiring a stable v1 API today: the project exposes v1beta2, while graduation to v1 remains aspirational.

Setup reality

The controller itself is easy to install with one server-side kubectl apply, followed by a small example that creates a ClusterQueue, LocalQueue, and ResourceFlavor. Production design is the harder part: model each tenant's quota, borrowing cohort, priority, preemption policy, resource flavors, and admission checks, then prove those rules under load. Monitoring, webhook certificates, upgrades, and integration-specific feature gates add operational work. MultiKueue is another tier entirely, requiring Kueue on multiple clusters, credentials, matching resources, and careful failure testing.

Admission control for jobs, not another pod scheduler

Kueue sits one level above the normal Kubernetes scheduler. It decides whether a whole job may start and how much quota that job reserves. Once admitted, Kubernetes still decides where the pods run. This separation is the key to understanding both the appeal and the limits. Kueue can stop ten large training jobs from flooding a cluster, preserve capacity for higher-priority work, and let teams share unused quota without replacing the scheduler that already understands nodes.

The core objects form a sensible hierarchy. A namespace-scoped LocalQueue is where a team submits work. It points to a cluster-scoped ClusterQueue that owns resource quotas and admission policy. ResourceFlavors distinguish kinds of capacity, such as GPU models or spot and on-demand nodes. Cohorts let ClusterQueues borrow unused quota from one another, while priorities, fair sharing, and preemption determine who moves first when demand exceeds supply.

This is most valuable when capacity is expensive and jobs are elastic in time. A web service cluster with ample headroom may gain little. A shared AI cluster where one training run can consume every GPU is exactly the problem Kueue was made to solve.

Broad integrations without forcing a new job API

Kueue works with ordinary Kubernetes Jobs and also has built-in integrations for Kubeflow training jobs, RayJob, RayCluster, JobSet, plain pods and pod groups. It can manage Deployments and StatefulSets alongside batch workloads, although serving workloads bring different interruption expectations. Partial admission can reduce a job's parallelism to fit available quota, and dynamic reclaim returns quota as pods finish.

AdmissionChecks provide an extension point before a workload starts. Kueue uses them for cluster-autoscaler ProvisioningRequests and MultiKueue, and external controllers can add policy or provisioning decisions. The WaitForPodsReady mechanism adds an all-or-nothing safeguard: if admitted pods do not become ready within the configured window, Kueue can release capacity so another workload gets a chance. Version 0.19 enables this behavior by default with 30-minute wait and recovery timeouts, so upgrades need an explicit review of long-starting jobs.

Kueue also understands topology-aware scheduling constraints, useful when distributed training depends on racks, blocks, or other failure and network domains. These advanced paths are powerful, but they multiply the states an operator must explain. Start with quotas and a small set of flavors before enabling fair sharing, topology, dynamic devices, elastic jobs, and multi-cluster dispatch together.

A quick install followed by policy engineering

The supported v0.19 path requires Kubernetes 1.34 or newer. Installation is a server-side apply of the published manifest, placing the controller in kueue-system. A minimal example creates the queue resources, after which a labeled Job can be submitted. That is enough to prove the admission loop in a test cluster.

A real rollout needs workload measurements. If CPU, memory, GPU, sidecar, or pod-overhead accounting is wrong, quota decisions will also be wrong. Administrators must choose nominal quotas, borrowing limits, queue strategy, priorities, preemption rules, namespace access, and behavior for jobs without a queue name. Then they need dashboards and alerts for pending workload age, failed admission checks, evictions, quota use, and pods that fail to become ready.

Prometheus metrics and a pending-workloads visibility endpoint are built in. One long-running open issue, updated August 8, says controller-runtime workqueue metrics appear on the visibility server rather than the expected manager metrics endpoint. The data exists, but operators copying a standard controller dashboard may need to scrape the second endpoint or adjust their setup.

MultiKueue is useful, not free federation

MultiKueue sends an admitted workload from a manager cluster to one or more worker clusters, then keeps the copy on the worker that accepts it. Version 0.19 supports all-at-once and incremental dispatch, with ordered clusters enabling patterns such as trying on-premises capacity before cloud spillover. It covers many of the same integrated job types as single-cluster Kueue.

The documentation is candid about boundaries. The manager cannot currently act as one of its own workers. For StatefulSet and LeaderWorkerSet, manager-side status may not reflect actual worker status. An external dispatcher must use the expected field manager or enable a merge-patch feature gate, otherwise workloads cannot be admitted. Production credentials should use ClusterProfile or Secret; path-based kubeconfigs are marked for development, and v0.19 tightened path handling after a security fix.

This is a capable dispatch layer, but it requires matching APIs, queues, resource flavors, credentials, and observability across clusters. Test network partitions, stale credentials, manager restarts, duplicate work, and status lag before trusting it with costly jobs.

Strong release discipline, with change still arriving fast

Kueue describes a two-to-three-month release cycle, uses a v1beta2 API under Kubernetes deprecation rules, publishes extensive unit, integration, end-to-end, topology, MultiKueue, and performance test grids, and lists production adopters. Version 0.19 arrived July 22, 2026. The repository was pushed on August 8, with active issue and pull-request discussion that same day. GitHub search showed 393 open issues, while the repository reported 581 combined issues and pull requests. Much of that queue tracks features, design proposals, and development-branch work across a very large surface.

The latest release notes deserve respect: they explicitly demand reading upgrade actions, including manual cleanup for earlier KueuePopulator Helm resources, changed resource accounting for Ray sidecars, a security restriction for MultiKueue kubeconfig paths, and new readiness defaults. This is healthy transparency, not plug-and-forget stability.

Kueue is ready for production when a platform team owns the queue policy and release process. Its documentation and test coverage make that job possible. The project still calls workflow-framework integration and v1 graduation long-term aspirations, so buy what exists now: excellent Kubernetes-native admission and quota control, not a promised workflow platform.

Alternatives

ProjectWhat it isPick it when
VolcanoA Kubernetes batch system with its own scheduler, queues, gang scheduling, and job controller.pick this instead when you want admission and pod scheduling in one batch platform, including scheduler plugins and Volcano-native jobs.
Apache YuniKorn Kubernetes ShimA Kubernetes integration for Apache YuniKorn's application-aware resource scheduler.pick this instead when hierarchical queue scheduling and direct scheduler control fit better than Kueue's admission layer.
Flux OperatorA Kubernetes operator for running Flux-managed high-performance computing workloads.pick this instead when the workload is HPC-first and needs a nested Flux scheduler rather than a general Kubernetes admission queue.

What people are saying

  1. [github-trending] kubernetes-sigs/kueue

Sources

  1. Kueue README
  2. Kueue overview
  3. MultiKueue concepts and limitations
  4. Kueue 0.19.0 release
  5. Controller metrics endpoint report