One control plane manages clusters, models, and GPU instances
GPUStack separates the server from the machines doing inference. The server exposes the API and UI, stores state, schedules model instances, and reconciles desired replicas. Workers detect accelerators, start inference containers, and report metrics. Higress routes client requests to those instances. The result is one place to manage models and issue SSH-accessible GPU instances instead of treating every GPU host as a hand-built snowflake.
The README lists 9 accelerator families, including NVIDIA, AMD, Ascend, Hygon, MetaX, MThreads, Iluvatar, Cambricon, and T-Head hardware. Support depth varies. The requirements page marks MThreads, Iluvatar, Cambricon, and T-Head experimental, and available inference backends differ by vendor. NVIDIA gets vLLM, SGLang, VoxBox, and custom backends; some other devices have one built-in option plus custom integration. Check the vendor table before assuming a named accelerator can run the engine you want.
Linux workers need privileged access and a wide port range
The quick start makes the server look simple: run a published container, expose port 80, persist /var/lib/gpustack, and read the generated admin password from that volume. A CPU-only host can run the server. The worker is the real infrastructure job. The documented Docker command uses privileged mode, host networking, the host Docker socket, a vendor runtime, a worker token, and an advertised address. Security teams should review that command before a trial.
Worker nodes support Linux on AMD64 or ARM64. Windows users are directed toward WSL2, and macOS cannot act as a worker. Distributed vLLM also needs worker-to-worker connectivity and reserves TCP 41000 through 41999 for Ray services; inference uses 40000 through 40063. NTP is required so node state stays synchronized. These are ordinary cluster requirements, yet they are far beyond the effort implied by starting the server container alone.
Scheduling filters for fit before it scores placement
The scheduler first removes workers with the wrong cluster, labels, status, backend support, selected GPU, or local model path. It then checks whether RAM and VRAM can hold the model. Candidate strategies proceed from one GPU to several GPUs on one worker, then distributed placement, with CPU or partial offload available only for supported model and backend combinations. This ordering is documented clearly enough to diagnose why a model never receives a placement.
After filtering, GPUStack scores placement and model-file locality. Binpack fills fewer GPUs; spread distributes replicas for fault tolerance. Existing files raise a candidate's score, including draft-model files used for speculative decoding. Scale-down uses health, offloaded layers, and placement to decide which replica goes first. The logic is more useful than a generic least-loaded rule, but estimates from GGUF parsers or backends still need confirmation on the exact 8 GB, 24 GB, or larger devices in your fleet.
What happened when we ran it
Our fresh Python 3.12 sandbox installed commit 76dc841 in 52 seconds. The environment gained 157 packages and occupied 525 MB. Building succeeded in 7 seconds, and pip-audit reported 0 known vulnerabilities. The repository itself had 1,362 files, about 228,133 source lines, and an 88.2 MB checkout. Six CI workflow files and a tests directory were present, while no Dockerfile was found.
Tests returned exit code 1 after 237 seconds. Pytest reported 3,738 passed and 19 failed out of 3,757, with 71 skipped. The log tail places the listed failures in tests/k8s/test_bootstrap_manifest.py. Scalar-type and ownership cases raised FileNotFoundError for paths below /work/repo/gpustack/ui/static/charts/. The log does not say why those chart files were missing, so the defensible finding is that this commit did not pass its full suite in our stated container.
API key storage deserves a security review
GPUStack creates an admin account at bootstrap and stores its initial password under the data directory. Users can create API keys with platform-management or model-access permissions, optionally limited to selected models. Auto-generated keys contain 128 bits from a cryptographic random source. Custom strings can be weak, and the documentation warns that their unsalted hashes may be published into gateway configuration and diagnostic bundles unless custom-key gateway authentication is disabled.
Open issue 6173 concerns deletion outside the supported API. It reports that the fallback lookup omitted a deleted_at filter: a raw-SQL soft delete continued authenticating, while a hard or cascade deletion could remain cached until the default 600-second TTL expired or the process restarted. The report notes that the product did not have a normal soft-delete trigger for the first case. This is still worth testing anywhere identity lifecycle or database automation can bypass the normal delete route.
Version 2.2.3 is maintained, with active operational reports
Release v2.2.3 shipped on July 31, 2026, with a fix for Claude Code requests routed through vLLM. GitHub recorded another push on September 10. The repository had 5,655 stars and 684 open issues and pull requests when fetched, so that combined number should not be read as 684 confirmed bugs. Same-day activity included issue 6197, which reports intermittent 409 conflicts when a deleted model name is immediately reused.
GPUStack fits a team that has outgrown one inference container and is ready to own a control plane. Apache-2.0 licensing, multiple backends, an explicit scheduler, external database options, and OpenAI-compatible APIs are credible foundations. The costs are equally concrete: privileged worker access, vendor-specific drivers, broad networking, a 525 MB development environment, and 19 failing Kubernetes tests in our run. A proof of concept should include worker loss, key revocation, model replacement, and one distributed deployment alongside a successful chat request.