KServe v0.20 splits predictive models and LLMs across two APIs
KServe v0.20 has two main serving resources. InferenceService handles predictive models and basic LLM deployments, while LLMInferenceService adds distributed placement, prefill and decode separation, and inference-aware routing for larger language models. Built-in runtimes cover TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, and vLLM. Model storage can come from object stores, persistent volumes, OCI images, or Hugging Face, with the controller turning each declaration into Kubernetes workloads and routes.
The scope is visible before a cluster starts. commit c1adffc occupied 469.4 MB and contained 3,124 files with roughly 1,205,516 lines of source. The repository includes controllers, Python model servers, storage helpers, Helm charts, generated CRDs, release installers, samples, and the tf2openapi utility. That size is reasonable for a shared inference platform, but excessive for a team that needs one HTTP endpoint around one model. Repository metadata identifies its license as Apache-2.0.
What happened when we ran it
Our measurement setup used commit c1adffc in a fresh unprivileged Debian container with 3 CPUs, 8 GB of RAM, no secrets, and the golang:1.24-bookworm image. Installation succeeded in 213 seconds and brought in 1,132 packages. The build also succeeded, taking 339 seconds. The checkout contained a Dockerfile, a tests directory, and 51 CI workflow files.
Tests failed with exit code 1 after 139 seconds. Go reported 81 passed and 23 failed out of 104. The supplied log tail names failing cases in tools/tf2openapi/cmd, including flag, input, and file-output cases, plus failures in tools/tf2openapi/generator and tools/tf2openapi/types. The tail does not say why those assertions failed, so we cannot responsibly blame the Go image, missing system software, or the code itself without more evidence.
Kubernetes 1.32 starts a longer platform setup
The v0.20 quickstart requires Kubernetes 1.32 or newer, kubectl, Helm, Git, and a working kubeconfig. Kind and Minikube are offered for local experiments, and the page warns that quickstart environments are not production installations. A real rollout also needs ingress or Gateway API decisions, DNS, certificates, resource quotas, model-storage credentials, and observability. The first LLM tutorial assumes a GPU and shows a Hugging Face token path for gated model access.
Repository development adds its own version wrinkle. The c1adffc go.mod declares Go 1.26.8, while our supplied container image was Go 1.24; the 213-second install and 339-second build still completed. Cluster installation is a separate exercise that our sandbox did not perform. We did not install CRDs, create a Kubernetes cluster, pull model weights, provision GPUs, or send inference traffic, so the successful build says nothing about model startup time or serving capacity.
Standard mode gives up scale-to-zero, while Knative targets predictive traffic
The v0.20 administrator guide assigns Standard InferenceService to predictive and generative workloads where direct resource control matters. Knative mode is the scale-to-zero route for predictive traffic. LLMInferenceService is the specialized choice for prefix routing and distributed language-model serving, while ModelMesh targets dense fleets of changing models. These are meaningful architectural divisions. Moving from a simple predictor to distributed LLM serving changes networking, scheduling, GPU topology, caching, and the custom resources an operator must understand.
Documentation has one material inconsistency. The current README says Standard installation does not support canary deployment or request-based autoscaling with scale-to-zero. Release v0.20.0 lists canary rollout support for RawDeployment among its changes. The 51 workflow files and large sample tree suggest serious engineering attention, but they do not resolve which combinations are supported for a buyer. Verify the exact controller mode and resource type on a disposable cluster before promising rollout behavior to an application team.
A v0.20 gateway regression can report unready while requests return HTTP 200
Open issue 5986 documents a precise v0.20 regression: an LLMInferenceService using a bring-your-own gateway, a managed scheduler, and no managed route remains Ready=False even though the data plane returns HTTP 200. The reporter ties the behavior to readiness evaluation and says the same shape worked on v0.19. A linked correction was under review when checked. That issue matters because automation often treats readiness, rather than a manual completion request, as the source of truth.
Other current reports show operational edges beyond model execution. Issue 5754 says OCI Helm charts use v-prefixed versions that some GitOps discovery tools will not accept. Issue 6167 describes unbounded inference-logger work when a downstream log endpoint stalls, eventually exhausting memory in the reported deployment; an associated fix was open. GitHub listed 221 combined open issues and pull requests, while our own suite had 23 failures. Read the queue by subsystem instead of treating that combined number as a bug count.
Three alternatives cover smaller or more specialized serving estates
KServe spans 2 serving APIs plus several deployment modes, which is exactly why smaller teams should compare it with narrower tools. BentoML starts from packaging a model or AI application as a service. Seldon Core is the direct Kubernetes platform comparison for organizations already familiar with its deployment model. KubeRay fits teams whose models and distributed jobs already run through Ray. None is automatically simpler once your requirements include shared GPUs, advanced routing, autoscaling, and several model frameworks.
Our 1,132-package install and 339-second build show that KServe asks for platform-level commitment before the first model reaches a cluster. The 81 passing tests prove much of the checked path worked, while 23 failures keep this commit from earning a clean bill of health. Adopt it when one Kubernetes team owns inference for many internal users and can test each chosen mode. For a single endpoint, the 469.4 MB checkout and multi-controller operating model are unnecessary weight.