A traffic controller for local models
llama-swap solves a problem that appears after the local AI demo works. A workstation may have enough disk space for many models but only enough RAM or GPU memory to run one or two. Applications still want a stable API address and familiar model IDs. llama-swap stands between those applications and the inference servers. It reads the requested model, starts the matching command, waits for it to become ready, unloads conflicting processes, and proxies the request.
This approach is refreshingly backend-neutral. The README lists llama.cpp and its forks, vLLM, stable-diffusion.cpp, audio.cpp, and ComfyUI, while saying any OpenAI-compatible server that runs from the command line can be managed. OpenAI-style routes cover chat, completions, responses, embeddings, speech, transcription, voices, and images. Anthropic-compatible message and token-counting routes are supported too. llama.cpp-specific reranking, infill, properties, and completion paths fill gaps that a generic proxy would miss.
llama-swap is therefore not a model runtime. It is the process supervisor and routing layer around runtimes you choose. That separation is its chief strength and the source of most setup work.
The small configuration claim holds up
Installation options are excellent for a young infrastructure tool. There are release binaries for Linux, macOS, Windows, and FreeBSD, plus Homebrew, MacPorts, WinGet, Docker, and Podman paths. The Go binary has no runtime dependencies. Building from source adds Go and Node.js because the interface must be compiled.
The minimum configuration contains a model ID and a command such as llama-server with a model path and the supplied ${PORT} variable. A request naming that ID causes llama-swap to launch the server on an assigned port. One model runs at a time by default. Time-to-live settings can unload idle models, aliases can present familiar names, and environment variables or macros reduce repetition.
That is genuinely simple once the upstream command works. llama-swap does not download the right weights, select quantization, calculate context memory, or repair an incompatible GPU stack. You must first prove each server command at the shell, then transfer its flags and paths into YAML. The README sensibly recommends containers for Python-based servers so termination signals and dependency isolation behave predictably.
The recommended unified container bundles several inference tools and now includes audio.cpp, llama-bench, and a vLLM wrapper. It reduces assembly work for CUDA or Vulkan users, but is not universal. A fresh report against the unified image describes a CUDA forward-compatibility failure and core dumps on older NVIDIA hardware. Test the exact image tag and driver combination before treating it as an appliance.
Swapping is good, scheduling is manual
For a single GPU and mutually exclusive large models, llama-swap's behavior is exactly right. A client changes the model value and the machine makes room. The web interface shows available and running models, request activity, token metrics, captured requests and responses, logs, and manual load controls. Profiles can change model mappings at runtime, which is useful when a fixed client cannot easily change IDs.
Concurrent placement uses a matrix DSL. You describe variables and allowed combinations, giving llama-swap explicit knowledge of which servers may coexist. This is more flexible than a fixed one-model rule, but it is not an automatic memory scheduler. Open issue 1004 comes from a user with eight models who finds enumerating combinations difficult and still unloads models manually near the memory limit. The requested global RAM budget and per-model footprint declarations do not exist yet.
Treat the matrix as policy you own. Measure each command under its actual context and offload settings, reserve headroom, and update the expressions when models or flags change. Teams expecting a cluster scheduler will find this too manual; home-lab owners may appreciate the direct control.
Useful operations, limited authorization
The built-in interface is more than decoration. It exposes live and buffered logs, model-specific streams, health, Prometheus metrics, profiles, running processes, unload actions, and direct upstream access. Request filters can strip or set parameters before forwarding. Behind nginx, the README warns that response buffering must be disabled for server-sent events and streaming completions, even though llama-swap also emits the relevant anti-buffering header.
Basic API keys protect endpoints, but the current authorization model is too broad for untrusted clients. Open issue 971 says each valid key can access every local model, peer, log, unload action, upstream path, and the UI. Logs may contain prompts, while unload rights can disrupt other users. Another open request notes that changing keys in YAML reloads configuration and can discard prompt caches; keys may also come from environment variables, but watched key files are only a proposal.
Keep the service bound to localhost or a trusted network unless another gateway supplies TLS, scoped identity, rate limits, and audit policy. The built-in keys are a useful lock for a personal server, not a tenant boundary.
Very active, with visible rough edges
The repository was pushed on August 14, 2026, and release v250 arrived the same day. The twelve releases from v239 through v250 landed between July 11 and August 14. GitHub showed 78 open issues and pull requests, with 47 true open issues. Current reports and merged work cover containers, new endpoints, model metadata, UI capabilities, client compatibility, and routing behavior. This is energetic maintenance with substantial outside contribution.
The pace also means operators should pin builds and read release notes. Open reports include opaque client errors when an upstream process exits, an absent official ARM64 container, and edge cases around client timeouts and proxy behavior. None cancels the core value, but each argues against unattended upgrades.
For a trusted local network, llama-swap is an easy recommendation. It turns a drawer full of server commands into one API and makes scarce hardware much less annoying. For shared production access, place it behind a policy-aware gateway, or choose a system designed around tenant controls from the start.