mrkeyoor.comAsk

mostlygeek/llama-swaptested 13 Oct 2026commit a025e03

llama-swap review

llama-swap is worth a local evaluation when you already operate inference servers and want request-driven switching behind one endpoint.

  • Install 106 s
  • Build 196 s
48/51passed, run exited non-zero
  • 724 packages
LLM Toolsevaluationupdated 01 Nov 2026
+28stars / 7d

Choose llama-swap if you want one endpoint to start and switch between configured local inference servers, and you are prepared to manage those servers yourself. llama-swap selects an upstream server from the requested model ID and replaces the wrong running upstream. Our run on October 13, 2026, at commit a025e03 recorded successful install and build steps, but the test command failed. We did not test model inference, GPU compatibility, swap latency or deployment.

Verdict

llama-swap is worth a local evaluation when you already operate inference servers and want request-driven switching behind one endpoint. Our recorded build succeeded, but the test command failed, so this run provides no clean-test endorsement. Validate one upstream, reproduce the relevant tests and check streaming and shutdown on your intended setup. Treat the later capacity-router work as a separate version check, not a result measured at a025e03.

We ran it

Lab card: what happened when we ran llama-swap
Lab card from our sandbox run
Screenshot of llama-swap (github.com/mostlygeek/llama-swap)
github.com/mostlygeek/llama-swap
Install✓ · 106s724 packages
Build✓ · 196s
Tests✗ · 72s48 passed · 3 failed of 51 (go test)
Repo757 files~93,784 lines of source · 7.3 MB · 11 CI workflows

Answers from our run

Does llama-swap build from source?

Dependencies installed in 106 seconds (724 packages), and the build succeeded in 196 seconds. We cloned commit a025e03 into a clean Debian container with 3 CPUs and no project-specific setup.

Do llama-swap's tests pass?

No. The project's own test command finished in 72 seconds and reported 48 passed, 3 failed (go test).

Who should not use llama-swap?

Readers seeking a measured inference-speed or memory-savings comparison: our run establishes neither.

What are the alternatives to llama-swap?

Ollama, LocalAI. llama-swap is worth a local evaluation when you already operate inference servers and want request-driven switching behind one endpoint.

Setup4/5Strong documented installation options and successful recorded install/build steps; configuring and validating upstream servers remains operator work.
Docs5/5Comprehensive within the reviewed scope: the pinned README explains installation, model commands, routes, lifecycle controls and Nginx streaming caveats.
Community5/5Assesses substantive public collaboration only: issue 1004 contains user scenarios, contributor proposals, owner feedback and completed work. Adoption and support responsiveness were not measured.
Maturity4/5Documented lifecycle, logging and monitoring controls plus successful install/build outcomes support this narrow assessment; the recorded test command failed and deployment reliability remains untested.

Who it’s for

Developers who already have local inference servers and want to route requests through one address.
Operators willing to configure model commands, shutdown behavior and concurrent model combinations.
Local AI users who need documented idle unloading, manual load controls and upstream logs.

Who it’s NOT for

Readers seeking a measured inference-speed or memory-savings comparison: our run establishes neither.
Users expecting the proxy configuration to replace installing and validating their inference backends and model files.
Teams requiring a clean test result from this recorded sandbox run.
Operators expecting resource requirements to be discovered automatically: the supplied capacity-router discussion describes configured resource limits and model requirements.

Setup reality

Our sandbox recorded an install step of 106 seconds, a build step of 196 seconds and a failed test step of 72 seconds on October 13, 2026, at commit a025e03. The command exit codes were 0, 0 and 1 respectively; none of those steps timed out. The build tail ends with vet-exit:0. These are command outcomes, not evidence that a model server or inference workload ran successfully.

The sandbox used 3 CPUs, 8 GB of memory and the golang:1.24-bookworm image. The install tail records downloading Go 1.27.1, so the image name alone does not establish the compiler version used throughout the run.

The pinned README requires Go and Node.js when building from source. Its minimum configuration identifies a model and the command that starts its upstream server, with ${PORT} supplying an assigned port. Validate that upstream and its model files before relying on proxy routing. If you use containers, select an image appropriate to your hardware and configure its mounts and shutdown behavior.

Our run built successfully, but tests failed

Our run on October 13, 2026, at commit a025e03 recorded successful install and build steps, but the test command failed. The full recorded commit is a025e0373addc2cd76e77a5064d60c517f05dcf8. These observations come from our stored sandbox record; this correction involved no new execution.

The install, build and test steps took 106, 196 and 72 seconds respectively. Our harness records each command's success flag, exit code, elapsed time and timeout status, alongside parsed output. Here, install and build returned zero, while tests returned one without timing out. A zero build exit does not establish that a model loaded, an inference request completed or a deployment worked.

The attached stored parser summaries report 48 passing and 3 failing package results, with a total of 51. They do not establish exhaustive unique test coverage. The supplied tail repeats package results under a summary section, so counting those lines again would inflate the evidence. No complete-log reconciliation was performed, and the supplied record does not establish a skipped-test count.

Separately, the raw tail names TestStart_GpuRestartWaitResetsAfterStats in internal/perf, with the message timed out waiting for GPU stats from collector 6. That identifies one visible failure; it does not explain every failure in the stored parser summary or establish the underlying cause. Entries marked [no test files] are not failed tests.

We did not test inference quality, tokens per second, model-loading delays, memory savings, GPU drivers or an exposed service. The record contains no audit result or scanned-package coverage, so it supports no security conclusion. The testing-methodology page explains the publisher's recording approach and limits; its current text does not prove that this older run used today's procedure.

What the proxy takes over

llama-swap selects an upstream server from the requested model ID and replaces the wrong running upstream. In the basic configuration, it handles one model at a time; the pinned README documents a matrix for concurrent models. It also documents idle unloading through ttl, startup hooks, aliases and manual unload controls.

That makes it useful when your client should keep one endpoint while different configured servers handle requests. The pinned README lists OpenAI and Anthropic routes, along with upstream-specific routes for several text and media backends. Those documented routes are capabilities to validate with your client and backend, not compatibility results from our sandbox.

The operator still supplies the upstream command and model path. llama-swap's lifecycle controls do not establish that your chosen quantization, context size or GPU configuration fits the machine. Start by checking one backend directly, then route the same request through the proxy before adding more models.

Scheduling changed after the measured commit

Issue 1004 was closed as completed on October 24, 2026, after capacity-based routing work. The issue discussion describes resource pools, configured model capacities and eviction preferences. It therefore no longer supports describing capacity scheduling as merely an outstanding request.

That work postdates our October 13 run at a025e03. Its completion commit, 957cc76, is not our measured commit. The supplied evidence does not establish that our run exercised the completed capacity router, and it does not establish automatic measurement of model resource requirements. Check the configuration documentation for the version you intend to deploy.

Check streaming through your actual endpoint

The pinned README recommends disabling Nginx response buffering for streaming endpoints. It also recommends explicit buffering configuration even though llama-swap sends X-Accel-Buffering: no on SSE responses. Test streaming through the address your client will use, including any reverse proxy.

API-key support, logs and request inspection are documented, but our run did not validate a public deployment or access-control policy. Decide who can reach the API and interface, and check what request data your deployment exposes through logs.

When to choose a different service

Ollama provides a REST API for running and managing models. Consider Ollama when that model-running and management workflow answers your need without configuring a proxy around separate upstream commands. We did not compare its installation effort or runtime performance with llama-swap.

LocalAI pulls separate backend images when a model needs them. Consider LocalAI when you want the service to manage those backend components. Its supplied README describes a modular system, so calling it a bundled collection of every backend would misrepresent the current design. These are alternative operating approaches, not measured winners.

How to read the editorial scores

Scores describe the narrow dimensions named here, not an overall production-readiness rating. The scale is 1 for substantial evidenced obstacles, 2 for major operator work, 3 for a workable path with material gaps, 4 for strong evidence with a specific limitation, and 5 for comprehensive evidence within the stated dimension. Community assesses the substance of the supplied public collaboration, not adoption or support response times. Maturity assesses the documented operational controls and recorded development checks, not deployment reliability.

The setup score reflects documented binary and configuration options plus successful install and build steps, with upstream setup still required. Documentation covers installation, configuration, routes, shutdown and streaming caveats. The community evidence includes a detailed scheduling discussion involving users, contributors and the owner, followed by completed work. Maturity receives credit for lifecycle and monitoring controls, but the failed test command remains an unresolved limitation.

Alternatives

ProjectWhat it isPick it when
Ollama gh↗A model runner with a REST API for running and managing models.Choose it when its model-running and management workflow meets your needs without configuring separate upstream commands behind llama-swap; no comparative setup or performance result was measured here.
LocalAI gh↗An AI service with separate backend images pulled when models need them.Choose it when you want the service to manage backend components rather than supplying each upstream process command yourself; this is an architectural choice, not a measured performance advantage.

What people are saying

  1. [github-trending] mostlygeek/llama-swap

Sources

  1. llama-swap repository
  2. llama-swap v251 release
  3. Memory-constrained scheduling request
  4. API key file request
  5. Kilo Code timeout report
  6. Nginx streaming issue

More llm tools reviews