mrkeyoor.comAsk

MakazhanAlpamys/Souptested 29 Sept 2026commit bb709b6

Soup review

Soup is worth a scoped trial if you want a YAML-based fine-tuning workflow and can validate the backend you intend to use.

  • Build 21 s
  • Tests timed out
104 sto install
  • 147 packages
AI Toolsevaluationupdated 26 Oct 2026
+124stars / 7d

Soup is a Python CLI for fine-tuning, evaluating, and exporting language models through YAML recipes. Its README documents LoRA training, data tools, a local web interface, serving, and optional backends. Layer streaming keeps the frozen base outside GPU memory and moves decoder layers onto the GPU as needed; it remains beta. Our recorded run checked setup, build, and part of the test suite, not model training.

Verdict

Soup is worth a scoped trial if you want a YAML-based fine-tuning workflow and can validate the backend you intend to use. Our September 29, 2026 run at bb709b6 recorded successful install and build steps, 6,104 MB of dependencies, and a test timeout at 900 seconds. It did not verify an installed CLI, training speed, model quality, or a complete passing suite. Start with a short LoRA job and retain its configuration, model revision, output, and evaluation results before expanding to streaming or distributed training.

We ran it

Lab card: what happened when we ran Soup
Lab card from our sandbox run
Screenshot of Soup (trysoup.dev)
trysoup.dev
Install✓ · 104s147 packages · 6104 MB
Build✓ · 21s
Tests✗ timed out · 900sran, no count parsed
Known vulns0(pip-audit)
Repo1776 files~536,859 lines of source · 31 MB · 8 CI workflows · Dockerfile · tests dir

Answers from our run

Does Soup build from source?

Dependencies installed in 104 seconds (147 packages), and the build succeeded in 21 seconds. We cloned commit bb709b6 into a clean Debian container with 3 CPUs and no project-specific setup.

Do Soup's tests pass?

We could not finish them: the suite was still running after 15 minutes in our container.

Does Soup have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use Soup?

Teams that need independently verified training throughput or model quality from this review: our run measured neither.

What are the alternatives to Soup?

Axolotl, LLaMA-Factory. Soup is worth a scoped trial if you want a YAML-based fine-tuning workflow and can validate the backend you intend to use.

Setup3/5The September 29, 2026 run at bb709b6 recorded successful install and build steps in 104 and 21 seconds, with 6,104 MB of dependencies. Installed CLI usability and a complete test result were not verified.
Docs4/5The supplied README extracts distinguish light and training installs, Python ranges, beta streaming, version changes, and measurement caveats. Documentation breadth does not verify every advertised backend.
Community3/5The v0.73.3 release notes report 24 pull requests from eight contributors other than the maintainer. This supports an active contribution example, not a current adoption or support-response rating.
Maturity3/5Recorded build completion and documented fixes support a scoped trial. Streaming remains beta, the FSDP fix's live eight-H100 acceptance run was deferred, and our full test run timed out.

Discussed on

  1. hnShow HN: Fine-tune an 8B model on a 4 GB laptop GPU139 points
  2. hnShow HN: A Soup-inspired runtime for a real fruit-fly connectome =)4 points

Who it’s for

Developers who understand fine-tuning and want configuration, data preparation, training, and export commands in one project.
Small teams willing to validate one model, dataset, and backend before adopting the wider workflow.
Laptop GPU owners prepared to check the beta streaming path against their own sequence length and hardware.
Contributors who can allow more than 15 minutes for full-suite verification on modest CPU resources.

Who it’s NOT for

Teams that need independently verified training throughput or model quality from this review: our run measured neither.
Users expecting the current release to support Python 3.10. The current README specifies Python 3.11 through 3.13; the measured September checkout documented 3.10 through 3.12.
Teams treating the closed FSDP dtype issue as proof that the 70B recipe passed its live eight-H100 acceptance run. That run was explicitly deferred.
Users treating a published 4 GB streaming example as a guarantee for another model, context length, or software stack.
Contributors who require a completed full-suite result within 900 seconds on a 3-CPU, 8 GB container.

Setup reality

In our recorded run on September 29, 2026, at commit bb709b6723468c3dd2138ffc0b055095116f16a5, the harness marked its install step successful after 104 seconds and its build step successful after 21 seconds. It recorded 147 dependencies and 6,104 MB of installed dependencies. The install tail identifies Python 3.12.12. The supplied record does not include the install or build command text, an installed-binary check, or a training workload, so these results do not establish that the Soup CLI was installed and usable.

The test step timed out after 900 seconds with exit 124. Its supplied tail reached 11% and contains progress dots and skip markers, without a named failure or traceback. There is no reconciled passing-test count or completed suite result. Pip-audit recorded an advisory total of 0; audit coverage is unknown.

The recorded container had 3 CPUs and 8 GB of memory. The checkout summary lists 1,776 files, approximately 536,859 source lines, 31 MB, 8 CI workflows, Dockerfile, Compose, and a tests directory. These are recorded inventory figures, not evidence of training performance.

The README distinguishes the light soup-cli package from soup-cli[train], which adds training dependencies. Match the Python range and extras to the version you install, then check soup doctor and run a short representative training job before reserving longer compute time.

A recipe is a useful starting point, but training needs its own check

Soup brings configuration, data preparation, fine-tuning, evaluation, and export into one Python project. It suits developers who understand the training task and want to manage it through a reviewable YAML recipe. Our recorded setup and build results support trying that workflow; they do not establish that a particular model will train correctly or fit your hardware.

The README documents templates for supervised training, preference methods, reasoning, vision, audio, and embeddings. It also lists chat, serving, export, and evaluation commands. Treat that list as documented capabilities, with separate validation needed for the backend and recipe you choose.

What our recorded run establishes

Our stored run began on September 29, 2026, at commit bb709b6723468c3dd2138ffc0b055095116f16a5. The configured container had 3 CPUs and 8 GB of memory, using ghcr.io/astral-sh/uv:python3.12-bookworm. The install tail identifies the actual Python environment as 3.12.12.

The harness marked the install step successful after 104 seconds and the build step successful after 21 seconds. It recorded 147 dependencies occupying 6,104 MB. The supplied record omits the command text and contains no installed-binary check. A successful step therefore establishes command completion within the harness, rather than proof that an installed Soup executable worked.

The test step timed out after 900 seconds with exit 124. The supplied tail reached 11%, with progress dots and skip markers. It contains no named failure or traceback. This is an incomplete suite result, not evidence that Soup is broken. The record supplies no reconciled passing-test count, and the percentage is a progress indicator rather than a measure of unique coverage.

Pip-audit recorded an advisory total of 0. Its coverage is unknown, so that result cannot establish that every dependency was checked or that the environment is secure.

The checkout summary records 1,776 files, approximately 536,859 source lines, and 31 MB. It also records 8 CI workflows, Dockerfile, Compose, and a tests directory. We did not measure dependency download time separately, training throughput, GPU memory, model quality, serving performance, or export correctness.

Match setup advice to the version

The README at the measured commit documents Python 3.10 through 3.12. The current repository README describes v0.77.0 and specifies Python 3.11 through 3.13. It says Python 3.10 resolves to v0.76.x. Neither Python range should be presented as applying to every Soup version.

The light soup-cli package handles configuration and data tools without PyTorch; soup-cli[train] adds the training stack. The README recommends an isolated application environment through pipx or uv tool, with pip also documented for an existing virtual environment, notebook, or container. Additional extras cover other capabilities.

Use soup doctor to inspect the environment. The current README also recommends checking a configuration before upgrading because v0.77.0 refuses settings that earlier versions warned about or ignored. CPU training is described as experimental and very slow. Our test timeout does not measure CPU training speed.

The 4 GB example has a specific scope

Soup's layer streaming moves frozen decoder layers onto the GPU while training an adapter. It is not evidence of full-parameter fine-tuning on a 4 GB card.

The README reports a pre-repair v0.72.2 example using Llama-3.1-8B-Instruct with NF4, LoRA, batch 1, and sequence length 512 on an RTX 3050 Laptop: 119.6 tokens per second and 3.32 GB peak GPU memory. The current README also reports a post-repair run on another RTX 3050 Laptop at 208.6 tokens per second and 2.40 GB peak. It explicitly warns that the higher GPU clock and changed memory pattern prevent a direct speed comparison. These are repository-reported measurements, not our sandbox results.

Issue 395 is closed, but its findings still limit the memory claim. The original RTX 3050, Windows, torch 2.5.1 measurements under-predicted memory at longer sequences. A second-stack investigation on an A10G, Ubuntu, and torch 2.13 did not reproduce that under-prediction. The closing comment says the mechanism remains unidentified. Closure does not turn the estimator into a guarantee across stacks.

For a trial, keep the model, quantization, sequence length, batch shape, and software versions fixed, then record peak memory and whether the job completes. A short-context example cannot establish fit at a longer context.

Closed issues change the adoption advice

Issue 350 originally documented FSDP plus four-bit dtype failures using Soup 0.72.4 and a Qwen2.5-72B stand-in at the recipe's shape. It was closed on August 28, 2026, after PR 588 added dtype compatibility and regression coverage. The maintainer explicitly deferred the live eight-H100 acceptance run. The supported conclusion is that the implementation fix merged, with that hardware acceptance check still unverified in the supplied evidence.

Issue 23 was closed on September 9, 2026. Its thread records real Apple Silicon runs, a macos-14 smoke job, adapter-load work, and callback and validation-loss changes. It no longer supports saying that the MLX training loop has never been tested end to end. Those project records also do not establish production reliability for every Apple Silicon configuration, and our sandbox run did not test MLX.

The distinction matters when choosing a trial: check the current status and the remaining hardware limits rather than treating an issue's original description as a permanent defect.

MCP controls are documented, not independently audited here

The v0.73.3 release notes describe soup mcp serve --allow-execute and planned training or export execution behind a short-lived, single-use, server-generated confirmation token. They also describe configuration snapshots and content-based protected-path digests. These are release-note claims about execution controls. Our recorded run did not exercise MCP or assess those controls.

Release v0.73.3 remains a historical reference: its notes report 24 pull requests from eight contributors other than the maintainer. That is evidence of contributions to that release, rather than a current adoption count or a reliability measurement.

How to choose a trial

Try Soup when its combined recipe and command workflow matches your needs. Start with one ordinary LoRA job and verify the resulting adapter and evaluation output before adding optional backends.

Axolotl is a relevant alternative for a YAML-based fine-tuning pipeline; its README documents preprocessing, training, evaluation, quantization, and inference under one configuration. LLaMA-Factory is relevant when you want a browser interface for fine-tuning as well as CLI commands. The supplied primary-source extracts support those roles, but provide no controlled comparison of setup effort, speed, or model quality against Soup.

How the ratings work

These editorial ratings use a five-point rubric: 1 means substantial documented barriers; 2 means limited evidence or unresolved validation; 3 means enough evidence for a scoped trial; 4 means broad documented support with relevant validation; 5 means comprehensive evidence for the assessed use. They are judgments about the supplied evidence, not benchmark results. Unknown adoption and untested compatibility do not earn a high score.

Setup earns 3 because the recorded install and build steps completed, while CLI usability and a complete test result remain unverified. Documentation earns 4 for explicit extras, version boundaries, correction notes, and measurement caveats. Community earns 3 for concrete outside contributions to v0.73.3, without a verified current adoption measure. Maturity earns 3 because fixes and validation records support a trial, while beta streaming and deferred hardware acceptance limit broader conclusions.

Before committing a longer training run, verify the installed executable, run a representative workload, and save the exact configuration and output. Those checks answer the questions our recorded setup and partial test run could not.

Alternatives

ProjectWhat it isPick it when
AxolotlAn open-source LLM fine-tuning framework whose README documents YAML configuration across preprocessing, training, evaluation, quantization, and inference.Choose Axolotl when you want to evaluate a YAML-based fine-tuning pipeline with documented full fine-tuning, LoRA, QLoRA, and distributed training options. The supplied evidence does not establish that it is faster or easier than Soup.
LLaMA-Factory gh↗A fine-tuning framework whose README documents a CLI, a web interface, multiple model families, and several training methods.Choose LLaMA-Factory when a browser interface is part of your fine-tuning workflow. Its README documents LLaMA Board for training and provides CLI examples for training, chat, and export.

What people are saying

  1. [github-trending] MakazhanAlpamys/Soup
  2. [producthunt] Soup CLI

Sources

  1. Soup repository and README
  2. Soup v0.73.3 release
  3. FSDP2 four-bit recipe failure, issue 350
  4. MLX end-to-end test gap, issue 23
  5. Layer-streaming memory under-prediction, issue 395

More ai tools reviews