A recipe is a useful starting point, but training needs its own check
Soup brings configuration, data preparation, fine-tuning, evaluation, and export into one Python project. It suits developers who understand the training task and want to manage it through a reviewable YAML recipe. Our recorded setup and build results support trying that workflow; they do not establish that a particular model will train correctly or fit your hardware.
The README documents templates for supervised training, preference methods, reasoning, vision, audio, and embeddings. It also lists chat, serving, export, and evaluation commands. Treat that list as documented capabilities, with separate validation needed for the backend and recipe you choose.
What our recorded run establishes
Our stored run began on September 29, 2026, at commit bb709b6723468c3dd2138ffc0b055095116f16a5. The configured container had 3 CPUs and 8 GB of memory, using ghcr.io/astral-sh/uv:python3.12-bookworm. The install tail identifies the actual Python environment as 3.12.12.
The harness marked the install step successful after 104 seconds and the build step successful after 21 seconds. It recorded 147 dependencies occupying 6,104 MB. The supplied record omits the command text and contains no installed-binary check. A successful step therefore establishes command completion within the harness, rather than proof that an installed Soup executable worked.
The test step timed out after 900 seconds with exit 124. The supplied tail reached 11%, with progress dots and skip markers. It contains no named failure or traceback. This is an incomplete suite result, not evidence that Soup is broken. The record supplies no reconciled passing-test count, and the percentage is a progress indicator rather than a measure of unique coverage.
Pip-audit recorded an advisory total of 0. Its coverage is unknown, so that result cannot establish that every dependency was checked or that the environment is secure.
The checkout summary records 1,776 files, approximately 536,859 source lines, and 31 MB. It also records 8 CI workflows, Dockerfile, Compose, and a tests directory. We did not measure dependency download time separately, training throughput, GPU memory, model quality, serving performance, or export correctness.
Match setup advice to the version
The README at the measured commit documents Python 3.10 through 3.12. The current repository README describes v0.77.0 and specifies Python 3.11 through 3.13. It says Python 3.10 resolves to v0.76.x. Neither Python range should be presented as applying to every Soup version.
The light soup-cli package handles configuration and data tools without PyTorch; soup-cli[train] adds the training stack. The README recommends an isolated application environment through pipx or uv tool, with pip also documented for an existing virtual environment, notebook, or container. Additional extras cover other capabilities.
Use soup doctor to inspect the environment. The current README also recommends checking a configuration before upgrading because v0.77.0 refuses settings that earlier versions warned about or ignored. CPU training is described as experimental and very slow. Our test timeout does not measure CPU training speed.
The 4 GB example has a specific scope
Soup's layer streaming moves frozen decoder layers onto the GPU while training an adapter. It is not evidence of full-parameter fine-tuning on a 4 GB card.
The README reports a pre-repair v0.72.2 example using Llama-3.1-8B-Instruct with NF4, LoRA, batch 1, and sequence length 512 on an RTX 3050 Laptop: 119.6 tokens per second and 3.32 GB peak GPU memory. The current README also reports a post-repair run on another RTX 3050 Laptop at 208.6 tokens per second and 2.40 GB peak. It explicitly warns that the higher GPU clock and changed memory pattern prevent a direct speed comparison. These are repository-reported measurements, not our sandbox results.
Issue 395 is closed, but its findings still limit the memory claim. The original RTX 3050, Windows, torch 2.5.1 measurements under-predicted memory at longer sequences. A second-stack investigation on an A10G, Ubuntu, and torch 2.13 did not reproduce that under-prediction. The closing comment says the mechanism remains unidentified. Closure does not turn the estimator into a guarantee across stacks.
For a trial, keep the model, quantization, sequence length, batch shape, and software versions fixed, then record peak memory and whether the job completes. A short-context example cannot establish fit at a longer context.
Closed issues change the adoption advice
Issue 350 originally documented FSDP plus four-bit dtype failures using Soup 0.72.4 and a Qwen2.5-72B stand-in at the recipe's shape. It was closed on August 28, 2026, after PR 588 added dtype compatibility and regression coverage. The maintainer explicitly deferred the live eight-H100 acceptance run. The supported conclusion is that the implementation fix merged, with that hardware acceptance check still unverified in the supplied evidence.
Issue 23 was closed on September 9, 2026. Its thread records real Apple Silicon runs, a macos-14 smoke job, adapter-load work, and callback and validation-loss changes. It no longer supports saying that the MLX training loop has never been tested end to end. Those project records also do not establish production reliability for every Apple Silicon configuration, and our sandbox run did not test MLX.
The distinction matters when choosing a trial: check the current status and the remaining hardware limits rather than treating an issue's original description as a permanent defect.
MCP controls are documented, not independently audited here
The v0.73.3 release notes describe soup mcp serve --allow-execute and planned training or export execution behind a short-lived, single-use, server-generated confirmation token. They also describe configuration snapshots and content-based protected-path digests. These are release-note claims about execution controls. Our recorded run did not exercise MCP or assess those controls.
Release v0.73.3 remains a historical reference: its notes report 24 pull requests from eight contributors other than the maintainer. That is evidence of contributions to that release, rather than a current adoption count or a reliability measurement.
How to choose a trial
Try Soup when its combined recipe and command workflow matches your needs. Start with one ordinary LoRA job and verify the resulting adapter and evaluation output before adding optional backends.
Axolotl is a relevant alternative for a YAML-based fine-tuning pipeline; its README documents preprocessing, training, evaluation, quantization, and inference under one configuration. LLaMA-Factory is relevant when you want a browser interface for fine-tuning as well as CLI commands. The supplied primary-source extracts support those roles, but provide no controlled comparison of setup effort, speed, or model quality against Soup.
How the ratings work
These editorial ratings use a five-point rubric: 1 means substantial documented barriers; 2 means limited evidence or unresolved validation; 3 means enough evidence for a scoped trial; 4 means broad documented support with relevant validation; 5 means comprehensive evidence for the assessed use. They are judgments about the supplied evidence, not benchmark results. Unknown adoption and untested compatibility do not earn a high score.
Setup earns 3 because the recorded install and build steps completed, while CLI usability and a complete test result remain unverified. Documentation earns 4 for explicit extras, version boundaries, correction notes, and measurement caveats. Community earns 3 for concrete outside contributions to v0.73.3, without a verified current adoption measure. Maturity earns 3 because fixes and validation records support a trial, while beta streaming and deferred hardware acceptance limit broader conclusions.
Before committing a longer training run, verify the installed executable, run a representative workload, and save the exact configuration and output. Those checks answer the questions our recorded setup and partial test run could not.

