mrkeyoor.com_
Mon 19 Oct 12:53 UTC
LLM Toolsevaluationupdated 15 Oct 2026

LlamaFactory review

LlamaFactory lets you fine-tune open language and multimodal models through a command line or browser interface. It puts dataset configuration, training methods, and model export in one project so you can run experiments without writing each trainer yourself.

+112stars / 7d
Verdict

Our LlamaFactory run installed 161 packages into 6,340 MB, then ended with 327 passing tests and 3 collection/setup errors, so adopting it means owning a substantial training environment. Choose it when its shared GUI and model configurations save repeated trainer work. Investigate the 63 audit findings and reproduce the full suite before standardizing on this commit.

We ran it

Lab card: what happened when we ran LlamaFactoryScreenshot of LlamaFactory (llamafactory.readthedocs.io)
Install✓ · 122s161 packages · 6340 MB
Build✓ · 5s
Tests✗ · 738s327 passed · 0 failed · 23 skipped · 3 errors of 330 (pytest)
Known vulns63(pip-audit)
Repo615 files~64,825 lines of source · 13.9 MB · 8 CI workflows · tests dir

Answers from our run

Does LlamaFactory build from source?

Dependencies installed in 122 seconds (161 packages), and the build succeeded in 5 seconds. We cloned commit 1fded91 into a clean Debian container with 3 CPUs and no project-specific setup.

Do LlamaFactory's tests pass?

No. The project's own test command finished in 738 seconds and reported 327 passed, 0 failed, 23 skipped, 3 collection errors (pytest).

Does LlamaFactory have known vulnerabilities in its dependencies?

pip-audit flagged 63 known advisories in the dependency tree at the time of our run.

Who should not use LlamaFactory?

Teams that require a clean complete suite before adoption: our run ended with 3 collection/setup errors despite 327 passing cases.

What are the alternatives to LlamaFactory?

Axolotl, TRL, Unsloth. Our LlamaFactory run installed 161 packages into 6,340 MB, then ended with 327 passing tests and 3 collection/setup errors, so adopting it means owning a substantial training environment.

Setup3/5Install and build passed, but 6,340 MB and 3 suite errors need attention
Docs4/5English and Chinese guides cover data, hardware, and platform exceptions
Community5/575,479 stars, an October push, and current issue and PR activity
Maturity3/5Broad training coverage, with suite errors and an image conversion report

Who it’s for

ML engineers comparing supervised fine-tuning and preference training across supported model families.
Developers who want the Gradio interface alongside reusable training configuration files.
Teams prepared to register custom datasets and maintain a pinned hardware and model configuration.

Who it’s NOT for

Teams that require a clean complete suite before adoption: our run ended with 3 collection/setup errors despite 327 passing cases.
Operators expecting a small, audit-clean Python environment: our install occupied 6,340 MB and pip-audit reported 63 known vulnerabilities without severity details.
Multimodal teams relying on transparent diagrams without preprocessing checks: issue 10887 reports black backgrounds obscuring image content at our measured commit.
Windows or Ascend users expecting the generic source instructions to cover their machine: the README gives separate GPU library or toolkit requirements.

Setup reality

Our sandbox installed commit 1fded91 in 122 seconds, adding 161 packages and using 6,340 MB. The build passed in 5 seconds. The test step failed after 738 seconds: pytest reported 327 passed, 23 skipped, and 3 collection/setup errors, with no failed assertions. Pip-audit reported 63 known vulnerabilities, without package or severity details in the supplied result.

Useful training needs model weights, correctly formatted data, and a matching configuration. Custom datasets require registration in data/dataset_info.json. Some datasets require Hugging Face login and access confirmation. Optional W&B or SwanLab logging needs the corresponding API key.

The Python 3.12 Debian run does not establish GPU readiness. Windows needs separate GPU library choices, and Ascend requires CANN tooling and kernels. The failing log names two API example files and a converter test but omits their exception details. Its accelerator warning does not explain those errors.

The Web UI saves trainer code, while configuration stays yours

Choose a training method before opening LlamaFactory's Gradio interface. The project supports full fine-tuning, LoRA adapters that update a smaller set of weights, and preference training such as DPO. A shared interface across those methods is its strongest reason to exist. You can change experiments without writing a different trainer each time, provided your model and method appear in its compatibility tables.

Our checkout contained 615 files and about 64,825 source lines. That breadth buys model templates, dataset handling, and inference integrations alongside training. It also gives you more combinations to check. The README's quick start separates training, chatting with the result, and merging an adapter into model weights. Those are useful distinctions for someone who needs a repeatable experiment rather than a notebook left running overnight.

What happened when we ran it

Our sandbox installed LlamaFactory commit 1fded91 in 122 seconds, adding 161 packages and occupying 6,340 MB on disk. The build succeeded in 5 seconds. We ran it on October 11, 2026, in an unprivileged Debian container using the uv Python 3.12 image, with 3 CPUs, 8 GB of RAM, and no secrets. Those results establish a working install and build on that environment.

The test step exited with code 1 after 738 seconds. Pytest's own summary reported 327 passed, 23 skipped, 3 expected failures, 4 unexpected passes, and 3 errors, with no failed test assertions. The errors named scripts/api_example/test_image.py, scripts/api_example/test_toolcall.py, and tests_v1/plugins/data_plugins/test_converter.py. Passing individual cases and completing the whole suite successfully are different outcomes. LlamaFactory achieved the former in our run.

The log tail shows a warning that pinned memory was enabled without an accelerator. It does not show the exceptions behind the 3 collection/setup errors, so blaming missing credentials, packages, or GPU hardware would go beyond the evidence. Pytest reported 708.16 seconds internally, while the enclosing test step took 738 seconds. We did not measure training throughput, model quality, or the memory needed for an actual fine-tuning job.

A 13.9 MB checkout became a 6,340 MB environment

The source checkout occupied 13.9 MB, but the installed environment reached 6,340 MB. That is a meaningful commitment on a development laptop or a small worker image, before budgeting for your chosen model and data. Pip-audit also reported 63 known vulnerabilities. The supplied result gives no package names, severities, or exposure analysis. It warrants investigation before adoption without establishing that a deployed service is exploitable.

The lab scan found 8 CI workflow files and a tests directory, but no Dockerfile. The README separately documents published images and Docker Compose paths. Those are different observations: the scan's signal describes our checkout inspection, while the documentation describes deployment routes. Neither the presence of CI nor a published image resolves the collection errors or explains the audit findings in this measured environment.

Custom datasets need registration before the GUI can help

LlamaFactory requires a custom dataset entry in data/dataset_info.json, and its data preparation instructions point readers to a separate format guide. You still need to map your records to the expected conversation or instruction format. I'd do that with a small sample before committing a training budget. A successful import cannot tell you whether the examples teach the behavior you want.

Our 161-package install used no secrets, which says little about access to your eventual training assets. The README recommends Hugging Face login for datasets requiring confirmation. It also supports local files and other model hubs. W&B and SwanLab logging use their own API keys when enabled. Decide where model assets and experiment logs may go before copying an example configuration into a company environment.

Windows and Ascend have their own setup instructions

The measured Python 3.12 Debian environment cannot validate every hardware route. Windows users get separate directions for GPU-enabled PyTorch, bitsandbytes, and FlashAttention. Ascend users need the CANN toolkit and kernels alongside extra requirements. Some model entries even ask for Transformers from its main branch and a disabled version check. Those exceptions make pinning a known configuration more useful than assuming every listed model shares one dependable setup.

A concrete multimodal concern appears in issue 10887. Its reporter used commit 1fded91 and described transparent image backgrounds becoming black during image conversion, obscuring black lines. A proposed fix appears in pull request 10888. This is an upstream report, separate from our sandbox failures. If your dataset includes transparent diagrams, inspect the converted examples before training on them.

October activity continues beyond the May release

GitHub recorded the last push on October 10, 2026, and showed 75,479 stars with 1,191 open issues and PRs. The latest release, v0.9.5, was published on May 30. Issue and pull request updates continued through October 14, including the image report and configuration fixes. The release gap alone doesn't establish abandonment. It does make the choice between a tagged release and a development commit worth documenting.

Axolotl is worth comparing if YAML configuration is already how you manage experiments. Hugging Face TRL suits developers who want direct trainer classes inside their own Python code, while Unsloth supplies model-specific notebooks. I'd choose LlamaFactory when its shared GUI and method coverage remove repeated work. For this checkout, budget the 6,340 MB environment and resolve the three suite errors before making a passing build your release criterion.

Alternatives

ProjectWhat it isPick it when
AxolotlA model training project built around reusable YAML configurations.pick this instead when your team prefers a configuration-driven workflow over choosing experiments through LlamaBoard.
TRLHugging Face trainer classes for supervised and preference-based model training.pick this instead when you want to compose training inside your own Python application.
Unsloth gh↗A training and inference project with model-specific fine-tuning notebooks.pick this instead when a documented notebook for your chosen model is the starting point you want.

Sources

  1. LlamaFactory README and platform requirements
  2. LlamaFactory repository facts
  3. LlamaFactory v0.9.5 release
  4. Transparent images become black, issue 10887
  5. Proposed transparency handling fix, PR 10888
  6. Recent LlamaFactory issue and PR activity

More llm tools reviews

AIStudio2API · GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold · livenerf · scion · opencode2api · IQuest-Q1 · the whole board →