Seven stages share one readable Transformer
The repository walks through 7 practical stages: data preparation, base pretraining, supervised fine-tuning, reward modeling, preference optimization, reinforcement learning, and evaluation. Its decoder-only Transformer is split into small PyTorch modules for attention, the feed-forward layer, a block, and the full model. Post-training then wraps that same backbone with loss masks, reward and value heads, and policy objectives. You can follow one set of weights from next-token prediction to a chat-formatted model.
That continuity is the best reason to choose this project over a folder of unrelated notebooks. The README points from each explanation to the real source file and command. JSON configurations cover the newer stages, while a Streamlit panel exposes Data, Pretrain, SFT, Reward, DPO, PPO, GRPO, Evaluate, and Chat pages. A separate documentation site explains the assumed theory. The code remains small enough to inspect, yet the path includes checkpoint resume, multi-GPU execution, evaluation, and interactive inference.
The 111-second install occupies 8,363 MB
Our sandbox installed 150 packages in 111 seconds, far longer than the other Python project in this batch. The resulting environment occupied 8,363 MB before adding training datasets or checkpoints. The package itself supports Python 3.9 and newer. Core dependencies include PyTorch, NumPy, HDF5, tiktoken, and compression and download utilities. Optional groups add public dataset tooling, Weights & Biases, Streamlit, pandas, Altair, and the MkDocs documentation stack.
The from-scratch label needs a useful boundary. The model, chat masking, reward head, DPO loss, PPO losses, group-relative advantages, and training loops are written directly in PyTorch rather than delegated to Transformers, TRL, or PEFT. Tokenization is not built here. The data scripts use OpenAI's r50k_base tokenizer through tiktoken, and training examples come from sources such as The Pile, Alpaca, Dolly, GSM8K, HH-RLHF, and UltraFeedback. That choice keeps attention on model and optimization code.
What happened when we ran it
We tested commit b995104 in an unprivileged Debian container with 3 CPUs and 8 GB of RAM. Installation succeeded in 111 seconds with 150 packages, and the build completed in 1 second. Pytest then ran for 42 seconds and reported 17 passed, 0 failed out of 17. Pip-audit found 0 known vulnerabilities in the installed environment. This is the only repository in this four-project batch whose measured test target ran and passed completely.
The checkout contained 186 files, roughly 6,998 source lines, one CI workflow, and a tests directory. The workflow builds and publishes the documentation rather than running the Python suite, so the 17 passing tests are useful local evidence but are not enforced by that GitHub Actions file. Tests cover checkpoint resume and post-training math and smoke behavior. Our CPU sandbox did not reproduce a real multi-GPU training session or judge the quality of generated text.
A 13M model teaches the path without proving scale
The smallest documented configuration has about 13 million parameters, while the README also shows examples around 77 million and 406 million. It says a free T4 can train the 13M version and makes clear that billion-parameter configurations need more capable GPUs. Mixed precision, gradient checkpointing, and gradient accumulation are opt-in memory controls. DistributedDataParallel supports several GPUs for the newer training path. These are enough to demonstrate the mechanics without pretending a teaching run matches commercial model development.
The data and compute bill continues after pretraining. SFT needs instruction rows and assistant-token masks. Reward modeling and DPO need preference pairs. PPO and GRPO need prompts, generated rollouts, rewards, and more checkpoints. The supplied smoke configurations shrink every stage enough for a quick CPU check, which is excellent for debugging. They do not make the main curriculum laptop-first. Open issue 38 specifically asks for a smaller CPU teaching track and bundled sample data for students with 4 to 8 GB of RAM.
Compiled multi-GPU checkpoints need an extra guard
Open issue 36 reports that combining torch.compile with DDP can add _orig_mod. prefixes to checkpoint keys. The current loader strips the DDP module. prefix but, according to the report, can miss the compiled prefix and load a model with 163 missing keys and random initialization. A proposed fix exists in pull request 37. The important operational lesson is simple: checkpoint loading needs an assertion that every expected tensor matched before post-training or inference begins.
That edge case does not contradict our 17 passing CPU tests. Our sandbox did not use torch.compile or DDP, which is the combination named in the report. Anyone scaling beyond the smoke path should add a save-and-reload test under the exact compilation and distributed settings planned for training. A run that finishes and writes a file is not enough. Reload it in a fresh process, compare the key set, and verify a known batch before spending compute on SFT or reinforcement learning.
August code and 7 open items show a maintained course
GitHub showed 10,075 stars and 7 combined open issues and pull requests. The default branch was last pushed on August 17, 2026, while issue 36 received activity on September 13. GitHub returned no latest release, so users should pin a commit instead of looking for a versioned training package. The open queue is small and includes both the compiled-checkpoint fix and a proposed CPU-friendly student mode.
Train LLM From Scratch works because it keeps one question in view: how does the loss change as a base Transformer becomes an assistant? The 17 passing tests make the answer easier to trust, and the 8,363 MB install states the cost plainly. Read the code, run the smoke path, then train the 13M configuration if you have suitable hardware. Move to a production library once your goal shifts from learning the algorithms to operating larger models.