Our source build succeeded, then pytest stopped before producing a passing test: Edge0 is worth a trial when memory limits the local model you can run. Use it if you can match a model, engine and hardware backend and work through their setup. Skip it if you need a commercial model service with unrestricted model licensing: the newer models require separate permission for that use, as the current README explains.
What people use Edge0 for
Edge0 is a local inference framework, the software that makes a language model produce answers. Its mixture-of-experts models divide work among specialist blocks and use only some of them for each generated token, a small piece of text. Keeping idle experts on storage makes a larger model possible without holding every weight in fast memory. That is the reason to consider it, rather than assuming the largest parameter count means the best answer. The project's research explanation describes both expert streaming and routing prediction.
| Who used it | What they did | Outcome |
|---|---|---|
| Lexlangel, GitHub #128 | Converted and loaded 8B on Windows with integrated Intel Iris Xe graphics and 16 GB RAM | Reported coherent casual chat after resolving setup issues |
| NenadSteric, GitHub #236 | Connected 145B NVFP4 to the pi coding agent on one RTX 3090 | Reported a correctly completed task across 531 turns, alongside repetitive reasoning |
| LightBrightLeftRight, r/LocalLLM | Compared 145B Q3_K with Qwen Flash Next IQ3_S through Strata at 32K | Reported faster decode and a shorter initial wait with Qwen |
| prabs_ma on X, crediting Sathyapriya31 | Tested 8B on iPhone 15 and OnePlus 11 | Reported Android decode of 10.43 tokens/s and iOS peak memory of 1044 MB |
The mobile post offers another useful data point, rather than a shared benchmark across phones: its iOS memory metric is undefined. In Data Science in your pocket's 35B video, the presenter discusses an iPhone demonstration and the model architecture. The transcript calls it a demonstration provided by the team. It belongs beside the project's demos, rather than beside the Windows reporter's completed conversion and interactive chat.
What happened when we ran it
Our November 21 source check installed Edge0 in 13 seconds and built it in 3 seconds, but pytest exited 3 after 7 seconds with six collection/setup errors.
We cloned commit da7a5ca and ran the Python project in ./edge0-strata/ with a recorded allocation of 3 CPUs and 8 GB RAM. The configured image was ghcr.io/astral-sh/uv:python3.12-bookworm. That label identifies the requested image, not the actual runtime version. The install completed, then the build completed. The test step was where the run stopped being a success.
| Source-check step | Our result | What you can conclude |
|---|---|---|
| Install | Succeeded in 13 seconds, 44 dependencies recorded, 121 MB on disk | The recorded dependency step completed |
| Build | Succeeded in 3 seconds | The recorded build step completed |
| Tests | Exit 3 in 7 seconds, 0 passed, 0 failed, 6 collection/setup errors | No passing test result emerged from this command |
The tail enters argparse.parse_args(), reaches the argument parser's error and exit methods, then ends in SystemExit: 2. Pytest prints 1 warning, 6 errors in 5.36s. Its own duration and the outer step's 7 seconds cover different scopes. The final lines show an argument-parser exit inside the test run. They don't identify which argument or importing module caused it, so blaming a Python version or missing package would be guesswork. Six collection/setup errors also aren't six failed assertions about inference.
The checkout scan found 4,007 files, approximately 820,859 source lines, 94.4 MB checked out, five CI workflow files and a tests directory. It found no Dockerfile. Those observations describe the repository check, not a model download or the memory consumed while generating answers.
A separate hands-on deployment attempt stopped at its Compose safety check because none of the required Compose filenames existed.
Its run header describes a November 21 deployment using the project's own Docker Compose with loopback-bound ports, but the actual safety step blocks that account of a completed deployment. The label "unsafe repo" is the harness's Compose requirement, not a security finding about Edge0. Teardown returned ok. There is no recorded chat response from that attempt to grade.
Where people get stuck: the fix depends on the engine
That result is more useful than calling the agent either successful or broken. The tool calls could still do the work while the model spent output on filler. The reporter used edge0-strata at 7b60076, then replayed detection from 257ff84. The opt-in recovery detector caught the longest loop but missed shorter repetitive turns. In follow-up experiments, removing repeated runs from reasoning sent back in the conversation prevented recurrence in those tests. A presence penalty helped inconsistently, and lowering temperature made it worse. This is a tested history-cleaning workaround; the closed thread doesn't document a shipped repair for what starts the repetition.
| Report | What changed or helped | Status you should use |
|---|---|---|
| Windows checks, #128 | Maintainer alexyzhou confirmed repository fixes in PR #135, including capturing both output streams and caching successful version probes | Fixed in the reported repository change |
| 8B initial delay, #110 | --prefill-ondemand avoids the whole-layer prefill path when weights won't stay cached |
Documented workaround for that Python engine |
| Two-GPU illegal access, #205 | Contributor patches changed per-device data and prompt handling; a maintainer changed embedding placement | Reporter confirmed illegal access disappeared at 257ff84 |
| Later-GPU memory pressure, #205 | Raise --vram-reserve-later-mib to 1400 or reduce context |
Headroom workaround; the thread still describes a planner gap and unmerged CUDA warm-up change |
| Replayed reasoning, #236 | Remove repetitive runs from replayed reasoning history | Reporter-tested mitigation, not a demonstrated root-cause repair |
JayOfTheKeyboard reported patched 120B serving on two RTX 3070 cards. Kotali-2019 then confirmed a previously stalled 823-token request returned successfully. Later comments distinguish that access fix from allocations needed when CUDA warms up its libraries. If your second card loads the model and fails on the first prompt, the remaining headroom problem is the relevant part of the discussion. A closed issue alone doesn't answer whether your chosen build includes each change.
The original M1 Pro report described long waits even for tiny prompts. The maintainer identified whole-layer reads of 4.06 GiB per request and an ineffective toggle. On-demand prefill trades some warm-cache speed for fewer cold reads. Don't transplant this flag into an unrelated native or Strata command just because the model also says Edge0.
Lexlangel's Windows report also includes ModuleNotFoundError: No module named 'yaml'. Installing PyYAML in the converter's exact Python environment resolved that reporter's error, but the maintainer says the shipped converter chain itself doesn't use PyYAML. It is a local workaround in #128, not grounds for declaring a missing dependency in every installation.
Match the model, backend and revision before downloading
The getting-started guide routes 35B to the native engine and 145B or 120B to edge0-strata.
edge0-infer is the common command-line front end. It chooses an engine from the model folder, rather than asking you to choose one for every request. The front end offers chat and serve for the large models; run is for the native engine. That distinction prevents an easy mistake: compiling the front end for CPU does not turn 145B into a CPU-only model.
| Model family | Engine and backend | Revision or documentation boundary |
|---|---|---|
| 35B | Native engine: Metal/CPU on Apple Silicon, CPU on Linux/Android, CPU/Vulkan on Windows, Metal on iOS | Current README platform table |
| 8B preview | Earlier platform-specific paths | Current README says native-engine support is planned, so don't infer it from older four-platform instructions |
| 145B/120B on Linux or Windows | edge0-strata, NVIDIA CUDA 13 | Getting-started guide |
| 145B/120B on Apple Silicon | edge0-strata, Metal, Q3_K and NVFP4 | v0.3.1 release, packages built from 43ceac6 |
| 145B/120B on AMD | Community HIP work | Open issue #197, PR #196 and contributor branches |
That release specifies macOS 26 or later for its packaged app. Older app build prerequisites aren't a substitute for that package requirement. The FAQ still says macOS offers only Q3_K, while the getting-started guide accepts both formats. Prefer the release contract for the package you download.
balor1eye also reports working 145B Q3_K inference and a small personal agent suite on an R9700 using additional gfx12 kernels. Those are useful contributor results, not an officially released AMD promise for every card with enough VRAM.
Memory savings still need SSD space and system RAM
After conversion, only the assets are needed for the engine. That temporary requirement matters if your laptop has room for the final model but little room left during preparation. The older app's approximately 23 GB released checkpoint and the native engine's approximately 22 GB assets describe different layouts. They aren't two contradictory measurements of the same installed folder.
Allocator memory counts the allocations managed by that library. RSS, the process's resident memory, is a different measure. The operating system, other apps and the growing conversation cache need room too. For a phone, use its platform requirements rather than subtracting the headline allocator number from advertised RAM.
For 145B, the current README explains that its 16 GiB VRAM plus 20 GiB system-RAM row is a constrained test on an RTX 5090 D, not a minimum specification. It separately records Q3_K residency of 23.14 GiB VRAM plus 31.28 GiB host RAM on that card. Those are project measurements under stated settings, not our sandbox results. Neither a 24 GB GPU nor a compact active-parameter count removes the host-memory and storage budget.
Local serving includes an access-control choice
It documents OpenAI- and Anthropic-compatible serving, and says the chat apps keep conversation history on the device. Compatibility lets an agent use a local endpoint instead of a cloud model API. It doesn't tell you what that agent does with files, commands or external tools. The FAQ also distinguishes offline generation after preparation from downloads, external links and feedback, which use the network.
The server guide reports request timing, weight activity and memory separately. That is useful when an apparent model slowdown is storage loading: read the time-to-first-token and bytes-read figures before treating decode speed as the whole experience. Keep loopback access for a client on the same machine; choose token-protected LAN access deliberately when another device needs it.
The code license does not cover every model use
The FAQ assigns Apache-2.0 to the code and 8B/35B previews, while 145B/120B have community-model terms.
The current README goes further: separate written permission is required for commercial model services and independent commercial coding or office assistants. It allows internal use and commercial use of generated outputs subject to upstream terms. If you're evaluating Edge0 for your own local work, that distinction leads to a different decision from selling a hosted endpoint or a standalone assistant. Read the model-license section for the intended product, including its Qwen terms, before building around a launch post's "open-source" label.
Edge0 vs Qwen, Ollama and mlx-lm
LightBrightLeftRight reported 56.2 tokens/s decode for Edge0-145B Q3_K versus 75.6 for Qwen Flash Next IQ3_S through Strata on a 3090 setup.
The same user's first-token times were 20.1 seconds for Edge0 and 12.2 for Qwen. This is an independent report with different model formats, not a universal engine ranking. It earns a narrow conclusion: don't choose Edge0 because you assume its smaller experts always make it faster.
| Alternative | Reason to consider it | What the comparison actually supports |
|---|---|---|
| Qwen Flash Next through Strata | Speed when your machine can accommodate the chosen model | One user's 32K run favored Qwen on decode and initial wait |
| Ollama | A smaller local model already answers your questions well | TechWaveArena recommends it for that situation, but its author explicitly didn't run Edge0 |
| Fully resident mlx-lm | An Apple Silicon model that fits without storage pressure | Edge0's paper compares against a memory-pressured resident configuration, not every mlx-lm deployment |
MLX LM is a Python package for text generation and fine-tuning on Apple Silicon.
Ollama documents downloads for macOS, Windows and Linux.
Those are real alternatives, but the decision should begin with a model and task. Comparing one model that fits against another that pages heavily mostly tells you about memory pressure. If you want to compare Edge0 with llama.cpp, match model family, quantization, context and hardware first; the community Qwen result above was through Strata, so calling it a llama.cpp head-to-head would mislabel the test.
The Edge0 team reports a five-benchmark 35B average of 79.2 against 83.2 for its fp16 base.
That evaluates the combined quantization, adapters and predicted routing, rather than quantization alone. The newer launch tables also report internal evaluations against named cloud models. Treat those as project evaluations, not a replacement for the coding errors, response times and memory use you care about on your own workload.
What the other reviews get wrong
Its "under 3GB" framing also leaves out the allocator-versus-process distinction. The research page defines that metric, and the current guide covers NVIDIA/Linux/Windows and native CPU/Vulkan paths.
ReviewRamble's Edge0 review describes two checkpoints and a unified framework still due later. The current repository has 120B/145B models and a working command interface documented in edge0-infer. TechWaveArena's Ollama comparison is explicit that its author didn't run Edge0; its conceptual memory example shouldn't become an Edge0 benchmark.
The November 13 eesel guide says macOS isn't supported for the large models. That describes a launch limitation. Maintainer linyubupa acknowledges in #201 that the Apple build didn't ship at launch, then announces Metal support through PR #214. Today's support doesn't make the original day-one availability claim accurate.
Two project statements also need care. The 35B model card's unchanged-directory promise conflicts with the FAQ's platform conversions. And its quality breakdown is published: five individual benchmark rows and a stated method, rather than merely a single average. Reproducing those results is a different question from whether the rows exist.
Questions people ask
Edge0 is a local model framework in the Edge0-AI/Edge0 GitHub repository, with separately downloaded models and engine choices determined by the model family.
What is Edge0, and how do its framework, GitHub repository and AI models fit together?
The GitHub repository contains the framework and applications; cloning it doesn't download model weights. An Edge0 AI model supplies those weights, while edge0-infer selects the engine from its model folder. Start with the current setup guide and the FAQ's source-versus-model explanation.
Does the search term edge01 refer to this Edge0 project?
The project is named Edge0 and its repository is Edge0-AI/Edge0. "edge01" appears in search suggestions alongside Edge0, but a suggestion doesn't prove it is an official alias. Use the repository identity to find this project.
Can I run Edge0 35B on a phone, and how do I install it on an iPhone?
Yes: the documentation covers 35B on iOS and Android. The iPhone preparation path uses Xcode and bundles the model before building the app. The FAQ says iOS 35B needs a file-layout conversion; "Not included" means rebuild and reinstall with the model bundled.
Can a 145B Edge0 model really run on a laptop or a single 24 GB GPU?
A single RTX 3090 ran the task reported in #236, but that machine also had 256 GB system RAM. That report proves a one-card run, not a 24 GB total-memory requirement. The current release documents large-model Metal execution and SSD streaming on Apple Silicon.
How much GPU memory, system RAM and SSD storage does Edge0 actually require?
The requirement depends on model, format, context and engine. Native 35B preparation needs roughly 42 GB temporary disk space according to the setup guide. The current README calls its 16 GiB VRAM plus 20 GiB RAM result a constrained test budget, not a minimum system requirement. Leave room for the OS and other runtime allocations.
Does Edge0's 2.9 GiB figure mean total process memory or only active allocator memory?
It means short-context MLX allocator peak for the project's 35B measurement. It excludes parts of total process memory and doesn't specify the minimum RAM a device needs, as the project's measurement notes explain.
How can Edge0 use only about 10B active parameters while being described as a 145B model?
A mixture-of-experts model uses only selected experts for each token, while the total parameter count includes weights outside that step's active set. The README architecture section describes routing each token to 10 of 512 experts plus a shared expert. Active compute doesn't mean the remaining weights need no storage.
Has anyone independently benchmarked Edge0 rather than repeated its launch numbers?
Yes: LightBrightLeftRight posted a 3090 speed comparison on r/LocalLLM, and Lexlangel reported completed Windows conversion and chat in #128. Those reports answer speed and operability questions under particular setups. They aren't an independent quality ranking across Edge0's model families.
Are Edge0's quality comparisons against quantized models or full-precision base models?
The 35B model card compares the complete int4-plus-adapters pipeline against its fp16 base using the team's OpenCompass settings. Its quality table states the method and individual results. The Reddit speed comparison uses different quantized formats and answers a different question.
How does Edge0 compare with Qwen and llama.cpp for speed and memory use?
One 3090 report favors Qwen Flash Next through Strata for decode and first-token time. That isn't a matched llama.cpp comparison. Edge0's storage-streaming design addresses weight residency, so compare both speed and memory under the same context and hardware limits.
Should I use Edge0 or Ollama when my model does not fit in RAM?
Edge0 deserves consideration for its supported sparse models when streaming experts solves your memory problem. If a smaller model already meets your needs, TechWaveArena recommends Ollama instead. That recommendation is conceptual: the author says they didn't run Edge0 for the comparison.
Can I run Edge0 120B or 145B on Apple Silicon through Metal now?
Yes: v0.3.1 documents both models on Metal with Q3_K and NVFP4 folders. Its packaged application requires macOS 26 or later. Follow that release's instructions rather than the launch-era platform table preserved in #201.
What we have not tested yet
Our run covers source install, build and a pytest command at da7a5ca, plus a separate Compose attempt blocked before a recorded model response. We have not measured chat quality, inference latency, VRAM, model-download footprint, native-engine behavior, mobile execution or the v0.3.1 packaged applications. The argument-parser tail is insufficient to diagnose all six collection/setup errors, and actual runtime versions require the raw toolchain logs.
The community reports aren't a broad independent quality assessment across model families and quantizations. Our review also doesn't establish sustained concurrency, production reliability, agent-tool isolation or network privacy. The iOS memory number in prabs_ma's post has no stated metric. In #140, a standardized iOS benchmark was proposed, but Prasad8830 stepped back without the required hardware; the closed issue doesn't demonstrate delivery.
For a useful next result, record the exact model folder, engine commit, backend and context, then run one task you can judge. Capture first-token wait, decode speed, host memory and GPU memory separately. Repeat after the conversation has grown: the long-session reasoning report shows why a correct first answer is an incomplete acceptance test.


