Our EvoVLM sandbox run at commit 16b56fd installed in 40 seconds and passed 132 pytest tests. That gives researchers a concrete reason to inspect the package's inference programs and preserved evidence. The useful decision is whether its research workflow matches your question: it changes the program around a frozen vision-language model, while model snapshots come from outside the repository.
The project reviewed here is zzzz7788990213-ops/EvoVLM, subtitled Evolving Multimodal Inference Programs. Its README identifies scientific chart understanding as the target task. Use the owner and subtitle to identify its documentation and experiment records. The practical starting point is the retained research directory, which contains the implementation, evaluation data, evidence and technical report described by the runtime guide.
What you can do with EvoVLM
EvoVLM searches prompts, image preprocessing, generation budgets and answer extraction while keeping model weights frozen. This makes the surrounding inference program the object of experimentation. A researcher can examine a prompt change, image transformation or answer-extraction rule alongside its recorded evaluation, rather than treating every improvement as a consequence of training new weights. The project overview explains that distinction before presenting the method and results.
The documented loop starts from manually designed programs for Qwen3-VL-2B-Instruct and Qwen3-VL-2B-Thinking. A local Qwen model ranks a frozen catalog of typed mutations, and a renderer turns the chosen configuration into Python. Candidates receive evaluations that record correctness, latency, errors and execution provenance. Pareto selection then retains useful accuracy and latency trade-offs. These steps give a reader specific artifacts to inspect: the starting program, selected mutation, generated program and associated evidence.
Start by comparing the manual seed with the selected program. That comparison follows the project's own experimental structure and helps distinguish the contribution of the human-designed starting point from subsequent search. Someone researching program selection can follow the mutation and evaluation records; someone researching model training needs a different experiment. The repository's stated method keeps Qwen weights fixed throughout this search.
What people use EvoVLM for
The repository documents evidence replay, constrained program search and fresh inference as distinct research activities. Its repository layout identifies exported programs, evaluation routines, experiment journals and packaged data subsets. These are author-documented uses of the package. They provide a practical way to choose an entry point according to whether you want to read existing evidence, inspect the search mechanism or generate new predictions.
For evidence inspection, the CPU instructions describe replaying preserved records and checking program interfaces. For search research, the method description explains typed mutations, evaluation and selection. For new predictions, the runtime guide describes external model snapshots and execution modes. Read the instructions for the task you intend to perform before installing inference dependencies or downloading models.
What happened when we ran it
Our November 13, 2026 sandbox run at commit 16b56fd passed 132 of 132 pytest tests in 8 seconds, with 0 failed. Installation took 40 seconds and the build took 3 seconds. The recorded configuration had 3 CPUs, 8 GB RAM and image ghcr.io/astral-sh/uv:python3.12-bookworm. The measured project directory was research/evolving-vlm-inference-main, and the harness recorded a total duration of 60 seconds.
| Recorded repository check | Outcome |
|---|---|
| Install | Succeeded in 40 seconds |
| Dependencies recorded | 90 |
| Installed disk footprint | 5,619 MB |
| Build | Succeeded in 3 seconds |
| Tests | Succeeded in 8 seconds; 132 passed, 0 failed of 132 |
| pip-audit | 0 known vulnerabilities |
| Checkout | 2,368 files, approximately 33,977 source lines, 39 MB |
| Repository signals | Tests directory present; 0 CI workflow files; no Dockerfile detected |
The 5,619 MB installed footprint is a useful planning figure alongside the 39 MB checkout. Reserve space for the recorded development setup and account separately for external model storage. Our measurements describe installation, build, tests and dependency auditing at this commit. The passing pytest result comes from the supplied harness, rather than a test count copied from the README or an inference score taken from archived experiments.
The separate deployment safety step failed because none of compose.yaml, compose.yml, docker-compose.yaml or docker-compose.yml existed. That procedure required a Compose file. Its failed prerequisite is an actual hands-on finding and belongs alongside the successful repository checks. The log identifies the missing prerequisite directly: the procedure's Compose requirement stopped this step before it could proceed to deployment.
The deployment summary says the November 13, 2026 run deployed with its own Docker Compose and bound every port to 127.0.0.1. That statement conflicts with the specific failed safety step. The recorded outcomes are therefore a failed prerequisite, a contradictory deployment summary and a teardown record reporting teardown: ok. Each record describes a different part of the attempt; the specific failure remains part of the result.
Where people get stuck: the default command requests a GPU run
Running reproduce.sh without a mode requests fresh GPU inference, according to the runtime guide. Supply a mode when following its CPU instructions, and run them from research/evolving-vlm-inference-main. The guide says some verification routines depend on that historical directory name. These documented command distinctions matter before you begin troubleshooting: replaying recorded evidence, checking interfaces and launching fresh inference perform different jobs.
| Situation | Documented route or explanation |
|---|---|
| You want to replay preserved evidence | Run python3 -B -m evaluation.portable_reproduce from the research directory |
| You want the explicit CPU checks | Use bash reproduce.sh check or bash reproduce.sh smoke as appropriate |
| You expected a CPU check from the bare script | Supply a mode; the no-argument default is fresh |
| A configured local model path is invalid | Correct that path and snapshot identity; the guide says the override fails closed |
| Snapshot files are symlinked | Use physical snapshot files; identity checks reject symlinked paths and critical files |
| You moved or renamed the research folder | Retain evolving-vlm-inference-main because verification routines bind to that name |
The runtime guide's resolution order starts with a per-model override, then a shared model root, then a package-local model directory, and finally the official model ID at its locked revision. An explicit local path is authoritative: an invalid path or identity fails instead of falling back. Check the configured source before changing caches, because a higher-priority override determines which snapshot the program attempts to use.
The recorded search starts from manually designed programs
The README describes two generations with at most six children per model population. Its recorded results separate Baseline Instruct, Manual Instruct, Evolved Instruct, Manual Thinking and Evolved Thinking. That separation matters when interpreting the experiment: the manual programs are part of the method, and the evolved programs follow them. A comparison between the original baseline and final selection combines changes made at both stages.
For Instruct, the README reports 78 correct answers out of 128 for both the manual and evolved programs, or 60.94%. Their recorded Speed32 median latencies are 15.44 seconds and 14.07 seconds respectively. The authors describe this as an 8.9% lower median latency with preserved recorded accuracy. The published result table is the source of those figures; they are archived experiment results, separate from our sandbox timings.
The README also reports Manual Thinking at 23 of 128 correct, 17.97% accuracy and 20.04 seconds median latency. Evolved Thinking records 44 of 128 correct, 34.38% accuracy and 17.91 seconds. Its stated changes are 16.41 percentage points in accuracy and 10.6% lower median latency. Speed32 takes a median across three repeats for each query, then a median across 32 queries, so preserve that aggregation when discussing the published timing figures.
External snapshots determine the inference setup
Fresh reproduction requires both pinned Qwen snapshots, including configuration, generation configuration, tokenizer, processor and chat-template files. The runtime guide names Qwen/Qwen3-VL-2B-Instruct and Qwen/Qwen3-VL-2B-Thinking and points to their critical manifests under evaluation/model_snapshots. Those manifests record revisions and critical file identities. Obtain the snapshots at those revisions rather than choosing another revision with the same display name.
The online and offline instructions describe different sources for model files. Online mode needs network access and a writable Hugging Face cache, while a package-local snapshot still takes precedence when present. Offline mode uses local snapshots. The launcher checks critical manifests before creating formal run outputs, starting lanes or initializing CUDA, and an offline launch without valid snapshots exits with an error.
The general dependencies live in requirements.txt, while requirements.lock.txt records the Linux/CUDA environment. The overview explicitly distinguishes that lockfile from a universal macOS setup. For research planning, preserve the selected program and snapshot identity together. Also keep the cost of evaluating search candidates separate from the cost of running the selected program: the method evaluates alternatives before selection, so these are different parts of the workflow.
Evidence integrity and reuse have specific documentation
Our recorded pip-audit result was 0 known vulnerabilities across 90 scanned packages. The README describes a separate set of evidence-integrity checks covering file hashes, critical model and processor identity, cross-bound attempt records and metric recomputation from per-query evidence. These statements address different objects: dependency advisories in our run and consistency controls for the archived experiments. Keep those purposes clear when deciding what to inspect next.
The attribution section identifies Qwen model families and CharXiv data and directs readers to third-party notices. It says the publication imposes no new repository-wide license and that existing third-party licenses continue to apply to their respective materials. Read the notices for the particular code, data and weights you intend to reuse. A license attached to another project called EvoVLM does not describe this repository.
EvoVLM vs EvoVLM-JP and EvoVLMA
EvoVLM studies frozen-weight inference programs, while Sakana AI's repository concerns model merging and EvoVLMA concerns vision-language model adaptation. Choose among them according to the object of your research. The supplied descriptions support a comparison of purposes and entry points: inference-program search, model merging or adaptation. Those distinctions help identify which repository's code and experiment records belong in your investigation.
| Project | What its supplied source describes | Choose it when |
|---|---|---|
| EvoVLM, zzzz7788990213-ops | Frozen-weight chart inference with constrained program mutations | You want to inspect or extend this inference-program experiment |
| Sakana AI Evolutionary Model Merge | Models and code for reproducing paper evaluations | You want the model-merging work associated with EvoVLM-JP |
| EvoVLMA, kding1225 | Evolutionary Vision-Language Model Adaptation | You want to investigate evolutionary vision-language model adaptation |
Sakana AI's repository includes models and code for reproducing its paper evaluations. Its Evolutionary Model Merge README lists EvoVLM-JP-v1-7B among its models. The supplied EvoVLMA introduction discusses adaptation through methods such as prompt tuning and adapters, then introduces Evolutionary Vision-Language Model Adaptation. The comparison table keeps those author-described research areas attached to the respective projects.
How to distinguish similarly named projects
The ICML listing identifies EvoVLM: Multimodal Evolutionary Feedback for Visual Symbolic Regression by Hyejin Lee and Junsuk Choe. Its symbolic-regression abstract describes an automated symbolic regression framework, giving readers another concrete identity to distinguish from the chart-inference repository. Match the owner, full title and method before transferring a command, benchmark or license into your research notes.
The npaka walkthrough explicitly tries EvoVLM-JP in Google Colab. In lmms-eval issue #449, a user asks which generation parameters were used for JMMMU evaluation of SakanaAI/Llama-3-EvoVLM-JP-v2 and reports a best accuracy of 28.48%. A collaborator suggests contacting the JMMMU or EvoVLM-JP-v2 authors. The closing comment cites prolonged inactivity and asks for updated reproduction details if the issue remains relevant. These are records about Sakana AI's model family.
FAQ
The runtime guide separates CPU evidence replay from fresh inference and names two fixed Qwen model IDs. Those details determine which files and execution path a reader needs. The questions below cover the supplied commands, model distribution and the EvoVLM-JP identity. The generation-parameter question comes directly from issue #449 and retains that issue's model attribution.
Does CPU evidence replay generate new predictions?
The runtime guide says CPU evidence commands do not generate new predictions. Its standalone evidence replay uses Python's standard library, while pytest is required for the test suite. Follow the CPU command list from the retained research directory. The commands address recorded evidence or program interfaces, which makes them a useful first step before preparing model snapshots for inference.
Does EvoVLM include the model weights?
Neither the GitHub repository nor the source archive includes model weights, according to the runtime guide. Code, packaged data subsets and evidence are supplied; authenticated snapshots and an appropriate execution environment come separately. The guide points to critical manifests for both Qwen models. Keep those revisions and required files together when preparing the external model dependencies.
Is EvoVLM the same project as Sakana AI's EvoVLM-JP?
Sakana AI's model card describes EvoVLM-JP-v1-7B as an experimental general-purpose Japanese VLM created using Evolutionary Model Merge. The EvoVLM-JP model card supplies that identity directly. The requested repository instead describes inference-program evolution around frozen Qwen models. Keep the full repository owner or model ID with saved instructions so that each command remains attached to its intended project.
What generation parameters were used for JMMMU evaluation of SakanaAI/Llama-3-EvoVLM-JP-v2?
The supplied issue extract mentions a maximum output length of 1,024 and temperature 0, then records a user's request for the remaining parameters. In issue #449's discussion, the collaborator directs the question to the JMMMU or EvoVLM-JP-v2 authors; the later closing comment requests updated reproduction details. That exchange concerns SakanaAI/Llama-3-EvoVLM-JP-v2, so it supplies no setup parameters for this frozen-Qwen inference-program repository.
What we have not tested yet
Our run at 16b56fd did not measure chart-answer accuracy, inference latency, GPU memory requirements, fresh search cost or sustained application operation. It measured repository installation, build, tests and dependency auditing. The command table describes supplied documentation rather than additional commands executed for this review. The failed Compose prerequisite and contradictory deployment summary provide no successful application deployment result. Actual runtime and operating-system versions require raw toolchain logs; the sandbox image label alone supplies neither.
The README identifies Dev128 as a public validation subset with two exact query overlaps and eight queries sharing two figures with the evolution-used set. It describes small-sample development results rather than generalization evidence or a full CharXiv leaderboard result, with latency tied to the recorded environment. The unmodified Thinking baseline was timeout-censored at 7,200 seconds, leaving no completed Dev128 accuracy for that baseline. Our run did not reproduce the archived Dev128 or Speed32 results, test a fresh holdout or compare alternatives on identical inputs and hardware.
The supplied synthesis contains no attributable outside adoption report for this repository, and the EvoVLMA extract supports its adaptation topic without specifying the detailed search targets. Dated repository metadata observed on November 14, 2026 records 5 stars, 0 open issues and pull requests combined, Python as the language and a last push of 2026-10-02T01:04:42Z. The null latest-release lookup leaves release status unresolved; no supplied response establishes a rate-limit cause. These metadata fields supply no measure of support capacity or adoption.
Execution security, redistribution permissions for every artifact, platform portability and English-language coverage beyond the supplied overview and runtime guide remain outside our assessment. The dependency audit and documented evidence checks answer narrower questions. Rendered layout, related links, recommendation modules and template behavior also fall outside this review's evidence. The next useful experiment would retain exact model snapshots, use a separate holdout and record search cost alongside selected-program performance.


