mrkeyoor.comAsk

zzzz7788990213-ops/EvoVLMtested 13 Nov 2026commit 16b56fd

EvoVLM review

EvoVLM is worth inspecting for research on inference programs around frozen vision-language models.

  • Safety failed
  • Install 40 s
132/132tests passed
  • Build 3 s
  • 90 packages
AI Toolsevaluationupdated 14 Nov 2026
+0stars / 7d

EvoVLM searches prompts, image preprocessing, generation budgets and answer extraction around frozen vision-language models for scientific chart understanding. The supplied overview and runtime guide are in English, and the overview includes a Chinese summary.

Verdict

EvoVLM is worth inspecting for research on inference programs around frozen vision-language models. Our run at 16b56fd passed all 132 recorded pytest tests after a 40-second install. Readers seeking bundled weights, a complete CharXiv image dataset, a repository-supplied Compose deployment or untouched-holdout validation should choose another starting point. Start with its preserved evidence and manually designed programs, then use the runtime guide to prepare a separate fresh inference experiment.

EvoVLM infographic: searches inference programs for scientific chart understanding, plus our test result and who it suits
Tested 13 November 2026: 132 of 132 source tests passed. Before that: source install 40 s, build 3 s.

We ran it

Lab card: what happened when we ran EvoVLM
Lab card from our sandbox run
Screenshot of EvoVLM (github.com/zzzz7788990213-ops/EvoVLM)
github.com/zzzz7788990213-ops/EvoVLM
Install✓ · 40s90 packages · 5619 MB
Build✓ · 3s
Tests✓ · 8s132 passed · 0 failed of 132 (pytest)
Known vulns0(pip-audit)
Repo2368 files~33,977 lines of source · 39 MB · 0 CI workflows · tests dir

Answers from our run

Does EvoVLM build from source?

Dependencies installed in 40 seconds (90 packages), and the build succeeded in 3 seconds. We cloned commit 16b56fd into a clean Debian container with 3 CPUs and no project-specific setup.

Do EvoVLM's tests pass?

Yes. The project's own test command finished in 8 seconds and reported 132 passed, 0 failed (pytest).

Does EvoVLM have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use EvoVLM?

Developers seeking bundled model weights or a complete CharXiv image dataset.

What are the alternatives to EvoVLM?

Sakana AI Evolutionary Model Merge, EvoVLMA. EvoVLM is worth inspecting for research on inference programs around frozen vision-language models.

Setup4/5Our source checkout installed in 40 s and built in 3 s with 90 dependencies, but we did not run a README task.Editor’s note: The harness records successful repository setup, but fresh inference setup and a working application deployment have no completed assessment.
Docs4/5The README has an install section (“Quick start”) and usage in the same section; we did not run its quickstart ourselves.Editor’s note: The supplied overview and runtime guide support the documented routes, but command accuracy across all CPU and GPU modes has no completed assessment.
Community1/5Last push on 2026-10-02 and 5 stars.Editor’s note: Dated repository metadata is available, but attributable outside adoption and maintainer support quality have no supplied assessment.
Maturity1/5No tagged GitHub release, repository created 2026-10-02, its test suite passed in our sandbox (132 of 132 passed) and no licence file GitHub recognises.Editor’s note: Repository tests passed, but sustained inference operation, search cost and production reliability have no completed assessment.

How we score

Who it’s for

Researchers studying constrained evolutionary search over chart-understanding inference programs.
Engineers inspecting preserved experiment evidence before obtaining external Qwen model snapshots.

Who it’s NOT for

Developers seeking bundled model weights or a complete CharXiv image dataset.
Teams seeking a repository-supplied Compose deployment.
Researchers requiring an untouched evaluation holdout: the README documents overlap in Dev128.
macOS users seeking a universal lockfile: the recorded lock includes Linux/CUDA packages.

Setup reality

Our November 13, 2026 sandbox run at commit 16b56fd installed in 40 seconds, built in 3 seconds and passed 132 of 132 pytest tests in 8 seconds, with 0 failed. It recorded 90 dependencies, 5,619 MB on disk and 0 known vulnerabilities from pip-audit. The sandbox configuration listed 3 CPUs, 8 GB RAM and image ghcr.io/astral-sh/uv:python3.12-bookworm.

The README documents CPU evidence replay without model weights. Fresh inference uses external pinned Qwen snapshots and inference dependencies. Keep research/evolving-vlm-inference-main intact and supply an explicit reproduce.sh mode: the default requests fresh GPU inference. A separate deployment procedure failed its safety prerequisite because no Compose file existed.

Our EvoVLM sandbox run at commit 16b56fd installed in 40 seconds and passed 132 pytest tests. That gives researchers a concrete reason to inspect the package's inference programs and preserved evidence. The useful decision is whether its research workflow matches your question: it changes the program around a frozen vision-language model, while model snapshots come from outside the repository.

The project reviewed here is zzzz7788990213-ops/EvoVLM, subtitled Evolving Multimodal Inference Programs. Its README identifies scientific chart understanding as the target task. Use the owner and subtitle to identify its documentation and experiment records. The practical starting point is the retained research directory, which contains the implementation, evaluation data, evidence and technical report described by the runtime guide.

What you can do with EvoVLM

EvoVLM searches prompts, image preprocessing, generation budgets and answer extraction while keeping model weights frozen. This makes the surrounding inference program the object of experimentation. A researcher can examine a prompt change, image transformation or answer-extraction rule alongside its recorded evaluation, rather than treating every improvement as a consequence of training new weights. The project overview explains that distinction before presenting the method and results.

The documented loop starts from manually designed programs for Qwen3-VL-2B-Instruct and Qwen3-VL-2B-Thinking. A local Qwen model ranks a frozen catalog of typed mutations, and a renderer turns the chosen configuration into Python. Candidates receive evaluations that record correctness, latency, errors and execution provenance. Pareto selection then retains useful accuracy and latency trade-offs. These steps give a reader specific artifacts to inspect: the starting program, selected mutation, generated program and associated evidence.

Start by comparing the manual seed with the selected program. That comparison follows the project's own experimental structure and helps distinguish the contribution of the human-designed starting point from subsequent search. Someone researching program selection can follow the mutation and evaluation records; someone researching model training needs a different experiment. The repository's stated method keeps Qwen weights fixed throughout this search.

What people use EvoVLM for

The repository documents evidence replay, constrained program search and fresh inference as distinct research activities. Its repository layout identifies exported programs, evaluation routines, experiment journals and packaged data subsets. These are author-documented uses of the package. They provide a practical way to choose an entry point according to whether you want to read existing evidence, inspect the search mechanism or generate new predictions.

For evidence inspection, the CPU instructions describe replaying preserved records and checking program interfaces. For search research, the method description explains typed mutations, evaluation and selection. For new predictions, the runtime guide describes external model snapshots and execution modes. Read the instructions for the task you intend to perform before installing inference dependencies or downloading models.

What happened when we ran it

Our November 13, 2026 sandbox run at commit 16b56fd passed 132 of 132 pytest tests in 8 seconds, with 0 failed. Installation took 40 seconds and the build took 3 seconds. The recorded configuration had 3 CPUs, 8 GB RAM and image ghcr.io/astral-sh/uv:python3.12-bookworm. The measured project directory was research/evolving-vlm-inference-main, and the harness recorded a total duration of 60 seconds.

Recorded repository check Outcome
Install Succeeded in 40 seconds
Dependencies recorded 90
Installed disk footprint 5,619 MB
Build Succeeded in 3 seconds
Tests Succeeded in 8 seconds; 132 passed, 0 failed of 132
pip-audit 0 known vulnerabilities
Checkout 2,368 files, approximately 33,977 source lines, 39 MB
Repository signals Tests directory present; 0 CI workflow files; no Dockerfile detected

The 5,619 MB installed footprint is a useful planning figure alongside the 39 MB checkout. Reserve space for the recorded development setup and account separately for external model storage. Our measurements describe installation, build, tests and dependency auditing at this commit. The passing pytest result comes from the supplied harness, rather than a test count copied from the README or an inference score taken from archived experiments.

The separate deployment safety step failed because none of compose.yaml, compose.yml, docker-compose.yaml or docker-compose.yml existed. That procedure required a Compose file. Its failed prerequisite is an actual hands-on finding and belongs alongside the successful repository checks. The log identifies the missing prerequisite directly: the procedure's Compose requirement stopped this step before it could proceed to deployment.

The deployment summary says the November 13, 2026 run deployed with its own Docker Compose and bound every port to 127.0.0.1. That statement conflicts with the specific failed safety step. The recorded outcomes are therefore a failed prerequisite, a contradictory deployment summary and a teardown record reporting teardown: ok. Each record describes a different part of the attempt; the specific failure remains part of the result.

Where people get stuck: the default command requests a GPU run

Running reproduce.sh without a mode requests fresh GPU inference, according to the runtime guide. Supply a mode when following its CPU instructions, and run them from research/evolving-vlm-inference-main. The guide says some verification routines depend on that historical directory name. These documented command distinctions matter before you begin troubleshooting: replaying recorded evidence, checking interfaces and launching fresh inference perform different jobs.

Situation Documented route or explanation
You want to replay preserved evidence Run python3 -B -m evaluation.portable_reproduce from the research directory
You want the explicit CPU checks Use bash reproduce.sh check or bash reproduce.sh smoke as appropriate
You expected a CPU check from the bare script Supply a mode; the no-argument default is fresh
A configured local model path is invalid Correct that path and snapshot identity; the guide says the override fails closed
Snapshot files are symlinked Use physical snapshot files; identity checks reject symlinked paths and critical files
You moved or renamed the research folder Retain evolving-vlm-inference-main because verification routines bind to that name

The runtime guide's resolution order starts with a per-model override, then a shared model root, then a package-local model directory, and finally the official model ID at its locked revision. An explicit local path is authoritative: an invalid path or identity fails instead of falling back. Check the configured source before changing caches, because a higher-priority override determines which snapshot the program attempts to use.

The recorded search starts from manually designed programs

The README describes two generations with at most six children per model population. Its recorded results separate Baseline Instruct, Manual Instruct, Evolved Instruct, Manual Thinking and Evolved Thinking. That separation matters when interpreting the experiment: the manual programs are part of the method, and the evolved programs follow them. A comparison between the original baseline and final selection combines changes made at both stages.

For Instruct, the README reports 78 correct answers out of 128 for both the manual and evolved programs, or 60.94%. Their recorded Speed32 median latencies are 15.44 seconds and 14.07 seconds respectively. The authors describe this as an 8.9% lower median latency with preserved recorded accuracy. The published result table is the source of those figures; they are archived experiment results, separate from our sandbox timings.

The README also reports Manual Thinking at 23 of 128 correct, 17.97% accuracy and 20.04 seconds median latency. Evolved Thinking records 44 of 128 correct, 34.38% accuracy and 17.91 seconds. Its stated changes are 16.41 percentage points in accuracy and 10.6% lower median latency. Speed32 takes a median across three repeats for each query, then a median across 32 queries, so preserve that aggregation when discussing the published timing figures.

External snapshots determine the inference setup

Fresh reproduction requires both pinned Qwen snapshots, including configuration, generation configuration, tokenizer, processor and chat-template files. The runtime guide names Qwen/Qwen3-VL-2B-Instruct and Qwen/Qwen3-VL-2B-Thinking and points to their critical manifests under evaluation/model_snapshots. Those manifests record revisions and critical file identities. Obtain the snapshots at those revisions rather than choosing another revision with the same display name.

The online and offline instructions describe different sources for model files. Online mode needs network access and a writable Hugging Face cache, while a package-local snapshot still takes precedence when present. Offline mode uses local snapshots. The launcher checks critical manifests before creating formal run outputs, starting lanes or initializing CUDA, and an offline launch without valid snapshots exits with an error.

The general dependencies live in requirements.txt, while requirements.lock.txt records the Linux/CUDA environment. The overview explicitly distinguishes that lockfile from a universal macOS setup. For research planning, preserve the selected program and snapshot identity together. Also keep the cost of evaluating search candidates separate from the cost of running the selected program: the method evaluates alternatives before selection, so these are different parts of the workflow.

Evidence integrity and reuse have specific documentation

Our recorded pip-audit result was 0 known vulnerabilities across 90 scanned packages. The README describes a separate set of evidence-integrity checks covering file hashes, critical model and processor identity, cross-bound attempt records and metric recomputation from per-query evidence. These statements address different objects: dependency advisories in our run and consistency controls for the archived experiments. Keep those purposes clear when deciding what to inspect next.

The attribution section identifies Qwen model families and CharXiv data and directs readers to third-party notices. It says the publication imposes no new repository-wide license and that existing third-party licenses continue to apply to their respective materials. Read the notices for the particular code, data and weights you intend to reuse. A license attached to another project called EvoVLM does not describe this repository.

EvoVLM vs EvoVLM-JP and EvoVLMA

EvoVLM studies frozen-weight inference programs, while Sakana AI's repository concerns model merging and EvoVLMA concerns vision-language model adaptation. Choose among them according to the object of your research. The supplied descriptions support a comparison of purposes and entry points: inference-program search, model merging or adaptation. Those distinctions help identify which repository's code and experiment records belong in your investigation.

Project What its supplied source describes Choose it when
EvoVLM, zzzz7788990213-ops Frozen-weight chart inference with constrained program mutations You want to inspect or extend this inference-program experiment
Sakana AI Evolutionary Model Merge Models and code for reproducing paper evaluations You want the model-merging work associated with EvoVLM-JP
EvoVLMA, kding1225 Evolutionary Vision-Language Model Adaptation You want to investigate evolutionary vision-language model adaptation

Sakana AI's repository includes models and code for reproducing its paper evaluations. Its Evolutionary Model Merge README lists EvoVLM-JP-v1-7B among its models. The supplied EvoVLMA introduction discusses adaptation through methods such as prompt tuning and adapters, then introduces Evolutionary Vision-Language Model Adaptation. The comparison table keeps those author-described research areas attached to the respective projects.

How to distinguish similarly named projects

The ICML listing identifies EvoVLM: Multimodal Evolutionary Feedback for Visual Symbolic Regression by Hyejin Lee and Junsuk Choe. Its symbolic-regression abstract describes an automated symbolic regression framework, giving readers another concrete identity to distinguish from the chart-inference repository. Match the owner, full title and method before transferring a command, benchmark or license into your research notes.

The npaka walkthrough explicitly tries EvoVLM-JP in Google Colab. In lmms-eval issue #449, a user asks which generation parameters were used for JMMMU evaluation of SakanaAI/Llama-3-EvoVLM-JP-v2 and reports a best accuracy of 28.48%. A collaborator suggests contacting the JMMMU or EvoVLM-JP-v2 authors. The closing comment cites prolonged inactivity and asks for updated reproduction details if the issue remains relevant. These are records about Sakana AI's model family.

FAQ

The runtime guide separates CPU evidence replay from fresh inference and names two fixed Qwen model IDs. Those details determine which files and execution path a reader needs. The questions below cover the supplied commands, model distribution and the EvoVLM-JP identity. The generation-parameter question comes directly from issue #449 and retains that issue's model attribution.

Does CPU evidence replay generate new predictions?

The runtime guide says CPU evidence commands do not generate new predictions. Its standalone evidence replay uses Python's standard library, while pytest is required for the test suite. Follow the CPU command list from the retained research directory. The commands address recorded evidence or program interfaces, which makes them a useful first step before preparing model snapshots for inference.

Does EvoVLM include the model weights?

Neither the GitHub repository nor the source archive includes model weights, according to the runtime guide. Code, packaged data subsets and evidence are supplied; authenticated snapshots and an appropriate execution environment come separately. The guide points to critical manifests for both Qwen models. Keep those revisions and required files together when preparing the external model dependencies.

Is EvoVLM the same project as Sakana AI's EvoVLM-JP?

Sakana AI's model card describes EvoVLM-JP-v1-7B as an experimental general-purpose Japanese VLM created using Evolutionary Model Merge. The EvoVLM-JP model card supplies that identity directly. The requested repository instead describes inference-program evolution around frozen Qwen models. Keep the full repository owner or model ID with saved instructions so that each command remains attached to its intended project.

What generation parameters were used for JMMMU evaluation of SakanaAI/Llama-3-EvoVLM-JP-v2?

The supplied issue extract mentions a maximum output length of 1,024 and temperature 0, then records a user's request for the remaining parameters. In issue #449's discussion, the collaborator directs the question to the JMMMU or EvoVLM-JP-v2 authors; the later closing comment requests updated reproduction details. That exchange concerns SakanaAI/Llama-3-EvoVLM-JP-v2, so it supplies no setup parameters for this frozen-Qwen inference-program repository.

What we have not tested yet

Our run at 16b56fd did not measure chart-answer accuracy, inference latency, GPU memory requirements, fresh search cost or sustained application operation. It measured repository installation, build, tests and dependency auditing. The command table describes supplied documentation rather than additional commands executed for this review. The failed Compose prerequisite and contradictory deployment summary provide no successful application deployment result. Actual runtime and operating-system versions require raw toolchain logs; the sandbox image label alone supplies neither.

The README identifies Dev128 as a public validation subset with two exact query overlaps and eight queries sharing two figures with the evolution-used set. It describes small-sample development results rather than generalization evidence or a full CharXiv leaderboard result, with latency tied to the recorded environment. The unmodified Thinking baseline was timeout-censored at 7,200 seconds, leaving no completed Dev128 accuracy for that baseline. Our run did not reproduce the archived Dev128 or Speed32 results, test a fresh holdout or compare alternatives on identical inputs and hardware.

The supplied synthesis contains no attributable outside adoption report for this repository, and the EvoVLMA extract supports its adaptation topic without specifying the detailed search targets. Dated repository metadata observed on November 14, 2026 records 5 stars, 0 open issues and pull requests combined, Python as the language and a last push of 2026-10-02T01:04:42Z. The null latest-release lookup leaves release status unresolved; no supplied response establishes a rate-limit cause. These metadata fields supply no measure of support capacity or adoption.

Execution security, redistribution permissions for every artifact, platform portability and English-language coverage beyond the supplied overview and runtime guide remain outside our assessment. The dependency audit and documented evidence checks answer narrower questions. Rendered layout, related links, recommendation modules and template behavior also fall outside this review's evidence. The next useful experiment would retain exact model snapshots, use a separate holdout and record search cost alongside selected-program performance.

Alternatives

ProjectWhat it isPick it when
Sakana AI Evolutionary Model MergeSakana AI's repository includes models and code for reproducing its paper evaluations.Choose this when your research concerns model merging or Sakana AI's EvoVLM-JP model family.
EvoVLMAThe supplied introduction describes evolutionary vision-language model adaptation and discusses prompt tuning and adapters.Choose this when your question concerns evolutionary vision-language model adaptation rather than EvoVLM's chart-inference program search.

Sources

  1. EvoVLM project overview
  2. EvoVLM runtime guide
  3. Sakana AI Evolutionary Model Merge
  4. EvoVLMA supplied introduction
  5. Sakana AI EvoVLM-JP model card
  6. Separate symbolic-regression EvoVLM
  7. Sakana model evaluation issue
  8. EvoVLM-JP Colab walkthrough

More ai tools reviews