mrkeyoor.comAsk

roboterax/video-prediction-policy-2tested 14 Nov 2026commit a103a1e

video-prediction-policy-2 review

Our VPP2 sandbox installed successfully in 63 seconds and built in 1 second, but the authors withhold the distilled planner and Stage-3 policy behind their reported real-world results.

  • Build 1 s
63 sto install
  • 106 packages
AI Toolsevaluationupdated 14 Nov 2026
+12stars / 7d

Video Prediction Policy 2 (VPP2) predicts how a robot manipulation task should unfold, then uses video features to generate actions. Its public release provides video-prediction and simulation-policy workflows, while the authors withhold components used for their reported real-world zero-shot experiments.

Verdict

Our VPP2 sandbox installed successfully in 63 seconds and built in 1 second, but the authors withhold the distilled planner and Stage-3 policy behind their reported real-world results. Use the public release for video-prediction research and LIBERO or RoboDojo work. Skip it if your immediate goal requires reproducing that real-world policy, and review the 13 dependency advisories before adopting the environment.

video-prediction-policy-2 infographic: uses video prediction to generate robot actions, plus our test result and who it suits
Tested 14 November 2026: uv pip install vpp2 failed after 1.1 s. Before that: source install 63 s, build 1 s.

We ran it

Lab card: what happened when we ran video-prediction-policy-2
Lab card from our sandbox run
Screenshot of video-prediction-policy-2 (robert-gyj.github.io/video-prediction-policy-2)
robert-gyj.github.io/video-prediction-policy-2
Install✓ · 63s106 packages · 6203 MB
Build✓ · 1s
Testsn/ano test script
Known vulns13(pip-audit)
Repo226 files~24,815 lines of source · 8 MB · 0 CI workflows

Answers from our run

Does video-prediction-policy-2 build from source?

Dependencies installed in 63 seconds (106 packages), and the build succeeded in 1 seconds. We cloned commit a103a1e into a clean Debian container with 3 CPUs and no project-specific setup.

Does video-prediction-policy-2 have tests you can run?

Not through a standard command: the project exposes no test script or target that our harness could run.

Does video-prediction-policy-2 have known vulnerabilities in its dependencies?

pip-audit flagged 13 known advisories in the dependency tree at the time of our run.

Who should not use video-prediction-policy-2?

Teams that need to reproduce the headline real-world policy: the authors have no plans to release its distilled planner and Stage-3 components.

What are the alternatives to video-prediction-policy-2?

π0.5, Video Prediction Policy (VPP). Our VPP2 sandbox installed successfully in 63 seconds and built in 1 second, but the authors withhold the distilled planner and Stage-3 policy behind their reported real-world results.

Setup2/5In our 2026-11-14 run “uv pip install vpp2” failed with “╰─▶ Because vpp2 was not found in the package registry and you requir…”, though our source checkout installed in 63 s and built in 1 s.Editor’s note (scored 3/5): Successful recorded installation and build support a usable repository path with material constraints: 6,203 MB on disk, skipped tests and separate policy assets. GPU deployment was outside the measured scope.
Docs4/5The README has an install section (“Installation”) and a usage section (“Get started”); we did not run its quickstart ourselves.Editor’s note (scored 3/5): The README links video, LIBERO and RoboDojo guides and lists paired assets. Important release and inference distinctions require issue replies, including #10 and #11.
Community4/5Last push on 2026-10-19, 376 stars and maintainers answered or closed 10 of the 10 most-discussed issues we sampled.Editor’s note: The supplied issues show author responses, but do not establish support capacity, response-time expectations or production adoption.
Maturity1/5No tagged GitHub release and repository created 2026-10-08.Editor’s note (scored 2/5): The public simulation workflow has released assets, but the real-world distilled planner and Stage-3 policy are withheld. The recorded scan found no CI workflows or tests directory, and the harness skipped tests.

How we score

Who it’s for

Robotics researchers studying instruction-conditioned manipulation video prediction.
Teams adapting policies to LIBERO or RoboDojo with paired video and action checkpoints.
Researchers who can separate the released simulation recipe from the paper's real-world experiment.

Who it’s NOT for

Teams that need to reproduce the headline real-world policy: the authors have no plans to release its distilled planner and Stage-3 components.
Developers expecting a tested drop-in policy environment: our harness skipped tests because it found no test script or target.
Teams expecting the released RoboDojo checkpoints to include subtask planning: the authors say they were trained on full instructions.
Operators needing a complete moving-base calibration procedure: issue #5 leaves that deployment question unresolved.

Setup reality

Our November 14, 2026 sandbox run at a103a1e installed successfully in 63 seconds, recording 106 dependencies and 6,203 MB on disk. The build succeeded in 1 second. Tests were skipped because no tests script or target was found, and pip-audit reported 13 known vulnerabilities. The scan found 0 CI workflows, no Dockerfile and no tests directory.

The README loads VPP2 from the checkout and documents CUDA/PyTorch setup before requirements. Policy evaluation needs matching video/action checkpoints, shared encoders and normalization assets. Hugging Face downloads are public; ModelScope requires an account with access. The unreleased real-world components are outside that download path.

The recorded sandbox had 3 CPUs and 8 GB RAM with image label ghcr.io/astral-sh/uv:python3.12-bookworm. That label does not identify actual runtime versions. The README starts with Python 3.10; our install/build measurements are repository checks, not a completed GPU policy deployment.

Our sandbox installed Video Prediction Policy 2 in 63 seconds and completed the build step in 1 second. Run the released project if you want to study manipulation video prediction or adapt policies to LIBERO and RoboDojo. Walk away if your immediate requirement is reproducing its headline real-world robot policy: the authors have withheld the distilled planner and Stage-3 policy. The useful question is which experiment you can reproduce with the assets actually released.

What people use video-prediction-policy-2 for

The authors use VPP2 for instruction-conditioned video prediction and robot manipulation, while a community experiment exposes the difference between the paper's planner and released RoboDojo checkpoints.

Who used it What they did What happened
The authors Deployed on real ALOHA manipulation tasks without task-specific fine-tuning Reported 58.5% mean success across 10 task categories on an embodiment seen during training. ALOHA evaluation
The authors Judged whether predicted manipulation videos followed instructions Reported 0.80 success for human-hand prompts and 0.90 for robot prompts using a GPT-based evaluator. Video evaluation
The authors Post-trained policies for simulation benchmarks Reported 45.0% on LIBERO-Pro, 63.9% on LIBERO-OOD and 32.26% on RoboDojo. Updated paper
The issue #9 reporter Added a reconstructed VLM subtask controller to the released RoboDojo policy Original instructions scored 49 versus 40.5 with subtasks across 10 tasks, with one episode per method per task. Community comparison

The reporter used one episode per method per task, rather than repeated trials. Evaluation design

For a robotics researcher, the video model is an interesting starting point because VPP2 trains predictions around a complete manipulation event before adapting them to shorter future chunks. The instruction describes a specific intended action, rather than merely asking for plausible motion. The paper then conditions an action expert on video features. That gives you separate questions to investigate: whether the predicted event follows the instruction, and whether the resulting actions complete the task. The authors' architecture

The issue #9 experiment makes that distinction concrete. In hang_mugs, the reporter's subtask caption stayed active for all 19 action chunks, yet the benchmark recorded full-task success after 445 actions. Their progress checker never advanced the caption. A completed task therefore wasn't evidence that their reconstructed planner had successfully tracked subtask transitions. The authors replied that the released RoboDojo checkpoints were trained on full instructions and contain no subtask-planning components. Start your comparison with the original instructions. Reporter and author response

What happened when we ran it

Our run at commit a103a1e on November 14, 2026 installed dependencies successfully in 63 seconds, with 106 dependencies recorded and 6,203 MB on disk.

Check in our sandbox Recorded result
Install Succeeded in 63 seconds
Build Succeeded in 1 second
Tests Skipped: no tests script or target
Dependency audit pip-audit reported 13 known vulnerabilities
Repository scan 226 files, approximately 24,815 source lines, 8 MB checked out
Automation and packaging signals 0 CI workflow files, no Dockerfile, no tests directory

The recorded configuration was 3 CPUs and 8 GB RAM, with image label ghcr.io/astral-sh/uv:python3.12-bookworm. The small checkout and much larger dependency footprint are a practical reason to give this project its own environment. A successful build step makes the source easier to evaluate, but the skipped test step supplies no regression evidence. Keep those two outcomes separate when deciding whether to modify the policy code.

The one-second build result is the recorded harness step, not the time needed to train a model or make a robot move. Likewise, 6,203 MB describes the recorded installed environment. The README separately lists downloadable video and action checkpoints plus shared encoders and normalization assets. Those are additional setup decisions for the workflow you choose. Released asset list

Where people get stuck: checkpoint names hide different training paths

RoboDojo's single-forward video inference comes from maximum-noise post-training, while the reported real-world planner uses a separate consistency-distillation route.

Symptom or question What the record says Practical response
A registry lookup cannot find the project A separate package-discovery attempt reports that vpp2 was not found in the package registry Follow the README's checkout-based loading instructions. This diagnostic record is separate from the successful sandbox measurements above.
More video sampling steps produce worse frames TangGuohh reported blurrier frames with 10 steps than with one. The authors confirmed maximum-noise-only RoboDojo post-training Use the intended single-forward video path for that checkpoint. Issue #11
Adding a planner fails to improve scores consistently The issue #9 reporter's reconstructed controller scored below original instructions on the mean native score Establish the released full-instruction baseline before adding caption switching. Issue #9
A short future window ends before the captioned task finishes Haodong-Yan says Stage-2 windows reuse the source event's detailed caption Let the observation indicate progress. Completion within every window is not required. Issue #4
A short prompt behaves differently across workflows Robert-gyj says detailed-caption training calls for prompt enhancement, with RoboDojo an exception Use original prompts for RoboDojo training and inference. Follow prompt enhancement for the general video-model route. Issue #2

The sampling report is especially useful because "try more steps" sounds like a reasonable quality adjustment. Here the author response supplies a specific reason to stop: intermediate noise levels were outside RoboDojo post-training. Treat video sampling and action denoising as separate settings. A checkpoint designed for a single video forward pass does not become the paper's consistency-distilled planner when you change a sampler option. Sampling clarification

The missing-recipe discussion has a different outcome. In issue #3, a reader asked how the history-conditioned robodojo_his10k initializer had been produced before joint policy training. The authors explained its pre-distillation Stage-2 origin and promised a recipe. The README subsequently announced the history-conditioned training code and Video-10k guide on October 12, 2026. That closes the documentation complaint about the preceding stage; it doesn't turn the withheld real-world policy into a released checkpoint. Provenance discussion, README announcement

The 8 MB checkout is only the start of the environment

Our sandbox's 6,203 MB installed footprint is the useful disk-planning observation, and the README requires separately selected model assets for policy evaluation.

The README starts with a Python 3.10 Conda environment and points to CUDA/PyTorch setup before repository requirements. Its scripts load VPP2 from the checkout. The model download route matters too: Hugging Face is public, while ModelScope requires an account with access. You can choose the public route without treating a ModelScope account as a universal prerequisite. Installation and downloads

For policy evaluation, pair the matching video and action checkpoints, rather than collecting filenames that look approximately right. The README also lists shared Wan encoders, CLIP, UMT5 and tokenizer assets, plus normalization files. Save an asset manifest alongside the configuration you use. That recommendation follows directly from the number of separate components involved: changing a checkpoint while retaining another benchmark's normalization files would make the resulting experiment hard to interpret. Policy assets

Captioning is another resource decision when preparing your own data. Haodong-Yan says the project reuses an event caption across Stage-2 windows and warns that finer window-specific annotation can introduce inconsistent descriptions and higher annotation costs. Before paying to recaption overlapping windows, establish what the existing caption-and-observation pairing gets wrong on your data. An event caption supplies the goal, while the observed frame shows how far the task has progressed. The released design already expects a window to cover only part of an event. Caption reuse explanation

Security: 13 dependency advisories need a separate decision

Our pip-audit result reported 13 known vulnerabilities, so the successful installation is insufficient grounds to approve this environment for a shared research machine.

Keep the dependency environment isolated and review the advisory details before connecting it to valuable datasets or robot-control systems. The supplied summary gives a total, rather than a package-by-package remediation map. It would be irresponsible to name a vulnerable library, prescribe an upgrade or claim a remote exploit from that total alone. The scan also found no CI workflow files and no tests directory, so your own change checks will matter if you carry patches.

The README licenses the code under MIT and explicitly gives model weights, datasets and simulator assets their own terms. A team planning commercial use should inventory those assets alongside dependencies. The code license is useful permission for the code itself, but it doesn't settle permission for every downloaded component. License boundaries

Video Prediction Policy 2 vs π0.5, Fast-WAM and Cosmos3

The authors report a higher ALOHA average for VPP2 than for π0.5 or Fast-WAM, while Cosmos3 is a video-prediction comparison with a different outcome measure.

Alternative Supplied comparison How to use it in a decision
π0.5 Author-reported ALOHA mean: VPP2 58.5%, π0.5 40.0%. Fold favors π0.5, 80% versus 70% Include π0.5 when folding resembles your target task. The average conceals a category where it wins. ALOHA results
Fast-WAM Author-reported ALOHA mean: VPP2 58.5%, Fast-WAM 20.5% Relevant to comparing action policies under the reported ALOHA setup. ALOHA results
Cosmos3-64B The paper reports an 11.0 percentage-point VPP2 advantage in video instruction following Compare instruction-faithful predictions, rather than reading this as a robot task-success margin. Paper abstract
Temporal Ratio The supplied analysis reports LIBERO-OOD success of 63.9% for VPP2 versus 59.4% A useful benchmark reference. The comparison does not isolate which mechanism produced the difference. LIBERO-OOD comparison
Original VPP The predecessor also uses predictive video representations; no direct score comparison is supplied Study it when the research question concerns what VPP2 changes in prediction and action learning. Predecessor discussion

The fixed-duration VPP2 ablation provides a more focused research comparison than a larger model's parameter count. The supplied analysis reports instruction-following results of 0.48/0.42 for human/robot prompts with fixed-step prediction, against VPP2's 0.80/0.90. That supports examining event-level prediction for this evaluation. It still calls for a carefully matched experiment before attributing every gain to caption detail alone. Event-level comparison

What the other reviews get wrong

The v1 paper's RoboDojo figure is 29.47%, while the supplied updated paper reports 32.26% success and the README separately labels 39.26 as Score.

Use the updated 32.26% success rate when discussing the supplied updated results. Do not replace it with 39.26%: the README's native Score and SR are different metrics. This matters especially when comparing the issue #9 reporter's partial-progress scores with the paper's success rates. The former can credit incomplete task progress. Original v1 figure, updated paper, README metric labels

The project page's "11%" Cosmos3 advantage also needs a unit. The abstract specifies 11.0 percentage points in video instruction-following success. That is the wording to carry into a comparison, rather than turning the difference into a relative percentage or a robot-control result. Project-page wording, abstract's units

More consequential than either number is the release boundary. Robert-gyj states in issue #10 that video distillation and Stage-3 training serve the real-world zero-shot experiments, that there are no plans to release them, and that the current code uses no Stage-3 checkpoints. A summary of the paper's full architecture needs that sentence alongside any recommendation to download the repository. Release clarification

Questions people ask

The authors' issue replies answer checkpoint provenance and inference settings more directly than the headline benchmark descriptions.

Will the consistency-distilled Stage-2 checkpoint and Stage-3 action policy be released?

The authors say they currently have no plans to release the distilled video model or Stage-3 training components. Their response identifies these with the real-world zero-shot experiments. A closed issue here records an answer about release scope, rather than delivery of the requested artifacts. Issue #10

Which reported evaluations use Stage-3 action-pretrained weights?

The authors identify the real-world zero-shot experiments as using video distillation and Stage-3 training. They also state that the current public code uses no Stage-3 checkpoints, and explain why simulation tasks use a different action-learning path. Issue #10

Are the LIBERO and RoboDojo video models initialized before or after consistency distillation?

RoboDojo starts from the Stage-2 video checkpoint before distillation, according to issue #3. For LIBERO, issue #11 says it is also initialized from the Stage-2 checkpoint, without explicitly naming the distillation status. The replies support a precise RoboDojo answer and a narrower LIBERO provenance statement. Issue #3, issue #11

What training recipe produced RoboDojo Video-10k?

The authors describe standard video-model fine-tuning on RoboDojo from the pre-distillation Stage-2 checkpoint, adding history frames for memory-dependent tasks. The README's October 12 announcement points to the released history-conditioned Video-10k recipe. Provenance reply, recipe announcement

What consistency-distillation settings are needed to reproduce the unreleased model?

The authors disclose lambda of 0.5 and 6,000 training steps in issue #11. That reply gives specific settings, rather than a complete reproducibility package. The same discussion says the distilled checkpoint will be withheld. Distillation response

Why can 10-step sampling produce blurrier RoboDojo videos than one-step inference?

The authors confirm that RoboDojo post-training deliberately uses only the maximum noise level so inference takes one forward pass. TangGuohh's report of worse multi-step output fits that training distinction. Use the checkpoint's intended inference path before treating extra steps as an improvement. Sampling report and reply

How should RoboDojo history slots be filled at episode startup?

The authors say to repeat available frames until there are eight when fewer than eight are available. Use that explicit startup instruction for the history slots. It answers an episode-initialization question, rather than requiring the robot to wait while accumulating a full history. Startup reply

How does action training align a predicted video trajectory with ground-truth actions?

The authors say detailed captions constrain possible futures, and RoboDojo uses history to reduce ambiguity. Their response acknowledges the multimodal-trajectory problem raised by LilyLove0810. Shared observations and a task description can still permit more than one plausible future, which is why the question matters. Alignment discussion

Does the action expert use predicted future dynamics or mainly pretrained visual representations?

Issue #1 raises exactly this distinction, but the supplied author reply addresses ambiguity through captions and history rather than demonstrating which information the expert relies on. Treat the question as an experiment to run with feature interventions, not as a settled interpretation of the benchmark score. Representation question

Have ground-truth future features or shuffled and removed video KV caches been tested as ablations?

The supplied issue #1 response reports no results for those requested ablations. A reader proposes them to distinguish dependence on future dynamics from dependence on learned visual features. They would answer a narrower causal question than comparing VPP2's overall success with a different policy. Requested ablations

Which training stages use detailed instructions?

The authors say detailed captions are used in both event-level training and chunk-level video/action learning. RoboDojo post-training is their stated exception. Carry that exception into any training-data plan instead of applying the general caption recipe indiscriminately. Caption-stage reply

Is VLM prompt enhancement required at inference?

The authors say the general video model requires prompt enhancement to match detailed-caption training. RoboDojo training and inference instead use original prompts without enhancement. This is a workflow-specific requirement, so identify the checkpoint and benchmark before adding a VLM call. Inference-prompt reply

What we have not tested yet

Our measurements cover dependency installation, a build step and a dependency audit, not video generation, simulator task success or physical robot operation.

We did not measure GPU memory, inference latency, training throughput, deployment cost or performance under concurrent workloads. The recorded 8 GB RAM configuration is a sandbox allocation, not observed peak memory or a model-serving requirement. The container image label does not identify the actual runtime or operating-system versions. Those require the raw toolchain logs. No test count can be inferred from the skipped test target.

The ALOHA, LIBERO and RoboDojo performance numbers remain author-reported. The community controller comparison is a single-episode-per-task pilot, not a repeated-trial benchmark. Neither establishes reliability for a new robot. The authors' ALOHA evaluation used an embodiment seen during training, and their issue #6 response says they currently have no plan for RoboDojo's real-world track. That track must not be silently substituted with ALOHA results. Evaluation scope, real-world-track response

Cross-embodiment deployment also needs more work than a checkpoint download. In issue #5, the authors describe dataset-specific heuristic workspace and end-effector alignment. The supplied discussion leaves moving-base handling, camera dependence, calibration files and the complete output-coordinate convention unresolved. We have not assessed those deployment questions, alignment guarantees, the requested feature ablations, or whether window-specific captions improve results. Calibration discussion, feature questions, caption tradeoff

The English README and issue replies are readable sources for these workflows, but they supply no evidence of worldwide production adoption. Our ratings concern the documented evaluation path: on a 1-to-5 scale, 3 means usable with material constraints, and 2 means substantial unresolved barriers. Setup receives 3 for successful recorded installation and build steps, documentation 3 for workflow guides plus issue-dependent clarifications, and maturity 2 for withheld real-world components and absent automated-test evidence. Community support capacity is unassessed.

For a first trial, choose the released RoboDojo or LIBERO workflow, record the paired checkpoints and establish the original-instruction baseline. If your deliverable is the reported real-world zero-shot policy, stop at the release boundary before allocating robot time.

Alternatives

ProjectWhat it isPick it when
π0.5An action-policy baseline in the authors' ALOHA comparison, with 40.0% mean success versus VPP2's 58.5%.Include it in a folding-focused evaluation: π0.5 wins Fold in the reported comparison, 80% versus 70%.
Video Prediction Policy (VPP)The predecessor also conditions robot policies on predictive video representations.Study it when you want to compare VPP2's event-level prediction approach with its predecessor; the supplied evidence has no direct score comparison.

Sources

  1. Official VPP2 README
  2. Updated VPP2 paper
  3. Stage-3 release clarification
  4. Video sampling and distillation answers
  5. Community subtask-controller comparison

More ai tools reviews