AgentGarten installed in 50 seconds and built in 1 second in our sandbox at commit 06bb462. Its released renderer is aimed at researchers who can supply CUDA hardware and geometry inputs. For teams seeking the demonstrated hide-and-seek experiment, the release distinction matters: the renderer is available, while the worlds and agent practice loop are announced release items.
AgentGarten separates a world's rules from its appearance. A simulator or game engine maintains objects, state and interactions; a neural renderer turns exported geometry into the agent's visual observations. MirroS uses this arrangement to study pretrained agents that write playbooks after each round and pass those lessons to later agents. The repository contains renderer training recipes and streaming inference, making that component the starting point for integration work. AgentGarten README
What people use AgentGarten for
MirroS demonstrates hide-and-seek, sheep herding and quarry-loader tasks in which agents carry written lessons into later rounds. These examples give researchers concrete goals, action sequences and playbooks to inspect. They also explain the engine-renderer split: task rules stay in executable code while agents respond to camera observations generated from the world's geometry. MirroS experiment protocol
| Who used it | Task | Reported outcome |
|---|---|---|
| MirroS researchers | Hide-and-seek with inherited playbooks | Hiders built shelters by round 4; seekers crossed walls with ramps by round 10. Official blog |
| MirroS researchers | Two sheepdogs guiding four sheep | Round 1 ended with three penned; rounds 2 through 4 penned all four. Project demonstration |
| MirroS researchers | Quarry loader staging rocks, delivering one and parking | Round 4 completed the sequence with 31 simulated seconds remaining. Project demonstration |
The sheepdog task gives each dog an eye-height view. Sheep move away from an approaching dog and scatter under excessive pressure, so the agents must guide the flock while keeping it together. The project reports three sheep penned in round 1 and all four in rounds 2 through 4. This supplies a specific behavior to examine alongside successive playbooks. Herding demonstration
The loader has three jobs in order: stage two rocks, deliver one behind a wall and park. Its first round cleared the rocks and got no further; round 4 completed the sequence with 31 seconds left. The technical report offers another use case: constructing a living-room scene from an image, then adding connected rooms as authored extensions beyond the original view. Loader demonstration, Living-room construction
What happened when we ran it
Our run at commit 06bb462 on November 14, 2026 installed the repository in 50 seconds and built it in 1 second. We measured 75 dependencies and 5,381 MB on disk. The checkout contained 97 files, about 17,730 source lines and 1.1 MB, so the installed environment accounts for substantially more storage than the source tree.
| Recorded check | Our result | What it tells you |
|---|---|---|
| Installation | Succeeded in 50 seconds | The recorded installation step completed. |
| Build | Succeeded in 1 second | The recorded build step completed. |
| Dependency inventory | 75 dependencies; 5,381 MB on disk | The installed environment is much larger than the checkout. |
| Tests | Skipped: no tests script or target | The harness recorded a skipped test step. |
| Dependency audit | pip-audit: 0 known vulnerabilities | The harness records 74 scanned packages, 75 reported packages and one skipped package. |
| Repository scan | 0 CI workflow files; no Dockerfile; no tests directory | The scan found none of these components. |
The repository harness used a recorded configuration of 3 CPUs, 8 GB RAM and image ghcr.io/astral-sh/uv:python3.12-bookworm. The separate quickstart used 2 CPUs and 4 GB RAM, a fresh unprivileged container with the same image label, no published ports and no secrets. It recorded commit 06bb4621deb9 and selected the PyPI package name from project metadata.
| Quickstart question | Answer | Verdict | Why |
|---|---|---|---|
| Did the clone step complete? | The step took 2 seconds. | Completed | The quickstart recorded its clone step. |
| What did the installation command install? | uv pip install wm installed wm 0.0.6 in 13.5 seconds. |
Completed | The command used a fresh virtual environment and the package name from project metadata. |
| How large was that installation? | Its site-packages occupied 5.4 MB. | Recorded | This figure belongs to the PyPI quickstart environment. |
| How many other packages were installed? | 8 packages besides wm. | Recorded | The quickstart recorded this dependency count. |
| Did the README inference block run? | The unedited 24-line block failed in 0.1 seconds with exit 1 at import torch: ModuleNotFoundError: No module named 'torch'. |
Failed | Execution stopped at the first import. |
| Did teardown complete? | Teardown was recorded as ok. | Completed | The teardown record reports success. |
Where people get stuck: package identity and missing inputs
The separate PyPI quickstart stopped at the torch import, before model loading. Its finding record labels the README snippet as failed, and the traceback identifies the missing module. The README documents cloning MirroS-Lab/AgentGarten and installing the source with its extras on a CUDA host. These are distinct installation routes, with different recorded outcomes. Source installation instructions
| If you hit this | Evidence | Practical next step |
|---|---|---|
| The example stops on importing torch | Our separate quickstart failed before model loading | Follow the documented repository installation and PyTorch requirements. |
| The renderer checkpoint is present but the video codec is missing | The model card says the Wan 2.2 VAE is excluded | Download the VAE separately and set the local paths. |
| The inference block has undefined inputs | The README expects anchor, first_condition and blocks from the caller | Prepare the initial latent and geometry conditions for the streaming loop. |
| You are looking for the physics-world programs | Issue #1 asks about their release; the README lists the worlds as a release item | Follow the announced release item and issue. |
The model card gives separate downloads for the renderer and Wan 2.2 VAE. The README's inference block loads a local artifact and starts a stream with caller-supplied inputs. Its comments identify anchor as the clean first-frame latent and describe conditions containing text context, depth and normal latents. Those preparations precede the denoising loop. Renderer model card, Streaming example
Issue #1 asks in Chinese whether the physics-modeling programs will be made public. The supplied record lists it as open with 0 comments. That question concerns the world programs, which the English README lists alongside the agent practice loop in its release checklist. Readers tracking those components have a specific issue to follow. Physics-program release question, Issue record
CUDA and caller-supplied geometry define the setup
The README specifies Python 3.11+ and PyTorch 2.9+ on a CUDA host. Its streaming interface loads the renderer artifact, prepares serving kernels and starts from an initial latent and structured conditions. Setup therefore involves model assets and tensor inputs as well as Python dependencies. Integration connects an engine's camera geometry to the renderer's expected conditions. AgentGarten requirements and inference
The technical report describes colorized depth or surface normals as structured conditions. An engine maintains object poses and task variables, exports camera geometry, and leaves the renderer to generate appearance using a reference image, text and visual history. Serving uses Triton kernels, cuBLAS and CUDA graphs. The README says compilation and graph capture happen on first use and instructs callers to warm input shapes before serving. Technical report, Serving instructions
AgentGarten vs conventional simulators and video world models
AgentGarten maintains persistent state and transition rules in an engine and generates observations with a shared neural renderer. That architecture gives readers a comparison criterion: decide whether the immediate task is building simulation rules, authoring a playable environment or studying learned visual observations. Godot and Chrono address parts of that work through their own engine and simulation interfaces. Technical report
| Option | Reason to choose it | Difference that matters |
|---|---|---|
| Conventional simulators and game engines | You want explicit objects, rules and task scoring | AgentGarten couples a code-controlled world to a shared learned renderer. Official explanation |
| Video world models | Generated visual history is central to your experiment | The report contrasts their implicit state with AgentGarten's executable transitions. Technical report |
| Code-based physics worlds using PyChrono | Physics simulation is the main job | The supplied overview distinguishes physics simulation code from an engine coupled to learned rendering. Architecture comparison |
| OpenAI's 2019 hide-and-seek experiment | You are studying self-play reinforcement learning | It trains policies from scratch on object state; MirroS uses pretrained visual agents and written playbooks. Official protocol comparison |
Godot is a cross-platform engine for creating 2D and 3D games. Its README describes a unified interface and common game-development tools, making it relevant when the immediate task is authoring a playable environment. Chrono provides multibody simulation and Python and C# APIs. Choose it when physical-system modeling is the main requirement. These tools address different parts of a research stack. Godot README, Chrono README
The reported rendering speed is an H100 result
StackSweep reports 36.5 frames per second at 480 by 832 resolution on one H100. Its breakdown gives 438.8 milliseconds per 16-frame block, including 414.4 milliseconds for the four-step transformer. Those figures describe rendering blocks. An application also has an agent selecting actions and an engine updating the world, so renderer throughput is one component of the interaction loop. Reported renderer timings
The README describes a bounded key-value cache that carries generated history between blocks. Each block is denoised and committed before later conditions arrive, allowing the next observation to depend on the preceding action. This explains the streaming interface: the application supplies conditions incrementally instead of preparing an entire future video window before generation. Streaming interface, Official interaction explanation
Code license and renderer asset inventory
The supplied repository metadata identifies the code license as Apache-2.0. The renderer card lists a generation tower, a frozen Cosmos3-Nano text tower, its tokenizer and configuration files. It also identifies the Wan 2.2 VAE as a separate download. An integration inventory should distinguish repository code, renderer files and the separately obtained video autoencoder. AgentGarten repository, Renderer asset inventory
MirroS's experimental action interface accepts short Python programs for movement and object interaction. The agent then receives generated observations of the resulting changes. That interface makes program execution part of environment design, alongside task scoring and geometry export. A developer adapting the design needs to decide where action programs execute and what resources the execution service exposes. Official agent action protocol
What the other reviews get wrong
OpenTrain calls MirroS-Lab/AgentGarten an adjacent implementation, but the repository identifies itself as the official AgentGarten repository. OpenTrain also says it found no trustworthy direct Hugging Face artifacts, while the supplied official renderer card documents model downloads. These corrections concern implementation identity and artifact availability: readers should begin with the official repository and its renderer release. OpenTrain implementation record, Official repository, Renderer downloads
Papers with Code summarizes inherited-playbook learning as four rounds instead of millions. The official blog gives separate milestones: shelters by round 4 and ramp crossings by round 10. It distinguishes pretrained agents keeping written playbooks from reinforcement-learning policies trained from scratch. The WISE AI article says ten games per round; MirroS specifies five games across sampled layouts and random seeds. Papers with Code summary, WISE AI description, Official procedure
Dataist describes exact replay as adding no peak-memory cost, while StackSweep reports 0.2% more memory. StackSweep also reports a 5.4% speed improvement and removal of a 3.99% relative error in the earlier replay approach. These accounts agree on the purpose of blockwise replay but differ on literal zero memory overhead. The reported values preserve that disagreement. Dataist replay account, StackSweep replay figures
Questions people ask
The renderer code and weights are available; the README lists the worlds and agent practice loop for a later release. Its training recipes and streaming inference describe the renderer side of the system. The questions below come from the supplied issue, community discussion and research synthesis, covering release status, agent observations and training replay. Official repository, Renderer model card
Will the physics-modeling programs be open-sourced, and when?
The README says the code worlds and agent practice loop will be released in the repository. Issue #1 asks whether the physics-modeling programs will be made public, and the supplied issue record is open with 0 comments. The release checklist names both the worlds and the rounds-of-play workflow, giving readers specific components to track. README release checklist, Issue #1
Does bitwise-identical replay guarantee deterministic agent-policy rollouts?
The report's bitwise-identical result concerns renderer-training recomputation of a sampled trajectory. Its blockwise replay preserves the rollout's execution structure while allowing later losses to update history encoding. A commenter on r/deeplearning distinguishes this renderer-training method from deterministic policy rollouts. The operation described in the report is replaying renderer computation, rather than choosing an agent's next action. Exact-replay explanation, Community discussion
What can I download and run today?
The released artifacts include renderer code and weights, with a separate Wan 2.2 VAE download. The model card documents downloading renderer files, and the README supplies a streaming loop and training recipes. Those are the renderer-side components for connecting prepared inputs to an engine exporting geometry, rather than a packaged version of the demonstrated games. Model downloads, README inference and training
Does the agent see the engine's hidden state?
MirroS's hide-and-seek agents receive rendered first-person observations. The official protocol excludes object coordinates, hidden world state and the opponent's private observations. Agents submit short Python action programs and inspect the resulting visual changes. Each role then reviews its own programs and permitted visual evidence, adding lessons or code to the playbook used in later rounds. Official observation and review protocol
Can AgentGarten render a code-built world during interactive flight?
MirroS_ai published a wingsuit demonstration of neural rendering through a code-built landscape. The supplied post describes flight from snowy peaks to forest valleys and presents the demonstration as an example of the renderer in action. It provides a visual example of the engine-renderer arrangement alongside the task-based hide-and-seek, herding and loader demonstrations. MirroS_ai wingsuit post
When do shelter-building and ramp-use strategies emerge?
MirroS reports shelter construction by round 4 and ramp crossings by round 10. Each hide-and-seek round contains five games across sampled layouts and random seeds. Agents begin with empty playbooks, review their games after a round and carry accumulated skill files forward. These procedural details explain what the round numbers count and how experience reaches the next agents. Official strategy milestones and round protocol
Documentation earns 3 out of 5 within this review's scope
Documentation receives 3/5 because the README supplies hardware requirements, asset paths, training commands and streaming code, while the complete demonstrated system is still an announced release item. The rubric assigns 1 to unusable guidance, 3 to concrete instructions with material gaps and 5 to instructions corroborated across the reviewed workflows. This rating assesses the supplied guidance and release coverage. README instructions and release checklist
Setup ease, community support and maturity receive N/A. Their evidence consists of successful repository installation and build steps, a failed alternative quickstart, one unanswered release issue and a repository scan. Those observations leave different questions for each score: supported GPU setup, maintainer responsiveness and operational reliability. The cards explain each evidence gap instead of assigning a numerical average to unrelated observations.
What we have not tested yet
Our repository run completed installation and build without rendering a world or reproducing the learning results. The separate PyPI quickstart failed at the missing torch import; its log establishes neither a renderer defect nor a CUDA or Python incompatibility. Actual runtime and operating-system versions require raw toolchain logs rather than the configured image label. GPU memory, target-machine throughput, simultaneous-session capacity and action-execution security also lack measurements here.
Weights licensing and redistributed-component licensing were not verified: the supplied readable model-card extracts contain no license passages establishing their coverage or redistribution terms. The code repository's Apache-2.0 metadata supplies no basis for assigning that license to model assets. Before adopting the renderer, review the actual terms covering the generation tower, text tower, tokenizer and separately downloaded VAE. Repository metadata and README, Supplied renderer card
The supplied release evidence gives no date or complete executable reproduction path for the worlds, agent loop and interactive frontend. The model listing reports no Inference Provider deployment. The demonstrations and wingsuit post supply no independent reproduction of learning gains or autonomous-flight evaluation. Differences in pretraining, observations, actions and learning procedures prevent interpreting the hide-and-seek milestones as a controlled efficiency ratio against reinforcement learning. Release question, Model listing, Official experimental comparison
The supplied analysis describes appearance drift after occlusion and loss of thin structures; its visual-quality comparison is qualitative. We have no measurements of those effects, replay-memory overhead or their impact on task success. Probe views, trial rollouts and engine checks describe the authors' scene-refinement process, while fully automatic selection of solvable, useful worlds remains unresolved. The authored living-room extensions also supply no ground-truth reconstruction of unseen rooms. Limitations analysis, Scene construction
The supplied records contain no matched evaluation isolating the contributions of the renderer, pretrained agent, action interface and inherited notes, or a matched cost comparison with conventional rendering. Bitwise renderer replay supplies no policy-determinism guarantee, and a dependency audit supplies no security assessment of generated action programs. Rendered layout, assets, accessibility, related links and recommendation modules also fall outside this text-and-evidence review. The next useful experiment is a focused renderer integration using prepared geometry and the documented assets.


