Should you use OSWorld?
Use OSWorld when desktop interaction is the capability you need to evaluate and your team can own the environment setup and result review. Its README documents desktop providers, agent execution, recorded trajectories and public evaluation. Our sandbox results help estimate installation friction; they do not establish whether an agent can complete its benchmark tasks.
OSWorld saves screenshots, actions, and video recordings of agent task completion. Those artifacts give researchers something to inspect alongside a score. The README also provides a manual examination workflow for checking individual tasks and evaluation metrics.
What our dated sandbox run established
Our run on September 28, 2026, used commit b138d34 (full SHA: b138d348256078fa634fc3b73567a7337c793e6b). This correction uses that stored run and introduces no new execution.
Our sandbox recorded a dependency footprint of 304 packages and 6,911 MB. The harness stores each command step's status, exit code, elapsed time and output tail, together with parsed test counts. Its installation step records exit 0 and 122 seconds, while the retained output reports an agp-client build error because the local directory lacked pyproject.toml and setup.py. Read both parts of the record: the successful step status alone cannot establish that every component installed correctly.
The build step returned exit 0 in 5 seconds and retained SyntaxWarning messages about invalid escape sequences. That outcome does not establish that a desktop provider or agent workload ran successfully.
Pytest returned exit 1 in 19 seconds. The attached stored parser summaries report 8 passed, 0 failed, 10 errors, 0 skipped and 18 total. They are not verified counts of exhaustive unique tests. The raw footer separately reports 8 passed, 10 errors and 4 passing subtests in 10.19 seconds. The footer duration and harness step duration describe different recorded timings; the passing subtests should not be added to the stored aggregate.
Seven named debug-system modules reported a missing DISPLAY variable. The Modal smoke test reported a missing optional modal dependency, and the Maestro provider test reported a missing zhipuai module. The footer also names a FastVM smoke-test error, but the supplied tail does not show its cause. These errors do not share one demonstrated explanation.
The supplied record does not establish a desktop-image launch, an agent evaluation, completed benchmark tasks or a model score. The MrKeyoor testing methodology explains the publisher's command-status and measurement conventions. That page is provenance context, not evidence that this September run used today's procedure or completed an additional workload.
Plan for provider and task configuration
The README pinned to our measured commit documents VMware and VirtualBox, Docker with recommended KVM support, and Modal, Daytona and AWS paths. Daytona's documented approach uses the sandbox itself as the desktop and does not require host KVM or the Docker qcow2 flow. Select the provider before treating package installation as the main setup task.
The baseline example requires a model API key. Some tasks require Google account access and OAuth configuration; others may require proxy settings. The README warns that missing configuration can prevent corresponding tasks from executing correctly and lower evaluation scores. Check the requirements of the tasks you intend to report.
For verified leaderboard results, the README describes a maintainer-run evaluation with disclosed implementation evidence, or monitoring data and trajectories from a trusted institution. Public verification therefore requires more than producing a local score.
Review grading assumptions before publishing scores
Issue 567 reports that command-based graders obtain ground truth from inside the agent-controlled VM. Its demonstration targets commit 091f5ef1d5544bc74953c77875d5feb5bed30108, rather than our measured commit. The report also describes apostrophe-containing output crashing a JSON parsing path and crafted text passing a check incorrectly. These are the issue author's reported demonstrations, not findings reproduced in our sandbox, and the supplied evidence does not establish their status at b138d34.
Before publishing a comparison, record the commit, provider, image, model and task configuration. Preserve trajectories and account for attempted tasks, errors and missing results separately. Inspect the graders relevant to your evaluation rather than treating an aggregate score as sufficient evidence.
Pip-audit's 111 recorded advisories are a separate observation. The audit summary associates them with 7 packages, but supplies no scannedPackages value. Scan coverage is unknown, and neither the advisory total nor the grader report establishes the overall security of OSWorld.
Choose alternatives by the environment you need
BrowserGym is a framework for web agent research. Its README documents Chromium setup and additional setup for individual benchmarks. Choose it when browser interaction answers your research question; it does not replace evaluation of desktop applications outside the browser.
Windows Agent Arena provides a Windows environment for testing desktop agents. Choose it when Windows behavior is the target. Its documented setup includes Docker and preparation of a Windows VM, so choosing a narrower platform does not remove environment work.
These can replace OSWorld for a narrower evaluation or complement it in a study covering several environments. Their supplied documentation does not establish a direct score conversion or equivalent task coverage.
How to read the ratings
These are editorial assessments, not benchmark results. The scale is 1 for substantial barriers, 2 for limited support, 3 for usable support with material caveats, 4 for substantial documented support or participation, and 5 for comprehensive support demonstrated by the supplied evidence. Each dimension uses the evidence named in its note. Community measures visible participation here, not adoption; maturity measures evaluation readiness, not repository age.
Setup ease is 1 because the large dependency footprint, conflicting installation status and raw error, pytest collection errors and provider requirements leave substantial work. Documentation is 4 because the README covers several provider paths, task configuration, manual examination and public evaluation. Community is 4 because the README identifies participating institutions and contributors, while issue 567 supplies a detailed external report; this does not measure user numbers or maintainer response speed. Maturity is 3 because the documented evaluation workflow is usable research infrastructure, but our run did not validate the desktop workload and the reported grading assumptions need review.