OSWorld measures agents inside resettable desktop machines
OSWorld evaluates agents on tasks carried out in real desktop applications. An agent observes screenshots or other configured inputs, sends mouse and keyboard actions, and works inside a virtual machine. The environment records screenshots, actions, and videos, then task-specific graders decide whether the requested state was reached. This makes it useful for studying failures that a text-only benchmark never sees.
The benchmark covers ordinary computer work across browsers, office applications, media tools, and operating-system settings. OSWorld-Verified is the current result track described in the README, and the maintainers ask users of the latest environment to compare against that updated track. Public verification may involve a scheduled run, disclosure of agent implementation under the framework, or shared monitoring data and trajectories.
The VM provider is part of every reported result
A local desktop or bare-metal machine can use VMware or VirtualBox. A server path can use Docker with KVM, while Modal, Daytona, and AWS have separate integration guides. These are not interchangeable details. Virtualization support, image state, network location, display behavior, and provider timing can change whether an agent sees or completes a task.
Some tasks need Google account access and OAuth configuration. Others may need a proxy because websites react differently to the evaluation network. The baseline example also needs a model API key. Keep secrets outside reusable VM images and inject them for the run, as the README advises for host-side credentials. A benchmark machine gives the agent broad control, so use dedicated accounts with minimal data.
What happened when we ran it
Our sandbox cloned commit 84aee65 and installed 308 Python packages in 111 seconds. Dependencies occupied 6,915 MB, which is unusually large for a 20.3 MB checkout. The repository had 1,190 files and about 189,390 source lines. Pip-audit reported 81 known vulnerabilities in the resolved environment.
The build completed successfully in 8 seconds. Pytest then exited 1 after 24 seconds. It reported 8 passed, 0 failed, and 10 collection or setup errors out of 18 discovered items, plus 4 passing subtests. A run with collection errors is a failed test step even though none of the executed assertions failed.
Seven debug-system modules stopped because the DISPLAY environment variable was absent. Errors also came from a FastVM smoke test, a Modal smoke test, and a Maestro provider test. The log does not show one common cause for all 10 errors. Our unprivileged headless Debian container was not a configured desktop benchmark host, and the test layout did not isolate those environment-dependent modules from generic collection.
The repository had no CI workflow files and no Dockerfile in our scan, though the README documents Docker as a provider for running virtual machines. A tests directory was present. Our run did not download a desktop image, start an agent, execute benchmark tasks, or calculate a model score. It only established the package, build, audit, and pytest outcomes above.
Current grader reports can change how scores are interpreted
Open issue 567 argues that some graders obtain ground truth by running commands through a server inside the same VM the agent controls. The report demonstrates how altered in-VM output could produce a passing score without the intended machine-state change. It also describes a JSON parsing path where an apostrophe can crash evaluation or crafted text can pass incorrectly. This is a current report, not a finding from our sandbox.
Issues 568 and 570 describe a simpler scoring problem: missing VLC configuration or an unavailable desktop shortcut file can crash a grader instead of recording a zero. The reports say crashed tasks can disappear from result output. Anyone publishing comparisons should pin the OSWorld commit, save raw trajectories, count attempted tasks independently, and inspect missing results rather than trusting one aggregate number.
Three thousand stars do not make results automatically comparable
GitHub recorded 3,107 stars, 197 combined issues and pull requests, and a last push on August 21, 2026. The latest tagged release in GitHub is v0.1.16 from June 26, 2024, while the repository and README have continued to change through 2026. The old tag alone is not evidence of abandonment because current pushes and issue activity are visible.
For internal research, OSWorld is valuable because it provides a common environment and detailed artifacts. For public claims, the environment is only one part of the method. Report the exact commit, provider, image, model, observation type, action space, credentials, proxy setup, failed or missing tasks, and any grader patches. Without that context, two scores can describe different experiments.
Use it for desktop research, not routine application QA
OSWorld makes sense when an agent must operate across applications and the research team can maintain virtual machines as lab equipment. The 8-second build is the easy part. A 6,915 MB dependency environment, desktop images, model calls, external accounts, and task-specific graders make each serious run an experiment that needs records and review.
A browser-only agent should start with BrowserGym or another web-focused suite. Windows-only and Android-only research have narrower alternatives too. OSWorld earns the extra setup when cross-application desktop work is itself the capability under test, and when researchers inspect the trajectory instead of treating the final score as self-explanatory.