Oxide's new Voxel lab runs its test workloads as operating-system zones because its virtual server sleds can't nest another hypervisor. Developers can examine a rack's networking and control services on one host, but ordinary guest VMs remain outside that test environment. The distinction is explicit in the operator guide.
In an October 14 technical post, Oxide product assurance engineer Steve Karam describes Voxel, short for Virtual Oxide Labs, as a way to test the company's rack software without waiting for scarce physical equipment. The project is available under the Mozilla Public License 2.0. Its audience includes developers working on Oxide's stack and operators who want to study how the pieces behave together.
Karam reports running six virtual sleds on a Minisforum MS-A2 with a 32-thread Ryzen 9 9955HX and 96GB of RAM, arranged as one rack or multiple racks. The roughly ten-minute launch in his account is a result from that setup, rather than a benchmark for other machines. He also describes emulating the firmware that manages the servers.
A rack has more than server CPUs
On physical Oxide hardware, a server sled's main processor runs customer workloads and control services. A separate service processor handles chassis functions, including thermal control, and provides interfaces for power management and software upgrades. Oxide's rack architecture documentation says these service processors work alongside a root of trust. Each board type has its own firmware image.
Networking adds another dependency. A production rack contains two Sidecar switches, each with a Tofino switching chip, and every sled connects to both. Adjacent server sleds control the switches through PCIe connections. Those details in the hardware guide explain why a collection of server VMs alone would leave substantial parts of rack operation untested.
Voxel assembles its virtual rack with Omicron, Oxide's control-plane software, running across Propolis VMs. Falcon creates the network topology, SoftNPU supplies software switches, and FRR supplies routers. The repository lets users select sled counts and multiple racks, with BGP or static routing. That gives a networking change somewhere to run before it reaches physical equipment.
Omicron's component guide shows what needs to agree during that process. Nexus exposes the APIs, while a per-server sled agent manages local resources. A bootstrap agent establishes trust during initialization. The management gateway bridges the rest of the control plane to the service processors. A failure between those services can matter even when each program passes its own tests.
Where firmware emulation changes the test
Voxel normally uses a simulated service processor. Running actual service-processor and root-of-trust firmware requires its optional sp-emu backend, enabled with voxel launch --emu. The Voxel README says firmware can come from the control-plane image's release artifacts, with explicit overrides available for testing different images. A default launch and a firmware-emulation run therefore exercise different implementations beneath the management interface.
The sp-emu project is a Rust interpreter for the Cortex-M7 and Cortex-M33 processors used by the relevant controllers. It boots unmodified Hubris firmware and models the peripherals that firmware touches. The service processor and root of trust can communicate through an emulated SPI link, and the root of trust reports measurements of the flashed image. These are behaviors the firmware executes inside the emulator.
Debugging also reaches into that running firmware. According to the sp-emu README, Oxide's Humility debugger can inspect task state and stack traces, halt execution, and step through it. Timing has a firm limit: peripheral accesses don't consume the clock cycles they would on silicon. An emulated boot can reveal a software interaction without establishing that the same operation meets a hardware timing requirement.
The firmware itself has a separate development workflow. Hubris's documentation describes application-specific images built with its Cargo xtask commands, rather than an ordinary cargo build. It also warns that linker differences can produce different images across build platforms. For someone investigating a firmware regression, recording the image and how it was built belongs alongside the virtual rack configuration.
Choose the amount of simulation the question needs
A zone provides an isolated environment for applications, with restrictions on which processes and devices they can access. The illumos manual also describes separate network port spaces and configurations with independent IP stacks. That makes a zone useful as a source or destination for network tests. It doesn't supply the additional guest-machine execution that Voxel's already-virtualized sleds would need to test a normal VM workload.
Oxide already offers a lighter way to develop control services. Its simulated Omicron guide provides a substitute sled agent that runs on Linux, macOS and illumos. Developers can exercise control-plane flows without real resources to manage. The guide directs anyone needing actual instance, storage and network management to the full stack.
That distinction helps decide whether Voxel's host requirements are justified. A change to an API flow may be testable with the simulated agent, which the Omicron guide pairs with services including Nexus and its databases. Questions involving the rack's network layout or firmware interactions call for more of the system. More emulation is useful when it brings the suspected failure into the test.
Operators also have telemetry to inspect while a test runs. Oxide's control-plane documentation describes Oximeter collecting measurements exposed by software producers and storing them in ClickHouse. The data can describe service behavior as well as hardware-related measurements. In a virtual lab, a developer still needs to identify what produced a measurement before treating it as evidence about a physical device.
The underlying VM machinery has its own constraints. Propolis is the userspace virtual-machine monitor for illumos bhyve and requires recent illumos components because of its close connection to the kernel interface. It works best on AMD hosts, where its CI tests run, though Intel is also supported. Its dependency on illumos ties this part of the lab to the host operating system.
Budget for the host and the configuration
The operator guide requires bare-metal Helios on x86_64, at least 64GB of memory and 500GiB of flash storage. It recommends more memory for larger topologies. Helios is Oxide's illumos distribution, with prebuilt installation media available through the linked engineering-environment project. A reader can use those images without building the operating system from source.
The defaults in Voxel's configuration reference allocate 8GB to each of four sleds and 4GB to each of three routers. That's 44GB of guest RAM before host overhead. These are separate nodes with adjustable resources. Increasing the topology means budgeting memory for the additional machines, even though the lab occupies one physical computer.
External networking deserves a deliberate choice too. The parameter reference defaults to attaching external node interfaces to the host's LAN. Its isolated mode instead constructs a host-side network segment with NAT through a physical uplink. The operator guide explicitly warns against putting a LAN-mode deployment on an untrusted or public network.
Image compatibility is another setup constraint: the operator guide requires the control-plane image to match the Omicron revision pinned by the Voxel checkout. It also documents an extra step when using release artifacts: the sled agent built for production switches needs a matching development package to run the SoftNPU switch role. Successful image creation alone doesn't establish a usable switching configuration.
Oxide's announcement describes continuing work on Voxel's configuration and management tools. Watch for reproducible failures tied to a specific software revision, firmware image and topology. Those reports would let another developer check what the lab captured. A passing zone probe still leaves guest execution and chip timing untested.