mrkeyoor.com_
Mon 09 Nov 15:40 UTC
Open Source3 min read

Sem86 pinpoints a toy malware sample's CPU-based sandbox check

The research emulator swaps instruction rules to isolate an evasion check. Its controlled demonstration stops short of evidence against real malware.

Researchers Jos Craaijo, Freek Verbeek and Binoy Ravindran have used Sem86 to locate the CPU instruction that lets a toy malware sample detect an emulator. Their approach gives analysts a way to investigate a specific source of sandbox evasion: differences in CPU behavior that the x86 architecture leaves undefined.

The work was presented at QRS 2026, held July 22-25 in Florence. It is an existing research result, rather than a newly announced November release. The authors' evaluation uses a constructed sample, so it doesn't establish how often real malware uses this technique.

Change the instruction rules during execution

Sem86 loads descriptions of what CPU instructions do from an input file. Researchers can change those descriptions while a program runs, then search for the execution point where the outcome changes. The implementation paper describes that search as bisection. Saved machine states let it resume an experiment without booting the operating system again for every attempt.

That separation has limits. The execution loop, paging and system hardware remain hardcoded, according to the implementation paper. Loading instruction rules from a file doesn't make every part of the emulated computer configurable.

A multiplication instruction gives the sample away

In the experiment, the sample runs the IMUL multiplication instruction and inspects the zero flag afterward. Its value is undefined for this instruction. The authors compare a rule that updates the flag with one that leaves it alone, causing the sample to display different messages. Bisection identifies IMUL as the instruction responsible.

One message says the sample is deleting the C drive. The demonstration only displays a message box. For an analyst, the useful result is the link between a changed instruction rule and changed program behavior. Whether that method finds evasive checks in real malware still needs evidence beyond this constructed example.

A research tool with a clear next test

The AGPLv3 repository documents Linux and Android support and explicitly says Sem86 isn't ready for production use. Users supply their own operating-system disk image. MrKeyoor reviewed the paper and documentation; we did not build the emulator or reproduce the experiment.

The project's longer-term goal is to load instruction behavior inferred automatically by libLISA, allowing emulation of a particular CPU's quirks. Today's rules are handwritten, and the project says libLISA lacks the required observers for 16-bit and 32-bit x86. Watch for that integration and evaluations against real malware: those would extend the evidence beyond the controlled case reported here.

Sources

  1. Sem86 research paper: abstract and publication details
  2. Sem86 paper: evaluation
  3. Sem86 paper: implementation
  4. Sem86 source repository and usage documentation
  5. Sem86 project page