mrkeyoor.com_
Sat 26 Sept 21:07 UTC
Tech7 min read

Loongson's LA664 Loses Atomic Updates, Breaking Safe Rust

A CPU erratum can drop atomic updates on Loongson's LA664 core. Tests reached 100% failure in one setup and turned safe Rust into heap corruption.

A safe Rust program using Arc can end in heap corruption on Loongson's 3A6000 and 3C6000/S processors even when its source follows Rust's rules. The failure sits below the language and compiler: under a specific mix of cross-core atomic operations and memory reads, the LA664 CPU core can lose an update that the instruction promises to preserve. In one controlled setup, the fault appeared in every trial. The investigators' report turns an obscure Debian packaging timeout into a warning for anyone trusting reference counts on affected machines.

A counter that could never reach its target

The first visible symptom was dull enough to ignore. In February 2026, Debian's community-maintained LoongArch port could not package Normaliz, a mathematical program, because one of its tests ran forever. Normaliz processed a list in parallel and incremented two counters with OpenMP atomic operations. Every item had been marked as processed, yet one counter remained below the number of items, so the exit condition could never become true. The package was skipped while maintainers looked for a tractable explanation, according to Jiajie Chen's account of the investigation.

The code was behaving as if two threads had read the same counter and one write had overwritten the other. That is the lost-update race an atomic read-modify-write instruction exists to prevent. Disassembly showed that the compiler had emitted LoongArch's amadd.d instruction as expected. Control counters built with C++ atomics drifted too, and the amount changed between runs. The team had moved below OpenMP and Normaliz, but a simple stress test of atomic addition still would not reproduce the failure.

Six months later, the missing ingredient turned out to be an ordinary memcpy. Glibc selects a vectorized implementation when the processor supports LASX, LoongArch's 256-bit SIMD extension. The copy's xvld loads, interleaved with barrier-free atomic operations on another address, made the failure reproducible. Loongson's toolchain conventions identify lasx as the 256-bit extension and list LA664 as a supported tuning target. That connection matters because source code does not need to contain an explicit vector intrinsic. A routine library copy can supply the trigger.

The failure needs three conditions

The researchers narrowed the erratum to the LA664 core used in the 3A6000 and 3C6000/S. They did not reproduce it on the earlier LA464 core found in the 3A5000. Their tests required threads on different physical cores, barrier-free atomic operations against the same address, and memory reads interleaved with those atomic operations. Two simultaneous threads on the same physical core were outside the reproduced condition. Those boundaries come from experiments, not from a public microarchitectural explanation from Loongson.

LASX vector loads were the reliable trigger. A later test found that scalar reads could also provoke the fault when the read address and atomic address had a particular positional relationship, though at a lower probability. That makes disabling LASX an incomplete safety measure. It explains one confusing observation as well: AOSC OS stopped reproducing the bug after its Core 13 release accidentally disabled glibc's multi-architecture build, sending memcpy away from the LASX path. Debian still used the accelerated path and still failed.

For the published benchmark, the team used two physical cores on a 3C6000/S, copied 2,208 bytes per point, ran 200 rounds per trial, and repeated each case 30 times. The table reports the share of trials with at least one lost operation:

Atomic instruction Both threads use LASX copy Both use LASX reads One uses LASX copy
amadd.d 67% 100% 53%
amadd.w 73% 100% 67%
amcas.d 77% 97% 17%
amcas_db.d 0% 0% 0%
ammax.d 43% 100% 50%
amswap.d 53% 100% 53%

The data-barrier form of compare-and-swap, amcas_db.d, had no failures across those 30-trial sets. That result supports the proposed compiler workaround, but it does not prove that every data-barrier atomic is immune under every workload. The firmware test is broader evidence for the affected machines because it addresses the processor behavior without asking each application to change its generated instructions.

Safe Rust makes the cost visible

Reference counting turns a missed increment into more than a wrong metric. If two threads acquire references and the CPU drops one increment, the stored count falls below the number of live owners. A later decrement can reach zero early and free an object that another thread still holds. The investigation produced safe Rust programs using Arc and cloned channel senders that crashed with SIGABRT or glibc heap-corruption reports. The researchers traced the result to use-after-free.

Rust's documented contract makes the mismatch plain. Arc::clone creates another pointer to the same allocation and increases the strong reference count. The standard library's channel counter performs this relaxed increment when a sender is cloned:

let count = self.counter().senders.fetch_add(1, Ordering::Relaxed);

That line comes directly from Rust's std::sync::mpmc counter source. Relaxed weakens ordering around other memory operations, but the increment itself must still be atomic. On the affected LA664 behavior, the compiler maps such operations to a barrier-free atomic instruction that can lose an update under the reproduced conditions. The source has no unsafe block to audit away, and changing application-level ordering everywhere would misplace responsibility for a CPU erratum.

The report does not describe an easy cross-process attack. Triggering the failure requires threads inside the same process to contend on the same object while the relevant reads occur, so process isolation keeps a remote party from simply racing a victim's counter. That still leaves a correctness problem. A low-rate hardware fault can surface as a hang. Premature frees and heap damage may appear far from the instruction that failed, leaving logs to implicate the allocator or application after the counter went wrong.

AI helped only after humans supplied the hard clue

The report also records the AI coding assistant's limits. On its first pass, the model noticed the infinite loop but did not blame the atomic instruction. Even after being told that the problem appeared only on LoongArch, it reached no definite cause. The researchers then supplied their strongest prior finding, that atomic addition was involved, and asked for a minimal reproducer. The model focused on memcpy, whose optimized implementation lived outside the visible Normaliz code, and produced a stable small test about two days later.

That contribution shortened the search after people had already isolated the suspicious operation and constrained the architecture. Humans expanded the test across compare-and-swap, maximum, swap, vector loads, and scalar reads. They measured the failure rates, reported the erratum, and checked the proposed firmware. The model was useful as a minimizer once the search had a firm boundary. Its result still needed controlled tests and a vendor fix before the diagnosis was settled.

The fix has to reach each motherboard

The researchers reported the issue to Loongson on August 26 and received test firmware on September 9. On both the 3A6000 and 3C6000/S, their reproducer stopped failing after the update. The fix sets bit 13 of an undocumented internal register called MCSR24. Their performance checks found no single-core loss and only a small multi-core drop, though the report does not publish a full benchmark suite or percentage for that cost.

Loongson told the researchers that public firmware was expected before October 1. As of the report's September 24 publication, that was a schedule rather than confirmation that every board image was available. Loongson's firmware repository instructions tell users to match an image to the machine or motherboard and verify it against the published SHA-256 file before updating. Operators should wait for an image and release note that explicitly cover their hardware instead of assuming any newer firmware contains this fix.

A compiler can avoid the faulty path by emitting atomic instructions with a data barrier or by using load-linked/store-conditional loops. Existing binaries would need rebuilding, which makes that route expensive for a distribution. Setting MCSR24 bit 13 from the Linux kernel produces the same effect as the tested firmware, according to the investigators, but an undocumented control bit is poor material for an improvised production tweak. Disabling LASX through GLIBC_TUNABLES=glibc.cpu.hwcaps=-LASX stopped the high-probability trigger in the Normaliz case. The scalar-read result means it cannot close the issue.

The next evidence should be mundane and public: firmware images for affected 3A6000 and 3C6000/S boards, checksums, release notes naming the erratum, and distribution guidance for systems that cannot update promptly. Until those pieces arrive, the Normaliz timeout is the wrong health check. Affected operators need the minimal reproducer, run across physical cores, because an atomic instruction can fail without leaving a helpful signature anywhere else.

We reviewed this

  1. atomic — our honest review
  2. vector — our honest review
  3. register — our honest review

Sources

  1. One CPU Atomic Instruction, One Packaging Infinite Loop: The Story of the Lost Update on LA664
  2. LoongArch Toolchain Conventions
  3. Rust Arc documentation
  4. Rust standard library mpmc counter source
  5. Loongson Firmware update instructions