A default connection-pool policy taught OpenAI's storage service to send fresh work back toward its slowest Python workers. After a traffic burst, overloaded processes returned their connections later, so aiohttp's last-in, first-out reuse selected those connections again. Load kept collecting where the service was already struggling. OpenAI says a switch to first-in, first-out reuse broke the loop. For developers, the incident is a sharp warning: a harmless-looking pool policy can turn a brief slowdown into a failure that sustains itself. OpenAI described the incident in its Habitat engineering report.
That bug sits inside an unusually large application storage system. Habitat now handles more than 70 million requests per second, serves over 500 petabytes, and supports OpenAI products used by more than 1 billion people each week across almost 40 regions. The same report says the Python service peaked above 20 million requests per second. A Rust replacement now carries 95 percent of production requests after a rewrite by two engineers working with Codex and GPT-5.5. OpenAI reports sixfold CPU efficiency and fifteenfold memory efficiency for the new service. Those figures come from OpenAI's own measurements, so they should be read as operator-reported results rather than an independent benchmark.
The pool favored slow workers
Habitat sits between products such as ChatGPT, Codex, and the API and a collection of storage resources. It handles schema lookup, routing, authorization, encryption, serialization, request shaping, and connection management. One user action can cause hundreds of database calls, according to OpenAI's account of the request path. That makes the tail of the latency distribution more important than the average. A single slow lookup can become the delay a person notices.
Python's asyncio could overlap the service's network waits, but it could not make CPU work run in parallel under the interpreter's global lock. Habitat also compressed and encrypted data, calculated checksums, checked downstream health, and ran request shadowing. OpenAI found high-percentile requests whose database response had already arrived while the responsible coroutine waited to run again. By measuring the difference between scheduled and actual execution times, the team saw event-loop delays reach hundreds of milliseconds and, in some edge cases, several seconds. The report explains its event-loop measurement rather than treating CPU utilization as a sufficient health signal.
One source of synchronized delay was the feature-flag client. Statsig polled for a large configuration file every minute without jitter, and each pod ran as many as eight Python processes. All the workers could stop handling requests at roughly the same moment to parse the same JSON. The fix was plain: send a smaller configuration, refresh it less often, and spread refreshes over time. OpenAI says live CPU profiling exposed the pattern. The episode is useful because the expensive work was routine control-plane housekeeping, not a database query.
Connection reuse created the nastier failure. Some processes were handling five to ten times the average concurrency, and the degradation continued even after the bursty client stopped. The team capped connection-reuse duration, then traced the behavior to aiohttp's LIFO selection. OpenAI connected the pattern to a class of metastable failures documented years earlier at Facebook. In Meta's account of a related production failure, most-recently-used connections gradually concentrated queries on congested network paths until an external intervention changed the state.
OpenAI patched the pool to use FIFO, which distributed requests more evenly and stopped the feedback loop. Habitat now relies mostly on Istio and Envoy for pooling and server-aware balancing. Envoy also upgrades the Python service's HTTP/1 connections to HTTP/2, multiplexes traffic, and gives operators one place for circuit breakers and rate limits. OpenAI says this connection fan-in limits the number of downstream connections created by its many Python processes. That matters during deployments, when connection cycling alone can create heavy CPU load or saturate a network address translation device.
Python bought time for the interface
Habitat began in mid-2024 as a Python library connected to ChatGPT's main server and Azure Cosmos DB. Product teams used it without a central migration order because it hid recurring storage work behind a common client. Developers could add caching, compression, or encryption to the shared package. The model worked while the number of clients was small enough for coordinated upgrades. OpenAI dates that first library design to mid-2024 and says adoption spread while teams could still update together.
By the middle of 2025, client-side control had become an operational liability. A plan to move sensitive data sets across regional Cosmos DB accounts required new routing code, staged feature flags, and request shadowing across dozens of services. Each correction took days to propagate. One team later rolled back an unrelated service and restored a buggy Habitat client, creating the outage the rollout was meant to prevent. That sequence led OpenAI to extract Habitat into a service, with centralized deployments and observability.
The service boundary also became an enforcement point. Habitat can apply access policies, record audits, and restrict direct access to storage resources in one layer. OpenAI explicitly includes requests from internal systems and AI agents in the threat model described in its storage architecture. Central control makes policy changes faster, though it also means faults in that layer can reach many products. The team's decision was to accept that concentration and invest in isolation, routing, and request shaping at the shared boundary.
Keeping the first service in Python was deliberate technical debt. OpenAI says the team expected the language's CPU and memory costs to become unacceptable at one hundred times the initial scale, but an immediate rewrite would have delayed the service boundary and left the interface unsettled. The team chose to stabilize the API first. It also expected future coding models to reduce the work of moving languages. The report presents that sequence as a conscious trade, with Python buying roughly a year before the Rust migration.
A smaller API made the load predictable
Habitat does not expose arbitrary SQL. Clients define object types and directed edge types, then use a small NoSQL interface whose operations are intended to have bounded cost. Direct-edge lookups are supported; broad graph traversals are left to callers. OpenAI says the data model was inspired by Facebook's TAO. The restriction is part of the capacity plan because one application team cannot slip an unbounded table scan or join into a hot request path.
That choice came from experience with Postgres. When OpenAI's product and engineering organization was smaller, reviewers could inspect query and schema changes before release. Growth made that process hard to sustain, and the report says a single expensive query on a busy path could take down the database. Habitat moves the cost into the API design: complex joins require extra work in the client, where their expense is easier to see. OpenAI's explanation of the limited query surface frames developer inconvenience as protection for shared online capacity.
The objects and their outgoing edges are stored together in a partition, but the objects at the far end of those edges may live in another Cosmos DB account or region. Multi-hop traversal is therefore inefficient by design. Teams that need search or analytical queries receive a separate Rockset view fed through change data capture, and each team scales its own Rockset instance. The architecture isolates those read-heavy jobs from the storage path used to render live products. It also gives product engineers a visible cost for choosing a more flexible query.
The Rust numbers need their denominator
In the second quarter of 2026, two engineers used Codex and GPT-5.5 while rewriting the full Habitat service in Rust. OpenAI says the replacement is six times as CPU-efficient, fifteen times as memory-efficient, and faster at both average and tail latency. It now handles 95 percent of production requests, with Python due to be retired in the following weeks. The migration figures appear in a short section of the report; OpenAI does not publish the code, total core reduction, dollar savings, or a workload-normalized benchmark there.
The 70-million-request headline also needs careful parsing. OpenAI assigns that rate to the Habitat platform, which includes the storage layer and supporting components, while it says the Python service itself peaked above 20 million requests per second. The post does not say that one Rust process, one cluster, or the rewritten service alone handles all 70 million. OpenAI separates the platform total from the service migration. That distinction keeps an architecture number from turning into a language shootout.
Nor does a two-engineer rewrite establish how much labor the coding models saved. OpenAI identifies Codex and GPT-5.5 as tools used in the project and says its earlier wager on coding models worked, but supplies no control project, estimate of generated code, defect rate, or review hours. The engineering post supports the tool attribution, not a general productivity multiplier. The stronger evidence is operational: the Rust version is already serving most traffic, and OpenAI reports large resource-efficiency gains under its own workload.
The next evidence should fill in what this first installment leaves open. OpenAI says a follow-up will cover multi-tenant reliability, layered read optimization, and its work with Azure Cosmos DB. It also plans a separate account of lessons from the Rust migration. Those promised reports can show whether the measured efficiency survives the last five percent of traffic, what reliability costs accompany 500 petabytes, and how much of the rewrite process is repeatable outside OpenAI. Until those numbers arrive, Habitat's most transferable lesson is smaller: event-loop delay and connection-pool ordering deserve first-class production metrics.