OpenAI explains how Habitat — its online storage platform — grew from a small Python client library into a centralized service that today supports more than 1 billion weekly users. Habitat handles over 70 million requests per second, serves over 500 petabytes of data, and operates across nearly 40 geographic regions.
Origin: a small Python client library
Habitat began in mid-2024 as a lightweight Python client used by ChatGPT’s main server. Its purpose was to spare product engineers from database management details: the library handled routing, authorization, encryption, serialization, connection pooling and translated product operations to Azure Cosmos DB under the hood.
Why Habitat became a service
By mid-2025 the client-side approach hit limits. Backward-compatible protocol changes and feature rollouts required coordinated deployments across dozens of services, which took days and was brittle. One migration to regionally distributed Azure Cosmos DB accounts — implemented with a client-side feature flag and shadowing — showed how fragile the process could be when teams rolled back independently and caused the outage the change aimed to avoid. To reduce that operational fan-out, the team pulled Habitat into its own centralized service, enabling single-point control for deployments, observability, and platform improvements.
Centralization also improves security
Running Habitat as a centralized service provides a choke point to enforce access control policies, perform audit logging, and limit direct access to underlying storage like Azure Cosmos DB. This centralization is an important part of protecting user data against external, internal, and agent threats.
Why they stayed on Python initially
Moving to a service introduced the choice of language. Running a high-throughput service in Python increases network latency and raises CPU and memory scaling costs versus a local library, and a full migration away from Python was expected to be necessary eventually. Nevertheless, the team accepted Python as a strategic short-term technical debt to unblock product teams and stabilize the platform quickly. They also bet that improved internal coding models (Codex, GPT) would simplify a future rewrite — a bet that later proved correct.
Python-specific challenges: asyncio and tail latency
At this scale the main operational challenge with Python was managing tail latencies caused by asyncio scheduling delays. While asyncio enables concurrent I/O, Python’s GIL prevents true CPU parallelism, and Habitat performed many CPU-heavy tasks (routing, compression, encryption, checksumming, downstream health checks, shadowing, hedging). During high utilization, expensive background tasks and CPU work caused coroutines to be rescheduled late, producing p99 and worse latencies. To detect this, the team measured event loop scheduling delay by recording the difference between expected and actual execution times of periodic background tasks.
Their mitigation strategy was to keep each process serving only a small number of concurrent requests and massively scale out the number of Python worker processes.
Concrete tuning: Statsig configuration parsing
Live CPU profiling identified a root cause of high asyncio delay: periodic JSON parsing of Statsig feature flag configurations. Statsig polled every minute (no jitter) and the config included all production rules for every service. Running up to 8 Python processes per pod meant that once a minute each process would stall to parse a very large configuration, delaying in-flight requests.
The fix was to deploy smaller, targeted configs, increase the refresh interval, and add jitter to background tasks.
Load balancing and connection pooling
Good load balancing across server processes proved critical. Client-side connection pooling concentrated load onto a few server processes: a client with many concurrent requests could reuse a small set of connections and therefore hit only a few workers. This led to some processes serving 5–10× more concurrent requests than average and to metastable failure modes where overloaded pods attracted more traffic and degraded further.
They traced the issue to aiohttp’s TCPConnector default behavior: LIFO connection reuse. During bursts, connections returned from slower servers were selected more often, concentrating traffic on those slower servers. Switching the client-side pool to FIFO reuse broke that feedback loop and reduced request variance. Ultimately, Istio and Envoy are used across OpenAI to provide connection pooling and server-aware balancing that avoids the issue.
Envoy, HTTP/2 multiplexing and downstream protection
Having many Python processes also increased the risk of overwhelming downstream resources (a “thundering herd”): frequent deployments or connection churn could cause CPU spikes or saturate NAT gateways. The team relies on Envoy to upgrade HTTP/1 to HTTP/2 for multiplexing, extend connection lifetimes, and pool connections. Envoy also provides a centralized place for rate limits and circuit breakers that would be less effective inside each Python process.
API design: predictable NoSQL requests
A key reason Python could be scaled as far as it was is Habitat’s constrained API, which keeps request cost predictable. Instead of allowing arbitrary SQL queries that could cause expensive table scans or joins, Habitat exposes a simple NoSQL API modeled around client-defined object and edge types (inspired by TAO). Clients predefine object and edge types; Habitat colocates an object with its edges in a partition to make simple operations efficient.
Complex graph traversals are intentionally inefficient in this model because hops may require fetching from different Azure Cosmos DB accounts in different regions. For teams that need richer querying, Habitat provides an offline secondary view via Rockset: change data capture streams online updates to isolated Rockset instances that each client team must provision and scale themselves. This adds friction, but isolates the online system from read-heavy analytics and search workloads.
Moving off Python: rewrite in Rust
Delaying a full rewrite allowed the team to focus on urgent issues during hypergrowth. Habitat became the second largest service at OpenAI by core count and fourth by Envoy footprint; Python at peak served over 20 million requests per second. In Q2 2026, two engineers, aided by Codex and GPT‑5.5, rewrote the entire service in Rust. The new Rust service now handles 95% of production requests and the Python implementation will be deprecated in the coming weeks. Telemetry shows the Rust service is 6× more CPU efficient and 15× more memory efficient than Python, with substantially lower average and tail latencies.
Next: part two of the series
This article is the first of two. The second installment will detail multi-tenancy reliability at scale, layered read-optimization strategies, and how the Azure Cosmos DB partnership was scaled to meet unprecedented demand.
Careers
OpenAI’s Habitat team is hiring engineers interested in building OLTP systems at frontier scale.
