OpenAI Habitat Part II: How ChatGPT Storage Runs on Azure Cosmos DB at 20M QPS

OpenAI has published Part II of its Habitat engineering series, and it is the clearest look yet at how the database layer under ChatGPT actually works. The headline numbers from the OpenAI post: Habitat answers in milliseconds for over 1 billion users across 40 geographic regions, holds more than 500 petabytes, and sits on an Azure Cosmos DB account that OpenAI says stores tens of petabytes and serves more than 20 million queries per second. The page description credits the work with "12x faster reads for ChatGPT."

The most useful part is not the scale. It is the story of how OpenAI went from one shared container, to many isolated ones, and back to one shared container, and why. If you run any multi-tenant system, that circle is the lesson.

This follows Part I, where OpenAI rewrote Habitat's serving layer. Part II goes one level down.

TL;DR: what OpenAI says in Part II

What is Habitat, and where does Cosmos DB fit?

Habitat is described as the online storage platform that lets OpenAI products "quickly and reliably access needed information." Clients such as ChatGPT, the API, Codex and internal services talk to Habitat. Habitat handles caching, authorization, data residency and placement, encryption, multi-tenant isolation, rate limiting and routing. Below it sit storage resources: Azure Cosmos DB and a second system called Nanobase for online storage, Valkey for caches, blob storage, and change data capture feeds into Databricks, Rockset and Kafka.

Cosmos DB is the main online store. The mechanics matter for what follows, and OpenAI gives a short primer. An account defines containers and a replication policy. Each container is provisioned for throughput in request units per second (RU/s). Data is sharded by a logical partition key, with documented per-key limits (10K RU/s), and Cosmos DB maps logical keys onto physical partitions, splitting them when storage or throughput demands it. Microsoft's own partitioning documentation covers the same model.

Phase I: one default container and three problems

OpenAI first let any client load through and relied on the per-key limit as back pressure, with a single default container per account. Three problems showed up:

• Logical partition key collisions. Early API choices exposed partition key selection to clients, so different products used the same popular keys such as user_id or workspace_id. They then shared the same per-key limits, which capped throughput for a given user.
• Noisy neighbors. The limit is really enforced per physical partition, so a logical key never reaches 10K RU/s in practice; it shares with whatever else landed on the same physical partition. OpenAI says workloads for niche features disrupted core login and signup flows.
• Weak partitions. A container's provisioned throughput is split roughly evenly across physical partitions. When data grows, partitions split, but total throughput does not, so each partition's share shrinks. For blob-like, storage-heavy use cases, growth alone cut the capacity available to active workloads while request rates stayed flat. Think 10K RU/s becoming two partitions at 5K each.

OpenAI also notes this was a familiar pain from its earlier Postgres days: one hot organization or key could take out a whole database, and teams spent significant effort caching hot keys and adding leases on cache misses so a miss storm would not flatten the database.

Phase II: isolation helped, then hurt

The fix was separate containers for the most critical new use cases, plus a script that kept each container's provisioned throughput in line with its physical partition count. That "mostly avoided" the early problems and, OpenAI says, saved it from many site-wide outages by limiting blast radius to feature or service degradation.

But legacy critical workloads were stuck in the shared default container without a data migration, so teams were told to create separate containers for anything new. Overhead piled up:

• No stable aggregate growth. In a big shared pool, one new feature going from 0 to 1 million QPS barely moves the total. On a new dedicated container, the same jump could be 1,000x the dogfooding baseline. OpenAI's own illustration: a 1K RU/s load test uses 0.005% of a 100M RU/s shared container's headroom and 5% at 1M RU/s launch, but against a fresh 10K RU/s dedicated container the launch load is 10,000% of capacity.
• Launch-day emergencies. Product teams shipped and forgot to pre-scale, and infrastructure engineers "sat in SEV rooms on launch day attempting to scale up containers."
• Capacity planning by everyone. Scaling dedicated containers correctly required product knowledge of future traffic, so essentially every product team had to take part, with wildly different diligence.

This is the classic single-tenant versus multi-tenant trade. Single-tenancy gives clean ownership and small blast radius but forces every team to plan capacity and over-provision. Multi-tenancy centralizes expertise but invites noisy neighbors. OpenAI ended up choosing multi-tenancy, but only after getting the platform to fix its failure modes.

Phase III: shared by default, with three fixes from Azure

Working with the Azure team, OpenAI moved back to one shared container using:

1. Hot key throttling. Throttle the hottest keys first so one tenant cannot drain a physical partition and cause a site-wide blast radius.
2. Request-tiering QoS. Offline work such as bulk inserts and extracts gets lower priority and is throttled first, protecting interactive, user-facing requests.
3. A better serverless mode. Cosmos DB's existing serverless mode would have solved weak partitions but scaled on storage only and capped each physical partition at 5K RU/s. The Azure team built a variant for OpenAI where every physical partition can scale to 10K RU/s and partitions also split on RU usage, not just storage. That ended manual monitoring and scaling.

The result, per OpenAI: one shared container, no upfront capacity planning for product teams, and "no provisioning or noisy neighbor incidents in the months since, despite accelerating product development." It adds that it is still early in its multi-tenancy story and has deferred investment by leaning on Cosmos DB primitives. Treat the no-incident claim as OpenAI's own self-report.

Two outages and the case for co-managed services

The post is unusually candid about reliability. At OpenAI's scale, it says, "managed services become less managed."

• Late 2024 power outage. A regional power outage hit the primary Azure region of one of its largest accounts. During failover OpenAI learned its largest accounts would take several hours to fail over to a secondary region, a path neither team had tested at that size.
• The failback outage, about a month later. Failing back triggered a connection stampede that overwhelmed the original region's Cosmos DB gateway. A quirk in how the gateway-mode client handled failovers made every client immediately connect to the original region, even though only some physical partitions had moved back.

Both teams ran incident processes and fixed root causes, but OpenAI's conclusion is organizational: the outages were symptoms of how the two teams operated and tested. Now Habitat and Azure engineers jointly monitor every layer from OpenAI's service tier down to Cosmos DB's physical storage, test changes against a production-sized non-production account, iterate on SDKs together, and can page each other directly without going through support tiers. Joint work over the past year added load protection and gateway metadata caching on Azure's side, and connection pooling and reuse, workload colocation and VPC isolation of critical versus offline workloads on OpenAI's side.

For context on how outages ripple to users, see our coverage of the ChatGPT and Codex outage in September and the August 31 ChatGPT Work outage.

Making reads fast across 40 regions

A stack of memory modules beside a server block, standing in for the cache layers in OpenAI Habitat's read path

By 2025 the system was stable and latency became the constraint. Compute and storage often sat in different regions, so cross-region requests were unavoidable, and product teams compensated with client-side caches, request batching and moving work off the critical path. That made products more complex and slowed iteration, the opposite of Habitat's goal of letting developers ignore the database.

Reads are over 90% of Habitat's requests, so OpenAI built a layered read strategy in 2026:

• Store data near users when capacity allows. Data residency users already get regional accounts. But outside residency cases, placement is constrained by physical capacity: many regions cannot hold their proportional share of data, and in regions short on datacenter capacity, power or cooling OpenAI must choose between colocated storage and extra serving compute. It describes placement as a product tradeoff between latency and raw capacity.
• Replicate only the smallest, most latency-sensitive datasets broadly. Authentication metadata needed on every request lives in highly replicated accounts, some replicating to all 40 serving regions and others to at least one region per continent. OpenAI had to build a transparent data movement and routing layer earlier than it wanted, to shift datasets between accounts with different replication topologies. This is capacity constrained, so it is reserved for a few hand-picked datasets.
• Integrated caching for everything else. Habitat workloads are overwhelmingly read-heavy, which is where caching does the rest of the work. The post's description attributes the headline 12x faster reads to this combination of multi-tenancy work, global caching and Cosmos DB tuning. Check OpenAI's page for the exact before-and-after latency figures and how they were measured.

What this means for builders

You do not need 20M QPS for these lessons to apply.

The post is also a window into AI economics. Storage, caching and region placement are being squeezed by the same power and cooling limits that shape model serving; see OpenAI's reported compute commitments and the wider fight over data center and AI infrastructure bills in California.

What to take with a grain of salt

• Self-reported. Every number comes from OpenAI. The "12x faster reads" figure, the 20M QPS and 500 PB claims, and the "no incidents in months" statement have no third-party verification.
• "Only company at this scale." OpenAI says "to our knowledge" it is the only Cosmos DB deployment of this size; that is a claim about others' systems it cannot fully see.
• Vendor relationship. The post is a co-engineering story with Azure, which is also OpenAI's long-standing infrastructure partner. Expect it to be favorable to both.
• Tradeoffs left open. OpenAI also mentions a second store, Nanobase, but this post does not detail it.

Related reading

• OpenAI rewrote Habitat from Python to Rust with two engineers (Part I)
• ChatGPT and Codex outage, paid limits reset
• ChatGPT Work outage, August 31
• OpenAI compute spend and capacity through 2030
• California data center and AI infrastructure bills
• Neocloud shakeout and Nvidia exposure

Details are accurate as of October 23, 2026, based on OpenAI's Part II post; figures may change as OpenAI updates its systems.