Meta Prometheus: Inside the Ohio Supercluster That Trained Muse Wonder, and the 1 Million GPU Target

Meta has published Inside Prometheus, an engineering deep dive on the Ohio supercluster that trained its Muse Wonder models. Some summaries read it as "Meta scales Prometheus to 1 million Nvidia GPUs." That is not what the post says. Meta says the first phase is complete, that Prometheus today holds "hundreds of thousands of GPUs" across multiple connected buildings, and that it is "working toward scaling it to comprise ~1 million GPUs."

The correction matters because the real news is somewhere else: not the headline GPU count but the reliability engineering. Meta reports a 99% reduction in training interruptions in about a year, and shares unusually specific numbers on goodput, failure domains and spare-GPU math. This post walks through what is confirmed, what is a goal, and what it means for the models you use, including Muse Wonder.

TL;DR: Prometheus, confirmed vs claimed

How big is Prometheus really?

Meta describes Prometheus as a "gigawatt-scale supercluster," built in phases. The first phase "has already been completed and was used to train our Muse foundational models, including the Muse Wonder models, our new frontier models." The post says the cluster is "tightly interconnected by miles of fiber and functioning as a single machine," and spans traditional data center buildings, AI-optimized buildings and "rapid deployment structures."

On scale, the sentences that matter are these two: today it comprises "an array of hundreds of thousands of GPUs," and "at full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep." The 1 million figure is future tense. We found no Meta statement that the cluster has reached it, and the post gives no timeline.

Likewise, "gigawatt-scale" is a scale label. The post cites 1,400W per GPU in its power-delivery section, so a million GPUs would imply well over a gigawatt for the accelerators alone, but that is our arithmetic, not a Meta figure.

Which Nvidia hardware is inside?

Meta names its platforms rather than just its chips:

• Catalina (GB200 NVL72). Meta deploys the Ariel variant, a two-rack configuration with 36 GPUs per rack linked by NVLink into a 72-GPU domain. Each Blackwell GPU pairs with a Grace CPU and two ConnectX-7 400G NICs at a 1:1 CPU-to-GPU ratio, giving 800G of I/O per GPU early. Per rack pair: 192 GB of HBM3e per GPU, 720 peak PFLOPS FP8 (1,440 at FP4), 130 TB/s NVLink bandwidth and roughly 120 kW.
• Clemente (GB300 NVL72). Blackwell Ultra GPUs with 288 GB of HBM3e each (20.7 TB per domain), over 1 exaFLOP of peak FP4, ConnectX-8 SuperNICs at 800G natively and a 2:1 GPU-to-CPU ratio that fits the full 72-GPU domain in a single rack at about 135 kW. Meta quotes a 5-10% performance improvement over GB200 NVL72.
• San Miguel (Vera Rubin NVL72). Described as "arriving soon." For how the next generation is landing elsewhere, see our coverage of CoreWeave's first multi-rack Vera Rubin cluster and Nvidia's open Vera Rubin MGX designs.

Closed-loop liquid cooling loop around a server block, representing the cooling used in Meta Prometheus

How does the network hold it together?

At this scale the network is part of the computer. Meta runs a dedicated Ethernet back-end network for GPU-to-GPU traffic, separate from its standard data center network. Each building-level fabric interconnects up to 20,736 GPUs. Meta started with a Disaggregated Scheduled Fabric, then moved to a Non-Scheduled Fabric using shallow-buffer 51.2T switches as NIC technology allowed endpoint packet re-ordering. Both run on Meta's FBOSS stack.

Buildings are linked by a Backend Aggregation network, a hub-and-spoke design spanning 2 km to over 100 km, built on Arista 7800-series chassis with Broadcom Jericho3 silicon and "multiple Pbps of bandwidth per region pair." Meta says pre-training over those distances needed a dedicated optimization effort with Meta Superintelligence Labs because of speed-of-light constraints. The post cites "millions of miles of fiber strand, tens of thousands of switches, and millions of optics," from short-reach pluggables to 800G-ZR coherent optics beyond about 10 km.

The real story: a 99% drop in interruptions

Large training jobs are only as good as their weakest host. Meta uses goodput, the fraction of time spent on productive training, as the headline metric. For Muse Wonder it reports more than 85% goodput for pre-training and more than 90% for reinforcement learning, with a wall-clock mean time between interruptions of about 47k GPU hours, since improved to more than 54k with firmware fixes.

The trend is the point. In mid-2025, during the first phase, interruptions peaked at "nearly 1,000 events per 8K GPUs daily." Meta says a year of cross-functional work delivered "a 99% reduction," enough to sustain "multiple training jobs of tens of thousands of GPUs in parallel." Its north star is two interruptions per 8K GPUs per day, about 100k GPU hours MTBI, so it has not claimed to be there yet.

What fixed it, per Meta:

• NVLink signal integrity. Initially the main bottleneck for GB200. With over 600,000 NVLink connections, marginal degradation triggered constant recovery cycles. Meta added continuous telemetry, automated flagging and diagnostic burn-in to catch weak links before production.
• Memory reliability. HBM and LPDDR faults grew with GB300's larger memory. Meta cites refined HBM firmware settings, memory scrubbing, an improved LPDDR error-correction algorithm and predictive retirement of degrading devices.
• Switch and fabric health. A single NVLink switch fault can compromise an entire 72-GPU domain, so Meta monitors fabric health at millisecond resolution.
• Process. A colocated triage team in New Albany, a stability sprint in March, mandatory quarantine after repairs, and a shared accountability model in which Meta handles scheduling and infrastructure while Nvidia handles firmware, signal diagnostics and memory.

Server blocks with airflow lines showing how thousands of racks are cooled in a Prometheus-scale supercluster

The 72-GPU failure domain, in numbers

The most instructive section covers why NVL72 changes capacity planning. Under the old NVL8 design, the tightly coupled scale-up domain was eight GPUs. With NVL72 it is 72, which Meta says "increases the fault blast radius by 9X."

Its answer is spares. Training parallelism maps best onto power-of-two groups, so each NVL72 rack runs a 64-GPU slice and keeps eight GPUs in reserve. When a GPU fails, a spare takes over and the job restarts from its last checkpoint instead of waiting for a repair. For rarer whole-rack faults Meta holds roughly 2% extra racks. Its worked example: a 32,768 active-GPU run requires 523 NVL72 racks, or 37,656 GPUs, against 32,768 active (roughly 15% more by our arithmetic). The 64-of-72 geometry alone accounts for 12.5% of overhead, since 72 divided by 64 is 1.125.

Two recovery mechanisms complement the sparing. Parallelism-Aware Fault Tolerance takes a single faulty data-parallel replica offline while the rest repeat the step and keep training. When a fault is bigger than a replica, the whole job restarts from a checkpoint, saved asynchronously with delta checkpointing and in-memory staging to limit the throughput hit.

Why this matters beyond Meta

Three takeaways are useful even if you never train a frontier model.

First, spare capacity is a real cost. Roughly 13 to 15% of GPUs bought for a big run sit as insurance. When you read about GPU counts, remember a share is reliability overhead, not throughput.

Second, reliability, not raw count, sets the pace. Meta says interruptions "directly impact our engineering iteration speed," meaning slower model releases. A 99% cut in interruptions can be worth more than adding GPUs.

Third, vendors are co-engineering. The embedded Nvidia teams and joint root-cause sessions are a different relationship from buying parts. It is also context for the supply picture in our coverage of Nvidia GB300 packaging and the Nvidia and OpenAI Ohio gigawatt financing.

There is also a community dimension. Data centers of this size are drawing resistance over power, water and cost, which we track in the AI data center backlash map and the Senate investigation into hyperscaler claims about data center costs. Meta's post emphasizes closed-loop liquid cooling for efficiency; the engineering post does not address local power or water impacts in the sections we read, so those questions remain open.

What people are asking

Is this the largest AI cluster in the world? Meta does not claim that. It describes Prometheus as its own most ambitious project. Other companies have announced gigawatt campuses, and rankings depend on how you count a "cluster," so we would not repeat a "world's largest" label without a primary source.

Does this explain Muse Wonder quality? Partly. Meta's Muse Wonder research post says the models were trained at Prometheus, and larger base models are credited for transfer gains. The infrastructure post explains how the run stayed efficient, not why the model scores as it does. For results and pricing, see our Muse Wonder launch coverage.

Will users see a difference? Indirectly, through faster iteration on the next Muse models, which Meta says Prometheus is being scaled for.

Related reading

• Meta Muse Wonder: full launch, pricing and benchmarks
• Meta Muse Spark Open: the 816B open-weight agentic coding model
• CoreWeave's first Vera Rubin cluster
• Nvidia OCP Vera Rubin MGX open designs
• AI data center backlash map
• Nvidia and OpenAI 10 GW Ohio financing
• Official: Inside Prometheus on Engineering at Meta

Figures are from Meta's October 19, 2026 engineering post and may be updated; the 1 million GPU number is a stated target, not a current count.