Groq 3 LPX Hits 3,400 tok/s — Nebius First Cloud Adopter

Agent loops live or die on decode speed — not parameter count.

At Hot Chips, August 24, 2026, NVIDIA announced Groq 3 LPX in full production: an interactive inference accelerator built for the Vera Rubin platform, designed to push token generation toward "speed of thought" interactivity on long contexts.

The headline number: ~3,400 output tokens per second on Gemma 4 31B with 100,000 tokens of input context — measured by Artificial Analysis, the fastest published score for that model at that context length.

Nebius is the first AI cloud adopting LPX for production via Nebius Token Factory.

TL;DR

Why LPX exists

NVIDIA's framing: generation determines responsiveness. Prefill can batch; decode is what users feel while an agent thinks, calls tools, and streams the next plan.

Agentic apps can use ~15× more tokens than traditional chat (NVIDIA product briefs). Factories optimized only for throughput-per-megawatt underserve tokens-per-second-per-user — the metric that makes Codex-class loops feel instant vs sluggish.

LPX specs (public materials):

Paired with Vera Rubin NVL72, NVIDIA claims up to 35× higher TPS/MW at 400 TPS/user vs GB200 NVL72 for premium interactive tiers — a separate product lane from raw batch inference.

How LPX decode differs from GPU batching

To understand why 3,400 tok/s matters, separate prefill from decode in transformer inference.

Prefill processes the entire input context in parallel — attention over all prompt tokens at once. GPUs excel here: wide matrix multiplies, high HBM bandwidth, batching many requests together amortizes fixed costs. This is why datacenter serving optimizes throughput per megawatt — fill the GPU, maximize tokens processed per second across tenants.

Decode generates one (or few) tokens at a time, autoregressively. Each new token depends on the full KV cache from prior tokens. Memory bandwidth becomes the bottleneck, not raw FLOPs. Users feel decode as streaming latency — the gap between tool results arriving and the model's next plan appearing.

Agentic workloads skew decode-heavy:

• Multi-step tool loops generate far more output tokens than a single chat reply
• Long scratchpads inflate KV cache size, which slows decode on memory-bound paths
• Replanning after tool errors adds generation rounds users experience sequentially

Groq's LPU (Language Processing Unit) architecture uses SRAM instead of HBM for weights and activations during decode. SRAM bandwidth (LPX: 40 PB/s across the rack) targets the memory-bound generation phase. Vera Rubin GPUs handle prefill and large-model serving; LPX racks specialize in fast sequential token emission.

NVIDIA's heterogeneous framing — Rubin GPU + Groq LPU in one platform — acknowledges that one chip topology does not win both batch economics and interactive decode. LPX is the decode lane.

Groq 3 LPX vs other fast-inference paths

LPX is not the only "fast inference" story in August 2026. Builders should compare at their context length and workload, not headline tok/s alone.

Artificial Analysis measured LPX at both 10K and 100K context with minimal speed variation (3,382 vs 3,431 tok/s median). That stability matters for agents: scratchpads grow across a session; decode speed that collapses at long ctx breaks UX even if short-context benchmarks look fine.

The agent loop math

Why NVIDIA cites ~15× more tokens for agentic apps vs traditional chat:

Consider a coding agent loop:

1. User message + repo context → prefill
2. Model generates plan → decode (200 tokens)
3. Tool call → external latency (not LPX)
4. Tool result injected → prefill again
5. Model revises → decode (400 tokens)
6. Repeat 3–5× per user request

Each decode phase is user-visible. Shaving 50ms off step 2 does not help if step 4 prefill takes 2 seconds — but when prefill is already optimized on Rubin GPUs, decode becomes the remaining bottleneck. LPX targets that remainder.

For builders on Codex-class harnesses: benchmark end-to-end loop latency, not single-request tok/s. LPX helps when generation dominates; it does not accelerate tool execution, network I/O, or human review.

Vera Rubin platform context

Groq 3 LPX is not a standalone product — it extends the NVIDIA Vera Rubin platform announced for datacenter AI infrastructure. Understanding the platform helps you place LPX in a procurement conversation.

Vera Rubin NVL72 targets high-throughput prefill and large-model serving — the workhorse tier for batch inference, training adjacency, and multi-tenant GPU economics. Groq 3 LPX adds a heterogeneous decode accelerator: 256 LPU chips per rack, 128 GB SRAM, 315 PFLOPS AI inference compute, 640 TB/s scale-up bandwidth.

NVIDIA's public claim — up to 35× higher tokens per megawatt at 400 TPS/user vs GB200 NVL72 — frames LPX as a premium interactive tier, not a replacement for batch-optimized racks. Datacenter buyers increasingly split fleets: batch economics on GPU-heavy rows, interactivity on LPX rows.

This connects to the broader NVIDIA infrastructure wave: Rubin-era hardware is designed for agentic AI's token volume, not just chat's. SpaceX's Starmind partnership pushes the same Vera Rubin generation toward orbital compute — LPX's decode specialization may matter less in space, but the platform bet is consistent: inference architecture splits by workload shape.

Pricing and access expectations

NVIDIA has not published LPX $/token pricing as of Hot Chips launch. Nebius's role as first cloud adopter suggests initial access via Token Factory tier selection — same API keys, different backend hardware class.

For budget planning:

• Expect premium pricing vs standard GPU tiers — you pay for decode SLA, not raw batch throughput
• Compare $/successful agent task, not $/million tokens in isolation — faster decode that reduces user abandonment may justify premium unit economics
• Keep batch workloads on open-weight Kubernetes economics; LPX is for loops where latency is product-critical

Builders without Nebius contracts should watch for second-cloud announcements — LPX as a Vera Rubin extension will likely appear on other Rubin-era hosts, but Nebius first-mover status means earliest production access flows through their Token Factory roadmap.

When LPX is worth it (and when it is not)

Use this decision tree before you chase 3,400 tok/s headlines:

LPX likely worth evaluating if:

• Your product is interactive — coding agents, copilots, live collaboration where users watch tokens stream
• Generation latency dominates your p95 loop time (profile first)
• You serve long context (50K–100K) where decode stability matters
• You already run on Nebius or plan Rubin-era infrastructure migration

LPX probably not your first move if:

• Workloads are batch/offline — report generation, overnight code migration, eval suites
• Tool and network I/O dominate loop time — faster decode will not move user-perceived latency
• You are cost-constrained on per-token spend — standard GPU tiers or open-weight routing may win on $/task
• You develop locally on laptop — LPX is datacenter rack hardware, full stop

The Artificial Analysis benchmark used Gemma 4 31B at 100K context deliberately: it is a demanding configuration that exposes decode bottlenecks. Your model and context may differ — replicate the methodology at your shape before assuming LPX headlines transfer.

FAQ addendum — infrastructure questions builders ask

Will LPX work with any model weights? NVIDIA's launch centers on Vera Rubin platform integration; model support follows inference stack compatibility on Rubin + LPX heterogeneous racks. Treat model list as vendor-published, not "bring any GGUF."

Does LPX replace my vLLM deployment? Different lane. vLLM optimizes GPU batch serving; LPX optimizes decode on LPU hardware. Many teams will run both — batch on GPU rows, interactive on LPX tiers.

How do I measure success after migration? Track p95 loop latency and task completion rate for 2–4 weeks A/B. Faster tokens that increase hallucination retries are not a win — same lesson as ATT model routing: optimize outcome economics, not a single metric.

What builders should benchmark before switching

If Nebius Token Factory exposes LPX through existing API endpoints, migration should be backend-tier selection, not a rewrite. Still validate:

ATT's model routing saved 56% by sending routine work to cheaper models — LPX is the inverse bet: pay for speed on the tier that needs interactivity, not blanket frontier spend. Route boilerplate to DeepSeek V4 Flash-class economics; reserve LPX for loops where decode latency sets UX.

What people are asking about LPX access

"Can I rent LPX on Nebius today?" Nebius announced first-cloud adoption at Hot Chips; production rollout timing follows their Token Factory release notes. Watch for LPX tier flags on existing endpoints — Shtan's quote emphasizes same API surface.

"Is LPX the same company as Groq (the API provider)?" NVIDIA acquired Groq's assets; Groq 3 LPX is NVIDIA's LPU product line on Vera Rubin. The standalone Groq API cloud is a separate commercial path — do not conflate rack hardware with consumer API access.

"Does 3,400 tok/s mean my agent feels instant?" Only for the generation phases. Tool calls, retrieval, and human-in-the-loop steps still dominate many loops. LPX removes one bottleneck; harness design removes others.

"Should I wait for LPX or optimize my prompt?" Both. Prompt compression and token budget planning reduce decode volume; LPX makes remaining decode faster. They compound.

Nebius as first mover

Nebius CTO Danila Shtan:

"Generation is the phase of inference that determines how responsive an AI system actually is… we're making sure every step of an agent's loop feels instant — through the same API developers are already using."

For builders already on Nebius for open-weight serving, LPX is an upgrade path — not a forked SDK.

What this means for your stack

This sits in the same infrastructure wave as SpaceX orbital Vera Rubin plans — compute moving wherever watts-per-interaction works.

What this means for builders (summary)

LPX does not change which model is smartest. It changes how quickly a capable model streams the next token during the generation phase — the metric agent harnesses feel as "responsiveness."

Related on explainx.ai

• SpaceX Q2 earnings + Nvidia Starmind Rubin partnership
• Gemma 4 31B — Cerebras fastest multimodal inference
• Codex as a platform — open agent harness
• Open-weight Kubernetes moment
• ATT model routing — 56% cost savings
• DeepSeek V4 Flash — cost per ARC task
• NVIDIA Nemotron 3.5 Lightning open MoE
• What is an agent harness?

Sources

• NVIDIA Newsroom — Groq 3 LPX full production — August 24, 2026
• NVIDIA Technical Blog — Inside Groq 3 LPX
• Artificial Analysis — 100K context benchmark methodology

Benchmark figures from NVIDIA and Artificial Analysis as of August 24, 2026. Follow @explainx_ai for inference infrastructure.