The most useful sentence in Jacob Peake's enormous AI Chip Architectures survey is not a specification. It is a diagnostic: to understand an AI chip, ask where data lives, how it moves to compute, what the compute units look like, and how chips communicate at scale.
That cuts through a lot of misleading comparison tables. Peak FLOPS matter, but a matrix unit sitting idle while it waits for model weights is not fast. A rack's aggregate bandwidth is not the same thing as bandwidth available to one token. A vendor's sparse or low-precision peak is not automatically comparable with a rival's dense number. And a chip that wins at batch-one decode may lose badly when a serving system batches hundreds of requests.
This guide compresses the architecture survey into a practical mental model. It covers the five approaches with meaningful deployment today: NVIDIA and AMD GPUs, Google's TPU, AWS Trainium, Cerebras's Wafer-Scale Engine, and Groq's LPU. The goal is not to crown one winner. It is to understand what each machine is betting on.
TL;DR
| Question | Answer |
|---|---|
| What problem are all these chips solving? | Keeping matrix-multiplication units fed without spending all the time and energy moving data |
| Why do GPUs dominate? | Programmability, CUDA's mature ecosystem, and a scale-up/networking stack that handles many changing workloads |
| What is the TPU bet? | Use large systolic arrays, software-managed memory, and whole-program compiler scheduling for efficiency |
| What is Trainium's bet? | Use a TPU-like compiler-first core inside an AWS-owned chip, network, runtime, and pricing stack |
| What is Cerebras's bet? | Do not cut the wafer; surround 900,000 small cores with local SRAM and eliminate most die boundaries |
| What is Groq's bet? | Put the entire schedule in the compiler, use on-chip SRAM, and make inference deterministic across chips |
| What should buyers compare? | Exact-model latency, throughput, utilization, memory capacity, interconnect, software effort, availability, and total cost |
The common problem: arithmetic is cheap, moving data is expensive
A transformer repeatedly performs matrix multiplication, with normalization, activation, residual addition, masking, and routing around it. But the shape of the work changes across three phases:
- Training passes many tokens through the same weights and then computes gradients. Large matrix-matrix operations provide substantial data reuse.
- Prefill processes the prompt before the first output token. It also stacks many tokens together, so it often looks compute-heavy.
- Decode produces one token after another. Each step must read model weights and, for attention, the growing KV cache. At low batch sizes, memory traffic dominates.
This is the memory wall. Compute throughput has increased much faster than the ability to move bytes. It also explains why RAM and HBM pricing matters to local AI builders, why quantization improves more than storage requirements, and why the fastest low-latency systems often devote unusual amounts of silicon to memory.
So every architecture is really answering the same question: how much flexibility will we trade for predictable data movement?
The architecture map
| Architecture | Where important data lives | Who schedules work? | Main strength | Main tradeoff |
|---|---|---|---|---|
| NVIDIA GPU | HBM, L2/L1, shared memory, registers, specialized tensor memory | Hardware warp schedulers plus programmer/compiler | Flexibility and ecosystem | More control and scheduling overhead |
| AMD GPU | HBM, large Infinity Cache, local caches/LDS | Hardware wavefront schedulers plus programmer/compiler | Memory capacity, open ROCm stack, chiplets | New-kernel support often trails CUDA |
| Google TPU | HBM plus explicit on-chip scratchpads | XLA compiler | Efficient large-scale scheduled execution | Less forgiving of unusual shapes and bad compiler schedules |
| AWS Trainium | HBM plus SBUF/PSUM scratchpads | Neuron/OpenXLA compiler | AWS-integrated price-performance and communication | Younger, cloud-specific ecosystem |
| Cerebras WSE | 44 GB of distributed on-wafer SRAM; external MemoryX for training | Data arrival plus compiled placement | Huge local bandwidth and low-latency decode | SRAM capacity and specialized deployment |
| Groq LPU | On-chip SRAM spread across many chips | Static compiler schedule | Predictable, fast batch-one inference | Models require many SRAM-only chips; narrow workload fit |
This table is a map, not a benchmark. Even the phrase “memory bandwidth” describes different things: HBM bandwidth at a GPU package, aggregate local SRAM ports across a wafer, and aggregate SRAM across a Groq rack are not interchangeable numbers.
NVIDIA GPU: keep the machine programmable
NVIDIA's core bet is that AI workloads will keep changing, so the accelerator must remain programmable. A GPU contains many Streaming Multiprocessors (SMs), each scheduling groups of 32 threads called warps. CUDA cores handle general and element-wise work; Tensor Cores handle tiled matrix multiplication.
The clever evolution is not merely “more Tensor Cores.” NVIDIA has progressively removed feeding and bookkeeping work from threads. Asynchronous copies, Hopper's Tensor Memory Accelerator, Blackwell's Tensor Memory, and descriptor-driven matrix instructions let one part of a kernel move the next tile while another computes. That overlap is why modern attention kernels are architecture-specific engineering projects rather than simple calls to a matrix library.
Blackwell pairs fifth-generation Tensor Cores with FP4 support and HBM3e delivering up to 8 TB/s per GPU, according to NVIDIA's architecture and inference material. But the deeper advantage is compounding software: CUDA, cuBLAS, cuDNN, CUTLASS, TensorRT-LLM, NCCL, PyTorch, Triton, profilers, and years of third-party kernels.
Choose this pattern when: model architecture changes quickly, custom kernels matter, the workload extends beyond transformer inference, or engineering portability across the broadest library ecosystem matters more than theoretical efficiency.
AMD GPU: similar programming model, different package bet
AMD Instinct is still a GPU: Compute Units run wavefronts, Matrix Cores execute matrix instructions, and ROCm/HIP expose a programmable kernel model. The differentiator is increasingly outside the individual compute unit—chiplets, HBM capacity, Infinity Fabric, and a large Infinity Cache.
The MI355X, for example, exposes 288 GB of HBM3E at 8 TB/s and supports MXFP4/MXFP6 on CDNA 4, per AMD's product specifications. That memory capacity is valuable when model weights and KV cache determine how many accelerators a serving replica needs.
The software question remains central. Common PyTorch, vLLM, and Triton workloads have become much easier to run on ROCm, but novel CUDA kernels can arrive first and require an AMD-specific port. The right comparison is therefore not only dollars per accelerator. Include porting time, missing kernel fallbacks, and achievable utilization on your model.
Choose this pattern when: HBM capacity and open software standards are attractive, your workload is already validated on ROCm, and the price-performance win survives the engineering needed for any unsupported tail.
Google TPU: make the compiler the scheduler
A TPU is organized around large systolic matrix arrays. Values move through a grid of multiply-accumulate cells, reusing operands in the wires rather than repeatedly loading them from a general cache. Vector and scalar units handle surrounding work. On-chip memory is software-managed, and XLA decides layouts, fusion, buffer placement, transfers, and collectives ahead of execution.
That deletes hardware structures a GPU uses to tolerate uncertainty. When the compiler's schedule matches the workload, the result can be extremely efficient. When dimensions underfill the array or a layout creates awkward transfers, there is less dynamic hardware to rescue the schedule.
Google's Ironwood system makes the system-level bet obvious: a superpod can connect 9,216 chips through ICI and expose 1.77 PB of HBM, according to Google Cloud's Ironwood announcement. JAX/XLA is the native path, while PyTorch/XLA, vLLM support, and Pallas-style custom kernels reduce ecosystem friction.
Choose this pattern when: you are already in Google's JAX/XLA stack, your model maps cleanly to supported operations and shapes, and pod-scale co-design matters more than dropping arbitrary CUDA code into the system.
AWS Trainium: compiler-first hardware as a cloud component
Trainium resembles the TPU at first glance: a systolic Tensor Engine, explicit on-chip buffers, and OpenXLA-based whole-graph compilation. But each NeuronCore also contains vector and scalar engines plus programmable GPSIMD processors for operations that do not fit the main tensor path.
The AWS-specific bet is vertical integration. Trainium is not sold as a merchant accelerator; it is one component inside EC2, Neuron, NeuronLink, EFA, Nitro, and AWS pricing. Trainium2 puts eight NeuronCores and 96 GiB of HBM on each chip; a Trn2 UltraServer connects 64 chips into 6 TB of HBM, according to the official Trn2 specifications.
The Neuron Kernel Interface is important because it gives developers a low-level escape hatch when automatic compilation is insufficient. AWS made NKI stable in April 2026 and added a CPU simulator, making custom kernel development less dependent on live accelerator access.
Choose this pattern when: AWS is already the deployment boundary, a supported model has measured cost advantages on Trainium, and your team is comfortable accepting a cloud-specific compiler and kernel ecosystem.
Cerebras WSE: remove the die boundary
Conventional manufacturers print many dies on a wafer, cut them apart, and then spend extraordinary engineering effort reconnecting them through packages, links, switches, and networks. Cerebras leaves the wafer intact.
WSE-3 contains 900,000 small cores and 44 GB of on-chip SRAM, according to Cerebras's announcement. Local SRAM bandwidth is the point: weights or activations can stay close to compute without the HBM round trip dominating each step. The design uses many dataflow cores rather than a few giant matrix engines.
But the strength is also the constraint. Forty-four gigabytes does not hold a frontier model plus a large KV cache. Cerebras trains large models by streaming weights from an external MemoryX system while activations remain on the wafer; inference shards resident weights across multiple systems. That can deliver extraordinary per-user decode speed, but it is a specialized system whose batching economics, context capacity, and software surface must be measured separately from latency.
For a current product-level example, see our CS-4 analysis, which distinguishes vendor peak claims from independently comparable results.
Choose this pattern when: interactive token latency is the product, the available model is supported, and the premium for low-batch speed is more important than minimizing cost per token at high utilization.
Groq LPU: schedule every cycle before execution
Groq removes another source of uncertainty. Its compiler statically schedules instructions, data movement, and chip-to-chip communication. The LPU has no conventional cache hierarchy; on-chip SRAM and a spatial “assembly line” of functional units make execution deterministic.
Groq describes more than 80 TB/s of on-chip SRAM bandwidth in its LPU architecture explanation. A compiled model is partitioned across enough chips for its weights to fit in the aggregate SRAM. The system can then stream a token through those stages with predictable latency.
This is excellent for the metric users feel directly—tokens per second for one interactive request. It is less obviously optimal for maximizing throughput per dollar through large batches, and the SRAM-only capacity model means a large model can require many chips even when their arithmetic units are lightly utilized during decode.
Choose this pattern when: batch-one responsiveness is a core feature—voice, agents, live interfaces—and the provider supports the exact model and context window you need at an acceptable per-token price.
What the original comparison gets right—and what buyers should add
Peake's survey gets the most important principle right: architectures are competing over data movement, not just multiplication. It also shows why “GPU versus ASIC” is too crude. NVIDIA is adding more explicit memory control; TPUs and Trainium expose custom-kernel escape hatches; Cerebras and Groq have cloud APIs. The poles are converging even while their physical bets remain different.
For an actual purchase or platform decision, add six questions the architecture diagram cannot answer:
- Does my exact checkpoint run? Include architecture, quantization, context length, tool calling, and speculative decoding.
- At what batch and latency target? A batch-one winner can lose on total throughput, and vice versa.
- What is independently measured? Separate vendor peak FLOPS and vendor demos from audited benchmarks and your own load test.
- What breaks portability? Count custom kernels, compiler-specific sharding, unsupported operators, and observability gaps.
- What is the real availability boundary? A spectacular pod specification does not help if capacity is unavailable in your region.
- What is total cost? Include hosts, networking, reserved capacity, engineering, idle replicas, and the number of copies needed to meet reliability targets.
The same discipline applies to emerging memory. Samsung's August 2026 zHBM is a concept, not a deployable HBM5 replacement: Samsung says vertically placing memory above the accelerator could provide about 8x interface performance, more than 10x density, and 3x energy efficiency, but those are roadmap targets from its FMS 2026 announcement. Treat it as evidence that packaging is becoming part of the architecture, not as a product benchmark you can buy today.
Bottom line
There is no single “AI chip.” There are different answers to four questions:
- Where does data live? HBM and caches, software scratchpads, or on-chip SRAM.
- Who decides what runs next? Hardware schedulers, a whole-program compiler, data arrival, or a static clock schedule.
- How much programmability survives? General kernels, compiler graphs, tile DSLs, or a closed supported-model surface.
- What happens after one chip? Coherent links, message-passing tori, wafer meshes, or a statically scheduled chip network.
NVIDIA's answer is to preserve programmability and amortize its overhead. AMD adds a strong memory-and-chiplet bet to the GPU model. Google and AWS trust compilers and system co-design. Cerebras removes the cut-wafer boundary. Groq removes runtime uncertainty.
Those are engineering philosophies, not leaderboard positions. Pick the constraint your product actually has—training throughput, prefill latency, decode speed, memory capacity, portability, or cost—and benchmark the whole system that serves it.
Related reading
- Cerebras CS-4: wafer-scale inference explained
- AMD acquires Taalas: weights etched into silicon
- Stanford MemoryDAX: 65 years of DRAM, HBM, and NAND prices
- RAM prices and the cost of local AI builds
- OpenAI's Jalapeño inference chip with Broadcom
- Mojo goes open source for heterogeneous AI kernels
- Cerebras-powered Ultrafast mode for GPT-5.6 Sol
Architecture details and linked vendor specifications were checked on August 24, 2026. Roadmaps, availability, pricing, software support, and performance claims change quickly; re-check the linked primary sources and benchmark your exact workload before making a deployment decision.