llama.cpp v0.9.0 and ggml 0.29.0: Native FP8, Multi-Node Tensor Parallelism and What Local-AI Users Get

llama.cpp tagged version 0.9.0 on November 4, 2026, built on ggml 0.29.0. The headline items in the ggml 0.29.0 release notes are a native OCP FP8 E4M3 tensor type, a new flash-attention variant for multi-sequence serving, batched tensor copy APIs and initial multi-node tensor parallelism through NCCL. This post explains what those mean for people who run models locally or on small clusters, and what to check before upgrading.

We built it from the ggml release notes and the llama.cpp releases list. The release notes are long and mostly kernel-level, so we concentrate on the parts that change what you can do.

TL;DR

FP8 E4M3: what it is

FP8 E4M3 is an 8-bit floating-point format with four exponent bits and three mantissa bits, as defined in the Open Compute Project's FP8 specification. It stores each number in one byte, half the size of the 16-bit formats still common for model weights. The ggml notes add GGML_TYPE_F8_E4M3 with float conversion helpers and a GGML_FTYPE_MOSTLY_F8_E4M3 file type. In practice that lets llama.cpp load and write models whose weights are natively FP8, without first converting through a larger type.

Most local users run integer-style quantizations such as Q4_K_M, which compress further and have long-standing kernels. FP8 is a different tradeoff. It keeps a floating-point range with modest compression, and newer GPUs and training pipelines use it, so it is a natural interchange format. This release adds the type and conversions. It does not say that every backend has a fast FP8 matrix-multiply kernel, so do not assume a speedup without testing. If you want to see how the project thinks about low-bit formats, our guide on what llama.cpp is and how to run models locally covers the quantization basics.

A memory budget example

Here is simple arithmetic, ours and not from the release notes. A model with 8 billion weights needs about 16 GB at 16 bits per weight, about 8 GB at 8 bits per weight, and about 4.5 to 5 GB at a typical 4-bit quantization once overhead is included. So FP8 sits in the middle. It halves the 16-bit size while keeping a floating-point format, which is often easier to convert from a training checkpoint than an integer quantization with calibration. For a 70B-class model the same math gives roughly 140 GB, 70 GB and about 40 GB. That is why a second node matters for large models, and why the multi-node work below arrives in the same release as the FP8 type.

The OCP 8-bit floating point specification defines the E4M3 and E5M2 variants. E4M3 trades range for precision: with three mantissa bits it resolves values more finely than E5M2, and it is the variant commonly used for weights and activations. The ggml release notes describe the new type as OCP FP8 E4M3 with float conversion helpers.

Where this sits in the project

The llama.cpp repository tags numbered builds continuously, so the first thing to know about v0.9.0 is that it is a named version marker on top of the build stream. The releases list shows the v0.9.0 tag published on November 4, 2026, right after a run of numbered builds (b12096 to b12110) from the same day. Whether you track tags or builds, the ggml library underneath carries the changes that matter: the new tensor type, the new ops and the communicator interface. If you build downstream software on ggml directly, such as a runtime or a language binding, the API changes are the important part. The new batched copy functions are additions to the backend API, so check whether your runtime can use them to cut copy overhead.

Flash attention rows: better multi-sequence serving

The new ggml_flash_attn_ext_rows() op is a variant of flash attention where each query slice attends to its own list of key-value rows, chosen through an index tensor. The release lists implementations on CPU, CUDA and Metal. The point is efficiency when one server handles several conversations at once. With a unified KV cache shared by many sequences, each sequence should look only at its own rows. This op expresses that directly. The release notes do not give speed numbers for it, so we cannot say how much faster it is. For a related deep-dive on Apple GPU attention work, see llama.cpp's Metal few-row MMA and speculative decoding on an M3 Ultra.

Multi-node tensor parallelism

Tensor parallelism splits each layer's math across several devices so that a model too large for one GPU can run on several. ggml 0.29.0 adds a multi-process communicator interface (ggml_backend_comm_init_rank_t) for collectives across separate ranks. On CUDA, the notes say NCCL support enables multi-node tensor parallelism through that interface, "used by the RPC server." The notes call it initial support. Vulkan and SYCL also gained tensor-parallel pieces: a Vulkan allreduce from shared host buffers and a fast SYCL comm exchange.

What this means for you: if you have more than one machine with GPUs and want to serve one large model, this is the first official building block. Expect tuning work, network sensitivity and bugs. The fast interconnect between nodes will decide whether it helps. For simpler multi-GPU cases on one machine, existing split modes remain the usual path.

Backend work, by platform

The notes list a long set of changes. A few that matter for common setups:

• CUDA: NCCL multi-node support, a fused MoE gate/up projection with GLU in MMQ, a fix for a Q8_1 correction sum in MMQ, a fix for premature eviction of CUDA graphs, and HIP changes including sparse attention on CDNA and an RDNA4 family for gfx1170 and gfx1171.
• Metal: faster MXFP4 matrix-vector, full-threadgroup launches for unary, binary and GLU kernels, and the rows flash-attention op.
• Vulkan: a tensor-parallel allreduce, better small-n split-K selection on RDNA3 and RDNA4, and an NVFP4 scale lookup bug fix.
• SYCL: a DPAS flash attention kernel for q4_0 and q8_0 KV caches on XMX, and vectorized dequantization stores.
• OpenCL: quantized GEMV and GEMM for Q2_0, IQ1, IQ2, IQ3 and IQ4_XS, plus Adreno-oriented kernels.
• Hexagon: new ops including the lightning indexer, top-k mask, conv2d and upscale, which matter for Qualcomm NPUs.
• CPU: new SVE 256-bit kernels for MXFP4 GEMV, Q6_K GEMM and q8_0 4x8 GEMV, and a gate so RISC-V vector code is not built for targets without the V extension.

Hexagon work is a nice pointer to where local AI is heading: phones and laptops with NPUs. If that interests you, see the memory math in our post on Qualcomm's 100B-parameter phones claim.

Core changes that affect safety and speed

Several lines are about correctness. ggml now bounds tensor views by the bytes their strides reach and validates strided views against the source extent. That tightens memory safety, which matters when you load untrusted model files. The GGUF writer can now write host-buffer tensor data straight to the output stream without an intermediate copy, which speeds up conversion and saves memory. Apple platforms use vm_allocate instead of posix_memalign for large allocations, and IQ4 quantization is faster through pre-computed halfway points.

How to upgrade safely

1. Read the full release notes for your backend, because many fixes are backend specific.
2. Rebuild from the tag, not from the moving master branch, so results are reproducible.
3. Run your own prompt set and compare output and speed before and after. Do not assume any speedup.
4. For multi-node work, start with two machines on a fast network and measure.
5. Keep your previous binary available for a quick rollback.

For Windows users, our look at the Llama Windows tray app shows an easy way to run recent builds, and our report on Transformers matching llama.cpp on GGUF shows how the format is spreading across tools.

What is not in the notes we read

We did not find benchmark numbers in the ggml release notes. Claims of speedups should be tested on your own hardware. We also have not verified any list of newly supported models for the llama.cpp v0.9.0 release itself, since that detail sits in the llama.cpp release page and we did not rely on it here. Check that page for the exact model list.

Related reading

• What is llama.cpp
• llama.cpp Metal and speculative decoding
• Llama Windows app
• Transformers and llama.cpp GGUF parity
• Qualcomm 100B-parameter phones

Details come from the ggml 0.29.0 release notes as read on November 4, 2026 and may change. Follow @explainx_ai for updates.