vLLM Push-Based and Bi-Directional KV Transfer: What AWS Measured on P5en

AWS has published two engineering posts on moving the KV cache faster in vLLM's prefill-decode disaggregated serving. A push-based NIXL connector cut time to first token (TTFT) by about 3x at 16k-token inputs in AWS's test. A bi-directional mode lets the prefill node reuse context the decode node already holds, cutting TTFT by 1.6x on Qwen3-32B and 3.3x on GLM-5-FP8. Both were measured on Amazon EC2 P5en instances, and both come with caveats worth reading before you copy the numbers into a capacity plan.

The two posts are from the AWS Machine Learning Blog (push-based) and its companion on multi-turn agentic workloads (bi-directional). The authors say the features were built and validated on AWS Trainium clusters first and then contributed to vLLM as accelerator-agnostic code; the GPU results are there to show the gains carry over.

TL;DR: the questions people will ask

First, what is disaggregated serving and why does KV transfer matter?

An LLM request has two phases. Prefill processes the whole prompt in one compute-heavy pass and builds the KV cache, the stored attention keys and values for every prompt token. Decode then generates tokens one at a time, which is limited by memory bandwidth. Disaggregation runs the two phases on separate instances so each can be sized and tuned for its own bottleneck.

The price is that the KV cache must travel from the prefill instance to the decode instance before the first token can appear. For a long prompt that is a lot of data, so the transfer sits directly on the TTFT critical path. If you want the basics of the cache itself, our KV cache explainer coverage of local runtimes touches on why long contexts get expensive, and our look at fast inference on AMD hardware shows how much serving-stack work now goes into squeezing latency.

In the transfer path AWS describes, KV blocks move over RDMA on AWS EFA, using GPUDirect RDMA on GPUs or Neuron Direct RDMA on Trainium, so the data goes device to device without staging in host memory.

Token cost graphic illustrating how a KV cache stores per-token state for vLLM disaggregated inference

How the default pull-based connector works

In vLLM's default NIXL connector, decode pulls. After prefill finishes, the router hands the prefill block IDs to the decode instance. Decode allocates destination blocks, prepares the transfer, and issues a NIXL READ. These steps run one after another, after prefill and before the first decoded token.

The AWS authors point to one specific cost: NIXL descriptor posting runs synchronously inside the decode worker's engine loop. Under load, that work stalls token generation for requests that are already decoding, which hurts TPOT as well as TTFT.

Push-based transfer: how it differs

Push flips the direction and moves setup off the critical path. The AWS post describes a three-step protocol.

1. Registration (decode side). When decode allocates blocks for a remote-prefill request, it sends the prefill instance a NIXL notification describing its identity and the reserved block locations.
2. Rendezvous (prefill side). The prefill producer matches that registration with prefill completion by request ID, in either order. This matching also handles preempted or retried requests.
3. Transfer (prefill side). The producer connects to the decode instance if needed and issues NIXL WRITEs into the decoder's registered memory, one per destination tensor-parallel rank. Prefill and decode may use different block layouts and block sizes; each side maps blocks to its own layout. The decoder counts one completion notification per producer rank and starts decoding only after all arrive.

Each prefill worker also gets a dedicated background thread. When the prompt KV is ready, the main loop hands the blocks to that thread and goes back to model execution. The thread reuses or establishes the NIXL connection, maps block IDs to memory descriptors, merges contiguous blocks and submits the write. Two benefits follow, per the authors: decode-side setup overlaps with prefill, and descriptor preparation leaves the decode engine loop.

The push numbers

The benchmark used Qwen3-32B on a 1P1D topology on P5en with EFA, input lengths from 1k to 16k tokens, request rates of 1 to 8 QPS and 128 output tokens. Prefix caching was disabled on the decode node (--no-enable-prefix-caching) so that every request had to fetch its full KV cache.

At low load and short prompts the two modes performed almost identically. That matters: the gain is a high-load, long-prompt effect, which is also where disaggregation is most attractive in the first place. Concurrency settings and software versions were not given in the post as we could read it, so treat the table as one data point rather than a general law.

Bi-directional transfer: why agents need it

Agent loops make prefill expensive in a particular way. Each turn, the client resends the entire conversation history. In ordinary disaggregation KV moves only from prefill to decode, so the decode node ends up holding KV for the prompt and the generated reply, while the prefill node cannot reuse it. On a prefill cache miss, the whole history is recomputed, and TTFT grows with every turn. The AWS design has four parts:

• NIXL connector changes. The connector was extended so prefill nodes can consume remote KV blocks, not just produce them. The transfer is a NIXL READ used in both directions.
• A KV-cache-aware router. A lightweight stateful component keeps an in-memory map from a client-provided conversation_id to the decode node's kv_transfer_params. It stores block locations only, never tensors, and learns them from the response stream. vLLM ships an example proxy, disagg_proxy_multiturn.py, that can be adapted.
• Block retention TTL. Decode keeps a finished request's blocks for decoder_kv_blocks_ttl (default 480 seconds) and publishes their location and expiry. A next turn that arrives in time reuses them; a later one falls back to normal prefill.
• A recompute threshold. kv_recompute_threshold (default 64 tokens) tells prefill to recompute when the remaining tokens are few, and to pull remote KV when the saving is large enough to cover the transfer.

A stopwatch graphic representing time to first token savings from bi-directional KV transfer in multi-turn agent serving

The bi-directional numbers

Tested on 1P1D P5en with EFA, with prefix caching disabled on the prefill node to emulate eviction or routing to a node that lacks the conversation's KV.

The reported reductions at the longest prompts are about 1.6x and 3.3x. The transfer itself is not free: AWS puts it at roughly 30 to 130 ms for Qwen3-32B and 29 to 110 ms for GLM-5-FP8. Total prefill time still fell, from 153 to 744 ms down to 128 to 414 ms for Qwen3-32B, and from 500 to 2,331 ms down to 316 to 541 ms for GLM-5-FP8. The authors attribute the larger GLM-5 gain to its higher per-token recompute cost, which is plausible: the more expensive recomputation is, the more a fixed-cost transfer saves.

What is claimed and what is not shown

This is vendor-run benchmarking on the vendor's own hardware, and both posts are candid about the scope. Things to keep in mind:

• One topology, one hardware type. Both tests used 1P1D on P5en with EFA. Other GPUs, other networks and larger prefill or decode pools may behave differently.
• Worst-case settings. Prefix caching was off in the stress configuration. With a healthy prefix cache and sticky routing, the baseline for multi-turn work may be much better than the "feature off" column, and the improvement correspondingly smaller.
• Routing trade-off. Push chooses the decode instance before prefill completes. Pull can pick after prefill using load or QoS information. The AWS post recommends pull for workloads that need bi-directional transfer or deferred decode selection.
• Fixed TTL. Retained blocks occupy decode-side accelerator memory, and the 480-second default is not renewed. The authors suggest a heartbeat-renewed lease as future work.
• Extra moving parts. The router is a stateful control-plane component someone has to deploy and maintain.
• Not yet merged for pull mode. AWS says it is exploring the same background-thread approach for pull mode in vLLM pull request 45211, which was not merged at the time of writing.

How to try it

For push mode, set both the prefill and decode servers to use NixlPushConnector in place of NixlConnector. For bi-directional mode, AWS says to use vLLM 0.21.0 or later with NixlConnector, enable bidirectional_kv_xfer on both servers, deploy a router that maps conversation_id to decode metadata, and tune decoder_kv_blocks_ttl and kv_recompute_threshold for your inter-turn gaps and interconnect. Both posts point to reproduction guides in the vLLM project website repository, and the vLLM documentation covers disaggregated serving and the NIXL connector.

A sensible rollout order: reproduce the vendor benchmark on a small pool, then replay a sample of your own traffic with prefix caching on and off, then compare TTFT and TPOT percentiles rather than averages. If your prompts are mostly under a few thousand tokens or your load is light, expect to see little.

Why this matters beyond one benchmark

Serving cost is increasingly set by how well the stack handles long, repeated context rather than by raw FLOPs. Agent workloads resend history constantly, and the way a framework moves or reuses KV decides whether you pay for that history once or every turn. The same pressure shows up in hardware and cloud strategy: see our pieces on Amazon's TorchNeuron claim, vLLM throughput claims for Kimi K3, and the neocloud shakeout for how capacity economics are shifting.

It is also worth noting where the work came from: Trainium first, then upstream, accelerator-agnostic. That is a pattern of cloud vendors pushing open-source inference engines to be less tied to one GPU vendor, which pairs with distributed efforts like mesh-llm on the hobbyist end and AMD's Claude bring-up deal on the industrial end.

Related reading

• Amazon TorchNeuron: PyTorch on Trainium, claimed vs verified
• Kimi K3 on vLLM: 842 tokens per second, claimed vs verified
• Fastest GLM-5.2 on AMD MI355X
• What is llama.cpp?
• mesh-llm distributed inference v1
• Neocloud shakeout and Nvidia exposure

Details reflect the AWS Machine Learning Blog posts and vLLM resources as of October 19, 2026; flags, defaults and version requirements may change as the PRs evolve.