Perplexity's Fast Embeddings on GPUs: Inside the Ivy, Tulip, ROSE Stack

If you run a RAG pipeline or a vector search index at any real scale, you already know embedding calls behave nothing like chat completions. A reindexing job hammers a model with millions of documents back-to-back; a live search query needs one embedding back in single-digit milliseconds. On September 4, 2026, Perplexity's engineering team published Fast Embeddings on GPUs, a technical breakdown of the serving infrastructure behind their embedding and ranking models — used across Search, Computer, and their API Platform, including their own pplx-embed model.

The headline insight for builders: Perplexity doesn't run a separate embedding-serving stack. They reuse their LLM inference stack almost wholly intact, because the compute-heavy part of a forward pass — the dense layers — is identical whether you're generating a token or producing an embedding. What differs is attention. That single architectural decision, plus two GPU-level optimizations, is the most transferable lesson in the post for anyone building their own retrieval or vector-search system.

TL;DR

Why embedding serving is a different problem than LLM chat serving

Perplexity's core framing is that embedding inference splits into two traffic shapes that pull serving infrastructure in opposite directions:

• Batch Embedding — bulk indexing or reindexing a corpus. This is throughput-focused and computationally similar to LLM prefill: you want to saturate the GPU with as many tokens as possible per second, and latency on any single request barely matters.
• Online Embedding — a per-query lookup at request time, the thing that fires every time a user types a search. This is latency-focused, similar to LLM decode: a slow embedding call blocks the entire user-facing request.

A generic inference server tuned for one of these shapes tends to underperform on the other. This is the practical reason a RAG pipeline that "just works" during a bulk reindex can still feel sluggish on live queries, or vice versa — the two workloads have opposite optimization targets even though they run the same model.

Reusing LLM inference kernels for embeddings

The non-obvious architecture choice in the post: Perplexity does not maintain a bespoke embedding engine. Dense-layer compute — the matrix multiplications that dominate FLOPs in a transformer forward pass — is identical between LLM and embedding inference, because tokens are processed independently through those layers regardless of whether the output is a next-token distribution or a pooled vector.

The difference is entirely in attention:

• Embedding inference uses ragged, unpadded inputs and needs no KV cache — there's no autoregressive generation step to cache for.
• LLM decoding needs paged KV-cache attention across a growing context.

Because of this, pplx-embed and Perplexity's Qwen3.5 LLM decoding path run through nearly the same kernels in ROSE, their Python model-serving engine. For a team building their own infrastructure, this is a real cost-saving pattern worth copying: if you already operate an LLM inference stack, extending it to serve an embedding model is closer to swapping an attention backend than building a parallel system.

The three-layer serving stack: Ivy, Tulip, ROSE

Perplexity's stack splits cleanly by responsibility, each layer in a different language chosen for what it's good at.

Ivy — the Rust HTTP gateway

Ivy handles all CPU-side request work: JSON parsing, tokenization, input templating, and batch splitting, before translating requests to gRPC for the layers downstream. Two details stand out:

• Ivy load-balances by splitting large-batch requests across replicas, specifically to avoid load imbalance when one caller sends a disproportionately large batch.
• Ivy runs Perplexity's own in-house unigram tokenizer, now fully rolled out, which the team says "drastically improves latencies" over off-the-shelf tokenizers. This is the same tokenizer work Perplexity open-sourced as part of pplx-garden, their broader inference-technology release.

Tulip — the Rust gRPC inference server

Tulip sits between Ivy and the model engine, built on tokio/tonic, and owns request scheduling and batching. Its scheduler is deliberately simple — first-come-first-served — and the reasoning is specific to small embedding models: latency is dominated by total token count (a linear dense-layer cost), not sequence count (the quadratic cost of attention, which is negligible at these model sizes). Once a batch on a sub-1B-parameter model hits roughly 512 tokens, the GPU is already saturated; packing in more sequences doesn't help. That threshold is a useful rule of thumb if you're tuning your own micro-batching logic for a small embedding model.

ROSE — Runtime-Optimized Serving Engine

ROSE is the Python engine implementing the actual model forward passes, kernels, and layers. It was originally built for LLM serving and extended to embeddings by sharing nearly all of that code — the difference, again, is ragged/unpadded attention in place of paged KV-cache attention.

CUDA graph capture: the biggest lever for small models

Launching kernels one at a time means the CPU has to issue a launch instruction for every single operation in a forward pass, and at small batch sizes that CPU-side launch overhead can dominate total latency more than the GPU compute itself. A CUDA graph captures an entire forward pass as one launchable unit, so the CPU issues a single "replay this graph" call instead of hundreds of individual kernel launches.

Perplexity found the inflection point where GPU cost starts to outweigh CPU launch overhead comes at thousands of tokens or tens of sequences for these small models — which is precisely why they capture the whole model as one graph rather than partial subgraphs. Getting there required upstreaming changes to attention kernels that previously relied on dynamic host-side inputs, which otherwise blocks graph capture outright.

They also use lazy graph capture: the first call for a given input shape runs eager (uncaptured) as a warmup pass; the second call captures the graph and replays it from then on. That spreads a multi-minute capture cost across hours of serving instead of paying it all up front at startup — a deliberate tradeoff of slightly worse p99 latency on a cold start in exchange for much faster fleet-wide startup and scaling. This is a generalizable ops lesson beyond embeddings specifically: any GPU-graph-based serving stack (see also how NVIDIA Dynamo's Shadow Engine treats cold-start cost as the thing to eliminate) has to decide where in the request lifecycle that capture cost gets paid.

LazyTensor: overlapping CPU and GPU work

The second optimization is a Rust-side async abstraction called LazyTensor. Instead of a step() call blocking until a GPU result is ready, LazyTensor tracks a pending result via page-locked host memory, a cudaMemcpyAsync call, and a CUDA event — and returns a future-like handle immediately.

The practical effect: Tulip can kick off the next batch's GPU work while it's still waiting on the CPU-side copy of the previous batch's result. That overlap of CPU and GPU work is what improves both latency and throughput simultaneously, rather than trading one for the other.

Attention backends: no single kernel wins everywhere

ROSE supports multiple attention backends for ragged attention — FlashInfer 2, FlashInfer 3, and FlashAttention 4 — because no one kernel is fastest across every case. FlashAttention 4 is generally faster, but FlashInfer 3 wins specifically on Qwen-based models at very long sequence lengths. Rather than picking one kernel and living with its worst case, Perplexity maintains multiple backends and selects per-case. This mirrors the broader pattern seen across the inference-serving landscape: kernel choice is a decision made per model family and per workload shape, not a one-time architectural commitment.

What the benchmarks actually measured

Perplexity compared their stack against vLLM v0.22.0, using BF16 precision and real model weights, across four scenarios:

That last scenario is the most representative of real production load, since it includes the CPU-side tokenization and network hops that a raw model-only benchmark skips. Perplexity's published charts on their engineering blog carry the actual numeric results — this post is describing the methodology, not restating figures it wasn't given, so go to the source for the hard numbers if you're evaluating against your own stack.

What's next: CPU protocol tuning and free-threaded Python

Perplexity says they're continuing to chase both sides of the latency budget: CPU-bound work (network protocol tuning inside Ivy and Tulip) and GPU-bound work (ROSE compute). They also flag free-threaded Python as a future lever for better Python-Rust interop, as that CPython mode matures — relevant to anyone gluing a Rust scheduler to a Python model engine the way Tulip and ROSE are wired together.

This is serving infrastructure, not model selection

Worth being explicit about: this post is about how Perplexity serves an embedding model efficiently at the infrastructure layer — not about which embedding model to pick for your own RAG pipeline. If you're choosing a model rather than architecting how to serve one, the practical companion piece is explainx.ai's top 10 open and closed-source embedding models roundup, or the more foundational what are embeddings and vector search guide if you're building your first retrieval pipeline. This post's lessons — reuse your existing inference stack, split batch and online traffic paths, use CUDA graph capture to cut launch overhead — apply once you already have a model and are trying to serve it well.

Perplexity's own Search API and their Mac hybrid-compute routing work are both downstream consumers of embedding and ranking infrastructure like this — the serving stack described here is what makes those products fast enough to ship.

FAQ

What is Perplexity's "Fast Embeddings on GPUs" post about? It's a September 4, 2026 engineering writeup describing the serving infrastructure — Ivy, Tulip, ROSE — behind Perplexity's embedding and ranking models, including pplx-embed, used across Search, Computer, and the API Platform.

Why is embedding serving a different engineering problem than LLM chat serving? Embedding traffic splits into Batch Embedding (bulk, throughput-focused, like prefill) and Online Embedding (per-query, latency-focused, like decode) — two shapes that pull infrastructure design in opposite directions.

Can I reuse my LLM inference stack to serve an embedding model? Yes, per Perplexity's own approach — dense-layer compute is identical, so only the attention path (ragged/unpadded, no KV cache) needs to differ from your LLM decoding path.

What is CUDA graph capture? Capturing an entire model forward pass as a single launchable GPU unit instead of launching each kernel individually, which matters most at small batch sizes where CPU launch overhead dominates.

What is lazy graph capture? Running the first call for a given shape eagerly as warmup, then capturing and replaying the graph from the second call onward — spreading the capture cost across hours of serving instead of paying it at startup.

Does this post include Perplexity's actual benchmark numbers against vLLM? No — it describes the benchmark methodology (BF16, real weights, four latency/throughput scenarios against vLLM v0.22.0) accurately; the numeric results live in Perplexity's own published charts.

Related reading

• What are embeddings? Vector search and semantic AI explained
• Top 10 open and closed-source embedding models (2026)
• RAG vs MCP: the complete comparison
• pplx-garden: Perplexity's open-source inference technology stack
• Perplexity Mac hybrid compute: local models for sensitive agent steps
• Perplexity Search API scores 80 in index debut
• NVIDIA Dynamo Shadow Engine: 39x faster LLM recovery
• Sentence Transformers v6.0: ColBERT late interaction for RAG
• Official source: Fast Embeddings on GPUs — Perplexity Engineering

Architecture details, benchmark configurations, and terminology in this post reflect Perplexity's engineering blog as published on September 4, 2026. Perplexity's serving stack may evolve after this date — check the official source for the latest version.