explainx / blog
Deep dive into TurboVec — the Rust-powered vector index built on Google Research's TurboQuant algorithm that shrinks AI memory requirements by 92% while beating FAISS on speed. Learn how it works, how to use it, and why it changes RAG architecture forever.
Jun 28, 2026
You have a 200-page PDF and you want to ask it questions in plain English. This step-by-step Langflow tutorial shows you how to build a working document Q&A bot in 30 minutes using drag-and-drop nodes — no code required.
Jun 21, 2026
Turso rewrites SQLite in Rust — keeping full SQL and file format compatibility while adding MVCC concurrent writes, async io_uring I/O, CDC, vector search, full-text search, and an MCP server mode. 20,000+ stars, 253 contributors, in beta. Here is the full breakdown and why it matters for the agentic era.
Jun 16, 2026
Every RAG pipeline, semantic search engine, and agent memory system is built on the same primitive: a list of floating-point numbers that encodes meaning. This guide explains embeddings from first principles — how they are trained, how similarity works mathematically, which vector databases handle them at scale, and why they remain indispensable even as context windows grow.
AI has a memory problem nobody talks about enough.
You fine-tune the model, deploy the API, and ship the product — and then the vector database bills arrive. A RAG pipeline over 10 million documents needs 31 GB of RAM just for the index. That's before your embedding server, your API layer, your caches, or your LLM inference. At scale, vector memory becomes the largest single line item in your AI infrastructure budget.
Google just shipped an answer: TurboVec.
Built on TurboQuant — Google Research's vector quantization algorithm presented at ICLR 2026 — TurboVec is an open-source vector index written in Rust with Python bindings that compresses that same 10-million-document corpus into 4 GB without sacrificing retrieval quality. And it does it while searching faster than FAISS.
This is not a marginal optimization. This is a fundamental rethinking of how vector search should work.
Every RAG system, semantic search engine, AI agent, and recommendation system ultimately depends on the same primitive: approximate nearest neighbor (ANN) search over high-dimensional embedding vectors.
The workflow is simple:
The problem is step 2. A single float32 vector at 1,536 dimensions is 6,144 bytes — about 6 KB. Multiply that by 10 million documents and you're at 61.4 GB in raw storage. Optimized indexes like FAISS reduce this, but a FAISS IndexFlatL2 still requires the full float32 representation: ~31 GB for 10M vectors at 1,536 dimensions.
This creates real-world constraints:
The industry response has been product quantization (PQ) — compressing vectors into smaller codes. FAISS ships PQ variants (IndexPQ, IndexPQFastScan) that work reasonably well. But they have a catch: they require training data.
Before you can index anything, PQ must analyze your corpus to build a "codebook" — a learned clustering of the vector space. New data can break the codebook. Changing your embedding model requires rebuilding everything from scratch.
Traditional PQ involves:
The training step adds complexity and latency to every pipeline change. If your corpus is dynamic — news articles, user documents, live data streams — PQ codebooks age poorly. You're constantly balancing index freshness against rebuild cost.
What if you could skip training entirely?
TurboQuant is a data-oblivious quantizer developed by researchers at Google Research and New York University, published at ICLR 2026 (arXiv:2504.19874). The paper proves that you can achieve near-optimal compression without ever looking at your data.
The key insight is a mathematical property of high-dimensional geometry.
Step 1: Normalize
Strip the length (norm) from each vector and store it as a single float. Now every vector is a unit direction on the high-dimensional hypersphere. Norms are tiny — one float per vector is negligible overhead.
Step 2: Random rotation
Multiply every vector by the same randomly-generated orthogonal matrix. This is a critical step. After rotation, a remarkable thing happens: every coordinate independently follows a Beta distribution that converges to Gaussian N(0, 1/d) as dimensionality increases.
This distribution is the same regardless of your input data. It doesn't matter if you're indexing medical papers, e-commerce products, legal contracts, or code. Once you rotate, the coordinate distribution is predictable from math alone.
Step 3: Lloyd-Max scalar quantization
Since the coordinate distribution is known analytically, you can precompute the optimal way to bucket each coordinate — the bucket boundaries and centroids that minimize mean squared error. This is the Lloyd-Max algorithm, applied not to empirical data but to the Beta distribution's closed-form statistics.
For 2-bit quantization: 4 buckets per coordinate. For 4-bit quantization: 16 buckets per coordinate.
These are computed once, hardcoded into the library. Zero data passes. Zero training time.
Step 4: Bit-pack
Each coordinate becomes a small integer. Pack them tightly into bytes.
A 1,536-dim vector goes from:
Search
At query time, rotate the query vector once into the same compressed domain. Score directly against codebook values using SIMD kernels — no decompression required.
The paper proves TurboQuant achieves distortion within 2.7× of the Shannon information-theoretic lower bound — you literally cannot do meaningfully better with any quantizer for a given bit budget.
For practitioners, the implications are profound:
# Traditional PQ workflow
index = faiss.IndexPQ(dim, M, nbits)
index.train(training_vectors) # ← requires training data, minutes/hours
index.add(all_vectors)
# TurboVec workflow
from turbovec import TurboQuantIndex
index = TurboQuantIndex(dim=1536, bit_width=4)
index.add(vectors) # ← no training step, just add
No training means:
TurboVec is the open-source implementation of TurboQuant as a production-grade vector index, written by Ryan Codrai. It ships as both a Rust crate and a Python package, and integrates natively with LangChain, LlamaIndex, and Haystack.
# Python
pip install turbovec
# Rust
cargo add turbovec
from turbovec import TurboQuantIndex
import numpy as np
# Create an index for 1536-dim vectors (e.g., OpenAI text-embedding-3-small)
index = TurboQuantIndex(dim=1536, bit_width=4)
# Add vectors — no training required
vectors = np.random.randn(10_000_000, 1536).astype(np.float32)
index.add(vectors)
# Search
query = np.random.randn(1, 1536).astype(np.float32)
scores, indices = index.search(query, k=10)
# Persist to disk
index.write("my_index.tq")
# Load later
loaded = TurboQuantIndex.load("my_index.tq")
For production use cases where documents are updated or deleted, TurboVec provides IdMapIndex — a wrapper that maps your external IDs to internal indices and supports O(1) deletes:
from turbovec import IdMapIndex
import numpy as np
index = IdMapIndex(dim=1536, bit_width=4)
# Add with your external IDs (e.g., database primary keys)
vectors = np.random.randn(1000, 1536).astype(np.float32)
doc_ids = np.array([1001, 1002, ..., 2000], dtype=np.uint64)
index.add_with_ids(vectors, doc_ids)
# Search returns your external IDs
scores, ids = index.search(query, k=10)
print(ids) # [1047, 1312, ...]
# Delete a document — no rebuild needed
index.remove(1312)
# Persist
index.write("my_index.tvim")
use turbovec::TurboQuantIndex;
let mut index = TurboQuantIndex::new(1536, 4);
index.add(&vectors);
let results = index.search(&queries, 10);
index.write("index.tv").unwrap();
let loaded = TurboQuantIndex::load("index.tv").unwrap();
TurboVec plugs into the major RAG frameworks with one-line installs:
pip install turbovec[langchain] # LangChain integration
pip install turbovec[llama-index] # LlamaIndex integration
pip install turbovec[haystack] # Haystack integration
# LangChain drop-in
from turbovec.langchain import TurboVecVectorStore
from langchain_openai import OpenAIEmbeddings
vectorstore = TurboVecVectorStore(
embedding=OpenAIEmbeddings(),
dim=1536,
bit_width=4
)
vectorstore.add_documents(documents)
docs = vectorstore.similarity_search("your query", k=5)
The headline number: 31 GB → 4 GB for 10 million 1,536-dim float32 vectors at 4-bit quantization.
| Configuration | Memory (10M vectors, d=1536) | Compression |
|---|---|---|
| Float32 (raw) | 61.4 GB | 1× |
| FAISS IndexFlatL2 | 61.4 GB | 1× |
| FAISS IndexPQFastScan (4-bit) | ~7.7 GB | ~8× |
| TurboVec (4-bit) | ~4.0 GB | ~15× |
| TurboVec (2-bit) | ~2.0 GB | ~30× |
The compression gains over FAISS PQ come from TurboQuant's more efficient bit-packing and the fact that no codebook storage is required.
TurboVec uses hand-written NEON intrinsics for ARM processors, with a nibble-split lookup table approach for maximum throughput.
Benchmarks: 100K vectors, 1K queries, k=64, median of 5 runs.
| Config | TurboVec (single-thread) | FAISS FastScan | Speedup |
|---|---|---|---|
| d=1536, 4-bit | faster | baseline | +12–20% |
| d=3072, 4-bit | faster | baseline | +12–20% |
| d=1536, 2-bit | faster | baseline | +12–20% |
On ARM, TurboVec beats FAISS IndexPQFastScan across every configuration tested.
TurboVec uses AVX-512BW kernels on modern x86 processors, with an AVX2 fallback for older hardware. Runtime feature detection via is_x86_feature_detected! — no recompilation needed.
| Config | TurboVec vs FAISS |
|---|---|
| 4-bit, single-thread | +1–6% (wins) |
| 4-bit, multi-thread | +1–6% (wins) |
| 2-bit, single-thread | within ~1% (ties) |
| 2-bit, multi-thread | -2–4% (narrow loss) |
The 2-bit multi-thread loss on x86 is a known limitation — the inner accumulate loop is too short for unrolling amortization to match FAISS's AVX-512 VBMI path. For most production workloads (4-bit is the recommended default), TurboVec is competitive or better across the board.
TurboQuant vs FAISS IndexPQ (LUT256, nbits=8) — 100K vectors, k=64.
On OpenAI embeddings (d=1536, d=3072):
On GloVe (d=200 — a harder, lower-dimensional regime):
Bottom line: For modern embedding models at 1,536+ dimensions, TurboQuant matches FAISS quality while using 2–4× less memory and running faster. For very low-dimensional embeddings, FAISS PQ maintains a quality edge.
Here's a complete example of a fully local, air-gapped RAG system using TurboVec — no managed services, no cloud APIs, no data leaving your machine.
pip install turbovec[langchain] sentence-transformers langchain
import numpy as np
from pathlib import Path
from sentence_transformers import SentenceTransformer
from turbovec import TurboQuantIndex
class LocalRAG:
def __init__(
self,
embedding_model: str = "all-MiniLM-L6-v2",
bit_width: int = 4,
index_path: str = "knowledge_base.tq"
):
self.embedder = SentenceTransformer(embedding_model)
self.dim = self.embedder.get_sentence_embedding_dimension()
self.bit_width = bit_width
self.index_path = index_path
self.documents = []
# Load existing index or create fresh
if Path(index_path).exists():
self.index = TurboQuantIndex.load(index_path)
print(f"Loaded existing index ({len(self.documents)} docs)")
else:
self.index = TurboQuantIndex(dim=self.dim, bit_width=bit_width)
print(f"Created new index (dim={self.dim}, {bit_width}-bit)")
def ingest(self, texts: list[str], batch_size: int = 512):
"""Embed and index documents in batches."""
for i in (, (texts), batch_size):
batch = texts[i:i + batch_size]
embeddings = .embedder.encode(
batch,
normalize_embeddings=,
show_progress_bar=
).astype(np.float32)
start_id = (.documents)
.index.add(embeddings)
.documents.extend(batch)
()
.index.write(.index_path)
()
() -> []:
query_vec = .embedder.encode(
[query],
normalize_embeddings=
).astype(np.float32)
scores, indices = .index.search(query_vec, k=k)
results = []
score, idx (scores[], indices[]):
idx < (.documents):
results.append({
: .documents[idx],
: (score),
: (idx)
})
results
rag = LocalRAG(embedding_model=, bit_width=)
() f:
documents = [line.strip() line f line.strip()]
rag.ingest(documents)
results = rag.search(, k=)
r results:
()
For a 10M-document corpus with 1,536-dim embeddings:
Float32 baseline: 61.4 GB RAM
FAISS FlatL2: 61.4 GB RAM
FAISS PQ (4-bit): ~7.7 GB RAM ← requires training phase
TurboVec (4-bit): ~4.0 GB RAM ← zero training, instant add
TurboVec (2-bit): ~2.0 GB RAM ← fits on a Mac mini
A Mac mini M4 with 16 GB RAM can now serve a 10M-document RAG system entirely in memory, with room left over for the embedding model and LLM inference.
For practitioners who want to understand the theory before trusting the library, here's the math at a workable depth.
Any lossy compression scheme for vectors trades off distortion against rate (bits per dimension). Shannon's rate-distortion theory tells us the theoretical minimum distortion achievable at a given bit budget. No algorithm can beat it — but many algorithms fall far short.
PQ-family methods (including FAISS PQ) are data-dependent: they fit the quantizer to the distribution of your specific corpus. This adaptation helps, but it introduces training cost and corpus lock-in.
TurboQuant's insight: for unit vectors in high dimensions, the post-rotation coordinate distribution is universal. You don't need data to characterize it. The Beta distribution that emerges after random rotation is the same regardless of what corpus you embed.
This means the Lloyd-Max quantizer can be derived from first principles once, baked into the library, and applied forever without retraining.
TurboQuant has a two-stage heritage:
Together they achieve the near-optimal distortion bound. The ICLR 2026 paper proves TurboQuant operates within a factor of ≈2.7 of the Shannon limit across all bit-widths and dimensions — meaning you're not leaving meaningful quality on the table.
The scoring kernel is where TurboVec's performance advantage lives. The key operation is: given a compressed query and compressed database vectors, compute approximate inner products as fast as possible.
TurboVec uses nibble-split lookup tables — the 4-bit code for each dimension is split into two 2-bit halves, each of which is scored against a precomputed 4-entry table. This maps perfectly onto NEON's vtbl instruction on ARM, which does 8 parallel table lookups in a single cycle.
On x86, AVX-512BW provides a similar vpshufb instruction. TurboVec's AVX-512 kernel adapts FAISS FastScan's pack layout and u16 accumulator strategy, which is why x86 performance is closely competitive with FAISS rather than blowing past it.
The ARM advantage comes from M-series chips' high NEON throughput and the fact that FAISS's x86 path is extremely well-tuned while its ARM path has historically received less attention.
If you're already on FAISS, TurboVec is designed as a drop-in:
# Before — FAISS
import faiss
index = faiss.IndexPQFastScan(dim, M, nbits)
index.train(training_data)
index.add(vectors)
D, I = index.search(query, k)
# After — TurboVec (no training step, better memory)
from turbovec import TurboQuantIndex
index = TurboQuantIndex(dim=dim, bit_width=4)
index.add(vectors) # same vectors, no training
scores, I = index.search(query, k=k)
For regulated industries (healthcare, finance, government), TurboVec's local-only architecture is a hard requirement:
# All processing stays on your hardware
from turbovec import TurboQuantIndex
# No API calls, no managed services
index = TurboQuantIndex(dim=1536, bit_width=4)
index.add(your_proprietary_embeddings)
results = index.search(query_embedding, k=10)
# Persist to encrypted volume
index.write("/mnt/encrypted/knowledge_base.tq")
TurboVec's zero-training architecture enables true streaming updates:
from turbovec import IdMapIndex
from kafka import KafkaConsumer
import json
index = IdMapIndex(dim=1536, bit_width=4)
consumer = KafkaConsumer("document-embeddings")
for message in consumer:
doc = json.loads(message.value)
embedding = np.array(doc["embedding"], dtype=np.float32)
doc_id = np.uint64(doc["id"])
if doc.get("deleted"):
index.remove(doc_id) # O(1) delete
else:
index.add_with_ids(embedding.reshape(1, -1), np.array([doc_id]))
# Checkpoint periodically
if message.offset % 10_000 == 0:
index.write("index.tvim")
For edge devices, IoT gateways, or Raspberry Pi deployments:
# 2-bit mode: 30x compression from float32
# Fits 1M docs in ~400 MB — viable for edge devices
from turbovec import TurboQuantIndex
index = TurboQuantIndex(dim=384, bit_width=2) # smaller embedding model too
index.add(corpus_embeddings)
# 1M × 384-dim vectors = 1.5 GB float32 → ~50 MB at 2-bit
# Runs on a Raspberry Pi 5 (8GB model)
TurboVec supports search-time filtering to restrict results to a subset of documents:
from turbovec import IdMapIndex
index = IdMapIndex(dim=1536, bit_width=4)
# Add documents with metadata (tracked externally)
for doc_id, embedding, metadata in documents:
index.add_with_ids(embedding, np.array([doc_id]))
doc_metadata[doc_id] = metadata
# Filter to only approved documents at search time
approved_ids = get_approved_document_ids(user_context)
scores, ids = index.search(query, k=100, filter_ids=approved_ids)
# Get top-k after filtering
results = [(scores[i], ids[i]) for i in range(len(ids)) if ids[i] in approved_ids][:10]
The AI industry has spent five years competing on scale: bigger models, larger context windows, more GPUs, bigger data centers. TurboVec represents a quieter but equally important trend: doing more with less.
The math matters here. A 92% memory reduction doesn't just make existing systems cheaper — it changes what's architecturally possible:
| Before TurboVec | After TurboVec |
|---|---|
| 10M docs requires dedicated 32GB server | 10M docs fits on a MacBook Pro |
| Private RAG needs $500/month cloud VM | Private RAG runs on local hardware |
| Real-time index updates require careful PQ retraining | Streaming updates with zero rebuild cost |
| Filtering requires over-fetch + post-filter | Native search-time filtering |
| Air-gap deployment = small knowledge base | Air-gap deployment = production-scale knowledge base |
The standard RAG pattern has been:
TurboVec makes a compelling case for self-hosted vector search at scale:
Managed vector DB (10M docs): ~$100–500/month
TurboVec on a VPS with 8GB RAM: ~$20–40/month
TurboVec on existing infrastructure: $0/month
This doesn't mean managed vector databases disappear — they offer persistence, replication, filtering, and operational simplicity that TurboVec doesn't provide out of the box. But for teams that already operate infrastructure and value data privacy, the calculus shifts.
While TurboVec targets vector search, TurboQuant's paper covers a broader application: KV cache quantization for large language model inference.
Attention's KV cache is itself a matrix of vectors — one per token, per layer, per head. At 128K context windows on large models, the KV cache alone can consume 10–20 GB of GPU memory.
TurboQuant achieves:
This means TurboQuant could compress the KV cache of a model running at 128K context from ~16 GB to ~2.7 GB while maintaining generation quality — dramatically expanding what fits in a given GPU budget.
The most underappreciated consequence of TurboVec is what it does for accessibility.
Right now, the teams who can run serious RAG systems are the ones who can afford serious infrastructure. A 10M-document knowledge base over proprietary company data — legal documents, customer records, internal wikis — costs real money to query at scale.
With TurboVec, a two-person startup can run that same system on a $40/month VPS, with data never leaving their infrastructure. A research lab can run 100M-document corpus experiments on a single workstation. A healthcare startup can build an air-gapped clinical knowledge base without HIPAA-constrained cloud infrastructure.
TurboVec is impressive, but it's not a universal replacement for all vector search infrastructure. Here's what to keep in mind:
The theoretical guarantees of TurboQuant rely on the Beta distribution approximation holding in high dimensions. For embeddings below ~256 dimensions (like GloVe d=200), the approximation is looser:
If you're using older, smaller embedding models, test recall carefully before deploying TurboVec in production.
TurboVec is a single-node library. It doesn't provide:
For massive corpora (>1B vectors) or high-availability requirements, managed vector databases or distributed systems like Milvus remain necessary. TurboVec is best for single-machine workloads — which, given 4 GB for 10M vectors, covers a lot of ground.
TurboVec implements flat quantized search (exhaustive search over compressed vectors), not HNSW (Hierarchical Navigable Small World graphs). HNSW offers sub-linear search time and is better for very large corpora with strict latency SLAs.
At 10M vectors, flat quantized search is fast enough for most applications. At 100M+ vectors, you'd want HNSW or IVF indexing on top of TurboQuant — which may come in future releases.
As noted in the benchmarks, TurboVec is 2–4% slower than FAISS on 2-bit multi-threaded x86 workloads. If you're on x86 and need maximum multi-threaded throughput at 2-bit, benchmark carefully. The 4-bit configuration is recommended for x86 production deployments.
Here's a decision tree for evaluating TurboVec for your use case:
Use TurboVec if:
Stick with your current solution if:
Before migrating, run this recall check against your own data:
import numpy as np
from turbovec import TurboQuantIndex
import faiss
dim = 1536
n = 100_000
# Generate or use your real embeddings
vectors = np.random.randn(n, dim).astype(np.float32)
faiss.normalize_L2(vectors)
queries = np.random.randn(100, dim).astype(np.float32)
faiss.normalize_L2(queries)
# Ground truth from exact search
flat_index = faiss.IndexFlatIP(dim)
flat_index.add(vectors)
_, gt = flat_index.search(queries, k=10)
# TurboVec at 4-bit
tv_index = TurboQuantIndex(dim=dim, bit_width=4)
tv_index.add(vectors)
_, tv_results = tv_index.search(queries, k=10)
# Compute recall@10
recall = np.mean([
len(set(tv_results[i]) & set(gt[i])) / 10
for i in range(len(queries))
])
print(f"TurboVec recall@10: {recall:.4f}")
# Expected: 0.90+ for d=1536
The past five years of AI progress have been measured in parameters. The next five may be measured in efficiency.
TurboVec is a clear signal that the compression frontier is just as important as the capability frontier. Google Research proved that you can derive near-optimal vector quantization from mathematics alone — no data, no training, no codebooks. Ryan Codrai built that into a production-grade Rust library with Python bindings that ships tomorrow.
The headline numbers are striking:
But the deeper implication is about access. The organizations that can now run serious vector search aren't just the ones with $10,000/month infrastructure budgets. They're the two-person startup on a VPS, the research lab on a workstation, the healthcare company that can't put patient data in the cloud.
Vector search at scale is no longer an enterprise-only capability.
Library:
pip install turbovecPapers:
Framework Integrations: