SubQ: SSA sparse attention, 12M context, and long-context evals
Subquadratic’s SubQ pairs a sub-quadratic sparse-attention stack (SSA) with a 12M-token positioning; official SSA post cites 52× prefill at 1M vs dense FA on B200s.
Subquadratic is shipping SubQ, an LLM positioned as the first fully sub-quadratic sparse-attention stack—branded SSA (Subquadratic Sparse Attention) in their technical article. The homepage advertises 12M-token reasoning, ~$1/5relative cost framing versus leading LLMs (comparison axes unstated), and two surfaces: a full-context API and SubQ Code for existing coding agents.
This article summarizes what Subquadratic publishes—not independent verification of every throughput or eval cell.
TL;DR
Topic
Takeaway
Architecture claim
SSA = content-dependent sparse attention; aims to avoid O(n²)all-pairsdense work as context grows
Context positioning
Public 12M-tokenreasoning window on subq.ai; technical post emphasizes functional vs nominal long context
SubQ Code marketed as a drop-in layer for Claude Code, Codex, Cursor with cost/latency claims
Disclosure gap
Model card and technical report described as coming—reasonable to wait before hard architecture commitments
Why long context is the stated problem
Subquadratic’s SSA article argues enterprise failure modes are usually distributed evidence: codebases, contracts, corpora, spreadsheets, and long-runningagent transcripts where answers require cross-references, not single-span lookup.
That distinction matches how practitioners already read needle-style benchmarks versus realrepo tasks—see also our LLM context window guide.
What SSA claims technically
In their narrative, dense attention pays quadratic cost even when most pairwise scores are negligible. FlashAttention improves execution of the sameall-pairs workload; SSA instead selects which positions receive exact attention.
Reported wall-clock prefill speedups vs their dense FlashAttention baseline on B200s (from their table):
Context
Speedup
128K
7.2×
256K
13.2×
512K
23.0×
1M
52.2×
They also publish attention FLOP reduction multiples vs standard quadratic attention, including 62.5× at 1M tokens.
Caveat: these are vendor-published numbers pending broader independent reproduction in your hardware/stack.
Benchmarks they highlight (read the tables in-source)
The technical page includes comparison tables against several namedfrontier models for:
RULER @ 128K
MRCR v2 (8-needle, 1M)
SWE-Bench Verified
Interpretation discipline:
MRCR is explicitly hard on multi-evidence retrieval; a mid-pack score can still accompany strong SWE if tasks differ.
SWE-Bench is not pure retrieval; it stresses patch quality and repo interaction assumptions.
If you run agent harnesses, pair model claims with agent harness engineering thinking: tool surface, traces, and evals dominate ship quality.
Doubling context to 2M tokens quadruples FLOPs to 16.4T and doubles KV memory to 17.2GB. This scaling makes million-token contexts impractical on consumer hardware and expensive even on datacenter GPUs.
Why FlashAttention Helps (But Not Enough)
FlashAttention (Dao et al., 2022) and FlashAttention-2 (2023) optimize the execution of quadratic attention by:
Tiling computations to fit in SRAM (reducing HBM round-trips)
Fusing operations (softmax, dropout, masking) into single kernels
Recomputing attention scores during backward pass to save memory
FlashAttention achieves 2-4x speedups and enables longer contexts than naive implementations, but it's still O(N²) in FLOPs—the fundamental quadratic cost remains.
At 12M tokens (SubQ's marketed window), dense FlashAttention would require:
~144 trillion FLOPs per forward pass (12M²)
~100GB+ KV cache (depending on model architecture)
This is why Subquadratic claims SSA is necessary for functional long context rather than just nominal capability.
SSA Technical Architecture: How It Works
While Subquadratic hasn't published a full academic paper (as of May 2026), their technical blog describes SSA's core mechanisms:
Content-Dependent Sparse Selection
Instead of computing attention for all N² token pairs, SSA selects a subset of relevant keys for each query based on content similarity.
Step 1: Query-Key Similarity Scoring
For each query token q_i, compute approximate similarity scores with all keys k_j using a fast hashing or clustering technique (likely Locality-Sensitive Hashing or k-nearest neighbors in embedding space).
Step 2: Top-K Selection
Retain only the top K most similar keys for full attention computation, where K << N. For example:
At 1M tokens, K might be 1,000 (0.1% sparsity)
At 12M tokens, K might be 10,000 (0.08% sparsity)
Step 3: Full Attention on Subset
Compute exact attention scores only for the K selected keys, reducing FLOPs from O(N²) to O(N × K).
Step 4: Residual Dense Attention
To prevent catastrophic forgetting of important but low-similarity tokens (e.g., structural tokens like section headers), SSA likely includes a fallback dense attention layer at coarser granularity or on downsampled representations.
Adaptive Sparsity
SubQ's claim of "content-dependent" sparsity suggests the sparsity level adapts per query:
Queries in dense information regions (e.g., middle of a paragraph) may attend to fewer keys
Queries at boundaries (e.g., start of a new section) may attend to more keys to gather context
This is more sophisticated than fixed sparse patterns (e.g., sparse transformers with fixed strides) and enables better recall on retrieval tasks.
Memory Optimization
Beyond FLOPs, SSA must also reduce KV cache memory. Techniques likely include:
Quantization: Storing keys and values in lower precision (INT8 or FP8 instead of FP16)
Pruning: Evicting low-attention keys from cache after each layer
Compression: Using learned compression codebooks (similar to vector quantization in CodeLLaMA)
Combined, these allow SubQ to fit 12M tokens in memory without requiring 100GB+ of VRAM.
Benchmark Analysis: What the Numbers Actually Mean
Subquadratic publishes three key benchmarks:
RULER @ 128K
RULER (Realistic Ultra-Long Context Evaluation) tests retrieval, reasoning, and aggregation across long sequences. At 128K tokens, SubQ's reported score is competitive with frontier models (exact numbers in Subquadratic's table).
What this means: SSA doesn't degrade significantly from dense attention at moderate context lengths, validating that sparse selection isn't "lossy" for typical reasoning tasks.
MRCR v2 (8-Needle, 1M Tokens)
MRCR (Multi-Needle Retrieval in a Haystack) is explicitly hard: find and connect 8 facts scattered across 1M tokens of distractor text.
SubQ's reported performance here is lower than on RULER, which Subquadratic frames as expected for multi-evidence retrieval at extreme scale. The key question: How much lower?
If SubQ scores 60-70% on MRCR vs. 85-90% on RULER, that's a significant gap. If it scores 75-80%, that's more acceptable. The exact numbers matter for production risk assessment.
Interpretation: MRCR is a stress test, not a typical use case. Most real-world agent tasks don't require pinpoint retrieval across 1M tokens—they involve reasoning over ~100K-500K token codebases or document sets where SSA's trade-offs are more favorable.
SWE-Bench Verified
SWE-Bench Verified measures code editing on real GitHub issues. SubQ's reported score (exact value in Subquadratic's table) is compared to GPT-4, Claude 3.5, and Gemini 2.0.
What this means: SWE-Bench isn't purely a long-context benchmark—it tests code understanding, reasoning, and patch generation. A competitive score here suggests SSA's sparsity doesn't hurt the model's core coding capabilities.
Caveat: SWE-Bench solutions often fit in <50K tokens, so this benchmark doesn't directly validate 12M-token performance. It confirms that SubQ is a competent coding model, but not that it's superior to alternatives because of long context.
Real-World Use Cases: Where 12M Tokens Matters
When would you actually use a 12M-token context window?
1. Codebase-Wide Refactoring
Scenario: You need to rename a function used across 500+ files in a monorepo.
Why 12M helps: Loading the entire codebase (5-10M tokens) into context allows the model to find all call sites, understand dependencies, and generate a coherent multi-file patch—without RAG chunking or vector search.
Alternative: Use grep + targeted edits. But SSA enables one-shot reasoning across the full dependency graph, catching edge cases that chunked approaches miss.
2. Legal Document Analysis
Scenario: A law firm needs to cross-reference a 3,000-page contract against 500 pages of regulatory text.
Why 12M helps: Loading both documents (combined ~1-2M tokens) allows the model to identify conflicts, cite specific clauses, and reason about legal implications without manual chunking.
Alternative: Traditional document review with keyword search. But SSA enables semantic reasoning ("Does clause 42 contradict section 18(b)?") that keyword search cannot.
3. Multi-Document Research Synthesis
Scenario: A researcher needs to summarize findings across 50 academic papers (combined ~500K tokens).
Why 12M helps: The model can read all papers in one pass, identify common themes, and generate a meta-analysis—avoiding the bias introduced by sequential chunking.
Alternative: Read each paper separately and manually synthesize. But SSA accelerates the process and reduces human error.
4. Agent Trace Debugging
Scenario: An agent runs for 100 turns, accumulating 2M tokens of conversation, tool calls, and outputs, then fails. You need to debug why.
Why 12M helps: Loading the full trace allows the model to backtrack, identify the failing decision point, and propose fixes—without truncating early context.
Alternative: Use logging and manual analysis. But SSA enables automated root-cause analysis at scale.
Economic Analysis: Is SubQ Actually 1/5 the Cost?
Subquadratic's homepage claims "1/5 the cost of leading LLMs" for target workloads. Let's unpack this:
Cost Drivers in LLM Inference
Total cost per request depends on:
FLOPs per token: Compute cost scales with model size and attention mechanism
Memory footprint: KV cache size determines max batch size and GPU utilization
Latency requirements: Faster responses require more expensive GPUs or larger batch sizes
Caching: Prompt caching can amortize prefill costs for repeated prefixes
SubQ's Claimed Advantages
52x prefill speedup at 1M tokens: If dense attention takes 10s, SSA takes ~0.2s—reducing GPU-hours and enabling higher throughput
Reduced KV memory: Smaller cache allows larger batch sizes, improving GPU utilization from ~30% (typical for long-context models) to ~60-70%
Lower FLOPs: Sparse attention reduces total compute per token, allowing cheaper GPUs (e.g., A100 vs. H100) for same throughput
Quality: If SSA requires more retries to achieve desired output, cost-per-success increases
When SubQ is Cost-Effective
SubQ's cost advantage is largest when:
Context lengths routinely exceed 500K tokens
Requests involve fresh, un-cached contexts (e.g., novel documents per request)
Batch sizes are high (benefiting from reduced KV memory)
SubQ is less cost-effective when:
Most requests are <100K tokens (sparse vs. dense FLOPs converge)
Heavy prompt caching is feasible (reducing prefill as a cost driver)
Ultra-low latency is required (SSA's adaptive selection adds overhead)
SubQ Code: Integration with Existing Agents
Subquadratic markets SubQ Code as a drop-in upgrade for existing coding agents (Claude Code, Codex, Cursor). How does this work?
Architecture
SubQ Code likely operates as a middleware layer that:
Intercepts requests: When a coding agent makes an API call, SubQ Code captures the prompt and context.
Routes intelligently: For short prompts (<100K tokens), SubQ Code may forward to the original provider (e.g., GPT-4). For long prompts (>500K tokens), it routes to SubQ's API.
Auto-redirects expensive turns: If a request would cost >$X on the default provider, SubQ Code substitutes SubQ and passes the result back.
Caching and optimization: SubQ Code may cache SubQ responses to avoid redundant long-context processing.
Integration Example
python
# Original Claude Code-style agent
response = client.messages.create(
model="claude-sonnet-4",
messages=[{"role": "user", "content": long_codebase + task}]
)
# With SubQ Code middleware
response = subq_code_client.messages.create(
model="claude-sonnet-4", # Fallback if context is short
messages=[{"role": "user", "content": long_codebase + task}],
subq_threshold=500_000, # Use SubQ for >500K tokens
)
Claimed Benefits
25% cost reduction: By routing long-context requests to SubQ instead of expensive dense-attention models
Faster exploration: SubQ's prefill speedup allows agents to iterate more quickly on large codebases
Seamless migration: No changes to agent prompts or workflows—just swap the client
Skepticism and Validation
Before adopting SubQ Code:
Measure actual cost savings: Track real-world spending before and after migration
Test quality on your tasks: SSA's sparsity may degrade performance on domain-specific code patterns
Monitor latency: Adaptive sparse selection adds overhead—ensure it doesn't hurt user experience
Check SLAs: SubQ is a new provider—validate uptime, rate limits, and support responsiveness
Challenges and Limitations
Despite promising benchmarks and cost claims, SubQ faces several hurdles:
1. Lack of Independent Validation
Subquadratic's benchmarks are self-reported. Independent replications by:
AI research labs (e.g., EleutherAI, HuggingFace)
Enterprise users (e.g., tech companies running their own evals)
Academic institutions (e.g., Stanford, MIT)
...are needed to confirm the 52x speedup and 1/5 cost claims.
Marketing ratios, benchmark conditions, and API availability change quickly. Treat this as May 6, 2026 context and reconcile numbers against Subquadratic's published tables and your own measurements before production budgets or architecture reviews.
