Nari Labs Hits Sub-50ms TTS at $2 per Million Characters

Most TTS latency conversations stop at "how fast is the model." Nari Labs' August 19, 2026 write-up, Pushing the Speed-Cost Frontier for Qwen3-TTS, argues the model was never the bottleneck — the serving stack around it was. Their Qwen3-TTS 1.7B CustomVoice implementation hits 10 requests/second with sub-50ms p95 time-to-first-audio (TTFA) on a single NVIDIA H100 SXM, and stays under 100ms even at 20 RPS under Poisson open-loop traffic — the load pattern that actually punishes queuing behavior instead of flattering a demo.

The post hit 100 points on Hacker News with a substantive technical thread, including the Nari Labs team answering follow-ups directly. This matters beyond TTS specifically: the scheduling techniques here — treating heterogeneous model stages as independently schedulable work, deadline-aware priority instead of strict FIFO, state-cached incremental decoding — are transferable lessons for anyone serving real-time voice agents or other multimodal inference pipelines.

Nari Labs is the YC-backed team behind Dia, an open TTS model with 2M+ downloads and the #1 spot on Hugging Face's TTS leaderboard at various points. The team includes ex-YC, ex-KRAFTON, and ex-NAVER engineers, with NeurIPS/ICLR publications and IOI/ICPC gold medals among them.

The headline numbers

The $2/1M-character figure is a compute-cost floor, not a delivered price — it excludes networking, idle GPU time, and ops overhead. Even generously padded for those, it's a different order of magnitude than the commercial APIs it's compared against.

Why default serving engines don't get you there

Nari Labs benchmarked four other serving paths against their own: vLLM-Omni, SGLang-Omni (flagged as having incomplete upstream Qwen3-TTS support at test time), VoxServe, and a research system referred to as "M*". At 1 RPS with each engine's default configuration, p95 TTFA looked like this:

After Nari Labs applied its own tuning — leading-silence trim and frame-accumulation tuning — to each competitor's engine, every one of them improved substantially at low concurrency (vLLM-Omni went from 277.9ms to 56.8ms at 1 RPS; VoxServe to 49.3ms; SGLang-Omni to 120.9ms; M* to 104.0ms). But all four still degraded past roughly 100ms by 6 RPS (vLLM-Omni climbing to 93.5ms, VoxServe to 363.2ms, SGLang-Omni to 273.7ms, M* to 179.5ms), while Nari Labs' own engine held sub-50ms all the way through 10 RPS. Tuning the existing engines closed most of the gap at 1 RPS; it did not close the gap under real concurrency.

The five techniques, explained

1. Leading-silence trimming

Qwen3-TTS, like most autoregressive TTS models, tends to emit a short stretch of near-silent audio before actual speech starts. Nari Labs runs short RMS (root-mean-square amplitude) windows over the generated samples in real time and dynamically detects where that silent lead-in ends, then starts streaming from there instead of from sample zero. This is a pure client-perceived latency win — it doesn't change model inference speed at all — worth roughly 80ms of TTFA on its own.

2. Frame accumulation tuning

TTS models generate audio as a sequence of codec frames, and a server has to decide how many frames to batch together before releasing a chunk to the client. Release too small a chunk and you get lower TTFA but risk playback underruns (the client's audio buffer runs dry) and pay more per-chunk decoder overhead; release too large a chunk and you batch efficiently but delay the very first audio the listener hears. Nari Labs ramps: start with small chunks to minimize TTFA on the first packet, then grow chunk size as the request continues and the buffer has more headroom.

3. A unified scheduler across three model modules

This is the structural change that makes the other techniques possible. Qwen3-TTS is architecturally three separate modules:

• Talker — predicts the first codebook token for each audio frame
• Code Predictor — generates the remaining 15 codebook tokens per frame
• Codec — converts codebook tokens into an audio waveform (this stage is causal)

Most serving implementations run Talker and Code Predictor together as one stage, then Codec as a second, separate stage — a 2-stage pipeline. Nari Labs instead exposes all three modules as independently schedulable tasks under a single scheduler, an approach inspired by the "M*" research paper. The payoff: the scheduler can interleave and reorder work across all three stages by urgency, rather than being locked into whatever order a fixed 2-stage pipeline dictates. Bundling Talker and Code Predictor together creates a non-preemptible unit of work — once that combined step starts, nothing more urgent can cut in front of it, even if another request's playback buffer is about to run dry. Splitting all three stages apart removes that failure mode.

4. Deadline-aware priority scheduling

The scheduler treats a request differently depending on where it is in its lifecycle. Before the first audio chunk goes out, every millisecond of delay is visible to the listener — this is the TTFA regime, and it gets maximum urgency. After the first chunk has shipped, a subsequent chunk only needs to land before the client's buffered audio runs out — a playback deadline, not a race for raw speed. The scheduler picks an "anchor" request that's most urgent right now and batches other compatible work around it, which is how it keeps both responsiveness (for the anchor) and GPU batching efficiency (for everything scheduled alongside it).

5. Code Predictor: CUDA graphs for a fixed-shape loop

The Code Predictor always runs exactly 15 steps per frame — a fixed, regular structure that's unusually friendly to aggressive optimization. Nari Labs preallocates its KV cache and captures the entire per-frame generation loop as a single CUDA graph, which removes CPU-side kernel-launch overhead that would otherwise accumulate across those 15 steps. They pair this with a Triton attention kernel specialized for the Code Predictor's short, bounded context window.

6. Codec: cached state instead of full replay

The Codec was rebuilt around cached state. Instead of replaying a request's entire utterance history through the Codec on every new chunk, Nari Labs retains the Transformer context and CNN convolutional state per request across chunks, so incremental decoding only has to process the newly arrived frames. Full decoding is still used for the very first audio chunk of a request, because initializing the state cache has its own overhead that would actually hurt TTFA if paid on every chunk — after that first chunk, it switches to the fast, state-cached incremental path.

Smaller wins that add up

A few additional details from the post round out the system: CUDA graphs are captured for a fixed set of batch sizes, and a batch that exceeds the largest captured size gets split across scheduling turns rather than falling back to slow eager-mode execution. The server also avoids unnecessary CPU-GPU synchronization by deferring EOS (end-of-sequence) termination checks while EOS is suppressed, so the CPU can keep submitting work instead of stalling on a sync point. And Qwen3-TTS supports streaming LLM tokens directly into the TTS model, so synthesis can start before an upstream LLM has finished generating its full response — directly relevant to anyone building a voice agent pipeline.

What Nari Labs' team said on Hacker News

The top comment came from a Nari Labs team member (posting as "toebee"), who summarized the post and then answered a stream of practitioner follow-ups directly — the kind of Q&A that's usually more useful than the original post for anyone deciding whether to actually deploy this.

Consumer GPU, not just H100. Asked whether any of this works outside data-center hardware, toebee confirmed an RTX 4090 handles around 10 concurrent requests at roughly 50ms TTFA after config changes — that card has no FP8 support, so it's a different numeric path than the H100 benchmarks, not a like-for-like repeat. Older Ampere (RTX 30-series) cards would likely need actual code changes to support the custom CUDA kernels, not just configuration tuning.

Quality parity, self-reported. On whether all this optimization degrades voice quality, toebee said the team continuously compared output against Qwen's original reference implementation throughout development and saw no regression. Worth stating plainly: that's the team's own claim, not an independent audit — treat it as a starting point for your own listening tests, not a settled fact.

Pretraining to inference infra, deliberately. Asked by commenter kamranjon whether Nari Labs has shifted focus from pretraining (the work that produced Dia) to inference and serving, toebee confirmed it directly: yes, the focus is now inference plus fine-tuning on top of open models, with no plans to return to pretraining.

LLM time-to-first-token is still a separate, open bottleneck. The most substantive thread came from commenter zuzululu, who made a point worth internalizing if you're building a voice assistant rather than a TTS demo: raw TTS TTFA is not the whole latency budget. Unless your upstream LLM emits speech tokens directly, you still pay full LLM inference latency before TTS synthesis even begins — so hitting a "conversationally invisible" total latency bar (zuzululu cited under ~150ms) requires solving LLM time-to-first-token too, not just TTS TTFA. toebee agreed this is a real, separate problem: Qwen3-TTS's streaming/websocket mode (feeding LLM tokens into the TTS model as they're generated) addresses part of it by letting synthesis start before the LLM finishes its full response, but LLM TTFT itself is still open and something they might tackle next.

On-device is unexplored territory. Commenter armcat, who has built their own voice assistant and tried Pocket TTS, Chatterbox, and Fish Audio S2 Pro, pushed on the on-device angle — summarizing the current state of local TTS as "so close, yet so far." toebee said Nari Labs hasn't tested on-device deployment but speculated that optimizing specifically for batch-size-1, concurrency-1 could get something fast running locally; they were clear that's outside their current area of focus.

The latency floor nobody talks about: too fast is also wrong

The most interesting comment on the thread wasn't about hitting a lower number — it was about whether a lower number is even always better. Commenter TZubiri pointed out that humans process conversational audio with roughly 200ms of natural auditory processing latency. A voice agent response that arrives faster than that window doesn't read as "impressively responsive" — it reads as an interruption of what the speaker said before their current sentence, not a reply to what they just finished saying.

Their example: someone says "I think murder is bad, but..." — and if a voice agent's response lands instantly after "but," a human listener perceives that as cutting off agreement with "murder is bad," not as a considered reply to the qualifying clause that was about to follow. Commenter nowittyusername, who has built a local voice agent for a year, added a related observation from direct experience: pushing latency down past a certain floor trades off against voice quality, cadence, and expressiveness — all of which matter enormously for whether a voice agent actually sounds good, independent of how fast it responds.

Put together, these two threads sketch a real design constraint for anyone chasing minimum TTS latency: the floor worth optimizing toward isn't zero, it's somewhere around natural human conversational timing — and going faster than that can make an agent feel worse, not better.

What this means if you're building a voice agent

If you're already running Kokoro on CPU for a low-throughput local project, none of this changes your setup — Nari Labs' numbers are a GPU-serving story, not a CPU one. But if you're serving TTS at real concurrency behind a product, the cost delta is the number that should reframe your buy-vs-build math: going from $100/1M characters (ElevenLabs V3) to a ~$2/1M compute floor is a 50x gap, even before accounting for the fact that Nari Labs' TTFA numbers beat Cartesia Sonic 3.5 at less than a quarter of its list price.

The techniques generalize past TTS, too. If you're building any real-time multimodal serving stack — not just voice — the unified-scheduler pattern (treat heterogeneous pipeline stages as independently schedulable work instead of a fixed sequence) and deadline-aware priority (protect the first response, then just beat the buffer) are systems-design lessons worth carrying into your own architecture, whatever the modality.

Nari Labs' own stated direction extends this playbook toward "simulate the world 1:1 through realtime multimodal inference" — image, video, and world models are the next targets, using the same serving philosophy.

FAQ

What did Nari Labs actually build? A custom serving stack for Qwen3-TTS that hits 10 RPS and sub-50ms p95 TTFA on one H100, open-sourced along with their benchmark methodology.

How much cheaper is this than commercial TTS APIs? Roughly $2 per 1M characters in compute cost, versus $100 (ElevenLabs V3) and $49 (Cartesia Sonic 3.5) — though the $2 figure excludes networking, idle time, and ops overhead.

Does it run on non-H100 hardware? Yes on RTX 4090 (confirmed, ~50ms TTFA at ~10 concurrent requests, no FP8), likely with code changes needed for older Ampere cards.

Is this a solved problem for voice agents now? No — LLM time-to-first-token remains a separate, unsolved bottleneck for full conversational latency, and human perception research suggests responses that are too fast can feel like interruptions rather than replies.

Related reading

• Hugging Face Speech-to-Speech Voice Agent Guide
• Kokoro: Local CPU-Friendly TTS
• Miso: One Voice Model for Real-Time TTS
• Voicebox: Open-Source AI Voice Studio
• OpenAI GPT Realtime 2 Voice Models
• Superwhisper S1-mini On-Device Transcript Normalizer
• What Are AI Benchmarks: Complete Guide

Official sources: Nari Labs — Pushing the Speed-Cost Frontier for Qwen3-TTS · Hacker News discussion

Figures, benchmark comparisons, and pricing in this post reflect Nari Labs' August 19, 2026 blog post and the Hacker News discussion that followed, as of publication. Serving benchmarks and third-party API pricing change quickly — verify current numbers before making infrastructure decisions.