Sophon-Realtime v1: Open Full-Duplex Voice With a 32B Brain and vLLM-Omni

Sophon-Realtime v1 is a new open full-duplex voice stack from Adventists AI, published on Hugging Face with Apache-2.0 weights and a GitHub serving repo. Unlike a speech-to-text → LLM → TTS pipeline, it listens while you talk, barges in when you interrupt, and marks laughs, sighs, and stress in the text before a speech model performs them.

For builders comparing 2026 voice agents, the release matters because it bundles benchmark tables, one-GPU economics, and a GPT-Live-shaped API in one package. explainx.ai recently covered Haru-4B without public evals and Shizuku-Duplex with Japanese-focused metrics. Sophon is the first open family in this wave to claim VoiceBench 82.8 beside MiniCPM, Moshi, and closed GPT-4o-audio baselines on the same card.

TL;DR

Three builds, one brain

Adventists ships one cognitive model and three speech renderers. The model card table is the quickest map:

NVV-SuperBench (non-verbal vocalizations) scores on the card: Tag hits 0.86 / 0.80 (zh / en F1), Flash 0.77 / 0.63, E2E 0.85 / 0.75. Tag and Flash share APIs so you can swap renderers without rewriting clients; E2E can run on its own engine or, since v1.1.4, on vLLM-Omni with the same surface.

Sophon-Realtime v1 open full-duplex voice model benchmark table on the model card Image: Adventists AI

Benchmarks: what the card actually claims

All numbers below come from the November 2026 model card unless noted. Treat vendor-run evals as directional until you reproduce them on your hardware and audio set.

Conversation quality (VoiceBench)

The card places Sophon-Realtime v1 at 82.8, equal to GPT-4o-mini-Audio and below GPT-4o-Audio at 86.8. Among open full-duplex entries in the same table, it ranks above MiniCPM-o 4.5 (76.1), Freeze-Omni (55.2), and Moshi (29.5).

Spoken reasoning (Big Bench Audio)

On 1,000 spoken reasoning questions, the card lists 81.8% accuracy for Sophon with open weights, versus 58% for Raon SpeechChat and 4% for Moshi. Sophon transcripts answers and grades with gpt-4.1, per the footnote.

Tool use under disfluent speech (Full-Duplex-Bench v3)

Here is the builder-relevant row:

Sophon does not beat the closed frontier on raw Pass@1, but the card stresses it always responds when a tool call is appropriate—a failure mode that breaks real agents.

Hearing tone (paralinguistic)

Listening uses DuplexJev, Adventists’ open speech-decision model. Combined LISTEN + StressTest + MSPB score: 48.6 for DuplexJev, ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on the card.

Two speech bubbles passing a turn-taking bead, illustrating full-duplex voice agent listening while speaking

How the stack works (mechanism, not marketing)

The card’s architecture section reads like a checklist of what Fish Audio Live 3 and OpenAI GPT-Live buyers already ask for:

1. Continuous listening — at each pause the model emits a single token deciding whether you finished, continued, back-channeled, or paused to think.
2. Early drafting — it starts a reply at your first pause and only releases audio when it takes the floor; if you keep talking, the draft is discarded silently.
3. Tagged paralinguistics — the brain inserts laughter, sigh, breath, cough, emphasis markers; renderers perform them instead of reading tags aloud.
4. Coupled text and audio streams — transcripts stay available for logging, tools, and downstream LLMs.
5. Interrupt-safe memory — history keeps audio both sides heard; partial assistant audio is trimmed on barge-in.

The E2E build replaces discrete speech tokens with continuous voice vectors from a ~350M talker—a design closer to research speech codecs than classic TTS chains.

Quick start: Tag on one GPU

The fastest path on the card:

pip install "sophon-realtime[trt]"
sophon-realtime serve

Requirements called out in the README path:

• Python 3.12, Linux, CUDA 13
• [trt] extra required — without TensorRT the project warns the voice path is about 3× slower under load
• First run downloads ~42 GB Tag weights, then listens on port 8000

GitHub scripts clone adventists-ai/sophon-realtime, run tag/install.sh, then GPU=0 PORT=8000 bash start.sh. Cold start can take 15–20 minutes the first time, 5–8 minutes later.

WebSocket API: ws://HOST:8000/v1/realtime for duplex sessions; REST: POST /v1/chat/completions with audio in and text+audio out. Example client: python client_example.py --wav my_question.wav --lang en --voice ryan_en.

GPT-Live gateway

For teams already running OpenAI-shaped clients:

cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh

Clients point at ws://HOST:8780/v1/live/sessions. Sophon keeps voice; when a request needs lookup, the gateway can say a short filler, call your backend LLM, and speak results. The card claims pairing with DeepSeek V4 Pro lifts Big Bench Audio from 81.2 to ~83 in their configuration—verify on your stack before quoting to customers.

Fine-tuning without retraining the voice

Adventists documents LoRA fine-tuning on the thinker backbone via ms-swift, keeping audio encoder and voice frozen:

pip install "ms-swift>=4.6"
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/finetune
modelscope download --model AdventistsAI/Sophon-Realtime-v1 --include 'base/*' 'adapters/*' --local_dir Sophon-Realtime-v1

Training continues the bundled tag LoRA so laughter and cough tags stay aligned. One 80 GB GPU is enough per the finetune README.

Open-weight model pieces distributed to trays, echoing Apache-2.0 Sophon builds builders can self-host

Comparisons: where Sophon fits in your stack

When to pick Sophon: you want Apache-2.0, self-hosted duplex on one H100-class GPU, with published VoiceBench and tool-call numbers and a GPT-Live migration path.

When to wait: you need sub-100 ms Japanese ticks (Shizuku’s pitch), Mac-first MLX (Haru’s path), or you cannot run CUDA 13 + TensorRT in production.

What people are asking

Are the benchmarks independent? VoiceBench and Full-Duplex-Bench rows mix official leaderboards with self-run Sophon jobs (Big Bench Audio, some NVV runs). Budget time to replicate on your audio before shipping SLAs.

Why three builds instead of one checkpoint? Speech quality and GPU multiplexing trade off. Flash sacrifices some NVV F1 in English (0.63 vs 0.80 on Tag) to hit 40 users per GPU.

Is DuplexJev separate weights? The card markets DuplexJev as the listening module inside Sophon; fine-tune docs focus on the thinker LLM only.

ModelScope vs Hugging Face? Deploy weights are linked per build page; finetune downloads use ModelScope paths in the script while marketing lives on HF.

HN traction? A Hacker News thread exists but had no substantive comments when we checked—hands-on reports may come later.

Cost and capacity planning

Capacity math on the card is blunt: one 80 GB GPU hosts 16 Tag users or 40 Flash users at recommended concurrency. If your contact center assumes P99 first audio under 500 ms, Flash’s 0.31 s cited first-audio figure is the build to model in load tests, not Tag’s ~0.6 s expressive path.

Download and storage also belong in finance models:

• ~42 GB Tag weights on first sophon-realtime serve
• TensorRT path effectively mandatory for production multiplexing per README warnings
• Finetune artifacts via ModelScope base/* and adapters/* bundles

Pair Sophon with the same observability you would run on GPT-Live API pilots: log barge-in rate, tool-call success, and truncated assistant audio after interrupts. The card’s 100% response rate on Full-Duplex-Bench v3 matters only if your gateway preserves those semantics under real network jitter.

Security and abuse notes for self-hosted voice

Self-hosted duplex endpoints are attractive prompt-injection and social-engineering surfaces when wired to customer CRMs. Treat the live/ GPT-Live gateway like any other externally reachable WebSocket: mutual TLS, per-session API keys, rate limits, and separate tool sandboxes for backend LLMs. explainx.ai’s MCP explainer is a useful template for how tool handoffs should stay scoped when duplex models call backends mid-call.

Limitations and licensing footnotes

• Hardware gate: Without sophon-realtime[trt], expect much slower speech under concurrent load.
• Weight size: Tag download ~42 GB; plan artifact mirrors and checksum policy like any frontier open model.
• Third-party licenses: The card’s NOTICE file keeps component-level terms; read before redistributing bundles.
• Business contact: jiejin@adventists.ai on the model page for commercial enquiries.
• Paper link: The Hugging Face page references arXiv 2609.21967 (NemotronLabs VoiceChat lineage) for readers who want architecture context beyond the marketing tables.

DuplexJev and the listening stack

Sophon’s listening path is not a generic Whisper front-end. The card brands DuplexJev as a speech-decision model that feeds the 32B thinker. That split explains why paralinguistic understanding scores are reported separately from VoiceBench conversation quality: one measures ears, the other dialogue.

If you are benchmarking against Qwen3-Omni or Gemini Live, reproduce LISTEN, StressTest, and MSPB tasks on your audio before claiming superiority—vendor F1 tables used Gemini 3 Flash as judge on NVV clips.

Related reading

• Haru-4B: open duplex without benchmark tables
• Shizuku-Duplex Japanese full-duplex
• Fish Audio Live 3 for duplex agents
• GPT-Live-1 in the OpenAI API
• GPT-Live in consumer ChatGPT
• Zyphra Zonos3 Mini open TTS for lightweight speech

Benchmark figures and install commands reflect the Adventists AI model card and GitHub README as accessed November 15, 2026. Re-run VoiceBench and latency tests on your own GPUs before production commitments.