Hugging Face Speech-to-Speech: Build Open-Source Voice Agents

Most "build your own voice agent" projects in 2026 mean stitching together three or four separate APIs and hoping the latency budget survives contact with a real conversation. Hugging Face's speech-to-speech takes a different approach: a single, modular pipeline — VAD → STT → LLM → TTS — exposed through a WebSocket API that speaks the OpenAI Realtime protocol natively. Swap the hosted OpenAI endpoint your app already talks to for this server, and the client code barely changes. It's not a demo — it's the production conversation backend for thousands of Reachy Mini robots today.

TL;DR

Why a "speech-to-speech" pipeline instead of separate APIs

The naive way to build a voice agent is to bolt together a hosted STT API, a hosted LLM API, and a hosted TTS API, gluing the round trips together yourself. That works, but every hop adds latency, and every provider swap means rewriting glue code. Hugging Face's framing is architectural: every stage runs in its own thread, connected by queues, with a shared protocol boundary (the OpenAI Realtime API) at the edge instead of at each internal stage. That means:

• The LLM stage can be hosted, self-hosted, or local — same server, same client code, different --llm_backend flag.
• STT and TTS backends are interchangeable per hardware — CUDA, CPU, or Apple Silicon each get sensible defaults.
• Any OpenAI Realtime-compatible client — including the stock openai Python SDK — can connect to a self-hosted instance by changing two URLs.

That last point is the practical payoff. The README's own example shows the entire migration:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)

with client.realtime.connect(model="local") as conn:
    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "instructions": "You are a helpful assistant.",
            "audio": {"input": {"turn_detection": {"type": "server_vad", "interrupt_response": True}}},
        },
    })
    for event in conn:
        print(event.type)

Swap base_url for a hosted OpenAI endpoint and this same code talks to OpenAI's actual Realtime API instead. That's the whole value proposition in one code block: the protocol boundary, not the vendor, is what your application depends on.

Quickstart

pip install speech-to-speech
export OPENAI_API_KEY=...
speech-to-speech

This starts an OpenAI Realtime-compatible server at ws://localhost:8765/v1/realtime, using Parakeet TDT for local STT and Qwen3-TTS for local speech output — only the LLM call leaves the machine, and only if you point it at a hosted provider. Talk to it from a second terminal with the bundled client script, or connect any Realtime-compatible client directly.

For a fully local LLM instead of OpenAI's hosted API, serve Gemma 4 with llama.cpp:

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

Then point the pipeline at it:

speech-to-speech \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

That's a complete voice agent — VAD, STT, LLM, TTS — running entirely on your own hardware, with the same client-side protocol as the hosted OpenAI path. This pairs naturally with explainx.ai's llama.cpp guide and top open-weight models for a laptop if you're picking the LLM to slot in.

The component matrix

Every stage has multiple backends, selected via CLI flags — a genuinely useful reference table for anyone evaluating hardware fit:

Pick a row per stage, pass the right --stt, --llm_backend, and --tts flags, and speech-to-speech -h prints the exact accepted values.

Two ways to hit the LLM stage: Responses API vs. Chat Completions

The LLM is the highest-latency link in the chain — a single forward pass through a large model can dominate total response time — so backend choice matters more here than anywhere else in the pipeline. Two API-based backends share the same --responses_api_* connection flags:

Both work with the same set of providers — swap only the base URL and key:

Routing through HF Inference Providers is a particularly clean option since it opens the door to third-party inference backends (Together, Groq, Cerebras) without leaving the Hugging Face ecosystem — the README's own examples show Qwen3.5-9B via Together and GPT-oss-20B via Groq, both through the same router.huggingface.co endpoint.

The LLM proxy: voice and background agents, concurrently

The newest addition (--enable_llm_proxy) solves a specific, previously awkward problem: what happens when your voice agent also needs to run background work — summarizing the conversation so far, generating a title, kicking off a tool-using sub-agent — without that work interrupting the live conversation every time new speech comes in?

With the proxy enabled, the realtime server exposes the same LLM it's already configured with as a plain OpenAI-compatible endpoint — /v1/chat/completions or /v1/responses, depending on which backend the server itself is running. A client can fire off tool-using, streaming requests to that endpoint fully concurrent with the live voice session:

from openai import OpenAI

llm = OpenAI(base_url="http://localhost:8765/v1", api_key="unused")
completion = llm.chat.completions.create(
    model="anything",  # ignored: server forces its configured --model_name
    messages=[{"role": "user", "content": "Summarize the conversation so far: ..."}],
)

Two things worth flagging before deploying this: requests are stateless (send the full message list every time), and the server performs no authentication or throttling of its own — the README is explicit that the proxy should run only on a trusted network or behind a gateway that owns access control. Hugging Face's own s2s-endpoint compute replica is offered as exactly that gateway, checking API keys against HF tokens and rate-limiting per user before requests ever reach the proxy.

Multi-language and running fully offline

Language coverage is a function of which STT and TTS backends you pick, not a pipeline-level constraint — worth checking before committing to a deployment language. Two usage patterns:

1. Single language: set --language <code> (default en).
2. Auto-detect and switch per turn: --language auto — the STT detects the spoken language per turn and forwards it to the LLM; add --enable_lang_prompt if a smaller LLM needs an explicit "reply in X" nudge (larger models usually infer it from context).

Combined with the fully-local llama.cpp path above, that's a complete, multilingual, fully offline voice agent — a meaningfully different deployment shape than most "build a voice bot" tutorials, which quietly assume a hosted LLM and hosted TTS the whole way through. For teams evaluating local vs. hosted tradeoffs more broadly, see explainx.ai's closed-source vs. local open-source alternatives guide.

Where this fits if you're building a voice product

For teams already running MCP servers or tool-calling agents, the Realtime protocol's native tool-call event support means voice can sit in front of the same agent stack you're already running elsewhere — the LLM proxy specifically exists to let that tool-calling work happen without fighting the voice session for the model's attention.

Summary

Hugging Face's speech-to-speech is a modular, fully open VAD → STT → LLM → TTS pipeline that speaks the OpenAI Realtime protocol — meaning any existing Realtime client can point at a self-hosted or fully local instance with a two-line change, no protocol rewrite required. Defaults (Parakeet TDT, Qwen3-TTS) run locally out of the box; only the LLM call needs to leave the machine, and even that's optional with a llama.cpp or vLLM backend. It's not a toy — it already runs as the production conversation backend for thousands of Reachy Mini robots, and the new LLM proxy lets voice sessions and background agent work run fully concurrently on the same server.

Related on explainx.ai

• What is llama.cpp? Run models locally
• Top 10 open-weight models for a laptop
• Closed-source AI vs. local open-source alternatives
• What is MCP? Model Context Protocol guide
• OpenAI GPT Realtime 2 — voice models and API
• FluidVoice — macOS open-source dictation
• VoxCPM2 — tokenizer-free TTS with voice cloning

Source: huggingface/speech-to-speech on GitHub

Feature set, benchmarks, and defaults reflect the repository as of July 31, 2026 (v0.2.10). Backend availability and default models change between releases — check speech-to-speech -h and the repo's release notes before deploying.