Respan Gateway: An Open-Source Rust AI Gateway Where Everything Is a Plugin

Respan (YC W24) launched an open-source Rust AI gateway on Hacker News on October 20, 2026, built so that "everything is a plugin." Providers such as OpenAI, Anthropic and Gemini, protocols such as chat, realtime voice and MCP, modalities such as images and video, and features such as fallbacks, smart routing and PII redaction are all separate pieces around a small core that dispatches events. The code is on GitHub under MIT or Apache-2.0, and the founders call it pre-alpha.

An LLM gateway is the layer many teams add once they use more than one model provider. If you read our guides on what OpenRouter is or routing for cost, this is the self-hosted cousin. Below: what Respan says it built and why, how the request flow works, what Hacker News commenters asked, and what to test before routing real traffic through it. The facts come from the founder's Launch HN post and the repository; we have not run it.

TL;DR

The problem they say they hit

The founder, Andy, writes that Respan has run a gateway on top of LiteLLM since 2024, "providing services for thousands of orgs." Each organization needed different API key settings, routes and limits, so the team kept wrapping LiteLLM. At one point the wrapper code exceeded 100,000 lines. In January, a routine LiteLLM upgrade caused intermittent connection errors and a rollback; the cause was never found, so they stopped upgrading directly and began manually backporting upstream fixes.

That is a one-company account, and the same story could be told about any large dependency you wrap heavily. It does, though, describe a real pattern: gateways accumulate features until adding one risks breaking another. Our post on LiteLLM Lens trace analysis covers the other side of that ecosystem.

How the design works

Inspired by Nginx, the gateway keeps a small core "only responsible for dispatching events." The request flow, as the founder describes it:

1. An adapter hands each HTTP or WebSocket connection to the first client format that recognizes it.
2. The format exposes the request as a two-way stream of typed input and output events.
3. The core runs plugins in registration order. A plugin can choose a provider, filter or wrap either stream, make additional calls, return a response, or refuse the request.
4. A provider format opens the upstream connection.
5. If the client and provider speak the same format and no plugin changes the payload, original bytes and protocol-specific fields pass through unchanged. When formats differ, they translate through shared types for messages, tool calls, results, streams and usage.

Two details stand out. First, preserving provider-specific fields when client and provider use the same API means the gateway can write them back rather than dropping them, which is a common gateway bug when providers add new fields. Second, the stream model covers both one-shot HTTP and long-lived bidirectional sessions, which is how an app built on the OpenAI Realtime API can reportedly use Gemini Live through the gateway, "though not every feature translates."

Providers as short files

Adding a provider is meant to be easy: "A provider is a short file, Mistral's is 22 lines." Provider quirks, such as Groq putting token usage in its own field, are plugins with their own tests. For comparison with a model-hosting approach to many providers, see OmniRoute, a free LLM proxy for Claude Code.

MCP in the gateway

A simple chat app can use MCP tools configured in the gateway without implementing MCP itself. The founder says they considered the rmcp Rust crate but implemented MCP themselves because the gateway needs hooks at every processing stage.

What Hacker News asked

Commenters raised the questions you would expect:

• Why plugins? One commenter, verdverm, said they use another gateway with all providers built in and worried a plugin ecosystem exposes users to supply chain events, asking why plugins beat a curated complete product (comment). The founder answered that Respan plugins are not arbitrary executables from a random repository; they are modular crates compiled into your app, which gives better interface boundaries, for example a PII feature that does not touch the logging, billing or provider subsystems (reply).
• Compile-time or interpreted? blakeashleyjr asked (comment). The founder said natively compiled crates, and that they considered WASM and libloading; WASM plugins often need access to infrastructure such as databases, which would require more host interfaces, and dynamic loading did not offer enough benefit (reply).
• Was it AI-written? m00dy asked whether the founders understand the Rust code (comment). The founder replied they used Claude to generate code but reviewed every line themselves (reply).

Note the supply-chain nuance: compiled-in crates avoid downloading a binary at runtime, but a plugin crate still pulls in its dependencies. The risk moves from runtime loading to your build and dependency review. Our look at the Bumblebee supply chain security scanner is relevant here.

How it compares

We have not benchmarked latency or throughput, and Respan has not published figures we could find. Rust does not automatically mean faster end to end, since gateway overhead is small compared with model latency; the benefit is more likely predictable memory use and safe concurrency.

When a gateway is worth having at all

Not every team needs one. A single-provider app can call the vendor SDK directly and add a retry. A gateway starts paying off when at least one of these is true:

• You use several providers and want one client format, one place for keys, and a fallback when a provider has an outage or rate limit.
• You need central controls: spend limits per team, allow-lists of models, logging, and PII redaction before prompts leave your network.
• You run agents with many tools, where a single MCP configuration in the gateway is easier to manage than the same tool list in every app.
• You want a seam for experiments, such as routing 5% of traffic to a new model and comparing quality and cost.

The cost is another hop and another system to operate. A gateway that goes down takes every model call with it, so health checks, timeouts and a bypass belong in the design from day one.

What to watch next

• Whether a plugin ecosystem forms. The founders say turning the plugin interface into an ecosystem is "basically our roadmap." Look at the number and quality of third-party crates over the next months.
• Stability of the plugin interface. Pre-alpha means breaking changes. Check release notes and the repository's issue tracker before upgrading.
• Real benchmarks. Published latency overhead, memory use and throughput under load would settle the Rust-versus-Python question with data.
• Security review. A gateway handles keys and prompts, so an independent review or fuzzing of the format parsers would be a strong signal.

Pre-alpha checklist before you try it

1. Run it beside, not in front of, production. Mirror a slice of traffic and compare responses with a direct provider call.
2. Test the translation you rely on. Tools, images, reasoning and streaming differ across providers. The founder concedes not every feature translates.
3. Read the plugin interface. The repository's PLUGINS.md explains how to write one. If you plan to add PII redaction or policy checks, check where in the event order they run.
4. Pin dependencies and review crates. Treat plugin crates like any dependency.
5. Keep a bypass. Make it trivial to point clients back at providers.
6. Log carefully. A gateway sees prompts, keys and outputs. Decide what to store and for how long.

For teams putting agents behind a gateway, a policy layer matters as much as routing. AgentBeam, from the explainx.ai team, is built to stop AI agents before they take dangerous actions, which complements, rather than replaces, request-level filtering in a gateway.

Why it matters

Multi-model setups are now normal: one coding agent, one cheap summarizer, one realtime voice model. The gateway has become critical infrastructure, and the market is crowded. Respan's bet is that a small, auditable core and an extension model beat a monolith for teams with unusual requirements. Whether that holds depends on whether a plugin ecosystem forms and whether the project survives its own growth. For now it is a promising pre-alpha with a candid origin story and a hosted business behind it.

Details come from the October 20, 2026 Launch HN post and repository; the project is pre-alpha and may change.

Related reading

• What is OpenRouter? An enterprise guide
• OpenRouter model routing for cost optimization
• OpenRouter vs direct provider APIs
• OmniRoute: free LLM proxy for Claude Code
• LiteLLM Lens: agent trace analysis
• Respan Span-01 behavior classifier
• Bumblebee supply chain security scanner
• Sources: Respan Gateway on GitHub, Launch HN discussion, Respan