TensorFold: Zig LLM Server for Metal and CUDA With an OpenAI-Compatible API

TensorFold is a Zig-native LLM inference engine that serves checkpoints on Apple Metal and NVIDIA CUDA from one binary, exposes OpenAI-style HTTP APIs on port 8080, and does not require Python or MLX at serving time according to its README.

The TensorFold GitHub repository trended in mid-November 2026 as developers looked for alternatives to Python-heavy GPU servers. Stars climbed quickly because the README is unusually concrete: pinned Hugging Face pulls, release tarballs, qualified model rows and copy-paste curl tests. This post explains what shipped, how to try the documented quick start, and where TensorFold sits next to llama.cpp, vLLM versus SGLang, and Ollama-style wrappers.

TL;DR: TensorFold quick answers

What TensorFold actually ships

The project's one-line pitch on GitHub is an LLM inference engine for Metal, CUDA and Vulkan. Under the hood the maintainers describe a Zig binary that owns checkpoint IO, tokenization and kernel dispatch. That design choice matters for operators who are tired of conda environments breaking CUDA wheels on upgrade weeks.

Serving surface. After tensorfold serve, clients can hit standard OpenAI chat completions, plus Responses and Anthropic Messages endpoints listed in the README. /v1/models advertises the served alias you choose with --name. Parallel request caps exist: --parallel N or --parallel auto (up to eight streams when memory allows). Health metadata exposes active and queued connections when authentication is off.

Distribution. Apple Silicon users get a Homebrew formula. Linux users pull gzipped release bundles with tensorfold-native as the entry binary. The README pins v1.2.7 in its example and demands NVIDIA driver R580 or newer for CUDA 13 kernels. Those constraints are strict: TensorFold refuses older drivers at startup rather than failing mid-request.

Ecosystem extras beyond bare serving.

• TensorFold Web — a local launcher for picking models (tensorfold web).
• Mac desktop app — beta DMG in the app/ tree bundling workers and metrics.
• Living Weights (--slide) — experimental on-device weight edits for Nemotron checkpoints; README warns about recall limits and speed tax.
• Starlight — enterprise multi-node governance in another repository.

None of those change the core news hook: a native server with first-class Metal and CUDA paths and an OpenAI-compatible wire format.

TensorFold local LLM inference serves open-weight models on Apple Metal or NVIDIA CUDA from a Zig binary with an OpenAI-compatible HTTP API

Quick start: reproduce the README path

The maintainers standardize on Qwen3.5-2B MLX 4-bit because it is small and checked on Metal plus GB10 class hardware.

Apple Silicon

brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Linux with NVIDIA GPU

Download the platform archive, verify sha256, extract, then alias tensorfold to tensorfold-native exactly as the README shows for aarch64 GB10 or x86_64 hosts. Pull the same pinned Qwen revision, then serve with the same flags.

Smoke test

curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'

If that returns JSON, your agent harness can point baseURL at http://127.0.0.1:8080/v1 the same way we document for llama-server in the llama.cpp guide.

Model qualification table: read before you benchmark

TensorFold is not a universal GGUF runner. The README's Pick a model section is a compatibility contract: each row lists download command, machine scope, and native recipe links.

Representative rows developers asked about in early November threads:

The table also marks conversions that still need Hugging Face downloads plus Python conversion steps before native serve. Treat those rows as two-phase workflows, not one command.

If you are evaluating Nemotron multimodal servers or multi-GB10 GLM-5.3 clusters, TensorFold's receipts are useful, but they are per-model receipts, not a blanket guarantee.

TensorFold vs llama.cpp vs Python servers

Versus llama.cpp. llama.cpp remains the default for GGUF on CPU, Metal, CUDA and Vulkan with enormous community quant coverage. TensorFold focuses on native MLX checkpoints and its own CUDA stack, with explicit model recipes and pinned pulls. Choose llama.cpp when your artifact is GGUF and you want maximum forum support. Choose TensorFold when you want the maintainer's qualified MLX or CUDA paths and are willing to live inside their compatibility table.

Versus vLLM / SGLang. Production GPU farms still lean on Python servers for MoE scale, radix caching and RL rollouts, as we covered in vLLM vs SGLang. TensorFold targets single-node or small-cluster operators who want a native binary without a conda stack. It is closer to "run this release zip on a DGX Spark" than "schedule 200 GPUs in Kubernetes."

Versus Ollama. Ollama optimizes pull-and-chat UX. TensorFold exposes lower-level flags (tensorfold capabilities --json, per-model tensorfold info REPO) and expects you to read recipes. That is a feature for SRE-minded builders, friction for casual chat users.

TensorFold Zig LLM server routes Hugging Face checkpoints through Metal or CUDA kernels to an OpenAI-compatible local API for coding agents

Parallel requests, auth, and production gaps

The README documents queueing when --parallel slots fill: /health reports live.connections and live.waiting. That is enough for a home lab or a single developer laptop, but production teams still need answers the README does not fully centralize:

• TLS termination — you front TensorFold with nginx or Caddy.
• API keys — optional key files exist for features like Living Weights; default local serve may run open on localhost.
• Multi-node — Starlight covers fleet policy; the base engine is not a full scheduler like Slurm plus NVIDIA recovery automation.

Treat November 2026 TensorFold as early-adopter infrastructure, not a drop-in replacement for every vLLM deployment.

Living Weights and experimental flags

The README devotes significant space to --slide, which attempts to write new facts into weights during serve. The maintainers label it experimental: interference between facts, recall decay and decode slowdown on Mac when thinking modes are on. Interesting for research demos; risky for production agents without rollback discipline (tensorfold slide rollback).

This is the same caution we apply when covering weight-editing research: cool science, heavy operational caveats.

Release cadence and verification habits

The README's Linux quick start pins tensorfold-1.2.7 tarballs with sha256sum -c before extract. That is the right habit for any binary you will put on a GPU host: verify checksum, keep the bin and library directories together, and read the startup line that states how many parallel streams the engine accepted. If the process refuses your driver, nvidia-smi is the first diagnostic; TensorFold's CUDA path targets R580+ because its kernels are built against CUDA 13.

tensorfold capabilities --json is the supported way to see what your build actually compiled in, which matters when a README row says "Metal only" or "CUDA only" for a given quantization path. Before you open a GitHub issue about a model, run tensorfold info REPO against the Hugging Face id; the README uses that command to print a memory floor so you do not serve a 70 GB checkpoint on a 24 GB card.

When you outgrow one machine, resist bolting auth and scheduling into the open port yourself until you read Starlight's install guide. Starlight is where TensorFold talks about OIDC, SCIM and fleet budgets. The base engine remains a deliberate single-node sharp tool.

Contributing and agent coordination

TensorFold's README asks contributors to run Agent Traffic Control (ATC) before coding so overlapping agent sessions do not collide. That pattern is new in open-source ML infra and worth watching: repositories that expect coding agents as first-class contributors need radar tooling, not just CONTRIBUTING.md prose.

If you automate benchmarking, pair ATC with the project's evaluating guide rather than ad hoc prompts.

What people are asking on GitHub and HN

Is Vulkan production-ready? The tagline mentions Vulkan; the quick start and qualification table emphasize Metal and CUDA. Treat Vulkan as advertised surface area with fewer published receipts today.

Windows? Native Windows builds live in docs/WINDOWS.md on the repository; Linux and Mac paths are the documented happy path.

Why Zig? Maintainers argue for a single static binary and controlled kernel ownership. Practically, that means faster cold start and fewer pip conflicts, at the cost of a smaller third-party ecosystem than Python.

How does this relate to xAI dropping JAX/SGLang? Musk's November Grok 4.8 post described rewriting RL and inference stacks in C++. TensorFold is unrelated product-wise, but it rides the same compile-the-serving-path zeitgeist we noted in Grok 4.8's soft rollout.

Security and supply-chain checklist

1. Pin pulls with @revision hashes as the README demonstrates; do not serve floating main in production.
2. Verify release checksums on Linux tarballs before extract.
3. Keep driver floors — CUDA 13 plus R580 is mandatory on NVIDIA paths.
4. Sandbox keys for --slide and API key files; README uses umask 077 examples.
5. Agent access — if you aim OpenCode or Claude Code at a local TensorFold endpoint, treat it like any local server: no inbound exposure without auth.

For broader agent hardening patterns, see explainx.ai coverage of Claude Code sandboxing and MCP dependency drift.

Related reading

• What is llama.cpp? GGUF and local APIs
• vLLM vs SGLang for GPU fleets
• Magnitude inference engine 2x claim checked
• TokenSpeed for agentic inference
• Transformers GGUF parity with llama.cpp
• Qwen3.5 local development stacks
• AI chip architectures guide

Commands and version examples reflect the TensorFold README and release tags as of November 12, 2026. Verify tensorfold --version, driver levels and model rows before production cutover.