explainx / blog
llama.cpp is the C/C++ inference engine behind most local LLM stacks — GGUF quants, Metal/CUDA/Vulkan/CPU, llama-server on :8080/v1. Install, run models, key flags, vs Ollama, and wire to OpenCode.
Jun 29, 2026
After HN front-page hype, hands-on tests say Qwen 3.6 27B dense is the local sweet spot — better code than the 35B MoE, runnable at Q8 on 48GB RAM. Full llama.cpp + OpenCode config inside.
Jun 29, 2026
MacBooks behave like a slow GPU with enormous shared RAM; dedicated cards are fast but VRAM-capped. The right buy depends on whether you wanted a laptop anyway, need privacy at 64k context, or need frontier-speed coding throughput.
Jun 4, 2026
Google DeepMind's Gemma 4 12B is a breakthrough open-source model that delivers flagship-level multimodal capabilities on consumer hardware. Here's everything you need to know about architecture, benchmarks, deployment, and use cases.
July 2, 2026: Every week another model ships as GGUF on Hugging Face, another HN thread argues Mac vs Nvidia for local LLMs, and another developer asks "do I need Ollama or something else?" The answer underneath most stacks is the same engine: llama.cpp.
Georgi Gerganov open-sourced it in March 2023 to run Meta's Llama weights on a MacBook CPU. Today the ggml-org/llama.cpp repo carries 118k+ GitHub stars, 450+ contributors, and a release cadence that shipped build b9829 on June 28, 2026. Ollama, LM Studio, and dozens of embedders sit on top of it — but when you want MTP speculative decoding, exact -ngl offload, router mode, or embedding endpoints, you run llama.cpp directly.
This post is explainx.ai's foundation guide: what llama.cpp is, how GGUF fits, install paths, llama-cli vs llama-server, copy-paste run commands, and how to hand the API to OpenCode or Codex OSS.
| Question | Answer |
|---|---|
| Do I need a GPU? | No — CPU works (slow). Metal on Apple Silicon, CUDA on Nvidia, Vulkan/ROCm elsewhere for speed. |
| What file format? | GGUF quants — see quantization guide. |
| Simplest run command? | llama-server -m model.gguf --port 8080 → browser UI + /v1 API. |
| vs Ollama? | Ollama = easy wrapper; llama.cpp = control plane. Same weights possible, different UX. |
| vs vLLM? | vLLM = multi-user production on big GPUs; llama.cpp = laptop to workstation breadth. |
| Best for coding agents? | llama-server + OpenAI-compatible /v1 — full local + OpenCode path. |
| Example model walkthrough? | Qwen 3.6 27B + MTP flags. |
Three layers, one project:
┌──────────────────────────────────────────┐
│ llama-server / llama-cli (user tools) │
├──────────────────────────────────────────┤
│ libllama (model load, decode, sampling)│
├──────────────────────────────────────────┤
│ ggml (tensor ops — CPU/GPU backends) │
└──────────────────────────────────────────┘
▲
│ reads
model.gguf (quantized weights on disk)
llama.cpp is not a model and not a chat app. It is an inference runtime — load weights, manage KV cache, sample tokens, optionally expose HTTP.
GGUF is the file format llama.cpp natively consumes. One file holds architecture metadata plus Q4/Q5/Q8 (or F16) tensors. That is why a 27B model that would need hundreds of GB in FP32 can run in ~18–48GB RAM depending on quant — the math in our quantization guide applies directly here.
Historical note: The project predates the current explosion of Chinese open weights (Qwen, GLM, DeepSeek). llama.cpp's role stayed constant: make the file on disk talk.
| Tool | Relationship to llama.cpp | When to pick it |
|---|---|---|
| llama.cpp | Core engine | Tuning, MTP, router, embeddings, exotic hardware |
| Ollama | Uses llama.cpp (and MLX on Mac) | Fastest first run, ollama pull, no flag soup |
| LM Studio | Embeds llama.cpp server | GUI model browser + local API toggle |
| MLX | Separate Apple stack | Pure Mac optimization; some models faster than llama.cpp on M-series |
| vLLM | Different codebase (PagedAttention) | Team server, many concurrent users on datacenter GPU |
explainx.ai read: Start with building a personal local AI system for the full layer cake. Drop to raw llama.cpp when Ollama's defaults leave performance on the table — especially on Apple Silicon with MTP (Qwen 3.6 benchmarks).
brew install llama.cpp
# verify
llama-cli --version
llama-server --version
Homebrew ships recent builds with Metal enabled on Apple Silicon.
Download the matching ggml-org/llama.cpp release asset for your OS/CUDA version from GitHub Releases, extract, add to PATH.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON # or -DGGML_METAL=ON on Mac
cmake --build build --config Release -j
# binaries in build/bin/
Use source builds when you need a specific CUDA arch, ROCm on AMD, or bleeding-edge server features before packagers catch up.
GGUF, publishers like unsloth, bartowski, MaziyarPanahi-hf flag — llama.cpp downloads/caches directly:llama-server -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 --port 8080
.gguf anywhere, pass -m /path/to/model.ggufPick quant by RAM: Q4_K_M for tight budgets, Q8_0 when you have 48GB+ unified memory and want coding quality — see VRAM/RAM tables.
llama-cli — terminal chatInteractive REPL in the shell. No HTTP. Best for smoke tests and prompt iteration.
llama-cli -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
-ngl 999 -c 8192 -fa on
Flags explained below. Exit with Ctrl+C or /quit depending on build.
llama-server — API + web UIThe production path for agents, Continue, Open WebUI, and OpenCode.
llama-server -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
--port 8080 -ngl 999 -c 8192 -fa on
| Endpoint | URL |
|---|---|
| Web chat UI | http://127.0.0.1:8080 |
| OpenAI models list | http://127.0.0.1:8080/v1/models |
| Chat completions | http://127.0.0.1:8080/v1/chat/completions |
Smoke test:
curl http://127.0.0.1:8080/v1/models
2026 server features (see server README): parallel decoding (-np), OpenAI + Anthropic-compatible chat routes, function calling, speculative decoding, multimodal input (experimental), embeddings and reranking endpoints, router mode for multiple models with LRU eviction, built-in MCP hooks in the web UI.
| Flag | Meaning | Typical value |
|---|---|---|
-m path.gguf | Model file | Local path |
-hf org/repo:quant | Hugging Face pull | unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 |
-ngl N | GPU layers to offload | 999 = all layers on GPU/Metal |
-c N | Context size (tokens) | 8192–65536; higher = more RAM |
-fa on | Flash attention | On when supported — faster long context |
--port N | HTTP port | 8080 (convention) |
-np N | Parallel slots / users | 4 on shared LAN box |
--spec-type draft-mtp | Multi-token prediction | Qwen 3.6 MTP GGUF builds |
-md draft.gguf | Speculative draft model | Smaller companion file |
--embedding | Embedding mode | Dedicated embed models only |
Sampling (quality vs creativity): tie to temperature, top-p, top-k guide — llama-server accepts the same params in API JSON as OpenAI.
Hardware cheat sheet (Mac vs GPU deep dive):
| Machine | Starting point |
|---|---|
| 32GB Mac | 7B–13B Q4, -ngl 999, -c 16384 |
| 48–64GB Mac | 27B Q4–Q8, MTP if available |
| 24GB Nvidia | 7B–14B Q4/Q5, watch KV cache vs -c |
| 48GB+ Nvidia | 27B–32B Q6/Q8, raise -np for roommates |
| CPU only | -ngl 0, tiny models (1B–3B), patience |
Good first pulls: Llama 3.2 3B Instruct, Qwen3 4B, Gemma 3 4B — any GGUF Q4_K_M under ~3GB download.
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M \
--port 8080 -ngl 999 -c 8192 -fa on
Open http://127.0.0.1:8080, ask: "Write a Python function that merges two sorted lists."
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": "Say hello in one sentence."}]
}'
(Model name string is often ignored locally — server uses the loaded GGUF.)
Add to ~/.config/opencode/opencode.jsonc:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llama": {
"name": "llama.cpp (local)",
"npm": "@ai-sdk/openai-compatible",
"options": {
"baseURL": "http://127.0.0.1:8080/v1",
"apiKey": "local"
},
"models": {
"local": { "name": "local-gguf" }
}
}
},
"model": "llama/local"
}
Full harness walkthrough: run open-source models in OpenCode. For a production coding default on 48GB+ hardware, jump to the Qwen 3.6 27B llama.cpp recipe with MTP flags.
Omit -m and point at a models directory — llama-server loads models on demand, evicts LRU when memory is full. Useful for a home lab with 3B for fast autocomplete and 27B for hard prompts. See router mode docs in server README.
llama-server -m nomic-embed-text.gguf --embedding --pooling cls -ub 8192 --port 8081
Pair with local vector DB in a personal AI workflow.
When upstream only ships safetensors:
llama-quantize input-f16.gguf output-q4_k_m.gguf Q4_K_M
Most users should download pre-made GGUF from trusted quantizers instead.
Load a small draft model (-md draft.gguf) or MTP weights (--spec-type draft-mtp) so the big model verifies multiple tokens per step — the ~32 tok/s vs ~18 tok/s gap in the Qwen 3.6 post.
| Issue | Reality |
|---|---|
| "Too many flags" | True vs Ollama. Keep a shell alias or Makefile for your daily server line. |
| Ollama drama | Some HN posts argue ethics of Ollama's packaging; llama.cpp is the neutral upstream. explainx.ai supports both — pick by friction vs control. |
| MoE sloppiness | Architecture matters more than runtime — dense vs MoE local coding. |
| Tool calling quality | llama-server supports tools; model must be trained for reliable function JSON. Weak local models → failed agent loops. |
| Windows friction | Builds exist; CUDA path is smoother on Linux. WSL2 + Nvidia is the usual Windows power-user route. |
| No training | Inference only. Fine-tune elsewhere (Unsloth GLM guide), infer here. |
Model (GGUF) → llama-server → /v1 API → OpenCode / Codex / Pi
↑
optional: Ollama wraps same engine for simpler pulls
After Fable 5 export controls, llama.cpp is the default on-ramp to enterprise open alternatives: weights stay on disk you control, API stays on localhost, harness swaps without re-downloading terabytes.
Add verification loops from explainx.ai loops when agents drive CI — e.g. ci-until-green.
Official: llama.cpp GitHub · llama-server README · GGUF on Hugging Face
Binary names, server routes, and star counts reflect the ggml-org/llama.cpp repo as of July 2, 2026 — verify release notes before production deployments. Last updated: July 2, 2026.