explainx / blog / topics
Local AI
Local AI means running models on your own laptop, workstation, or phone instead of a cloud API. It trades some capability for privacy, cost control, and offline use.
This page collects our coverage of local runtimes, on-device models, and the hardware that makes them practical.
87 stories · latest Nov 17, 2026
Start here
Together AI and Ollama: Prefix Cache Rules for Long Agent Sessions
On November 16, 2026 Together AI and Ollama co-authored a guide to prefix caching for agentic workloads. Coding agents resend the full session each turn; this post turns their harness and engine checklist into actions for builders shipping long-horizon tools.
192GB Local AI Without a $9,500 Mac: Hardware Paths and Real Trade-offs
You need about 192GB of fast memory to run quantized DeepSeek-class models with any prompt headroom. This guide maps the non-Mac options Martin Weinberg demos on YouTube and cross-checks them against Lucebox and Framework public benchmarks.
GLM-5.3 on Two DGX Stations: NVIDIA Shows 753B Running Locally with vLLM
NVIDIA published a tutorial for serving the 753-billion-parameter GLM-5.3 NVFP4 checkpoint across two DGX Stations with vLLM and tensor parallelism. A single GB300 GPU cannot hold the 464 GB of weights, so the two machines split them over a 400 Gb/s RDMA link.
Semantic Caching for LLM Apps: Exact Hash, Embeddings and Safe Rollouts
Prompt caching on APIs discounts repeated prefixes; semantic caching goes further by reusing whole answers when a paraphrased question is close enough. This guide covers exact-hash layers, embedding search, false-positive risk, and how to ship caching without breaking tool-calling agents.
What Is the KV Cache in LLM Inference? Memory, Context and Tuning
Every new token in an LLM forward pass needs attention over all prior tokens. The KV cache keeps those past key and value tensors in GPU memory so decode steps stay fast. That speed comes at a price: VRAM grows with context length, layer count, and concurrent sessions — the main reason long chats and big batches OOM on local and server GPUs alike.
Run LLMs on Apple Silicon With MLX: Complete 2026 Guide
Apple’s MLX framework is the native path for LLMs on M-series Macs. The mlx-lm package pulls Hugging Face models, quantizes them, exposes a chat REPL, and can serve OpenAI-style traffic — without renting a Linux GPU box for every experiment.
Timeline
November 2026
Nov 17
Mobile AI Factory: Agentic System That Builds Fine-Tuned SLMs for Your PhoneMobile AI Factory is an agentic system from UW researchers that searches model size, quantization, and inference backends using live Android measurements. On ten tasks and four phones it matches or beats five-shot frontier models on eight tasks while cutting GPU search cost.
Nov 16
Together AI and Ollama: Prefix Cache Rules for Long Agent SessionsNov 14
192GB Local AI Without a $9,500 Mac: Hardware Paths and Real Trade-offsNov 13
Cactus Mirage: On-Device Vision and Structured Image ExtractionCactus Compute added Mirage, a vision model for local describe and structured extraction, beside Whistle speech and Needle tool-calling. The Mirage weights trended on Hugging Face in November 2026 with more than 200 likes.
Nov 13
Edge0-145B: Open 145B Agent Model Built to Run on Your LaptopEdge0 released open Edge0-145B and Edge0-120B weights on November 12, 2026, targeting a single NVIDIA GPU with edge0-strata. The announcement claims frontier-class coding and agent scores at roughly 29GB peak memory on laptop-class hardware.
Nov 13
GLM-5.3 on Two DGX Stations: NVIDIA Shows 753B Running Locally with vLLMNov 13
InferSSD: Running 800B-Class MoE Models From NVMe on Consumer GPUsInferSSD is an open llama.cpp-based runtime that stores routed mixture-of-experts weights on NVMe while attention and KV cache stay in RAM or VRAM. WiCi Research published benchmark tables for DeepSeek-V4.1-Flash-1017 and Muse-Spark-Open on RTX 5090, MacBook Pro, and DGX Spark class machines.
Nov 13
llama.cpp v1.0.0: What a Stable libllama and libggml Mean for Local AIggml-org tagged llama.cpp v1.0.0 on November 13, 2026, after about 3.5 years of development. libllama and libggml now follow semantic versioning, so breaking C API changes show up as major-version bumps. Here is what that promise covers, what it does not, and how to upgrade safely.
Nov 12
gputaskman: Scott Hanselman GPU Task Manager for LLM VRAM and KV Cachegputaskman crossed 2,400 GitHub stars in November 2026 as a cross-platform GPU task manager aimed at developers running local LLMs. Here is what the README promises, which GPUs it supports, and when to use it beside nvidia-smi or llama.cpp logs.
Nov 12
TensorFold: Zig LLM Server for Metal and CUDA With an OpenAI-Compatible APITensorFold ships language models from a single Zig binary on Apple Metal and NVIDIA CUDA, with Vulkan and HIP paths in development. It pulls Hugging Face checkpoints, serves chat and Responses APIs on port 8080, and needs no Python or MLX install at serving time. The GitHub repo gained fast traction in November 2026. Here is how it works and who should try it.
Nov 9
Semantic Caching for LLM Apps: Exact Hash, Embeddings and Safe RolloutsNov 8
llama-modchip: 64 GB CPU Path for DeepSeek-V4 and 30B MoE GGUFThe synchopate/llama-modchip repository (v2.4.0) documents CPU-only inference for DeepSeek-V4-Flash Fractal GGUF near 57 GiB RAM at 256K context and Nemotron-3.5-30B-A3B at roughly 14–17 GiB, using custom Q2_0 tensors and Stockfish-PVS speculative decoding on llama.cpp forks.
Nov 8
What Is the KV Cache in LLM Inference? Memory, Context and TuningNov 7
Run LLMs on Apple Silicon With MLX: Complete 2026 GuideNov 7
vLLM vs SGLang: Which GPU Inference Server Should You Run in 2026?If you self-host models on NVIDIA GPUs in 2026, vLLM and SGLang are the two frameworks teams argue about most. Both expose OpenAI-compatible HTTP APIs, both support MoE models, and both ship fast attention kernels — but vLLM leads on ecosystem breadth while SGLang leads on structured generation and RL-heavy serving. Here is how to choose.
Nov 5
Mercedes-Benz and Liquid AI: On-Device Car Voice AI, but the Cloud Is Not Going AwayA November 5, 2026 Wall Street Journal report revived Mercedes-Benz and Liquid AI on-device voice models for MBUX. A viral headline says local AI replaces the cloud. The April partnership announcement says otherwise. Here is the evidence.
Nov 4
llama.cpp v0.9.0 and ggml 0.29.0: Native FP8, Multi-Node Tensor Parallelism and What Local-AI Users Getllama.cpp tagged v0.9.0 on November 4, 2026. The ggml 0.29.0 library underneath adds a native OCP FP8 E4M3 tensor type, a new flash-attention op for multi-sequence serving, batched tensor copy APIs and initial multi-node tensor parallelism through NCCL, plus kernel work across CUDA, Metal, Vulkan, SYCL, OpenCL and Hexagon.
October 2026
Oct 31
dasllama: A Game-Scripting-Language Inference Engine Claims 5x Faster Speech-to-Text Than llama.cpp on CPUA new local inference engine written entirely in a game scripting language reports 4x to 7x CPU speedups for speech models and mixed results for LLMs. Here are the numbers, the losses the authors admit, and how to test it yourself.
Oct 27
Offline Coding Agent on a Laptop: OpenCode, Ollama and Qwen3.5 9B With the Wi-Fi OffRittika Jindal, a principal engineer living where power and network outages are routine, tested whether a coding agent can work with the internet switched off. Her answer on a Mac laptop with OpenCode, Ollama and Qwen3.5 9B: yes for a small bug fix, with caveats. Here is the setup and what it does and does not prove.
Oct 27
Opera WebAI: A Proposed Web Standard for Running AI Models Locally in the BrowserOn October 27, 2026 Opera began shipping WebAI, a proposed web standard that lets websites and extensions call AI models running on your own computer. Here is what is confirmed, what is still unclear, and what it means for developers.
Oct 27
Sentient OS: The YC-Backed Open-Source Mac Agent That Works Overnight on a Local Gemma 4Sentient OS is an open-source, AGPL-3.0 Mac app from a YC F26 team that analyzes your mail, messages and files overnight with an on-device model, then offers to do the work. We checked the README and release notes for what is truly local.
Oct 27
Survival LLM: The Offline Emergency AI With a 930,000-Article Library, Tested on 22 QuestionsSurvival LLM, trending on Hacker News on October 27, 2026, packages a local model and an offline emergency library. explainx.ai covers what it does, the author's own test numbers, install costs, and why commenters question whether an LLM belongs in an emergency kit.
Oct 22
Underdog Bark 3.0: A 163 MB Open Speech Model That Matches ParakeetUnderdog released Bark 3.0, a 0.6B English speech recognition model in a 163 MB download under CC-BY 4.0. Its card says it averages 4.94% word error on seven Open ASR Leaderboard sets, edging NVIDIA Parakeet and beating Whisper large-v3. The numbers are self-scored.
Oct 20
PyTorch 2.15 on Mac: FP8, Faster MoE Decoding and an 8x Softmax on Apple SiliconA Hugging Face post by two PyTorch MPS contributors details what changes for Apple Silicon in PyTorch 2.15: FP8 tensors, faster dense and mixture-of-experts decoding, Metal kernels in place of MPSGraph, and a preview of 2.16.
Oct 16
Magnitude: Open-Source Local Inference Engine Claims 2x Faster Than OllamaMagnitude, a Y Combinator-backed open-source inference engine, says it decodes about 2.2x faster than Ollama on a Mac M4 Pro and on an NVIDIA DGX Spark by tuning kernels to each model and chip. Here is how it works, how to try it, and why the benchmark needs a careful read.
Oct 16
PS5LM: Run llama.cpp on a Jailbroken PS5 as a Home LLM ServerA new open-source app, PS5LM, runs llama.cpp on a jailbroken PlayStation 5 GPU and serves OpenAI and Anthropic-style APIs to your network. We go through the claimed speeds, the setup, and what to be careful about.
Oct 15
PrismML Ternary Bonsai 2: 2B, 4B and 9B Models From 0.43 GB, Apache 2.0PrismML added three small ternary models to the Bonsai 2 family, each more than 9x smaller than its full-precision counterpart. Here is what the company claims, what is unverified, and how to try them.
Oct 14
Llama Windows: The Free llama.cpp Tray App for Windows 11 and RTX SparkHugging Face and the ggml team shipped Llama Windows, a free open-source tray app that runs llama.cpp locally, downloads models from the Hub in one click and exposes an OpenAI-compatible API on localhost.
Oct 13
Janas: Run an 80B MoE Model on a 32 GB Laptop by Streaming Experts From SSDJanas is a new GPL-3.0 inference engine written in C that keeps mixture-of-experts weights on an SSD and streams only the experts each token needs. Its author reports an 80B-parameter model answering on a 32 GB laptop. Here is how it works, what the numbers do and do not show, and how to try it safely.
Oct 13
LFM2.5-24B-A2B: Liquid AI's Open 24B Agent Model With Only 2B ActiveLiquid AI released LFM2.5-24B-A2B, a 24B-parameter reasoning model that activates only 2B parameters per token. It targets local agents in OpenClaw, Hermes Agent, Pi and OpenCode. explainx.ai walks through the numbers, the architecture, how to run it and where it is weaker.
Oct 13
Qualcomm CEO: AI Labs Want 100B-Parameter Models on Phones by 2028. What It Would TakeQualcomm CEO Cristiano Amon said on October 8 that AI companies want phones that continuously run models around 100 billion parameters by 2028. It is a customer request, not a product promise. Here is what is verified and the memory and quantization math behind it.
Oct 12
MiniCPM-V 4.7: A 35B MoE Vision Model That Runs on an iPhoneOpenBMB has published MiniCPM-V 4.7, its first mixture-of-experts vision model. The 35B-A3B checkpoint is Apache-2.0 and, per the model card, runs on phones using Lookahead Expert Scheduling. Here is what is documented, what is self-reported, and how to try it.
Oct 11
Local AI With Hermes Agent and Qwen: What The Verge Diary Teaches BeginnersThe Verge laptop reviewer Antonio G. Di Benedetto started a local AI diary: Hermes Agent plus a 125-billion-parameter Qwen model on a 256GB M5 Ultra Mac Studio. His first tasks, a daily briefing, a Steam library clean-up and private data analysis, show what local agents are actually good for today and where they still fail.
Oct 10
llama.cpp Metal Gets Few-Row MMA Kernels: Up to 4x Faster Verification on M3 Ultrallama.cpp build b11404 adds Metal matrix kernels tuned for 2 to 16 rows, the shape speculative decoding produces when a model verifies draft tokens. A follow-up PR extends them to nearly every quantization type. Here is what the benchmarks actually show and what they do not.
Oct 9
Cactus Whistle: A 16.9 MB Speech-to-Text Model That Runs on a CPUOn October 2, 2026, Cactus Compute released Whistle, a 16.9 MB Apache 2.0 speech recognition model that runs on one CPU engine beside the Needle tool-calling model. It beats Whisper base on several benchmarks and loses on others. Here is the full picture.
Oct 9
Samsung LittleBit: A 13B LLM Under 1 GB, and What the Paper Really ShowsSamsung Research released LittleBit code that compresses LLMs to 0.1 bits per weight by factorizing and binarizing each weight matrix. The headline says a 13B model fits under 1 GB. The paper says so too, but the method needs training, and the license blocks commercial use.
Oct 8
Surface Laptop Ultra: Which Configuration to Buy for Local AIMicrosoft opened preorders for the Surface Laptop Ultra on October 7 with prices from $2,599.99 and shipping from October 16. The base 24GB model and the 128GB model are very different machines for local AI. This guide does the memory math, ties it to the new MAI-Code-1.1 Flash, and says who should buy which tier or wait.
Oct 7
Oki Home: A $1,799 "Memory Computer" With a Local Qwen 3.8 27B and a Swappable MemchipYC-backed Oki launched Oki Home on October 6, 2026: a 5-liter PC with an RTX 5060 Ti, 32 GB RAM and a credit-card-sized 2 TB Memchip that holds your photos, messages and files on one searchable timeline, with a local Qwen 3.8 27B model claimed at 106 tokens per second. Reservations are $59 and shipping is planned for mid-December. Here is what is claimed, what is not shown, and how it compares with doing it yourself.
Oct 5
Ghost Core: The $3,499 Personal AI Computer That Runs Models On DeviceGhost launched Core, a $3,499 personal AI computer that runs open-weight models entirely on device, ingests your apps, files and wearables, and acts without being asked. Batch 1 is already sold out. Here is what Ghost has said, what it has not, and how it compares with building your own local AI box.
Oct 3
DwarfStar 4: Can You Run ds4 on Your Mac This Week?Salvatore Sanfilippo's ds4 is back on the Hacker News front page. The useful question is whether your Mac, Spark, or Strix Halo can hold a project quant this week, and whether Claude Code or Codex can talk to it.
Oct 3
llama.cpp Adds Decision Model Support for Five Open ModelsOct 3
DGX Spark 64GB: What $4,999 Actually Buys on Oct 23
September 2026
Sep 18
PrismML Bonsai 2 27B: 9x Smaller, 98.2% of Full-Precision QualitySep 17
BITCOS: Breaking the 1.58-Bit Barrier for Ternary LLMsSep 10
Edge0 Open-Sources a 35B Model That Runs in 2.5GB of MemorySep 5
How AI Toothbrushes Work: Sensors, Models, and Dyson's CameraSep 3
Tether Released Offline Translation Models for 19 African LanguagesSep 2
Gemma 4 26B A4B Runs 2x Faster on Mac — Community MLX Optimization
August 2026
Aug 25
Apple M6 Mac mini: On-Device AI, Dual Neural Engine, and What 32GB Actually RunsAug 25
Mac Studio M5 Max and M5 Ultra: On-Device AI for Builders Who Outgrew the MiniAug 25
Can You Use MacBooks as Servers? Namespace’s Rack Video ExplainedAug 25
Xiaomi Xring O3: 44MB Cache and the Wide-Core Trend Daniel Lemire SpottedAug 24
FreeToken Runs a 753B MoE Locally — But the GPU Is Only Half the StoryAug 24
Pipette: Liquid AI’s Open On-Device Benchmark SuiteAug 24
OCR It: Pin a Region, Hotkey Through a Book, Feed Your LLM OfflineAug 20
S1-mini: Superwhisper's 0.6B On-Device Transcript CleanerAug 19
DFlash-MLX Brings Lossless Speculative Decoding to Apple Silicon — Up to ~189 tok/s on M5 MaxAug 19
RAM Prices Are Up 500% — What That Means for Local AI BuildsAug 17
Matic Cues: How a Home Robot Runs Voice, Vision, and Mapping On-DeviceAug 16
A 1.5B Model That Writes Your Shell Commands — On a Laptop CPU, No GPUAug 12
Unsloth Desktop: One Local App That Both Trains and Runs AI ModelsAug 5
LFM2.5-2.6B: Liquid AI's Biggest On-Device Agent Model YetAug 3
BitNet on a 1975 6502: A Language Model Inside 25KB
July 2026
Jul 30
TurboFieldfare: Gemma 4 26B in ~2 GB RAM on Apple SiliconJul 28
Framework Laptop 13 Pro: Modular Hardware for Local AI (2026)Jul 26
28.9M Params on an $8 ESP32 — How It FitsJul 23
Petals Resurfaces: Why BitTorrent-Style LLM Inference Still StrugglesJul 19
Moonshine Micro: Voice AI on 80-Cent MCUs — RP2350 VAD, STT, and Neural TTS (2026)Jul 17
LM Studio Bionic: Open-Model Agent for Code and Work ProjectsJul 14
Tencent Hy3 GGUF — 1-Bit and 4-Bit Quants for Single-GPU llama.cppJul 12
Mesh LLM v1.0: Split 235B Models Across Your LAN with iroh P2PJul 9
What Is Ollama? $88M Funding, 9M Builders, and the Open-Models Bet (July 2026)Jul 4
Local LLMs Keep Looping? Fix It With Samplers, Not More VRAMJul 4
SOTA LLMs Locally: Jamesob’s $46k RTX PRO 6000 Hardware GuideJul 3
Can You Self-Host Your Photos? Immich 3.0, Privacy, Costs, and When to Blur FacesJul 2
What Is llama.cpp? Install, Run GGUF Models, and Serve OpenAI-Compatible APIs
June 2026
Jun 29
MacBook vs dedicated GPU for local LLMs: how much RAM you really get, and when each wins in 2026Jun 29
Ollama 0.31: Gemma 4 Is ~90% Faster on Apple Silicon With Multi-Token Prediction (No Output Change)Jun 26
LFM2.5-230M: Liquid AI's 230M Model Built to Run Agents on Phones and RobotsJun 26
TREK: Self-Hosted Travel Planner with Real-Time Maps, Budgets, and AIJun 17
NVIDIA DGX Spark: The Best Setup for Running Local LLMs in 2026Jun 16
What Is AI Model Quantization? Running Frontier AI LocallyJun 15
Build Your Own Personal AI System: The Complete 2026 Guide to Local Models, Frameworks, and Workflows