explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

← All topics

explainx / blog / topics

Local AI

Local AI means running models on your own laptop, workstation, or phone instead of a cloud API. It trades some capability for privacy, cost control, and offline use.

This page collects our coverage of local runtimes, on-device models, and the hardware that makes them practical.

87 stories · latest Nov 17, 2026

Start here

Together AI and Ollama: Prefix Cache Rules for Long Agent Sessions

On November 16, 2026 Together AI and Ollama co-authored a guide to prefix caching for agentic workloads. Coding agents resend the full session each turn; this post turns their harness and engine checklist into actions for builders shipping long-horizon tools.

192GB Local AI Without a $9,500 Mac: Hardware Paths and Real Trade-offs

You need about 192GB of fast memory to run quantized DeepSeek-class models with any prompt headroom. This guide maps the non-Mac options Martin Weinberg demos on YouTube and cross-checks them against Lucebox and Framework public benchmarks.

GLM-5.3 on Two DGX Stations: NVIDIA Shows 753B Running Locally with vLLM

NVIDIA published a tutorial for serving the 753-billion-parameter GLM-5.3 NVFP4 checkpoint across two DGX Stations with vLLM and tensor parallelism. A single GB300 GPU cannot hold the 464 GB of weights, so the two machines split them over a 400 Gb/s RDMA link.

Semantic Caching for LLM Apps: Exact Hash, Embeddings and Safe Rollouts

Prompt caching on APIs discounts repeated prefixes; semantic caching goes further by reusing whole answers when a paraphrased question is close enough. This guide covers exact-hash layers, embedding search, false-positive risk, and how to ship caching without breaking tool-calling agents.

What Is the KV Cache in LLM Inference? Memory, Context and Tuning

Every new token in an LLM forward pass needs attention over all prior tokens. The KV cache keeps those past key and value tensors in GPU memory so decode steps stay fast. That speed comes at a price: VRAM grows with context length, layer count, and concurrent sessions — the main reason long chats and big batches OOM on local and server GPUs alike.

Run LLMs on Apple Silicon With MLX: Complete 2026 Guide

Apple’s MLX framework is the native path for LLMs on M-series Macs. The mlx-lm package pulls Hugging Face models, quantizes them, exposes a chat REPL, and can serve OpenAI-style traffic — without renting a Linux GPU box for every experiment.

Timeline

November 2026

  1. Nov 17

    Mobile AI Factory: Agentic System That Builds Fine-Tuned SLMs for Your Phone

    Mobile AI Factory is an agentic system from UW researchers that searches model size, quantization, and inference backends using live Android measurements. On ten tasks and four phones it matches or beats five-shot frontier models on eight tasks while cutting GPU search cost.

  2. Nov 16

    Together AI and Ollama: Prefix Cache Rules for Long Agent Sessions
  3. Nov 14

    192GB Local AI Without a $9,500 Mac: Hardware Paths and Real Trade-offs
  4. Nov 13

    Cactus Mirage: On-Device Vision and Structured Image Extraction

    Cactus Compute added Mirage, a vision model for local describe and structured extraction, beside Whistle speech and Needle tool-calling. The Mirage weights trended on Hugging Face in November 2026 with more than 200 likes.

  5. Nov 13

    Edge0-145B: Open 145B Agent Model Built to Run on Your Laptop

    Edge0 released open Edge0-145B and Edge0-120B weights on November 12, 2026, targeting a single NVIDIA GPU with edge0-strata. The announcement claims frontier-class coding and agent scores at roughly 29GB peak memory on laptop-class hardware.

  6. Nov 13

    GLM-5.3 on Two DGX Stations: NVIDIA Shows 753B Running Locally with vLLM
  7. Nov 13

    InferSSD: Running 800B-Class MoE Models From NVMe on Consumer GPUs

    InferSSD is an open llama.cpp-based runtime that stores routed mixture-of-experts weights on NVMe while attention and KV cache stay in RAM or VRAM. WiCi Research published benchmark tables for DeepSeek-V4.1-Flash-1017 and Muse-Spark-Open on RTX 5090, MacBook Pro, and DGX Spark class machines.

  8. Nov 13

    llama.cpp v1.0.0: What a Stable libllama and libggml Mean for Local AI

    ggml-org tagged llama.cpp v1.0.0 on November 13, 2026, after about 3.5 years of development. libllama and libggml now follow semantic versioning, so breaking C API changes show up as major-version bumps. Here is what that promise covers, what it does not, and how to upgrade safely.

  9. Nov 12

    gputaskman: Scott Hanselman GPU Task Manager for LLM VRAM and KV Cache

    gputaskman crossed 2,400 GitHub stars in November 2026 as a cross-platform GPU task manager aimed at developers running local LLMs. Here is what the README promises, which GPUs it supports, and when to use it beside nvidia-smi or llama.cpp logs.

  10. Nov 12

    TensorFold: Zig LLM Server for Metal and CUDA With an OpenAI-Compatible API

    TensorFold ships language models from a single Zig binary on Apple Metal and NVIDIA CUDA, with Vulkan and HIP paths in development. It pulls Hugging Face checkpoints, serves chat and Responses APIs on port 8080, and needs no Python or MLX install at serving time. The GitHub repo gained fast traction in November 2026. Here is how it works and who should try it.

  11. Nov 9

    Semantic Caching for LLM Apps: Exact Hash, Embeddings and Safe Rollouts
  12. Nov 8

    llama-modchip: 64 GB CPU Path for DeepSeek-V4 and 30B MoE GGUF

    The synchopate/llama-modchip repository (v2.4.0) documents CPU-only inference for DeepSeek-V4-Flash Fractal GGUF near 57 GiB RAM at 256K context and Nemotron-3.5-30B-A3B at roughly 14–17 GiB, using custom Q2_0 tensors and Stockfish-PVS speculative decoding on llama.cpp forks.

  13. Nov 8

    What Is the KV Cache in LLM Inference? Memory, Context and Tuning
  14. Nov 7

    Run LLMs on Apple Silicon With MLX: Complete 2026 Guide
  15. Nov 7

    vLLM vs SGLang: Which GPU Inference Server Should You Run in 2026?

    If you self-host models on NVIDIA GPUs in 2026, vLLM and SGLang are the two frameworks teams argue about most. Both expose OpenAI-compatible HTTP APIs, both support MoE models, and both ship fast attention kernels — but vLLM leads on ecosystem breadth while SGLang leads on structured generation and RL-heavy serving. Here is how to choose.

  16. Nov 5

    Mercedes-Benz and Liquid AI: On-Device Car Voice AI, but the Cloud Is Not Going Away

    A November 5, 2026 Wall Street Journal report revived Mercedes-Benz and Liquid AI on-device voice models for MBUX. A viral headline says local AI replaces the cloud. The April partnership announcement says otherwise. Here is the evidence.

  17. Nov 4

    llama.cpp v0.9.0 and ggml 0.29.0: Native FP8, Multi-Node Tensor Parallelism and What Local-AI Users Get

    llama.cpp tagged v0.9.0 on November 4, 2026. The ggml 0.29.0 library underneath adds a native OCP FP8 E4M3 tensor type, a new flash-attention op for multi-sequence serving, batched tensor copy APIs and initial multi-node tensor parallelism through NCCL, plus kernel work across CUDA, Metal, Vulkan, SYCL, OpenCL and Hexagon.

October 2026

  1. Oct 31

    dasllama: A Game-Scripting-Language Inference Engine Claims 5x Faster Speech-to-Text Than llama.cpp on CPU

    A new local inference engine written entirely in a game scripting language reports 4x to 7x CPU speedups for speech models and mixed results for LLMs. Here are the numbers, the losses the authors admit, and how to test it yourself.

  2. Oct 27

    Offline Coding Agent on a Laptop: OpenCode, Ollama and Qwen3.5 9B With the Wi-Fi Off

    Rittika Jindal, a principal engineer living where power and network outages are routine, tested whether a coding agent can work with the internet switched off. Her answer on a Mac laptop with OpenCode, Ollama and Qwen3.5 9B: yes for a small bug fix, with caveats. Here is the setup and what it does and does not prove.

  3. Oct 27

    Opera WebAI: A Proposed Web Standard for Running AI Models Locally in the Browser

    On October 27, 2026 Opera began shipping WebAI, a proposed web standard that lets websites and extensions call AI models running on your own computer. Here is what is confirmed, what is still unclear, and what it means for developers.

  4. Oct 27

    Sentient OS: The YC-Backed Open-Source Mac Agent That Works Overnight on a Local Gemma 4

    Sentient OS is an open-source, AGPL-3.0 Mac app from a YC F26 team that analyzes your mail, messages and files overnight with an on-device model, then offers to do the work. We checked the README and release notes for what is truly local.

  5. Oct 27

    Survival LLM: The Offline Emergency AI With a 930,000-Article Library, Tested on 22 Questions

    Survival LLM, trending on Hacker News on October 27, 2026, packages a local model and an offline emergency library. explainx.ai covers what it does, the author's own test numbers, install costs, and why commenters question whether an LLM belongs in an emergency kit.

  6. Oct 22

    Underdog Bark 3.0: A 163 MB Open Speech Model That Matches Parakeet

    Underdog released Bark 3.0, a 0.6B English speech recognition model in a 163 MB download under CC-BY 4.0. Its card says it averages 4.94% word error on seven Open ASR Leaderboard sets, edging NVIDIA Parakeet and beating Whisper large-v3. The numbers are self-scored.

  7. Oct 20

    PyTorch 2.15 on Mac: FP8, Faster MoE Decoding and an 8x Softmax on Apple Silicon

    A Hugging Face post by two PyTorch MPS contributors details what changes for Apple Silicon in PyTorch 2.15: FP8 tensors, faster dense and mixture-of-experts decoding, Metal kernels in place of MPSGraph, and a preview of 2.16.

  8. Oct 16

    Magnitude: Open-Source Local Inference Engine Claims 2x Faster Than Ollama

    Magnitude, a Y Combinator-backed open-source inference engine, says it decodes about 2.2x faster than Ollama on a Mac M4 Pro and on an NVIDIA DGX Spark by tuning kernels to each model and chip. Here is how it works, how to try it, and why the benchmark needs a careful read.

  9. Oct 16

    PS5LM: Run llama.cpp on a Jailbroken PS5 as a Home LLM Server

    A new open-source app, PS5LM, runs llama.cpp on a jailbroken PlayStation 5 GPU and serves OpenAI and Anthropic-style APIs to your network. We go through the claimed speeds, the setup, and what to be careful about.

  10. Oct 15

    PrismML Ternary Bonsai 2: 2B, 4B and 9B Models From 0.43 GB, Apache 2.0

    PrismML added three small ternary models to the Bonsai 2 family, each more than 9x smaller than its full-precision counterpart. Here is what the company claims, what is unverified, and how to try them.

  11. Oct 14

    Llama Windows: The Free llama.cpp Tray App for Windows 11 and RTX Spark

    Hugging Face and the ggml team shipped Llama Windows, a free open-source tray app that runs llama.cpp locally, downloads models from the Hub in one click and exposes an OpenAI-compatible API on localhost.

  12. Oct 13

    Janas: Run an 80B MoE Model on a 32 GB Laptop by Streaming Experts From SSD

    Janas is a new GPL-3.0 inference engine written in C that keeps mixture-of-experts weights on an SSD and streams only the experts each token needs. Its author reports an 80B-parameter model answering on a 32 GB laptop. Here is how it works, what the numbers do and do not show, and how to try it safely.

  13. Oct 13

    LFM2.5-24B-A2B: Liquid AI's Open 24B Agent Model With Only 2B Active

    Liquid AI released LFM2.5-24B-A2B, a 24B-parameter reasoning model that activates only 2B parameters per token. It targets local agents in OpenClaw, Hermes Agent, Pi and OpenCode. explainx.ai walks through the numbers, the architecture, how to run it and where it is weaker.

  14. Oct 13

    Qualcomm CEO: AI Labs Want 100B-Parameter Models on Phones by 2028. What It Would Take

    Qualcomm CEO Cristiano Amon said on October 8 that AI companies want phones that continuously run models around 100 billion parameters by 2028. It is a customer request, not a product promise. Here is what is verified and the memory and quantization math behind it.

  15. Oct 12

    MiniCPM-V 4.7: A 35B MoE Vision Model That Runs on an iPhone

    OpenBMB has published MiniCPM-V 4.7, its first mixture-of-experts vision model. The 35B-A3B checkpoint is Apache-2.0 and, per the model card, runs on phones using Lookahead Expert Scheduling. Here is what is documented, what is self-reported, and how to try it.

  16. Oct 11

    Local AI With Hermes Agent and Qwen: What The Verge Diary Teaches Beginners

    The Verge laptop reviewer Antonio G. Di Benedetto started a local AI diary: Hermes Agent plus a 125-billion-parameter Qwen model on a 256GB M5 Ultra Mac Studio. His first tasks, a daily briefing, a Steam library clean-up and private data analysis, show what local agents are actually good for today and where they still fail.

  17. Oct 10

    llama.cpp Metal Gets Few-Row MMA Kernels: Up to 4x Faster Verification on M3 Ultra

    llama.cpp build b11404 adds Metal matrix kernels tuned for 2 to 16 rows, the shape speculative decoding produces when a model verifies draft tokens. A follow-up PR extends them to nearly every quantization type. Here is what the benchmarks actually show and what they do not.

  18. Oct 9

    Cactus Whistle: A 16.9 MB Speech-to-Text Model That Runs on a CPU

    On October 2, 2026, Cactus Compute released Whistle, a 16.9 MB Apache 2.0 speech recognition model that runs on one CPU engine beside the Needle tool-calling model. It beats Whisper base on several benchmarks and loses on others. Here is the full picture.

  19. Oct 9

    Samsung LittleBit: A 13B LLM Under 1 GB, and What the Paper Really Shows

    Samsung Research released LittleBit code that compresses LLMs to 0.1 bits per weight by factorizing and binarizing each weight matrix. The headline says a 13B model fits under 1 GB. The paper says so too, but the method needs training, and the license blocks commercial use.

  20. Oct 8

    Surface Laptop Ultra: Which Configuration to Buy for Local AI

    Microsoft opened preorders for the Surface Laptop Ultra on October 7 with prices from $2,599.99 and shipping from October 16. The base 24GB model and the 128GB model are very different machines for local AI. This guide does the memory math, ties it to the new MAI-Code-1.1 Flash, and says who should buy which tier or wait.

  21. Oct 7

    Oki Home: A $1,799 "Memory Computer" With a Local Qwen 3.8 27B and a Swappable Memchip

    YC-backed Oki launched Oki Home on October 6, 2026: a 5-liter PC with an RTX 5060 Ti, 32 GB RAM and a credit-card-sized 2 TB Memchip that holds your photos, messages and files on one searchable timeline, with a local Qwen 3.8 27B model claimed at 106 tokens per second. Reservations are $59 and shipping is planned for mid-December. Here is what is claimed, what is not shown, and how it compares with doing it yourself.

  22. Oct 5

    Ghost Core: The $3,499 Personal AI Computer That Runs Models On Device

    Ghost launched Core, a $3,499 personal AI computer that runs open-weight models entirely on device, ingests your apps, files and wearables, and acts without being asked. Batch 1 is already sold out. Here is what Ghost has said, what it has not, and how it compares with building your own local AI box.

  23. Oct 3

    DwarfStar 4: Can You Run ds4 on Your Mac This Week?

    Salvatore Sanfilippo's ds4 is back on the Hacker News front page. The useful question is whether your Mac, Spark, or Strix Halo can hold a project quant this week, and whether Claude Code or Codex can talk to it.

  24. Oct 3

    llama.cpp Adds Decision Model Support for Five Open Models
  25. Oct 3

    DGX Spark 64GB: What $4,999 Actually Buys on Oct 23

September 2026

  1. Sep 18

    PrismML Bonsai 2 27B: 9x Smaller, 98.2% of Full-Precision Quality
  2. Sep 17

    BITCOS: Breaking the 1.58-Bit Barrier for Ternary LLMs
  3. Sep 10

    Edge0 Open-Sources a 35B Model That Runs in 2.5GB of Memory
  4. Sep 5

    How AI Toothbrushes Work: Sensors, Models, and Dyson's Camera
  5. Sep 3

    Tether Released Offline Translation Models for 19 African Languages
  6. Sep 2

    Gemma 4 26B A4B Runs 2x Faster on Mac — Community MLX Optimization

August 2026

  1. Aug 25

    Apple M6 Mac mini: On-Device AI, Dual Neural Engine, and What 32GB Actually Runs
  2. Aug 25

    Mac Studio M5 Max and M5 Ultra: On-Device AI for Builders Who Outgrew the Mini
  3. Aug 25

    Can You Use MacBooks as Servers? Namespace’s Rack Video Explained
  4. Aug 25

    Xiaomi Xring O3: 44MB Cache and the Wide-Core Trend Daniel Lemire Spotted
  5. Aug 24

    FreeToken Runs a 753B MoE Locally — But the GPU Is Only Half the Story
  6. Aug 24

    Pipette: Liquid AI’s Open On-Device Benchmark Suite
  7. Aug 24

    OCR It: Pin a Region, Hotkey Through a Book, Feed Your LLM Offline
  8. Aug 20

    S1-mini: Superwhisper's 0.6B On-Device Transcript Cleaner
  9. Aug 19

    DFlash-MLX Brings Lossless Speculative Decoding to Apple Silicon — Up to ~189 tok/s on M5 Max
  10. Aug 19

    RAM Prices Are Up 500% — What That Means for Local AI Builds
  11. Aug 17

    Matic Cues: How a Home Robot Runs Voice, Vision, and Mapping On-Device
  12. Aug 16

    A 1.5B Model That Writes Your Shell Commands — On a Laptop CPU, No GPU
  13. Aug 12

    Unsloth Desktop: One Local App That Both Trains and Runs AI Models
  14. Aug 5

    LFM2.5-2.6B: Liquid AI's Biggest On-Device Agent Model Yet
  15. Aug 3

    BitNet on a 1975 6502: A Language Model Inside 25KB

July 2026

  1. Jul 30

    TurboFieldfare: Gemma 4 26B in ~2 GB RAM on Apple Silicon
  2. Jul 28

    Framework Laptop 13 Pro: Modular Hardware for Local AI (2026)
  3. Jul 26

    28.9M Params on an $8 ESP32 — How It Fits
  4. Jul 23

    Petals Resurfaces: Why BitTorrent-Style LLM Inference Still Struggles
  5. Jul 19

    Moonshine Micro: Voice AI on 80-Cent MCUs — RP2350 VAD, STT, and Neural TTS (2026)
  6. Jul 17

    LM Studio Bionic: Open-Model Agent for Code and Work Projects
  7. Jul 14

    Tencent Hy3 GGUF — 1-Bit and 4-Bit Quants for Single-GPU llama.cpp
  8. Jul 12

    Mesh LLM v1.0: Split 235B Models Across Your LAN with iroh P2P
  9. Jul 9

    What Is Ollama? $88M Funding, 9M Builders, and the Open-Models Bet (July 2026)
  10. Jul 4

    Local LLMs Keep Looping? Fix It With Samplers, Not More VRAM
  11. Jul 4

    SOTA LLMs Locally: Jamesob’s $46k RTX PRO 6000 Hardware Guide
  12. Jul 3

    Can You Self-Host Your Photos? Immich 3.0, Privacy, Costs, and When to Blur Faces
  13. Jul 2

    What Is llama.cpp? Install, Run GGUF Models, and Serve OpenAI-Compatible APIs

June 2026

  1. Jun 29

    MacBook vs dedicated GPU for local LLMs: how much RAM you really get, and when each wins in 2026
  2. Jun 29

    Ollama 0.31: Gemma 4 Is ~90% Faster on Apple Silicon With Multi-Token Prediction (No Output Change)
  3. Jun 26

    LFM2.5-230M: Liquid AI's 230M Model Built to Run Agents on Phones and Robots
  4. Jun 26

    TREK: Self-Hosted Travel Planner with Real-Time Maps, Budgets, and AI
  5. Jun 17

    NVIDIA DGX Spark: The Best Setup for Running Local LLMs in 2026
  6. Jun 16

    What Is AI Model Quantization? Running Frontier AI Locally
  7. Jun 15

    Build Your Own Personal AI System: The Complete 2026 Guide to Local Models, Frameworks, and Workflows

May 2026

  1. May 28

    Coral Edge AI: Complete Guide to Google's Edge Computing Platform
  2. May 22

    Gemma 4 E4B and Argent: Local On-Device Automation for iOS
  3. May 17

    DreamServer: Your Complete Local AI Stack for LLM Inference, Chat, and Voice
  4. May 5

    Immich: self-hosted photo and video library (Google Photos–class, AGPL-3)

Other topics

  • Claude Code
  • OpenAI Codex
  • AI Coding Tools
  • Model Context Protocol (MCP)
  • Agent Skills
  • Decision Models
  • Anthropic and Claude
  • OpenAI and ChatGPT
  • Google Gemini and DeepMind
  • Meta AI
  • xAI and Grok
  • Microsoft, Apple and Amazon AI
  • Open-Weight Models
  • AI Agents
  • AI Safety and Alignment
  • AI Policy and Regulation
  • AI Security
  • AI Chips and Infrastructure
  • Robotics and Physical AI
  • AI Benchmarks and Evals
  • AI Research
  • AI Image, Video and Voice
  • Prompt Engineering
  • Learning AI and Careers
  • AI Tools and Apps
  • AI Industry and Business