explainx / blog
AirLLM loads large language model layers one at a time from disk, enabling 70B inference on a single 4GB GPU without quantization, distillation, or pruning. It also runs 405B Llama 3.1 on 8GB VRAM. Here is how the layer-by-layer technique works, what the real performance trade-offs are, and when to use it over standard approaches.
Jun 23, 2026
GLM-5.2 has 744B parameters but only 40B are active at any time — that's what makes it runnable locally. The 2-bit dynamic GGUF fits in 239GB of disk/RAM. With Unsloth Studio's web UI, you can run it on a Mac without touching the command line. Here is the full guide.
Jul 3, 2026
Pasting a YouTube link into ChatGPT reads the transcript, not the picture. Claude often rejects video files outright. Here is what actually works in 2026 — native multimodal APIs, local frame+transcript pipelines like claude-real-video, and transcript-first agents like video-use — with honest limits and cost math.
Jun 29, 2026
video-use is an open-source skill for Claude Code (and Codex, Hermes, Openclaw) that edits videos via natural language — no timeline scrubbing, no NLE menus. It reads footage as transcript text, reasons over word-level timestamps, calls ffmpeg, self-evaluates every cut, and outputs final.mp4. 11.6k GitHub stars in two months. Here is the full setup and how it works.
There is a persistent myth in the local AI community: to run a 70B model, you need 140GB of VRAM. A 4-bit quantized version requires ~40GB. Consumer GPUs top out at 24GB. Therefore, 70B at anything approaching full precision requires a multi-GPU server or an expensive cloud instance.
AirLLM breaks this assumption. Not by magic — by trading speed for accessibility in a principled way that the community has found genuinely useful (21,000+ GitHub stars).
The technique is not new. Loading model layers sequentially from disk was used in academic settings before AirLLM made it accessible. What AirLLM did was turn it into a clean open-source library that works with Hugging Face models and takes three lines of code.
A transformer language model is a stack of attention layers. During a forward pass, the computation flows sequentially:
Standard inference loads every layer into GPU memory before any computation starts. This is fast — everything is resident — but requires the full model to fit in VRAM simultaneously.
AirLLM's observation: at any point in the computation, only one layer's weights are actually being used. The rest are just waiting.
So why load them all at once?
AirLLM's approach:
Peak VRAM usage: one layer's weights + working activations. For a 70B model with ~80 layers, each layer at FP16 is roughly 1.75GB. A 4GB GPU can handle that.
Every forward pass reads the full model from disk — sequentially. That I/O is the bottleneck.
| Hardware | 70B speed estimate | Comment |
|---|---|---|
| NVMe SSD + GPU | 1–3 tokens/sec | Best-case local setup |
| SATA SSD + GPU | 0.5–1 tokens/sec | Meaningfully slower |
| HDD + GPU | < 0.3 tokens/sec | Not practical |
Compare to full-VRAM inference on appropriately-sized hardware: 15–30 tokens/sec for 70B on high-end multi-GPU. AirLLM is 5–30x slower depending on disk speed.
This is the honest trade-off. AirLLM does not make 70B fast on a 4GB GPU. It makes 70B possible on a 4GB GPU.
The use cases where this trade-off makes sense:
pip install airllm
from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("meta-llama/Llama-3-70B-Instruct")
input_text = ['Explain the trade-offs in distributed system design.']
input_tokens = model.tokenizer(
input_text,
return_tensors="pt",
return_attention_mask=False,
truncation=True,
max_length=MAX_LENGTH,
padding=False
)
generation_output = model.generate(
input_tokens['input_ids'].cuda(),
max_new_tokens=20,
use_cache=True,
return_dict_in_generate=True
)
output = model.tokenizer.decode(generation_output.sequences[0])
print(output)
Important: On first run, AirLLM downloads the model from Hugging Face and splits it into layer shards. This requires approximately 2x the model size in disk space during the splitting process (the original plus the shards). A 70B FP16 model needs ~280GB free during conversion. After conversion, set delete_original=True to reclaim half the space.
Layer-by-layer loading is slow because loading from disk is slow. Make the layers smaller, loading gets faster.
AirLLM's compression is block-wise quantization of weights only — not activations. This distinction matters:
Standard quantization quantizes both weights and activations because both contribute to compute bottlenecks. AirLLM's bottleneck is disk I/O, not compute. It can quantize only weights, which is easier to do accurately because you're not dealing with dynamic runtime values.
pip install -U bitsandbytes
pip install -U airllm
model = AutoModel.from_pretrained(
"meta-llama/Llama-3-70B-Instruct",
compression='4bit' # or '8bit' for a quality/speed balance
)
Reported speedup: ~3x. Accuracy impact: comparable to standard 4-bit quantization — very small for most tasks, slightly more on specialized domains.
With compression enabled, a 70B model's layer shards drop to ~18GB total (from ~140GB), which also means the loading-to-VRAM step is proportionally faster.
model = AutoModel.from_pretrained(
"model-id-or-local-path",
compression='4bit', # '4bit', '8bit', or None
profiling_mode=False, # True to log per-layer timing
layer_shards_saving_path=None, # Custom path for shards
hf_token=None, # For gated models
prefetching=True, # Overlap load + compute (~10% speed boost)
delete_original=False, # Remove original after splitting
)
The prefetching parameter is the most impactful free optimization — it starts loading the next layer into a buffer while the current layer is computing. Enabled by default for Llama models.
AutoModel.from_pretrained() detects the model class automatically. No need to import AirLLMLlama2 specifically.
Supported:
For gated models:
model = AutoModel.from_pretrained(
"meta-llama/Llama-2-7b-hf",
hf_token="your_hf_token"
)
Apple Silicon machines have unified memory — VRAM and system RAM are the same pool. A Mac Studio with 192GB unified memory can technically hold a 70B FP16 model in memory without AirLLM. A Mac with 16GB can't.
AirLLM works on Apple Silicon (M1/M2/M3 only). Install mlx and torch, ensure Python native is available (not Rosetta), and run the same code as Linux.
The relevant consideration: a 4GB limit on Mac corresponds to using an old MacBook Air or a non-unified-memory setup. Most modern Apple Silicon Macs have 16–192GB unified memory, which changes when AirLLM is useful. On those machines, the layer-loading approach may still be worthwhile for models larger than available unified memory.
AirLLM does not solve the problem for production serving. It solves the problem for individual inference where hardware is the constraint.
If you need:
...AirLLM is not the answer. Those requirements need hardware that fits the model.
AirLLM's value is exactly where its trade-off is acceptable: personal use, development workflows, research environments, and batch tasks where the alternative is not running the model at all.
At 21,000+ stars, the community has clearly found that constraint useful often enough.