quantization▌

The GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.

Fast, calibration-free weight quantization supporting 8/4/3/2/1-bit precision with multiple optimized backends.

4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss.