ColGrep-Agent-2B: A Local 2B Model That Cut Claude Code Tokens by 34%

LightOn has released ColGrep-Agent-2B, an open 2-billion-parameter code-search agent that runs on your machine and sits between your coding agent and the repository. In the authors' own test with Claude Code, delegating code search to it cut API tokens by 34% and cache-aware API cost by 40% relative to plain grep, with no statistically significant change in accuracy, according to LightOn's write-up on Hugging Face. The catch: each question took longer.

If you pay per token for a frontier coding agent, this is a concrete pattern worth understanding: let a small local model do the "where is the code" legwork, and keep the big model for reasoning and edits.

TL;DR: the questions people will ask

What it does

Before a coding agent can fix a bug it often spends several turns working out where to look: grep a name, open a file, follow a reference, try another search. Each turn goes through the frontier model and resends the growing conversation. ColGrep-Agent-2B takes over that exploration. Given a question like "Where are retries configured?", it searches with ColGrep, reads candidate files through a read-only terminal, and returns file paths, line ranges and the code itself.

ColGrep is the semantic code search engine, built on the NextPlaid search engine, combining natural-language search with regex filtering. ColGrep-Agent-2B is the model, and colgrep --agent is the command that runs the model with ColGrep and the terminal. The model card describes a 10-turn budget and three tools: colgrep, terminal and finish.

Local small model handing focused code locations to a larger coding agent, shown with delegation arrows for ColGrep-Agent-2B

How to install and try it

From the model card:

brew install lightonai/tap/colgrep && colgrep --install-agent
cd /path/to/project
colgrep init
colgrep --agent "where does the agent decide which GPU backend llama.cpp uses"

The install step downloads the default colgrep-agent-2B-Q8_0.gguf file, verifies it loads, and prepares a prompt cache. Inference uses the GPU when available (Metal on Apple Silicon, Vulkan on NVIDIA, AMD and Intel) or falls back to CPU. On an M3 Pro, LightOn reports roughly 20 seconds per question with Metal and 3 to 4 GB peak memory.

To connect it to an agent:

colgrep --install-claude-code
colgrep --install-codex
colgrep --install-opencode

For Claude Code these install hooks that suggest the search tool; they do not force Claude to use it. You can change the suggestion with colgrep settings --hook-mode: hybrid (default, balanced), agent (largest token savings) or colgrep (plain semantic search only). If you are new to the command surface, our Claude Code commands reference covers hooks and settings in general, and our guide to running open models locally with OpenCode covers the local side.

The test: how LightOn measured it

LightOn used Claude Code running Sonnet 5.5 on 52 code-search questions across 13 public repositories created in 2026, to reduce the chance the models had seen them in training. The repositories span Rust, Go, TypeScript and Python, from 17k to 1.8M lines. There were 39 cross-folder questions and 13 single-location ones, each run three times in four setups, for 624 sessions. Opus 5.5 graded answers without knowing the setup, and the authors computed bootstrap confidence intervals over questions.

Token totals count Claude Code's billed input and output tokens and exclude tokens generated by the local 2B model, since those are not billed.

Context compression funnel narrowing many search turns into one focused result, illustrating Claude Code token savings from ColGrep-Agent

Why it saves tokens

The savings come mainly from fewer Claude turns. Each extra turn resends the accumulated context, about 23k tokens per turn in these runs. With grep, Claude averaged 5.5 turns, 5.1 grep/find/ls commands and 2.0 file reads per question; with ColGrep-Agent it averaged 3.9 turns, 2.0 grep commands and 0.9 file reads.

Savings grew with repository size: 19% below 50k lines, 29% from 50k to 150k, and 40% from 150k to 600k. ColGrep-Agent saved tokens on all 13 repositories. The write-up's worked example is a CRM repo where the code says "field" but the question says "custom attributes": grep took six turns and 160.6k tokens, while the agent's first semantic search landed in the right module and the job finished in three turns and 79.2k tokens. The authors note that this question was a better-than-average case at 51% savings.

This matches a pattern we have covered before: spending less context is often the biggest lever on agent cost. See our pieces on context compression for agents and managing AI coding costs at scale.

How good is a 2B search agent on its own?

On the model card, LightOn adapted SWE-bench Lite and Multi-SWE-bench Flash into code-localization tasks (find the right files and lines, not write the patch). Scores are file F1 and line F1, averaged over five seeds with 300 instances per seed:

The 2B model beats several much larger models on line-level F1 on SWE-bench Lite and approaches them on file-level F1, which is the point: it is trained for one job. Training had two stages: a short supervised cold start using traces from Qwen3.5-27B as teacher, then 500 steps of GRPO reinforcement learning in a code-localization environment that rewards accurate file and line identification and penalizes unnecessary exploration. The paper is on arXiv as 2610.05099, titled "Small Agents with Semantic Search: Efficient Multilingual Code Localization".

The model-in-the-middle pattern

LightOn frames the agent as a "model in the middle": a small specialist that sits between the expensive model and the environment, so the expensive model sees results instead of raw exploration. The idea generalizes. Any step that is repetitive, well-defined and verifiable, such as finding files, summarizing logs or checking whether a test passed, can in principle be handed to a cheap local model, as long as the big model keeps the final say. The trace judge in the study supports the practical side: in head-to-head comparisons with grep, it ranked the agent's trace higher 75% of the time, and a common pattern was one delegated search replacing several rounds of guessing symbol names.

The vocabulary mismatch example shows why semantic search matters here. Grep is exact, so Claude must guess the project's naming before it can search. ColGrep's multi-vector retrieval can match a description of the behavior to code that uses different words, which is also where a local agent that can iterate on queries helps most.

How to run your own check

Do not rely on the published numbers alone. A quick evaluation on your own repository takes an hour:

1. Pick 10 to 20 real questions you have asked your coding agent recently, ideally some that span several folders.
2. Run each with your current setup and note tokens, time and whether the answer was right.
3. Switch the hook mode to agent, rerun, and compare. Then try hybrid.
4. Watch cache behavior: LightOn's cost figure is cache-aware, so compare your actual billing, not just raw token counts.
5. Keep whichever mode gives the best accuracy-per-dollar for your tolerance for waiting.

Honest limitations

• Self-run, search-only results. The 34% figure is from the authors' benchmark of code-search questions, not end-to-end coding tasks, as they say themselves.
• Accuracy interval is wide. The 2.6-point drop (80.8% vs 83.3%) is not significant, but the interval runs from minus 10.9 to plus 5.1, so small-sample noise could hide a real loss. The Balanced setup, which kept 83.3%, may be the safer default.
• It is slower. 21.6 seconds vs 13.0 seconds per question; local calls took 7 s median and 22 s at P90 on a GPU server. Token savings buy lower bills, not faster answers.
• Local compute is free only in dollars. You need roughly a 2.5 GB download and a few GB of memory.
• Hooks are suggestions. The Claude Code integration nudges Claude to use the tool; it may still choose grep.

Should you use it?

If your coding-agent bill is dominated by exploring large repositories, this is cheap to test: install it, run colgrep init on one repo, and switch --hook-mode between agent and hybrid while watching token usage. If you value latency over cost, or work in small repos where savings were smallest (19% below 50k lines), the benefit is thinner. For the retrieval ideas behind ColGrep, our explainer on multi-vector ColBERT retrieval in Sentence Transformers v6 covers why late-interaction search finds code by meaning.

Related reading

• Claude Code commands: complete reference
• Run open-source models locally with OpenCode
• Headroom: context compression for agents
• Managing AI coding costs at scale
• Sentence Transformers v6 and multi-vector retrieval
• Fork-and-Flush for long-running agents
• SubQ Snap decision model

Figures are from LightOn's October 12, 2026 write-up and model card and may change with new releases.