XERJ Autoindex: Local Search for AI Agents and the "40x Fewer Tokens" Claim

XERJ is a new open-source search engine built for AI coding agents, and it gained more than 700 GitHub stars in a single day this week. The project, from the xerj-org GitHub organization, is written in Rust, licensed Apache 2.0, and speaks the Elasticsearch API. Its pitch is blunt: instead of letting an agent burn tokens on grep and sed, index your data once and let the agent query it. The repository's About line says "40x fewer tokens than grep." The README's own measurements are smaller, and that gap is the most useful thing to understand before you install it.

If you are new to how agents burn context, start with our guides on why agents need documentation, not just memory and the viral Caveman token compression skill.

TL;DR: questions answered

What XERJ does

XERJ stores documents in an index and lets you search them with Elasticsearch-style queries, then packages that for agents. The README lists five use cases:

• Reference coding: before writing code, the agent retrieves how similar open-source projects solved the same problem.
• Codebase Q&A and RAG, with results cited by file and line.
• Security audits: query for dangerous patterns and secrets across file types.
• Log and incident analysis.
• Agent long-term memory through a /_memory/{namespace} API.

Because it implements the Elasticsearch API, existing clients and Kibana can connect. The README is careful to say it is "compatible with the API but is not a fork or reimplementation of internals", and reports passing 1,366 of 1,369 conformance cases. This is a practical choice: you can point tools you already use at it rather than learning a new query language.

How to try it

The README's quick start is short. On macOS or Linux, using the installer hosted on the project's site, xerj.org:

curl -fsSL https://xerj.org/get | sh
xerj --insecure --data-dir ./data &
xerj init
xerj autoindex ~/my-project
xerj search "where do we validate webhooks"

On Windows the installer is irm https://xerj.org/get.ps1 | iex. Building from source needs a stable Rust toolchain and cargo build --release -p xerj-server. Note the pattern of piping a remote script to a shell: it is common, but read the script first, or build from source, especially on a machine that holds credentials.

xerj init wires XERJ into Claude Code or Cursor with an MCP server plus a skill. For other clients, xerj mcp exposes 13 tools. If MCP is new to you, our MCP primer explains the protocol. Other useful commands in the README are xerj def for definitions and xerj gain, which reports token savings.

The claim versus the data

Here is where the headline and the README diverge. The repo description says 40x fewer tokens than grep. The README body reports these figures from a reference-coding study of 8 tasks with 16 runs per arm:

That is 2.7x fewer output tokens than grep, about 26x fewer than memory alone, and the README says users report roughly 5x in everyday work. Those are real and interesting numbers. They are also from the project's own study with a small task count, run by the authors, so treat them as a vendor-style benchmark until someone reproduces them. We found no independent replication as of today.

Two caveats stated in the README matter more than the multiplier. First, the savings depend on the model not having memorized the code: gains are "neutral to harmful" on popular public libraries, where the model already knows the answer. Second, the study measures output tokens, not the input tokens your harness spends re-reading context, which is where cache costs and context rot usually bite.

Performance claims against Elasticsearch

The README also compares XERJ with Elasticsearch 8.13.4 in a closed-loop run: 55 wins, 26 ties and 4 losses, 1.72x ingest throughput and a 1.61x smaller disk footprint. All four losses are read p99 latency under a concurrent high-rate write load. Code extraction is reported at about 1,500 files per second on a Lucene checkout, single-threaded. An optional rerank on the FiQA dataset moved nDCG@10 from 0.24 to 0.36 at a total cost of about $0.64.

These are the author's benchmarks. They are useful for understanding where XERJ is strong, ingest and footprint, and where it is weaker, tail latency under heavy writes. For agent workloads, which are read-heavy and bursty, the write-load losses may not matter, but check that on your own corpus.

Security and privacy: read before you run

The README is candid about its limits, which we appreciate:

• --insecure disables TLS and authentication. Every caller becomes a superuser. A fresh node binds to 127.0.0.1 only, so the dev setup is local, but do not expose that port. Anything on your machine that can reach localhost, including a malicious dependency or a prompt-injected agent tool, can read and write the index.
• Rerank leaves your machine. Optional rerank sends result text to TypeSafe's servers, per the README, which also says ranking quality with the real model is unverified and that Jev scores are not well calibrated. For background on the model, see our posts on TypeSafe's Jev and the Grep launch week.
• Default embedder is lexical. Neural mode needs model files staged locally.
• Other limits: scroll is a bounded snapshot, so queries whose totals exceed the window get a 400 instead of truncated results, and binary-serialized Unity assets are unreadable unless Force Text is enabled.

Indexing is also a data-concentration risk. An index of your codebase, logs and PDFs is a single searchable store of secrets; the README itself advertises querying for secrets. Treat the data directory like the repository itself, keep it out of backups you do not control, and add it to .gitignore.

If you let agents run commands against local tools like this, AgentBeam, the agent security platform from the explainx.ai team, stops AI agents before they take dangerous actions, which is relevant when an agent can both read your secrets and call out to the network.

How it compares with other approaches

XERJ is closest to a search server your agent can call, not a prompt trick. That makes it complementary to the output-side savings in Caveman and to the skills ecosystem described in our Claude plugin guide.

A one-week evaluation plan

Because the benchmarks are self-reported, the best evidence is your own. Here is a simple protocol that takes minutes to set up and gives you numbers you can trust more than a README table:

1. Pick three tasks you do often, such as "find where we validate webhooks", "add a field to this API", and "explain why this job fails". Write them down so you can repeat them.
2. Run each task twice with your usual agent: once with XERJ connected through xerj init, once without. Keep the model, prompt and repository identical.
3. Record total tokens, wall-clock time, and whether the answer was correct. Your harness usually shows tokens; xerj gain reports the tool's own estimate, which you should compare with the harness count rather than trust alone.
4. Repeat on a repository your model is unlikely to have seen, such as a private monorepo, and on a popular open-source one. The README predicts the second will show little or negative gain, and testing that prediction tells you whether the tool fits your work.
5. Watch for stale indexes. If the agent edits files, check that the index refreshes; an agent that searches an outdated index can confidently cite code that no longer exists.

Also read the project's Apache 2.0 repository issue tracker before adopting. At capture it listed 15 open issues and 19 open pull requests, which is a normal load for a week-old viral project but means the surface is moving. Pin the version you install, because flags such as --insecure and the MCP tool list may change.

Should you adopt it?

A reasonable plan: try it on one large, private repository your model has not memorized, run xerj gain over a week, and compare token spend with your current harness. Skip it for small repos where grep is fast, and for popular open-source code the model already knows. Do not run --insecure outside localhost, and leave rerank off for proprietary code.

The honest takeaway is that the product idea is sound, indexing beats rereading, and the repo's own data supports a moderate improvement, not 40x. The hype figure is a reminder to read the README body rather than the blurb.

Related reading

• Agents don't need memory, they need documentation
• Caveman token compression
• What is MCP?
• TypeSafe Jev and grep.ai workflow distillation
• Grep AI launch week
• Top 25 Claude plugins

Star counts, flags and benchmarks are as published in the repository README on October 15, 2026 and may change.