Cutoff Bench: Do LLMs Know Their Own Knowledge Cutoff? A Solo Test Says Not Always

Every large language model has a knowledge cutoff, the date after which it has seen little or no training data. Ask a model what its cutoff is and you get an answer. Cutoff Bench, a small open project posted to Hacker News on October 14, 2026, asks a sharper question: is that answer true?

The method, per the project's README, is simple. It asks models through OpenRouter, with no tools and no system prompt, what their knowledge cutoff is, then which models are state of the art. It compares the cutoff the model reports with the newest model it actually remembers. A gap between the two suggests the stated date is optimistic, or that the model's memory of its final months is thin.

This is one developer's experiment, not an established benchmark. It had 3 points on Hacker News when we checked, and we have not re-run it. But the logs are public and illustrate something every builder should know.

TL;DR

What the sample logs show

The repository includes short logs of single answers. We read the cutoff log and the state-of-the-art log. These are individual responses, not averages, so treat them as illustrations.

Two things stand out. First, several models gave different answers to the two questions. Gemini 4 Argon said March 2026 for its cutoff but described the landscape "as of mid-2024" when asked for state of the art. DeepSeek gave October 2023 in one answer and June 2024 in another. A model whose stated date and remembered landscape disagree is exactly what the benchmark is built to expose.

Second, even the best-informed answers lag the frontier. GPT-6.1 Sol reported December 2025 yet opened its list with o3. That does not prove its training data stops earlier, since a model may simply list familiar names first, but it is the kind of mismatch the project flags.

We cannot say these single answers are representative. The repository also contains a larger runs file of roughly 327 KB, which the page uses for its chart. We did not analyze it in full.

Why models misreport their cutoff

This is not a bug in one vendor's model. It follows from how training works.

The final months are thin. Opus 5.5's own answer in the log says its knowledge of the last months before the cutoff is likely thinner than for earlier periods "since less had been written about recent events when my training data was collected." The web keeps accumulating text about an event for months or years after it happens. A model trained soon after has seen only the first trickle.

Models are not told the date. Unless a system prompt provides it, the model infers its cutoff from patterns, and the inference is often biased early or late. Kimi K3's answer, noting that "the current session date is 2026-04-28, but that's separate from a training-data cutoff," shows a model separating the two correctly.

Post-training shifts things. Instruction tuning and reinforcement learning may add data of different vintages, so the effective cutoff for different topics can differ.

Refusing to answer is common. Grok 4.7 and Kimi K3 both declined to give a single date. That is arguably the most honest answer, but it makes the field harder to compare.

What this means for what you build

• State the date. Put today's date in the system prompt. Without it, the model may reason as if it were still its cutoff year.
• Do not trust a model's list of "latest" libraries or models. Our guide to grounding, RAG and fine-tuning covers when to inject current documents.
• Use search for anything time-sensitive. Services such as Cloudflare's web search API for agents exist because stale priors are a standing problem.
• Do not mistake the cutoff for recency of skill. A model with a later cutoff is not automatically better. See how the leaders compare in Muse Wonder vs GPT-6.1 Sol vs Opus 5.5.

How to run Cutoff Bench yourself

The repository is five small files and a data folder: ask.py (the query script), serve.py (serves the page locally), index.html, the data logs and a screenshot. To try it:

1. Clone the repository from GitHub.
2. Set OPENROUTER_API_KEY in your environment.
3. Run python3 ask.py "<your question>" gemini to query models whose names contain "gemini". Results append to data/runs.jsonl.
4. Run python3 serve.py and open the page locally to see the chart.

Cost depends on which models you query; check OpenRouter's pricing before running a large sweep. We have not run it ourselves.

Reported cutoff vs effective cutoff: a practical test

The useful idea in Cutoff Bench is the difference between two numbers. The reported cutoff is what the model says when asked. The effective cutoff is the latest period it can reliably talk about. They need not match, and for most work the second one is what counts.

You can approximate the effective cutoff for any model you depend on without special tooling. Here is a simple routine, in the spirit of the project:

1. Ask the model for its knowledge cutoff, with no system prompt, and record the answer verbatim.
2. Ask it to list the newest models, libraries or tools it knows in a field you care about, such as agent frameworks or your main programming language's package ecosystem.
3. Pick the newest item on that list and look up its real release date.
4. Compare. If the newest thing it knows is a year older than the cutoff it claims, plan on an effective cutoff of that older date for your use.
5. Repeat three to five times, because answers vary between runs.

Then adjust your prompts. If the effective cutoff is old for the area you work in, add current documentation to the context, use retrieval, or give the model search access. For fast-moving topics such as model names, API parameters and pricing, assume the model is out of date regardless of its reported date.

Questions people are asking

Is a later cutoff always better? No. Cutoff says how recent the training text is, not how good the model is at reasoning, coding or following instructions. A model with an earlier cutoff plus search can beat one with a later cutoff and no tools on current-events questions.

Should I trust a model that gives a precise date? A precise date is more specific, not more accurate. Several models in the sample gave exact months, while the model that hedged was at least explicit about uncertainty. Without an independent check of the kind Cutoff Bench attempts, a confident answer is just a confident answer.

Why do vendors not publish the cutoff clearly? Many do list a cutoff in model cards or documentation, and those are the better source than asking the model. The point of the benchmark is that the model's own claim and its behavior can diverge. We would treat documentation as the stated cutoff and behavior as the effective one.

Does this affect agents? Yes. Agents that write code or call tools rely on the model's picture of current APIs. Stale assumptions show up as calls to deprecated endpoints or missing newer libraries. Giving the agent the documentation it needs, as in the retrieval patterns in our grounding guide linked above, removes most of that risk.

Limits of the benchmark

• Sampling noise. Language models give different answers on repeat. The repo has a samples log, but a handful of runs per model cannot give tight error bars.
• Prompt sensitivity. Asking "what is your cutoff" and "which models are SOTA" are two prompts; changing the wording could change the result.
• Remembering is not the same as knowing. A model might know about a release but not list it when asked for a short summary.
• Moving target. Providers update models behind the same name, and OpenRouter routes may change.

These do not make the project useless. They make it a prompt for you to test the models you depend on, not a leaderboard to quote.

Related coverage

For how the headline models in the logs launched, see Gemini 4 Argon and Claude Opus 5.5 topping the Epoch Capabilities Index. For basics on how models work, see How to use AI: a fundamental guide.

Related reading

• Grounding, RAG and fine-tuning: a decision guide
• Cloudflare web search API for agents
• Gemini 4 Argon launch, benchmarks and pricing
• Claude Opus 5.5 and the Epoch Capabilities Index
• Muse Wonder vs GPT-6.1 Sol vs Opus 5.5 comparison
• How to use AI: fundamental guide
• DeepSeek V4 Pro benchmarks and pricing

Based on the Cutoff Bench repository and page as of October 14, 2026; results may change as the author adds runs.