If you backtest an LLM on historical Fed decisions or month-end stock prices, the model might recite outcomes instead of forecasting them. Practitioners try to prevent that with vendor cutoffs, "answer as of date X" prompts, or "do you know this?" checks.
A new paper, LLMs Don't Know What They Know (arXiv 2611.06808, posted on Hugging Face Papers), introduces FinDendro and argues those safeguards often fail behavioral tests. The authors measure what models answer and what they claim on the same dated facts.
TL;DR
| Safeguard | FinDendro finding (paper) |
|---|---|
| Vendor cutoff dates | Llama-3.3-70B accuracy falls to chance by mid-2021 while claims persist into 2023 |
| Self-reported knowledge | GPT-5 is right on 76% of controlled facts but says it knows only 38% |
| Historical-date instruction | Cuts GPT-5 "I know" claims from 39% to 1% with little accuracy change |
Why financial LLM backtests need temporal audits
Trading and macro demos on social media often show models "predicting" past crises with spooky accuracy. The uncomfortable explanation is hindsight leakage: weights may already encode the realized CPI print or the hike the Fed actually delivered.
Industry benchmarks like FinBen and StockBench score economic skill, but the FinDendro authors note many suites do not jointly test knowledge claims and factual recall on identically dated items. Our earlier Cutoff Bench post asked a simpler question — whether models repeat their card cutoff — without pairing know/guess language on macro facts.
FinDendro closes part of that gap by building 2,226 US facts from 1999 through 2026 across:
- Twelve monthly FRED macro series (first-release values)
- Month-end prices for 50 large US stocks (split-adjusted back to quoted history)
- Federal Reserve target-rate decisions
Each fact spawns four prompts: a straight value question, recall and predict variants that pair answers with know versus guess claims, and a predict variant with an explicit historical date instruction.
Chance accuracy stays near 0.27 after controls for persistence and dominant policy outcomes, per the abstract.
Timeline marker at a model knowledge cutoff boundary for dated financial probes
Three failure modes the paper reports
Self-report understates knowledge
On controlled facts, GPT-5 answers 76% correctly while claiming knowledge on only 38%, according to the paper abstract. Worse, it still answers 61% of items it labels as guesses correctly — so filtering on "I do not know" does not reliably drop leaked outcomes.
That matches a builder intuition many teams already have: models sound humble while citing exact digits.
Stated vendor cutoffs miss behavioral ends
The authors report Llama-3.3-70B factual accuracy collapses toward chance after mid-2021, before its December 2023 vendor cutoff, while knowledge claims continue into early 2023. If you backtest only after the card date, you may still be inside the model's effective memory window.
Date instructions re-label answers, not skill
When GPT-5 must predict as of a historical date, claimed prior knowledge falls from 39% to 1%, yet accuracy barely moves. The model sounds like it is forecasting while behaving like it is recalling — exactly the rhetoric risk compliance teams worry about in claimed-versus-verified finance posts.
Benchmark ceiling chart metaphor for LLM accuracy versus self-reported knowledge claims
What Hugging Face readers asked
On the paper discussion, commenter PolyHelperAI asked whether the historical-date instruction helped at all. The authors' framing in the abstract implies it mainly suppresses claims, not errors — worth reproducing on your own evaluator before you trust date-stamped prompts in production.
How practitioners should respond
FinDendro does not ban historical evaluation. It pushes behavioral measurement:
- Pair accuracy curves with claim curves on the same dated items instead of trusting a single cutoff field.
- Hold out recent months prospectively when you need clean skill estimates — same lesson as prospective agent benchmarks waiting for outcomes.
- Document which FRED release you score against; FinDendro uses first-release values, so revised series would change labels.
- Compare point-in-time fine-tunes separately; the paper evaluates small PIT families whose accuracy stays near chance on both sides of training cuts — specialized, not a default fix.
If you fine-tune open models on financial text, treat leakage tests as part of release QA beside perplexity and tool accuracy — parallel to how we discuss verification tax for agent output.
Task design: why four prompts per fact
FinDendro is built to test safeguards practitioners already use:
| Task | What it probes |
|---|---|
| Task 1 — value only | Behavioral knowledge without a claim |
| Task 2 — value plus know/guess | Whether self-report tracks accuracy |
| Task 3 — dated predict | Whether "as of" instructions reduce leakage or only re-label it |
The paper contrasts FinDendro with FinBen, StockBench, ForecastBench, HindsightBench, and SelfAware in a comparison table. FinDendro is the row that combines dated finance, cutoff estimation, claims, and guard evaluation — a niche overlap, but exactly the gap teams cite when a backtest looks too good.
Authors use first-release FRED observations via ALFRED, split-adjusted Yahoo prices as quoted at month-end, and Federal Reserve target-rate history. That matters because revised macro series would change labels; if you replicate, mirror their release policy.
Sampling is sparse before 2021 and dense from 2021 onward because frontier cutoffs cluster in that window. Of 2,226 facts, 1,340 count as controlled after persistence checks — the subset where repeating last month's value is not a winning strategy.
Numbers worth copying into your evaluator README
From the abstract and introduction we fetched:
- Chance accuracy near 0.27 after controls.
- GPT-5: 76% controlled accuracy versus 38% claimed knowledge; 61% correct on items it says it does not know.
- Historical-date instruction: claimed prior knowledge 39% → 1% for GPT-5 with little accuracy movement.
- Llama-3.3-70B: accuracy to chance by mid-2021 while claims extend into 2023, ahead of a December 2023 vendor cutoff cited in the paper.
Those figures are paper-reported; treat them like any external benchmark until you reproduce on your harness.
When FinDendro matters even if you never trade
You do not need a brokerage account for the lesson to land:
- Risk teams backtesting LLM macro summaries for internal memos still leak if the model saw COVID-era CPI in weights.
- Agent evals that score "forecasting" on historical quarters inherit the same bias as OR-License style operations tasks.
- RAG pipelines that filter documents by date still leave parametric memory untouched — FinDendro is a reminder to separate retrieval cutoff from weight cutoff.
Pair FinDendro with prospective benchmarks when you need marketing claims: hold out the last N months, publish predictions before outcomes, then score — slower, but legible to auditors.
Implementation ideas for ML platform teams
If you maintain an internal model garden:
- Snapshot FinDendro-style MCQ on release candidates weekly; track claim-accuracy gap as a regression signal.
- Alert when gap widens after a silent router change — same discipline as monitoring tool hallucination rate.
- Store vendor cutoff field separately from empirical cutoff you estimate from dated probes; show both in an internal model card.
- Ban "predict 2020 CPI" demos in customer decks unless the demo uses models trained with provable exclusion — marketing risk is reputational, not just statistical.
Limits and open questions
The benchmark is US-only and multiple-choice styled with single letter answers — cheap to run, but not identical to open-ended trading policies. The full pool and construction code are promised at release; until then, replicate tables from the PDF before you cite leaderboard ranks.
The paper evaluates a handful of frontier and open models; your fine-tune may behave differently. Still, the mechanism — claims diverging from accuracy — is the reusable lesson.
Open questions we will watch for in follow-on work: multilingual macro facts, intraday prices, options surfaces, and whether tool-using agents behave differently when they can fetch ALFRED at runtime (FinDendro prompts forbid browsing in the described setup).
Glossary alignment for eval teams
If your organization maintains a model card template, add two fields today: vendor_stated_cutoff and empirical_cutoff_fin_dendro (or your internal probe name). Require both before a model is marked approved for historical macro demos. That single bureaucracy line prevents sales engineers from citing vendor cards alone when FinDendro-style gaps appear in QA.
Researchers evaluating central bank communication tools should note FinDendro includes FOMC decision months as first-class facts. Stock pickers should treat the fifty large-cap tickers the same way: month-end labels are unforgiving when a model silently remembers earnings-week prices. A model that "predicts" past rate moves correctly may still be useless for live policy if the measured boundary shows recall years after the meeting — exactly the scenario monetary economists worry about when LLM summaries enter staff briefing pipelines without audit trails.
Related reading
- Cutoff Bench: reported versus remembered cutoffs
- OR-License benchmark for agent operations research
- AI verification tax at work
- Menlo enterprise AI survey and API share
- Coinbase fine-tuned Qwen3.5 9B on fraud
Sources
- arXiv 2611.06808, October 30, 2026 posting.
- Hugging Face Papers page, community discussion November 2026.
Benchmark scope and model names are accurate as of the paper text we fetched on November 10, 2026.