Microsoft open-sourced FreshSearch — a search-grounding benchmark in the web-grounding-evals GitHub repository — to measure whether AI search providers return fresh evidence and find newly published feed articles. The README frames the problem plainly: developers need to know if grounding keeps answers current, not only if models answer familiar questions.
The repo is a new Microsoft org release (candidate feed lists it November 18, 2026) with offline smoke tests that never call providers, plus authorized live evaluation paths that demand --live and --confirm-authorized-use flags.
Two panels: EF and FCD
Search-source threads linking a query to evidence cards, illustrating how grounding picks cited material
| Panel | Dataset | What it measures |
|---|---|---|
| Evidence Freshness (EF) | current-launch-ef120.json — 60 finance + 60 weather tasks | Age of evidence supporting time-sensitive answers |
| Fresh Content Discovery (FCD) | current-launch-fcd96.json — 96 feeds (4 paywalled feeds excluded at assembly) | Retrieval of ten newest eligible feed items, weighted toward newer posts |
Microsoft combines EF and FCD 20% EF / 80% FCD when cohort requirements in METHODOLOGY.md are met. Subsets are exploratory.
This is complementary to Google AI Mode temporary threads product news: FreshSearch is measurement, not a consumer feature launch.
Quick start without spending API dollars
A hand pulling the freshest search result card to the front of a stack
From the benchmark/ directory:
npm ci --ignore-scripts
npm test
npm run smoke:ef
npm run smoke:fcd
Expected offline scores: EF 62.5 and FCD 51/93 on synthetic fixtures. Outputs land in local-runs/ef/ and local-runs/fcd/.
Builders already running RAG pipelines can treat smoke tests as CI guards before wiring a provider into production agents.
Live evaluation: authorization and judge setup
Live runs require:
- Provider API keys (
WEBIQ_API_KEY,EXA_API_KEY, etc.) - A judge configuration (Azure OpenAI templates ship in
examples/) --confirm-authorized-use— your attestation that publisher and provider terms allow the crawl pace you set
Microsoft stresses robots rules are not licenses. Metadata-only output files cannot replay captured publisher text — important for compliance teams comparing FreshSearch to internal crawlers.
Budget example for five providers on the full cohort (from README):
- 4,800 FCD searches (96 feeds × 5 providers × 10 targets)
- 600 EF searches (120 tasks × 5 providers)
- Judge calls can exceed 1,200 with confirmation and repair steps
Finance tasks only run 10:00–16:00 America/New_York on exchange days — a detail agents automating benchmarks must schedule around.
Built-in search adapters
| Provider | Env key | Notes |
|---|---|---|
| WebIQ | WEBIQ_API_KEY | https://api.microsoft.ai/v3/search/web |
| WebIQ Auto | WEBIQ_API_KEY | Auto endpoint variant |
| Exa Auto | EXA_API_KEY | Third-party |
| Parallel | PARALLEL_API_KEY | Third-party |
| Perplexity | PERPLEXITY_API_KEY | Third-party |
Microsoft notes WebIQ is Microsoft-developed but not an official ranking. Compare with Bing grounding stories on how Microsoft ships eval culture internally.
Why freshness benchmarks matter for agents
A search bar feeding one answer card with source markers, like AI Overviews-style grounding
Coding agents increasingly call search tools (OpenAI research acceleration documents soaring internal agent spend). Stale snippets produce confident wrong answers on earnings, weather warnings, and policy shifts.
FreshSearch encodes head/torso/tail entity tiers so providers cannot overfit famous tickers alone. That mirrors jagged frontier language across OpenAI and Google science reports.
EF judge pitfalls
README warns a successful "return JSON" probe does not prove scoring compatibility. Pilot at least one scored Finance and one scored Weather task before full runs — categories use different judge budgets and evidence shapes.
Teams using Microsoft Aion browser models for local judging still need parity checks against Microsoft's reference judge configuration.
FCD timing constraints
Each feed run has a five-minute execution deadline. Provider searches must finish before reference material ages out; aggressive --search-delay-ms pacing can leave targets held rather than scored. README says never shorten pacing below account-required intervals to force scores.
How this fits Microsoft's grounding product line
FreshSearch arrives beside:
- HorizonDB for agents at Ignite — different layer (database versus search)
- Microsoft IQ and Foundry connectors at Ignite
Nothing in the repo ships a hosted leaderboard URL; you self-host HTML via node report.mjs.
Builder workflow
- Clone
github.com/microsoft/web-grounding-evals - Smoke offline in CI
- Authorize one feed and one EF task in a pilot directory
- Scale providers only after
summary.jsonshowsstatus: "complete" - Log
outputSha256and git SHA when publishing results internally
Step-by-step live pilot (authorized only)
Microsoft's README walks operators through a small live pilot before the full 4,800-search FCD marathon:
- List feed IDs offline with the bundled Node one-liner
- Pick one feed your contracts allow
- Run
evaluate.mjswith--max-fcd-calls 10,--max-ef-calls 1, and documented--search-delay-ms - Inspect
local-runs/live-pilot/summary.jsonforstatus: "complete"
Pilot failures categories — rate limits, judge truncation, held FCD targets — are documented under failures and partial runs in the runner guide. Do not treat a failed pilot as a zero score; treat it as incompatible pacing or judge config.
Weather versus Finance judge shapes
Weather EF tasks carry timezone and ICU versioning notes in the README because evidence timestamps depend on regional weather services. Finance tasks face NYSE window constraints and early-close calendar rules baked into the dataset through 2028.
Agents automating benchmark runs should tag runs with Node and ICU versions the way the README recommends; otherwise year-over-year comparisons lie.
Custom providers and BYO search
Microsoft ships examples/provider.mjs as a custom adapter contract. Enterprises with private crawl indices can plug internal search without publishing captures, but they still owe publisher authorization for any live FCD feed they include.
That pattern mirrors private RAG deployments (vector DB comparisons) where legal review precedes engineering sprints.
Security and data handling for eval teams
Live runs keep provider and publisher content in memory only during execution. Output artifacts are metadata-only and cannot rebuild HTML bodies. Teams sharing results.json externally still leak URLs, timings, and fingerprints — review before posting to blogs.
Built-in requests ask caches not to store responses; EF judge calls set store:false. Microsoft explicitly says that does not guarantee third-party retention — align with your DPA before pointing FreshSearch at customer-owned feeds.
When not to use FreshSearch
Skip FreshSearch as a daily CI job on live keys — costs explode. Use offline smoke for regression and schedule quarterly live runs when providers change ranking models.
Skip comparing FreshSearch scores to chatbot leaderboards; Microsoft disclaims ranking intent.
Agent harness integration
If your harness calls search via MCP:
- Log EF/FCD proxies on golden tasks even when not running the full benchmark
- Cache reference snapshots with TTL aligned to your product SLA
- Fall back to primary sources when FCD scores drop — pattern used in OpenAI research acceleration internal eval culture
Microsoft WebIQ in context
WebIQ endpoints (api.microsoft.ai/v3/search/web and /v3/auto) sit in the same ecosystem as Copilot grounding refreshes covered during Ignite 2026. FreshSearch is how Microsoft invites third parties to compare against its adapters — not how consumers see rankings in Bing.
Repository layout (benchmark directory)
| Path | Role |
|---|---|
datasets/ | EF120 and FCD96 JSON |
providers/ | Adapter modules |
examples/ | Synthetic captures and judge JSON templates |
run.mjs | Offline scorer |
evaluate.mjs | Live orchestrator |
report.mjs | HTML report builder |
Clone URL in candidates: github.com/microsoft/web-grounding-evals (README clone example also references grounding-evals naming — verify which repo your tarball uses before filing issues).
Head, torso, and tail entities
EF tasks bucket entities into head, torso, and tail frequency tiers so providers cannot optimize only for mega-cap tickers or flagship cities. METHODOLOGY.md warns subsets are not population estimates — interpret exploratory cuts carefully when blogging leaderboard takes.
Finance task window cheat sheet
| Situation | FreshSearch behavior |
|---|---|
| Weekday before 10:00 ET | Finance EF tasks held |
| Exchange holiday | Held per bundled 2026–2028 calendar |
| Early close day | Window may end 13:00 ET per README |
Custom --ef-input file | Order preserved; windows still apply to Finance category |
Schedule cron jobs in America/New_York, not UTC, when running unattended eval agents.
FCD five-minute deadline in plain language
Reference capture starts a clock. Each provider search must finish while the snapshot is fresh. Slow --search-delay-ms values or two-minute attempt timeouts can leave targets held — visible in issues.csv — rather than scored poorly. That is missing data, not a low FCD score.
Teams publishing internal leaderboards should attach held-rate percentages beside FCD points so executives do not confuse infrastructure timeouts with retrieval quality. Export those fields from issues.csv whenever an automated weekly report goes to leadership.
Contributing and license
The README points contributors to standard Microsoft open-source governance in-repo. Forks should not republish live capture metadata containing publisher text; respect the metadata-only live output rule.
Open questions for Microsoft
The repository does not yet publish a hosted results dashboard or monthly reference scores. Watch the GitHub releases tab for dataset version bumps to current-launch-ef120.json and current-launch-fcd96.json.
Comparison to other freshness evals
| Approach | FreshSearch | Typical RAG unit tests |
|---|---|---|
| Scope | Multi-provider search + feeds | Single index snapshot |
| Judge | LLM-scored evidence age | String match |
| Cost | Potentially thousands of paid calls | Cheap |
| Goal | Provider comparison | Regression guard |
Pair FreshSearch with application-level evals (Supercite shopping study) when your agent calls search mid-task.
Limitations Microsoft documents
- Combined scores need full cohort eligibility
- Interrupted runs may incur paid provider charges without complete scores
--continue-after-failureyieldspartialstatus with exit code 2- Timezone semantics bind Weather tasks — do not pool across ICU versions blindly
Related reading
- Vector database RAG compared
- Google AI Mode privacy threads
- OpenAI agent spend surge
- Microsoft GitHub AI findings pipeline
- Higher-scoring coding models natural experiment
- What is an agent harness?
Workshops
Teams benchmarking search for Copilot Studio agents should pair FreshSearch pilots with explainx.ai's Microsoft Copilot workshop for governance patterns.
