explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR
  • Why inclusive healthcare evaluation needed a new atlas
  • How benchmarks enter the Atlas
  • What the first Index numbers show
  • InclusionMed-DxFiles and the private benchmark choice
  • How to use InclusionMed this week
  • Contributing and staying current
  • Limits to keep in mind
  • Walking through the six audit principles in practice
  • How InclusionMed differs from single-suite healthcare leaderboards
  • Governance hooks for hospital innovation teams
  • Related reading
← Back to blog

explainx / blog

InclusionMed Index: A Healthcare AI Benchmark That Ranks Models by Role

Healthcare AI, Benchmarks, Open Source, Evaluation, Research

Part of AI Benchmarks and Evals

InclusionMed audits 388 medical AI benchmarks and scores 22 frontier models on five tasks. Overall Index spans 17.5–49.1 with role-specific rank changes.

Nov 17, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
InclusionMed Index: A Healthcare AI Benchmark That Ranks Models by Role

InclusionMed is a November 2026 open healthcare evaluation initiative that scores frontier language models on five audited benchmarks — and reports overall Index values from 17.5 to 49.1 out of 100, with rankings that change when you filter by clinician, patient, or administrator roles. The arXiv technical report, project site, and GitHub repository ship together.

For builders picking models for triage chat, prior auth, or clinical documentation, the headline is not a single leaderboard number. It is that the best aggregate model may still be the wrong choice for the hat your agent wears.

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

TL;DR

table · 2 cols
QuestionAnswer
What released?Technical report Nov 16, 2026; HF Paper of the Day Nov 17
Scope22 models, 5 benchmarks, 388 candidates audited
Score band17.5–49.1 overall Index
Structure3 healthcare roles × 3 AI capabilities grid (9 cells)
Open artifactsAtlas, Index, contribution guides on GitHub

Why inclusive healthcare evaluation needed a new atlas

Medical AI marketing still leans on single-number leaderboards from narrow exam-style tests. Real care settings split work across patients, clinicians, and administrators, and expect different knowledge, reasoning, and agency skills in each lane.

InclusionMed's response is structural:

  1. InclusionMed Atlas — maps which benchmarks occupy which role-capability cell.
  2. InclusionMed Index — equally weighted mean across included benchmarks, plus slices by role and capability.
  3. Community contributions — audited intake so future releases widen coverage instead of swapping one vanity metric for another.

The release landed the same week as OpenAI's healthcare computer-use agents post and Google's Gemini for Healthcare preview. Those announcements describe product pilots; InclusionMed describes how to compare models before you wire them into Epic, Cerner, or payer portals.

How benchmarks enter the Atlas

Clinical documentation motif linking healthcare evaluation to structured medical AI benchmarksClinical documentation motif linking healthcare evaluation to structured medical AI benchmarks

The GitHub README lists six audit principles applied to every candidate, public or private:

table · 2 cols
PrincipleWhat reviewers check
Healthcare relevanceTasks map to real care activities
TrustworthinessGround truth and scoring have professional basis
AvailabilityTasks, code, and licenses permit evaluation
Complementary coverageFills gaps in roles, languages, or settings
TimelinessContent still matches current guidelines
Discriminative capacityScores separate models under fair conditions

From 388 candidates, the team advanced four public benchmarks through structured audits and independent reproduction, then added InclusionMed-DxFiles, a private diagnostic reasoning suite, for five total tasks covering three of nine Atlas cells.

Hatched cells in project figures mark intentional gaps — places where inclusive healthcare still lacks a trusted test. That honesty is more useful than padding the Index with duplicate MedQA clones.

What the first Index numbers show

The paper's headline range — 17.5 to 49.1 on a 100-point equally weighted mean — signals that frontier models are far from saturated on this portfolio, even when vendors market near-perfect clinical demos.

Two subtler findings matter for product teams:

Rankings move when you change the hat

The Index publishes scores by healthcare role and by AI capability. A model leading the overall table can fall mid-pack on administrative agency or professional reasoning depending on the slice.

That behavior should inform routing in multi-agent clinic stacks: a cheap model tuned for patient education may beat an expensive general model on bedside tone while failing structured utilization review.

Patient communication beat clinical decision-making among leaders

Within-benchmark task-level tags show an uncomfortable pattern: for the ten highest Index models, patient communication scores exceeded clinical decision-making scores in every case.

If you ship a chat surface to discharged patients, benchmark that lane explicitly — do not assume a high USMLE-style number protects prescribing suggestions hidden in the same weights.

Cost diverges at similar Index scores

The project's cost–performance frontier chart (on inclusionmed.ai) plots estimated USD per task from recorded token usage and published API prices. Models with similar Index scores can sit orders of magnitude apart on cost.

For health systems watching OpenAI Luna pricing in computer-use benchmarks or running Apple's health AI usage studies, InclusionMed reinforces: pick per workflow, not per press release.

InclusionMed-DxFiles and the private benchmark choice

The Atlas marks InclusionMed-DxFiles with a dagger as a private diagnostic reasoning benchmark. Public reproducibility fans may chafe, but the authors argue some clinical items cannot ship openly without contaminating future evaluations.

Builders should note:

  • Public leaderboard claims that omit private cells are incomplete relative to the full Index.
  • Contribution pathways exist to add new public suites that fill hatched cells without waiting for the next corporate benchmark drop.

How to use InclusionMed this week

  1. Map your agent's job to a role-capability cell before you trust any single score.
  2. Open the interactive Atlas on inclusionmed.ai and record which benchmarks actually ran for your shortlisted models.
  3. Clone the GitHub checklist (evaluation-record requirements) into your internal eval CI so regressions are comparable release to release.
  4. Pair Index scores with live EHR tests — benchmarks do not replace Epic-connected ChatGPT pilots or HIPAA BAA constraints.

If you fine-tune open weights for chart summarization, compare InclusionMed slices with Sarvam's India-focused inference pricing or other regional hosts — the Index is model-centric, not deployment-centric.

Contributing and staying current

The collaboration invites four contribution types on GitHub:

  • Benchmarks with auditable materials
  • Benchmark ideas when you see a care gap but no dataset yet
  • Corrections with evidence for Index rows
  • Advisory partnerships scoped before announcement

Contact: info@inclusionmed.ai per the repository README.

For governance teams, InclusionMed complements policy news like OpenAI CSAM classifier warnings — one addresses clinical fitness, the other platform abuse. Both belong in enterprise risk reviews.

Limits to keep in mind

  • Five benchmarks cannot represent global healthcare; hatched cells are explicit admission of that fact.
  • Private DxFiles means external teams cannot fully replicate the published Index without collaboration.
  • Snapshot models — scores reflect November 2026 frontier weights; weekly vendor updates can reorder slices.

Treat InclusionMed as a directional compass, then run your own tasks on de-identified charts and payer PDFs.

Walking through the six audit principles in practice

The GitHub README's audit rubric is the backbone of InclusionMed's credibility claim. Here is how each principle changes what lands in the Atlas:

Healthcare relevance filters vanity NLP tasks dressed in scrubs. A benchmark must tie items to identifiable care work — triage questions, utilization management, patient education — not generic multiple-choice trivia with medical vocabulary sprinkled in.

Trustworthiness is where many public sets fail quietly. Reviewers examine annotation manuals, clinician involvement, and whether reference answers trace to guidelines or committee consensus. The initiative excludes benchmarks generated entirely by models without human professional review — a bar more suites should adopt.

Availability and permitted use stops the common trap of leaderboard papers that cite private items readers cannot run. InclusionMed still keeps DxFiles private, but marks it visibly; transparency about that choice is part of the audit.

Complementary coverage forces each new candidate to justify its cell in the 3×3 grid. Otherwise the Index would re-weight the same MedQA derivatives five times and call it inclusivity.

Timeliness and data independence matter because clinical guidelines shift — what was true for COVID triage in 2021 is not the right lens for 2026 agent copilots. Reviewers look for contamination guards when models may have trained on test items.

Discriminative capacity uses published score spreads to reject benchmarks where every frontier model ties. If nothing separates, the benchmark does not help procurement.

How InclusionMed differs from single-suite healthcare leaderboards

table · 3 cols
ApproachInclusionMedTypical vendor healthcare blog
Unit of scoreRole-capability slices + overall IndexOne pilot metric (e.g., inbox safety %)
Model count22 in first releaseOften one flagship model
CostPublished frontier on cost–performance chartSometimes omitted
GapsHatched Atlas cellsOften unmentioned

OpenAI's 99% clinical safety statistic on EHR inbox work is invaluable for deployment narrative — read it alongside InclusionMed when you ask whether the same model leads on administrative agency or patient-only chat.

Governance hooks for hospital innovation teams

  • Add InclusionMed slice requirements to vendor RFPs when a startup claims "SOTA clinical AI."
  • Mirror the evaluation-record checklist in your MLOps pipeline so fine-tunes do not ship without role-specific regression.
  • Pair Atlas gaps with internal task libraries — if the administrator × agency cell is empty, your billing agent needs a bespoke eval, not a generic chat guardrail.

Regulatory readers may also crosswalk with California's November litigation against model hosts: benchmarks do not replace safety testing, but they document fit-for-purpose claims procurement lawyers can audit.

Schedule quarterly reruns when vendors ship new healthcare-tuned weights; the Index is a snapshot, not a perpetual certificate.

When you cite InclusionMed in board slides, show the role slice that matches the workflow you are buying, not only the overall Index bar chart.

Related reading

  • OpenAI computer-use agents for healthcare
  • Google Gemini for Healthcare preview
  • ChatGPT Epic EHR integration
  • Apple health AI usage study with OpenEvidence
  • Mass General Brigham FAHR-Face cancer survival model
  • OpenAI Apollo embedded evaluation for agent scheming
Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 11, 2026

Longtail-SWE: Nebius Benchmark for 13 Underrepresented Coding Languages

Nebius released Longtail-SWE on Hugging Face: 202 real issue-resolution tasks across thirteen languages that rarely appear in SWE-bench style leaderboards. Every task includes executable environments, tests, and published trajectories.

Nov 11, 2026

WikiTrace: Evidence-Linked Agent Memory When Lessons Hurt Performance

Agent memory can make models fail tasks they would solve without memory — KAIST AI calls that memory-induced degradation. WikiTrace links each lesson to the run it came from so agents descend into evidence before acting. Code is on GitHub under Apache-2.0.

Nov 10, 2026

FinDendro Benchmark: LLMs Misreport What They Know in Financial Backtests

Researchers introduce FinDendro, a benchmark of 2,226 dated US macro, stock, and Fed facts from 1999 through 2026. Frontier models often answer correctly while insisting they are guessing, and historical-date instructions change claims more than accuracy.