InclusionMed is a November 2026 open healthcare evaluation initiative that scores frontier language models on five audited benchmarks — and reports overall Index values from 17.5 to 49.1 out of 100, with rankings that change when you filter by clinician, patient, or administrator roles. The arXiv technical report, project site, and GitHub repository ship together.
For builders picking models for triage chat, prior auth, or clinical documentation, the headline is not a single leaderboard number. It is that the best aggregate model may still be the wrong choice for the hat your agent wears.
TL;DR
| Question | Answer |
|---|---|
| What released? | Technical report Nov 16, 2026; HF Paper of the Day Nov 17 |
| Scope | 22 models, 5 benchmarks, 388 candidates audited |
| Score band | 17.5–49.1 overall Index |
| Structure | 3 healthcare roles × 3 AI capabilities grid (9 cells) |
| Open artifacts | Atlas, Index, contribution guides on GitHub |
Why inclusive healthcare evaluation needed a new atlas
Medical AI marketing still leans on single-number leaderboards from narrow exam-style tests. Real care settings split work across patients, clinicians, and administrators, and expect different knowledge, reasoning, and agency skills in each lane.
InclusionMed's response is structural:
- InclusionMed Atlas — maps which benchmarks occupy which role-capability cell.
- InclusionMed Index — equally weighted mean across included benchmarks, plus slices by role and capability.
- Community contributions — audited intake so future releases widen coverage instead of swapping one vanity metric for another.
The release landed the same week as OpenAI's healthcare computer-use agents post and Google's Gemini for Healthcare preview. Those announcements describe product pilots; InclusionMed describes how to compare models before you wire them into Epic, Cerner, or payer portals.
How benchmarks enter the Atlas
Clinical documentation motif linking healthcare evaluation to structured medical AI benchmarks
The GitHub README lists six audit principles applied to every candidate, public or private:
| Principle | What reviewers check |
|---|---|
| Healthcare relevance | Tasks map to real care activities |
| Trustworthiness | Ground truth and scoring have professional basis |
| Availability | Tasks, code, and licenses permit evaluation |
| Complementary coverage | Fills gaps in roles, languages, or settings |
| Timeliness | Content still matches current guidelines |
| Discriminative capacity | Scores separate models under fair conditions |
From 388 candidates, the team advanced four public benchmarks through structured audits and independent reproduction, then added InclusionMed-DxFiles, a private diagnostic reasoning suite, for five total tasks covering three of nine Atlas cells.
Hatched cells in project figures mark intentional gaps — places where inclusive healthcare still lacks a trusted test. That honesty is more useful than padding the Index with duplicate MedQA clones.
What the first Index numbers show
The paper's headline range — 17.5 to 49.1 on a 100-point equally weighted mean — signals that frontier models are far from saturated on this portfolio, even when vendors market near-perfect clinical demos.
Two subtler findings matter for product teams:
Rankings move when you change the hat
The Index publishes scores by healthcare role and by AI capability. A model leading the overall table can fall mid-pack on administrative agency or professional reasoning depending on the slice.
That behavior should inform routing in multi-agent clinic stacks: a cheap model tuned for patient education may beat an expensive general model on bedside tone while failing structured utilization review.
Patient communication beat clinical decision-making among leaders
Within-benchmark task-level tags show an uncomfortable pattern: for the ten highest Index models, patient communication scores exceeded clinical decision-making scores in every case.
If you ship a chat surface to discharged patients, benchmark that lane explicitly — do not assume a high USMLE-style number protects prescribing suggestions hidden in the same weights.
Cost diverges at similar Index scores
The project's cost–performance frontier chart (on inclusionmed.ai) plots estimated USD per task from recorded token usage and published API prices. Models with similar Index scores can sit orders of magnitude apart on cost.
For health systems watching OpenAI Luna pricing in computer-use benchmarks or running Apple's health AI usage studies, InclusionMed reinforces: pick per workflow, not per press release.
InclusionMed-DxFiles and the private benchmark choice
The Atlas marks InclusionMed-DxFiles with a dagger as a private diagnostic reasoning benchmark. Public reproducibility fans may chafe, but the authors argue some clinical items cannot ship openly without contaminating future evaluations.
Builders should note:
- Public leaderboard claims that omit private cells are incomplete relative to the full Index.
- Contribution pathways exist to add new public suites that fill hatched cells without waiting for the next corporate benchmark drop.
How to use InclusionMed this week
- Map your agent's job to a role-capability cell before you trust any single score.
- Open the interactive Atlas on inclusionmed.ai and record which benchmarks actually ran for your shortlisted models.
- Clone the GitHub checklist (evaluation-record requirements) into your internal eval CI so regressions are comparable release to release.
- Pair Index scores with live EHR tests — benchmarks do not replace Epic-connected ChatGPT pilots or HIPAA BAA constraints.
If you fine-tune open weights for chart summarization, compare InclusionMed slices with Sarvam's India-focused inference pricing or other regional hosts — the Index is model-centric, not deployment-centric.
Contributing and staying current
The collaboration invites four contribution types on GitHub:
- Benchmarks with auditable materials
- Benchmark ideas when you see a care gap but no dataset yet
- Corrections with evidence for Index rows
- Advisory partnerships scoped before announcement
Contact: info@inclusionmed.ai per the repository README.
For governance teams, InclusionMed complements policy news like OpenAI CSAM classifier warnings — one addresses clinical fitness, the other platform abuse. Both belong in enterprise risk reviews.
Limits to keep in mind
- Five benchmarks cannot represent global healthcare; hatched cells are explicit admission of that fact.
- Private DxFiles means external teams cannot fully replicate the published Index without collaboration.
- Snapshot models — scores reflect November 2026 frontier weights; weekly vendor updates can reorder slices.
Treat InclusionMed as a directional compass, then run your own tasks on de-identified charts and payer PDFs.
Walking through the six audit principles in practice
The GitHub README's audit rubric is the backbone of InclusionMed's credibility claim. Here is how each principle changes what lands in the Atlas:
Healthcare relevance filters vanity NLP tasks dressed in scrubs. A benchmark must tie items to identifiable care work — triage questions, utilization management, patient education — not generic multiple-choice trivia with medical vocabulary sprinkled in.
Trustworthiness is where many public sets fail quietly. Reviewers examine annotation manuals, clinician involvement, and whether reference answers trace to guidelines or committee consensus. The initiative excludes benchmarks generated entirely by models without human professional review — a bar more suites should adopt.
Availability and permitted use stops the common trap of leaderboard papers that cite private items readers cannot run. InclusionMed still keeps DxFiles private, but marks it visibly; transparency about that choice is part of the audit.
Complementary coverage forces each new candidate to justify its cell in the 3×3 grid. Otherwise the Index would re-weight the same MedQA derivatives five times and call it inclusivity.
Timeliness and data independence matter because clinical guidelines shift — what was true for COVID triage in 2021 is not the right lens for 2026 agent copilots. Reviewers look for contamination guards when models may have trained on test items.
Discriminative capacity uses published score spreads to reject benchmarks where every frontier model ties. If nothing separates, the benchmark does not help procurement.
How InclusionMed differs from single-suite healthcare leaderboards
| Approach | InclusionMed | Typical vendor healthcare blog |
|---|---|---|
| Unit of score | Role-capability slices + overall Index | One pilot metric (e.g., inbox safety %) |
| Model count | 22 in first release | Often one flagship model |
| Cost | Published frontier on cost–performance chart | Sometimes omitted |
| Gaps | Hatched Atlas cells | Often unmentioned |
OpenAI's 99% clinical safety statistic on EHR inbox work is invaluable for deployment narrative — read it alongside InclusionMed when you ask whether the same model leads on administrative agency or patient-only chat.
Governance hooks for hospital innovation teams
- Add InclusionMed slice requirements to vendor RFPs when a startup claims "SOTA clinical AI."
- Mirror the evaluation-record checklist in your MLOps pipeline so fine-tunes do not ship without role-specific regression.
- Pair Atlas gaps with internal task libraries — if the administrator × agency cell is empty, your billing agent needs a bespoke eval, not a generic chat guardrail.
Regulatory readers may also crosswalk with California's November litigation against model hosts: benchmarks do not replace safety testing, but they document fit-for-purpose claims procurement lawyers can audit.
Schedule quarterly reruns when vendors ship new healthcare-tuned weights; the Index is a snapshot, not a perpetual certificate.
When you cite InclusionMed in board slides, show the role slice that matches the workflow you are buying, not only the overall Index bar chart.
