| name | imaging-data-commons |
| description | Query and download public cancer imaging data from NCI Imaging Data Commons using idc-index. Use for accessing large-scale radiology (CT, MR, PET) and pathology datasets for AI training or research. No authentication required. Query by metadata, visualize in browser, check licenses. |
| license | This skill is provided under the MIT License. IDC data itself has individual licensing (mostly CC-BY, some CC-NC) that must be respected when using the data. |
| metadata | version: 1.4.0 skill-author: Andrey Fedorov, @fedorov idc-index: "0.11.14" idc-data-version: "v23" repository: https://github.com/ImagingDataCommons/idc-claude-skill |
Imaging Data Commons
Overview
Use the idc-index Python package to query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.
Current IDC Data Version: v23 (always verify with IDCClient().get_idc_version())
Primary tool: idc-index (GitHub)
CRITICAL - Check package version and upgrade if needed (run this FIRST):
import idc_index
REQUIRED_VERSION = "0.11.14"
installed = idc_index.__version__
if installed < REQUIRED_VERSION:
print(f"Upgrading idc-index from {installed} to {REQUIRED_VERSION}...")
import subprocess
subprocess.run(["pip3", "install", "--upgrade", "--break-system-packages", "idc-index"], check=True)
print("Upgrade complete. Restart Python to use new version.")
else:
print(f"idc-index {installed} meets requirement ({REQUIRED_VERSION})")
Verify IDC data version and check current data scale:
from idc_index import IDCClient
client = IDCClient()
print(f"IDC data version: {client.get_idc_version()}")
stats = client.sql_query("""
SELECT
COUNT(DISTINCT collection_id) as collections,
COUNT(DISTINCT analysis_result_id) as analysis_results,
COUNT(DISTINCT PatientID) as patients,
COUNT(DISTINCT StudyInstanceUID) as studies,
COUNT(DISTINCT SeriesInstanceUID) as series,
SUM(instanceCount) as instances,
SUM(series_size_MB)/1000000 as size_TB
FROM index
""")
print(stats)
Core workflow:
- Query metadata →
client.sql_query()
- Download DICOM files →
client.download_from_selection()
- Visualize in browser →
client.get_viewer_URL(seriesInstanceUID=...)
When to Use This Skill
- Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images
- Selecting image subsets by cancer type, modality, anatomical site, or other metadata
- Downloading DICOM data from IDC
- Checking data licenses before use in research or commercial applications
- Visualizing medical images in a browser without local DICOM viewer software
Quick Navigation
Core Sections (inline):
- IDC Data Model - Collection and analysis result hierarchy
- Index Tables - Available tables and joining patterns
- Installation - Package setup and version verification
- Core Capabilities - Essential API patterns (query, download, visualize, license, citations, batch)
- Best Practices - Usage guidelines
- Troubleshooting - Common issues and solutions
Reference Guides (load on demand):
| Guide | When to Load |
|---|
index_tables_guide.md | Complex JOINs, schema discovery, DataFrame access |
use_cases.md | End-to-end workflow examples (training datasets, batch downloads) |
sql_patterns.md | Quick SQL patterns for filter discovery, annotations, size estimation |
clinical_data_guide.md | Clinical/tabular data, imaging+clinical joins, value mapping |
cloud_storage_guide.md | Direct S3/GCS access, versioning, UUID mapping |
dicomweb_guide.md | DICOMweb endpoints, PACS integration |
digital_pathology_guide.md | Slide microscopy (SM), annotations (ANN), pathology workflows |
bigquery_guide.md | Full DICOM metadata, private elements (requires GCP) |
cli_guide.md | Command-line tools (idc download, manifest files) |
parquet_access_guide.md | Direct Parquet queries via GCS (no idc-index install needed) |
IDC Data Model
IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):
- collection_id: Groups patients by disease, modality, or research focus (e.g.,
tcga_luad, nlst). A patient belongs to exactly one collection.
- analysis_result_id: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections.
Use collection_id to find original imaging data, may include annotations deposited along with the images; use analysis_result_id to find AI-generated or expert annotations.
Key identifiers for queries:
| Identifier | Scope | Use for |
|---|
collection_id | Dataset grouping | Filtering by project/study |
PatientID | Patient | Grouping images by patient |
StudyInstanceUID | DICOM study | Grouping of related series, visualization |
SeriesInstanceUID | DICOM series | Grouping of related series, visualization |
Index Tables
The idc-index package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames.
Complete index table documentation: Use https://idc-index.readthedocs.io/en/latest/indices_reference.html for quick check of available tables and columns without executing any code.
Important: Use client.indices_overview to get current table descriptions and column schemas. This is the authoritative source for available columns and their types — always query it when writing SQL or exploring data structure.
Available Tables
| Table | Row Granularity | Loaded | Description |
|---|
index | 1 row = 1 DICOM series | Auto | Primary metadata for all current IDC data |
prior_versions_index | 1 row = 1 DICOM series | Auto | Series from previous IDC releases; for downloading deprecated data |
collections_index | 1 row = 1 collection | fetch_index() | Collection-level metadata and descriptions |
analysis_results_index | 1 row = 1 analysis result collection | fetch_index() | Metadata about derived datasets (annotations, segmentations) |
clinical_index | 1 row = 1 clinical data column | fetch_index() | Dictionary mapping clinical table columns to collections |
sm_index | 1 row = 1 slide microscopy series | fetch_index() | Slide Microscopy (pathology) series metadata |
sm_instance_index | 1 row = 1 slide microscopy instance | fetch_index() | Instance-level (SOPInstanceUID) metadata for slide microscopy |
seg_index | 1 row = 1 DICOM Segmentation series | fetch_index() | Segmentation metadata: algorithm, segment count, reference to source image series |
ann_index | 1 row = 1 DICOM ANN series | fetch_index() | Microscopy Bulk Simple Annotations series metadata; references annotated image series |
ann_group_index | 1 row = 1 annotation group | fetch_index() | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |
contrast_index | 1 row = 1 series with contrast info | fetch_index() | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |
volume_geometry_index | 1 row = 1 CT/MR/PT series | fetch_index() | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite regularly_spaced_3d_volume flag |
rtstruct_index | 1 row = 1 RTSTRUCT series | fetch_index() | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID |
Auto = loaded automatically when IDCClient() is instantiated
fetch_index() = requires client.fetch_index("table_name") to load
Joining Tables
Key columns are not explicitly labeled, the following is a subset that can be used in joins.
| Join Column | Tables | Use Case |
|---|
collection_id | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data |
SeriesInstanceUID | index, prior_versions_index, sm_index, sm_instance_index | Link series across tables; connect to slide microscopy details |
StudyInstanceUID | index, prior_versions_index | Link studies across current and historical data |
PatientID | index, prior_versions_index | Link patients across current and historical data |
analysis_result_id | index, analysis_results_index | Link series to analysis result metadata (annotations, segmentations) |
source_DOI | index, analysis_results_index | Link by publication DOI |
crdc_series_uuid | index, prior_versions_index | Link by CRDC unique identifier |
Modality | index, prior_versions_index | Filter by imaging modality |
SeriesInstanceUID | index, seg_index, ann_index, ann_group_index, contrast_index | Link segmentation/annotation/contrast series to its index metadata |
segmented_SeriesInstanceUID | seg_index → index | Link segmentation to its source image series (join seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID) |
referenced_SeriesInstanceUID | ann_index → index | Link annotation to its source image series (join ann_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID) |
SeriesInstanceUID | index, volume_geometry_index | Link series to its 3D geometry validation result (join index.SeriesInstanceUID = volume_geometry_index.SeriesInstanceUID) |
SeriesInstanceUID / referenced_SeriesInstanceUID | index, rtstruct_index | Join RTSTRUCT series to its metadata (index.SeriesInstanceUID = rtstruct_index.SeriesInstanceUID); use rtstruct_index.referenced_SeriesInstanceUID to find the source image series |
Note: Subjects, Updated, and Description appear in multiple tables but have different meanings (counts vs identifiers, different update contexts).
For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see references/index_tables_guide.md.
Clinical Data Access
client.fetch_index("clinical_index")
tables = client.sql_query("SELECT DISTINCT table_name, column_label FROM clinical_index")
clinical_df = client.get_clinical_table("table_name")
See references/clinical_data_guide.md for detailed workflows including value mapping patterns and joining clinical data with imaging.
Data Access Options
| Method | Auth Required | Best For |
|---|
idc-index | No | Key queries and downloads (recommended) |
| Direct Parquet (GCS) | No | Quick queries without installing idc-index; always uses latest data |
| IDC Portal | No | Interactive exploration, manual selection, browser-based download |
| BigQuery | Yes (GCP account) | Complex queries, full DICOM metadata |
| DICOMweb proxy | No | Tool integration via DICOMweb API |
| Cloud storage (S3/GCS) | No | Direct file access, bulk downloads, custom pipelines |
Cloud storage organization
IDC maintains all DICOM files in public cloud storage buckets mirrored between AWS S3 and Google Cloud Storage. Files are organized by CRDC UUIDs (not DICOM UIDs) to support versioning.
| Bucket (AWS / GCS) | License | Content |
|---|
idc-open-data / idc-open-data | No commercial restriction | >90% of IDC data |
idc-open-data-two / idc-open-idc1 | No commercial restriction | Collections with potential head scans |
idc-open-data-cr / idc-open-cr | Commercial use restricted (CC BY-NC) | ~4% of data |
Files are stored as <crdc_series_uuid>/<crdc_instance_uuid>.dcm. Access is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access. Use series_aws_url column from the index for S3 URLs; GCS uses the same path structure.
See references/cloud_storage_guide.md for bucket details, access commands, UUID mapping, and versioning.
DICOMweb access
IDC data is available via DICOMweb interface (Google Cloud Healthcare API implementation) for integration with PACS systems and DICOMweb-compatible tools.
| Endpoint | Auth | Use Case |
|---|
| Public proxy | No | Testing, moderate queries, daily quota |
| Google Healthcare | Yes (GCP) | Production use, higher quotas |
See references/dicomweb_guide.md for endpoint URLs, code examples, supported operations, and implementation details.
Direct Parquet access
All idc-index metadata tables are published as Parquet files to a public GCS bucket (idc-index-data-artifacts) with unrestricted CORS. This enables DuckDB or pandas queries without installing idc-index, including cross-table joins and queries against volume_geometry_index and rtstruct_index.
See references/parquet_access_guide.md for URL patterns, available files, and DuckDB query examples.
Installation and Setup
Required (for basic access):
pip install --upgrade idc-index
Important: New IDC data release will always trigger a new version of idc-index. Always use --upgrade flag while installing, unless an older version is needed for reproducibility.
IMPORTANT: IDC data version v23 is current. Always verify your version:
print(client.get_idc_version())
If you see an older version, upgrade with: pip install --upgrade idc-index
Tested with: idc-index 0.11.14 (IDC data version v23)
Optional (for data analysis):
pip install pandas numpy pydicom
Core Capabilities
1. Data Discovery and Exploration
Discover what imaging collections and data are available in IDC:
from idc_index import IDCClient
client = IDCClient()
query = """
SELECT
collection_id,
COUNT(DISTINCT PatientID) as patients,
COUNT(DISTINCT SeriesInstanceUID) as series,
SUM(series_size_MB) as size_mb
FROM index
GROUP BY collection_id
ORDER BY patients DESC
"""
collections_summary = client.sql_query(query)
client.fetch_index("collections_index")
collections_info = client.sql_query("""
SELECT collection_id, CancerTypes, TumorLocations, Species, Subjects, SupportingData
FROM collections_index
""")
client.fetch_index("analysis_results_index")
analysis_info = client.sql_query("""
SELECT analysis_result_id, analysis_result_title, Subjects, Collections, Modalities
FROM analysis_results_index
""")
collections_index provides curated metadata per collection: cancer types, tumor locations, species, subject counts, and supporting data types — without needing to aggregate from the primary index.
analysis_results_index lists derived datasets (AI segmentations, expert annotations, radiomics features) with their source collections and modalities.
2. Querying Metadata with SQL
Query the IDC mini-index using SQL to find specific datasets.
First, explore available values for filter columns:
from idc_index import IDCClient
client = IDCClient()
modalities = client.sql_query("""
SELECT DISTINCT Modality, COUNT(*) as series_count
FROM index
GROUP BY Modality
ORDER BY series_count DESC
""")
print(modalities)
body_parts = client.sql_query("""
SELECT DISTINCT BodyPartExamined, COUNT(*) as series_count
FROM index
WHERE Modality = 'MR' AND BodyPartExamined IS NOT NULL
GROUP BY BodyPartExamined
ORDER BY series_count DESC
LIMIT 20
""")
print(body_parts)
Then query with validated filter values:
results = client.sql_query("""
SELECT
collection_id,
PatientID,
SeriesInstanceUID,
Modality,
SeriesDescription,
license_short_name
FROM index
WHERE Modality = 'MR'
AND BodyPartExamined = 'BREAST'
LIMIT 20
""")
for idx, row in results.iterrows():
print(f"Patient: {row['PatientID']}, Series: {row['SeriesInstanceUID']}")
To filter by cancer type, join with collections_index:
client.fetch_index("collections_index")
results = client.sql_query("""
SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality
FROM index i
JOIN collections_index c ON i.collection_id = c.collection_id
WHERE c.CancerTypes LIKE '%Breast%'
AND i.Modality = 'MR'
LIMIT 20
""")
Available metadata fields (use client.indices_overview for complete list):
- Identifiers: collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID
- Imaging: Modality, BodyPartExamined, Manufacturer, ManufacturerModelName
- Clinical: PatientAge, PatientSex, StudyDate
- Descriptions: StudyDescription, SeriesDescription
- Licensing: license_short_name
Note: Cancer type is in collections_index.CancerTypes, not in the primary index table.
3. Downloading DICOM Files
Download imaging data efficiently from IDC's cloud storage:
Download entire collection:
from idc_index import IDCClient
client = IDCClient()
client.download_from_selection(
collection_id="rider_pilot",
downloadDir="./data/rider"
)
Download specific series:
series_df = client.sql_query("""
SELECT SeriesInstanceUID
FROM index
WHERE Modality = 'CT'
AND BodyPartExamined = 'CHEST'
AND collection_id = 'nlst'
LIMIT 5
""")
client.download_from_selection(
seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
downloadDir="./data/lung_ct"
)
Custom directory structure:
Default dirTemplate: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID
client.download_from_selection(
collection_id="tcga_luad",
downloadDir="./data",
dirTemplate="%collection_id/%PatientID/%Modality"
)
client.download_from_selection(
seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
downloadDir="./data/flat",
dirTemplate=""
)
Downloaded file names:
Individual DICOM files are named using their CRDC instance UUID: <crdc_instance_uuid>.dcm (e.g., 0d73f84e-70ae-4eeb-96a0-1c613b5d9229.dcm). This UUID-based naming:
- Enables version tracking (UUIDs change when file content changes)
- Matches cloud storage organization (
s3://idc-open-data/<crdc_series_uuid>/<crdc_instance_uuid>.dcm)
- Differs from DICOM UIDs (SOPInstanceUID) which are preserved inside the file metadata
To identify files, use the crdc_instance_uuid column in queries or read DICOM metadata (SOPInstanceUID) from the files.
Command-Line Download
The idc download command provides command-line access to download functionality without writing Python code. Available after installing idc-index.
Auto-detects input type: manifest file path, or identifiers (collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID, crdc_series_uuid).
idc download rider_pilot --download-dir ./data
idc download "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data
idc download "tcga_luad,tcga_lusc" --download-dir ./data
idc download manifest.txt --download-dir ./data
Options:
| Option | Description |
|---|
--download-dir | Output directory (default: current directory) |
--dir-template | Directory hierarchy template (default: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID) |
--log-level | Verbosity: debug, info, warning, error, critical |
Manifest files:
Manifest files contain S3 URLs (one per line) and can be:
- Exported from the IDC Portal after cohort selection
- Shared by collaborators for reproducible data access
- Generated programmatically from query results
Format (one S3 URL per line):
s3://idc-open-data/cb09464a-c5cc-4428-9339-d7fa87cfe837/*
s3://idc-open-data/88f3990d-bdef-49cd-9b2b-4787767240f2/*
Example: Generate manifest from Python query:
from idc_index import IDCClient
client = IDCClient()
results = client.sql_query("""
SELECT series_aws_url
FROM index
WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
""")
with open('ct_manifest.txt', 'w') as f:
for url in results['series_aws_url']:
f.write(url + '\n')
Then download:
idc download ct_manifest.txt --download-dir ./ct_data
4. Visualizing IDC Images
View DICOM data in browser without downloading:
from idc_index import IDCClient
import webbrowser
client = IDCClient()
results = client.sql_query("""
SELECT SeriesInstanceUID, StudyInstanceUID
FROM index
WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
LIMIT 1
""")
viewer_url = client.get_viewer_URL(seriesInstanceUID=results.iloc[0]['SeriesInstanceUID'])
webbrowser.open(viewer_url)
viewer_url = client.get_viewer_URL(studyInstanceUID=results.iloc[0]['StudyInstanceUID'])
webbrowser.open(viewer_url)
The method automatically selects OHIF v3 for radiology or SLIM for slide microscopy. Viewing by study is useful when a DICOM Study contains multiple Series (e.g., T1, T2, DWI sequences from a single MRI session).
5. Understanding and Checking Licenses
Check data licensing before use (critical for commercial applications):
from idc_index import IDCClient
client = IDCClient()
query = """
SELECT DISTINCT
collection_id,
license_short_name,
COUNT(DISTINCT SeriesInstanceUID) as series_count
FROM index
GROUP BY collection_id, license_short_name
ORDER BY collection_id
"""
licenses = client.sql_query(query)
print(licenses)
License types in IDC:
- CC BY 4.0 / CC BY 3.0 (~97% of data) - Allows commercial use with attribution
- CC BY-NC 4.0 / CC BY-NC 3.0 (~3% of data) - Non-commercial use only
- Custom licenses (rare) - Some collections have specific terms (e.g., NLM Terms and Conditions)
Important: Always check the license before using IDC data in publications or commercial applications. Each DICOM file is tagged with its specific license in metadata.
Generating Citations for Attribution
The source_DOI column contains DOIs linking to publications describing how the data was generated. To satisfy attribution requirements, use citations_from_selection() to generate properly formatted citations:
from idc_index import IDCClient
client = IDCClient()
citations = client.citations_from_selection(collection_id="rider_pilot")
for citation in citations:
print(citation)
results = client.sql_query("""
SELECT SeriesInstanceUID FROM index
WHERE collection_id = 'tcga_luad' LIMIT 5
""")
