imaging-data-commons

# Imaging Data Commons ## Overview Query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access. **Expected network access:** IDC metadata is reachable three ways — a bundled local DuckDB index (offline after installation; additional indices are fetched from GitHub and clinical tables from S3), or the hosted IDC service over MCP or REST (`api.imaging.datacommons.cancer.gov`, no authentication). File downloads use public GCS (`storage.googleapis.com`) and AWS S3 (`s3.amazonaws.com`) — no authentication required. DICOMweb access uses either the public IDC proxy (`proxy.imaging.datacommons.cancer.gov`, no auth) or the Google Cloud Healthcare API (`healthcare.googleapis.com`, requires GCP authentication). Optional BigQuery queries (`bigquery.googleapis.com`) also require GCP authentication. Citation resolution contacts DOI services. Public IDC routes require no credentials; optional Google clients use Application Default Credentials. **Reviewed 2026-09-30:** idc-index 0.12.5, idc-index-data 24.2.2, IDC v24; hosted API 3.0.0b3. Recheck at use time. **Choose the access path first.** There is no single default: the cheapest correct path depends on the session and the task. 1. **Session already has the IDC MCP server?** Route discovery and metadata there — see *IDC MCP Server*. 2. **Otherwise, is `idc-index` installed and current?** Run `python scripts/check_version.py`. If it passes, use `idc-index` for everything. 3. **Not installed, and the task is read-only metadata** — counts, attribute values, collection lookups, SQL under 10 000 rows, licenses, citations, viewer URLs? **Use the REST API over `curl`; do not install anything.** Installing costs ~77 MB of packaged index data plus pandas, pyarrow, and duckdb, which a metadata question does not need. See *Data Access Options*. 4. **Not installed, and the task needs more than metadata** — downloading files, pandas or plotting, pydicom/SimpleITK, pathology tiling, results past 10 000 rows, or a version-pinned script the user re-runs? Install `idc-index`: `check_version.py` exits non-zero and prints the exact install command for the running interpreter. Prefer a virtual environment, then restart Python. `idc-index` ([GitHub](https://github.com/imagingdatacommons/idc-index)) is still the most capable Python path, with query and download helpers in one client. `check_version.py` never installs anything itself — it also flags a newer `idc-index` or skill release when one exists. **Setup for the `idc-index` path:** use the intended interpreter for `scripts/check_version.py` and confirm its `meets pinned minimum` message before continuing. A launch failure is not a pass. ```python from idc_index import IDCClient client = IDCClient() # Verify IDC data version (should be "v24") print(f"IDC data version: {client.get_idc_version()}") ``` Download and image-processing examples are illustrative unless noted; see the reference review notes for verification scope. **Core workflow:** query metadata with `client.sql_query()` → download with `client.download_from_selection()` → visualize with `client.get_viewer_URL()`. Python examples below assume this `client`; *Data Access Options* has the REST equivalents. For current data scale, run the summary query in `references/sql_patterns.md` or `GET /v3/stats`. ## IDC MCP Server IDC operates a hosted MCP server at `https://api.imaging.datacommons.cancer.gov/mcp` (streamable HTTP, no authentication). Where it is available it complements — it does not replace — the `idc-index` workflow below. **Identify it** by the MCP resource `idc://guide`, or by three or more of the tool names `build_cohort`, `get_cohort_urls`, `list_analysis_results`, and `get_idc_version`. Generic names such as `run_sql` are not evidence on their own. If identification is ambiguous, use `idc-index`. **If this session has the server**, treat it as authoritative for discovery and metadata — IDC version, counts, attribute values, cohort building, metadata SQL — and follow the server's own instructions rather than re-deriving them from this file. Its data version is whatever the server reports: call `get_idc_version` instead of relying on the version pinned in this file. Return here for what the server does not do: downloading files, local pandas/notebook analysis, DICOMweb, BigQuery, digital pathology tiling, and reproducible scripts. Hand off by passing SeriesInstanceUIDs from the server to `client.download_from_selection(...)`, and run `scripts/check_version.py` at that point. **If it is not available**, the identical service is reachable with no configuration as a REST API at `https://api.imaging.datacommons.cancer.gov/v3` — use it for read-only metadata rather than installing `idc-index`, per the routing gate in *Overview*. Suggest connecting the MCP server at most once, only for repeated interactive discovery, and never change the user's configuration yourself. See `references/mcp_guide.md` for the tool inventory, handoff patterns, and per-host notes. ## When to Use This Skill - Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images - Selecting image subsets by cancer type, modality, anatomical site, or other metadata - Downloading DICOM data from IDC - Checking data licenses before use in research or commercial applications - Visualizing medical images in a browser without local DICOM viewer software ## Quick Navigation Inline below: the MCP/REST routing rules, the IDC data model, the index tables and how they join, the core API patterns (query, download, visualize, license, cite), best practices, and troubleshooting. **Reference Guides (load on demand):** | Guide | When to Load | |-------|--------------| | `index_tables_guide.md` | Complex JOINs, schema discovery, DataFrame access | | `use_cases.md` | End-to-end workflows: training datasets, batch downloads, DICOM reading with pydicom/SimpleITK, pipeline integration | | `sql_patterns.md` | Quick SQL patterns for filter discovery, annotations, size estimation | | `clinical_data_guide.md` | Clinical/tabular data, imaging+clinical joins, value mapping | | `licensing_and_citation.md` | Commercial-use questions, mixed-license cohorts, citation formats | | `cloud_storage_guide.md` | Direct S3/GCS access, versioning, UUID mapping | | `dicomweb_guide.md` | DICOMweb endpoints, PACS integration | | `digital_pathology_guide.md` | Slide microscopy (SM), annotations (ANN), pathology workflows | | `bigquery_guide.md` | Full DICOM metadata, private elements (requires GCP) | | `cli_guide.md` | Command-line tools (`idc download`, manifest files) | | `parquet_access_guide.md` | Direct Parquet queries via GCS (no idc-index install needed) | | `mcp_guide.md` | Hosted IDC MCP server: tool inventory, identification, handoff to `idc-index` | | `rest_api_guide.md` | Hosted IDC REST API: endpoints, filter syntax, SQL over HTTP, manifests | ## IDC Data Model IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance): - **collection_id**: Groups patients by disease, modality, or research focus (e.g., `tcga_luad`, `nlst`). Treat `(collection_id, PatientID)` as the patient key; do not assume PatientID is globally unique. - **analysis_result_id**: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections. Use it to find AI-generated or expert annotations, while `collection_id` finds original imaging data (which may itself include deposited annotations). **Key identifiers for queries:** | Identifier | Scope | Use for | |------------|-------|---------| | `collection_id` | Dataset grouping | Filtering by project/study | | `PatientID` | Patient | Grouping images by patient | | `StudyInstanceUID` | DICOM study | Grouping of related series, visualization | | `SeriesInstanceUID` | DICOM series | Grouping of related series, visualization | ## Index Tables The `idc-index` package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames. The REST API exposes the same tables through `GET /tables` and `POST /sql`. **Important:** `client.indices_overview` is the authoritative source for current table descriptions, available columns, and their types — query it when writing SQL or exploring data structure. It also answers "which table contains column X"; see `references/index_tables_guide.md` for that search pattern and full schema discovery. ### Available Tables Always call `client.fetch_index("table_name")` before querying any index table — it is safe and idempotent for all tables, including those loaded automatically at startup. | Family | Tables | Granularity | |--------|--------|-------------| | Core | `index` (primary metadata for all current data), `collections_index`, `analysis_results_index` | series / collection / analysis result | | Modality acquisition parameters | `ct_index`, `mr_index`, `pt_index`, `contrast_index` | 1 row = 1 series of that modality | | Derived objects | `seg_index`, `rtstruct_index`, `ann_index`, `ann_group_index` | 1 row = 1 series (or annotation group) | | Microscopy | `sm_index`, `sm_instance_index` | 1 row = 1 SM series / instance | | Geometry, clinical, history | `volume_geometry_index`, `clinical_index`, `version_metadata_index`, `prior_versions_index` | see guide | `references/index_tables_guide.md` has the full inventory with each table's columns and contents — load it when you need to know what a specialized table actually holds. **`prior_versions_index` contains historical series versions**, including revised versions whose DICOM SeriesInstanceUID still occurs in `index`. Pin `crdc_series_uuid` for historical content. For "what's new" in the current release use `series_init_idc_version` / `series_revised_idc_version` in the main `index` table, which are not equivalent to this table's `min_idc_version` / `max_idc_version`. ### Joining Tables **`SeriesInstanceUID` is the universal join key** for all series-level specialized tables: `sm_index`, `sm_instance_index`, `seg_index`, `ann_index`, `ann_group_index`, `contrast_index`, `volume_geometry_index`, `rtstruct_index`, `ct_index`, `mr_index`, `pt_index`. Always join these to `index` on `SeriesInstanceUID`. The exceptions below use different column names. | Join Column | Tables | Use Case | |-------------|--------|----------| | `collection_id` | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data | | `analysis_result_id` | index, analysis_results_index | Link series to analysis result metadata (annotations, segmentations) | | `source_DOI` | index, analysis_results_index | Link by publication DOI | | `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (`seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID`) | | `referenced_SeriesInstanceUID` | ann_index → index, rtstruct_index → index | Link annotation or RTSTRUCT to its source image series | **Note:** `subjects`, `updated`, and `description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts). A UID-only join to `prior_versions_index` can match several historical revisions; use CRDC UUIDs to distinguish them. For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see `references/index_tables_guide.md`. ### Clinical Data Access Clinical (non-imaging) attributes — staging, demographics, therapy — live in per-collection tables. `client.fetch_index("clinical_index")` loads the dictionary mapping columns to collections; `client.get_clinical_table(name)` returns one table as a DataFrame. See `references/clinical_data_guide.md` for the discovery workflow, coded-value mapping, and joining clinical data with imaging. ## Data Access Options | Method | Auth | Best For | Reference | |--------|------|----------|-----------| | `idc-index` | No | Downloads, pandas analysis, unbounded queries — the most capable path | This document | | IDC MCP server | No | Discovery, cohort building, metadata when the session already has it | `mcp_guide.md` | | IDC REST API | No | Metadata with no install, from any language or shell — the default when `idc-index` is absent | `rest_api_guide.md` | | Direct Parquet (GCS) | No | Version-pinned queries, or results past the REST row cap | `parquet_access_guide.md` | | Cloud storage (S3/GCS) | No | Direct file access, bulk transfer, custom pipelines | `cloud_storage_guide.md` | | DICOMweb via IDC proxy | No | Tool and PACS integration; daily quota, so testing and moderate use | `dicomweb_guide.md` | | DICOMweb via Google Healthcare | Yes (GCP) | The same DICOMweb API at production volume, without the proxy quota | `dicomweb_guide.md` | | SlicerIDCBrowser | No | 3D visualization and analysis in 3D Slicer | https://github.com/ImagingDataCommons/SlicerIDCBrowser | | BigQuery | Yes (GCP) | Full DICOM metadata, private elements, SR measurements — last resort | `bigquery_guide.md` | **The IDC Portal (https://portal.imaging.datacommons.cancer.gov/) is interactive only** — browser-based exploration, manual cohort selection, and download. Unlike every option above it has no programmatic interface, so point a user there to browse or click through data themselves; never use it as a step in a script or workflow. **REST API — the no-install metadata path** `https://api.imaging.datacommons.cancer.gov/v3`, no authentication: discovery, cohort counts and manifests, read-only SQL, clinical tables, viewer URLs, licenses, citations. It is the same service as the MCP server over plain HTTP, so it needs no configuration. It never moves image bytes — switch to `idc-index` to download, to get a DataFrame, or for results past 10 000 rows. ```bash B=https://api.imaging.datacommons.cancer.gov/v3 curl -s $B/version # idc_version, idc_index_data_version, api_version curl -s $B/stats # collections, patients, studies, series, instances, size_TB curl -s "$B/attributes/Modality/values?limit=5" # real filter values, with counts curl -s $B/sql -H 'content-type: application/json' \ -d '{"sql":"SELECT collection_id, COUNT(*) n FROM index GROUP BY 1 ORDER BY n DESC LIMIT 3"}' curl -s $B/cohort/counts -H 'content-type: application/json' \ -d '{"filters":{"terms":{"collection_id":["rider_pilot"]}}}' ``` **The filter object always goes under `filters`** — on `cohort/counts`, `cohort/manifest`, `cohort/manifest.txt`, `licenses`, and `citations` alike. A bare filter or an unrecognized key is a 422 naming the fix; an unfiltered series-enumerating request is a 400, not the whole archive. Filtered JSON responses echo `filters_applied` and `warnings` (inside `counts` for manifests) — read them, because they name any predicate the server dropped. A zero count with empty `warnings` therefore means the filter matched nothing, not that a value was miscased; miscasing produces a warning that says so. `POST /sql` takes one read-only `SELECT`/`WITH` over the tables `idc-index` exposes plus `clinical.<table>`; `max_rows` defaults to 5 000, caps at 10 000, and `truncated` flags clipping. `GET /attributes` lists the 19 filterable attributes — clinical values, segmented anatomy, and acquisition parameters are not among them and need SQL. Request limits still apply. **Use v3 only:** V1 and V2 are superseded and scheduled for shutdown, so port any `/v1/`- or `Modality_btw`-style example a user brings rather than extending it. Both sides build on `idc-index-data`, so compare the API's `idc_index_data_version` against local `idc_index_data.__version__` before mixing them: the **major is the IDC data release** (`24.x.y` serves `v24`); minor/patch index builds may correct metadata. If the API is a whole release ahead, `idc-index` **cannot download the extra series** — mixed selections can omit unrecognized UIDs, while wholly unmatched selections raise — so either upgrade it (run `scripts/check_version.py` for the right command) or transfer directly from the bucket with `s5cmd --no-sign-request`. See `references/rest_api_guide.md` for the endpoint reference, filter grounding, limits, and the manifest-based download flow. **Cloud storage organization** All DICOM files live in public buckets mirrored between AWS S3 and GCS, organized by CRDC UUIDs