TCIA_Radiology_Inventory.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OA_CwnKI1R4-"
      },
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/TCIA_Radiology_Inventory.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Radiology_Inventory.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KmXfYFZtja2F"
      },
      "source": [
        "# Summary\n",
        "\n",
        "This notebook can be used to generate reports that summarize TCIA's radiology datasets.  It also provides an option to compare the reports over time and assess changes in the number of patients, studies, series, images and disk space utilized.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqulqBEDMSS6"
      },
      "source": [
        "# 1 Setup\n",
        "\n",
        "Install the latest release of [**tcia_utils**](https://pypi.org/project/tcia-utils/) and import the necessary modules."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YPrZ5pc-yYQl"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# install tcia utils\n",
        "!{sys.executable} -m pip install --upgrade -q tcia_utils"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [],
        "id": "GJaQG42kLI1k"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "import pandas as pd\n",
        "from tcia_utils import nbia\n",
        "\n",
        "# set logging level to INFO in Google Colab (not necessary in Jupyter)\n",
        "if 'google.colab' in sys.modules:\n",
        "  import logging\n",
        "\n",
        "  for handler in logging.root.handlers[:]:\n",
        "      logging.root.removeHandler(handler)\n",
        "\n",
        "  # Set handler with level = info\n",
        "  logging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',\n",
        "                      level=logging.INFO)\n",
        "\n",
        "  print(\"Google Colab Logging set to INFO\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2uMbL1nwIJ5f"
      },
      "source": [
        "# 2 Create a Token (optional)\n",
        "This is necessary if you're planning on including restricted NBIA datasets in the reports.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jgn4YvSa9GBZ"
      },
      "outputs": [],
      "source": [
        "# create token\n",
        "nbia.getToken()\n",
        "\n",
        "# set api_url to include restricted collections\n",
        "api_url = \"restricted\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LHbh-qSHDhs3"
      },
      "source": [
        "# 3 Generate new report\n",
        "Next we'll generate  reports for what's currently in the system.  First we'll choose **ONE** of the two cells below based on whether you want to run a report on all collections or only specific collections.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0Cgh-_makmY9"
      },
      "outputs": [],
      "source": [
        "# For all collections, run this cell\n",
        "collections_json = nbia.getCollections(api_url = api_url)\n",
        "collections = [item['Collection'] for item in collections_json]\n",
        "\n",
        "print(collections)\n",
        "print(str(len(collections_json)) + \" collections were found.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r5L9W9NSkuPh"
      },
      "outputs": [],
      "source": [
        "# For specific collections, update and run this cell\n",
        "collections = [\"LIDC-IDRI\", \"VICTRE\", \"BREAST-DIAGNOSIS\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvHNC40wk8xU"
      },
      "source": [
        "Next we'll build Patient, Study and Series level inventories of what's currently available in those collections."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UkCLU70TlgWA"
      },
      "outputs": [],
      "source": [
        "# get inventory of studies\n",
        "\n",
        "studies = pd.DataFrame()\n",
        "\n",
        "for collection in collections:\n",
        "    studyDescription = nbia.getStudy(collection, api_url = api_url)\n",
        "    studies = pd.concat([studies, pd.DataFrame(studyDescription)], ignore_index=True)\n",
        "\n",
        "studies.to_csv('study_metadata_{}.csv'.format(datetime.date.today()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mQt8R7xcmX72"
      },
      "outputs": [],
      "source": [
        "# get inventory of series\n",
        "\n",
        "series = pd.DataFrame()\n",
        "\n",
        "for collection in collections:\n",
        "    seriesDescription = nbia.getSeries(collection, api_url = api_url)\n",
        "    series = pd.concat([series, pd.DataFrame(seriesDescription)], ignore_index=True)\n",
        "\n",
        "series.to_csv('series_metadata_{}.csv'.format(datetime.date.today()))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Summarize the inventory reports\n",
        "Here is the summary of those reports:"
      ],
      "metadata": {
        "id": "7BW3E2jCNWvw"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jiIk4fVONUAv"
      },
      "outputs": [],
      "source": [
        "# Total count of unique PatientID and StudyInstanceUID values in studies\n",
        "total_unique_patient_ids = studies['PatientID'].nunique()\n",
        "total_unique_study_instance_uids = studies['StudyInstanceUID'].nunique()\n",
        "\n",
        "# Total count of values in series\n",
        "total_unique_series_patient_ids = series['PatientID'].nunique()\n",
        "total_unique_series_instance_uids = series['SeriesInstanceUID'].nunique()\n",
        "image_count_sum = series['ImageCount'].sum()\n",
        "file_size_sum = series['FileSize'].sum()\n",
        "disk_space = nbia.format_disk_space(file_size_sum)\n",
        "\n",
        "# Print the summary statistics\n",
        "print(\"Summary Statistics:\")\n",
        "print(f\"Total currently available subjects (in series report): {total_unique_series_patient_ids}\")\n",
        "print(f\"Total currently available subjects (in study report): {total_unique_patient_ids}\")\n",
        "print(f\"Total currently available studies: {total_unique_study_instance_uids}\")\n",
        "print(f\"Total currently available series: {total_unique_series_instance_uids}\")\n",
        "print(f\"Total currently available images: {image_count_sum}\")\n",
        "print(f\"Total current FileSize: {file_size_sum} bytes or {disk_space}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0EdHnIZTF3Xt"
      },
      "source": [
        "# 4 Compare your results against previously generated reports (optional)\n",
        "\n",
        "Follow these steps if you want to compare what's currently available with older reports.  First, we'll import the old reports you want to compare.  Don't forget to update the file names to match your reports.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jpbyz-Wi-MqK"
      },
      "outputs": [],
      "source": [
        "# import results of getStudy()\n",
        "oldStudies = pd.read_csv(\"study_metadata_2023-11-14.csv\")\n",
        "\n",
        "# import results of getSeries()\n",
        "oldSeries = pd.read_csv(\"series_metadata_2023-11-14.csv\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZHLc9W4Iei21"
      },
      "source": [
        "This next step produces UID-level reports to allow one to look at the specifics of what changed between the current and previous study/series reports.  This accounts for new data additions as well as data that may have been removed.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AEe04x3V02K6"
      },
      "outputs": [],
      "source": [
        "# Merge the studies and oldStudies dataframes based on the PatientID and StudyInstanceUID columns\n",
        "studyDiff = pd.merge(studies, oldStudies, on=['PatientID', 'StudyInstanceUID'], how='outer', indicator=True)\n",
        "\n",
        "# Filter the merged dataframe to keep only the rows that are different\n",
        "studyDiff = studyDiff[studyDiff['_merge'] != 'both']\n",
        "\n",
        "# Drop the '_merge' column\n",
        "studyDiff = studyDiff.drop('_merge', axis=1)\n",
        "\n",
        "# Print the studyDiff dataframe\n",
        "display(studyDiff)\n",
        "studyDiff.to_csv(\"studyDiff.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dyrjIGsY91xx"
      },
      "outputs": [],
      "source": [
        "# Merge the series and oldSeries dataframes based on the PatientID and SeriesInstanceUID columns\n",
        "seriesDiff = pd.merge(series, oldSeries, on=['PatientID', 'SeriesInstanceUID'], how='outer', indicator=True)\n",
        "\n",
        "# Filter the merged dataframe to keep only the rows that are different\n",
        "seriesDiff = seriesDiff[seriesDiff['_merge'] != 'both']\n",
        "\n",
        "# Drop the '_merge' column\n",
        "seriesDiff = seriesDiff.drop('_merge', axis=1)\n",
        "\n",
        "# Print the studyDiff dataframe\n",
        "display(seriesDiff)\n",
        "seriesDiff.to_csv(\"seriesDiff.csv\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ldXoxgIIaBSk"
      },
      "source": [
        "## Comparison Summary\n",
        "Here we will summarize the overall changes between the current and previous reports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UPp0K5IpaSZl"
      },
      "outputs": [],
      "source": [
        "# Count of new PatientID values in studies but not in oldStudies\n",
        "new_patient_ids = studies[~studies['PatientID'].isin(oldStudies['PatientID'])]['PatientID'].nunique()\n",
        "\n",
        "# Count of new StudyInstanceUID values in studies but not in oldStudies\n",
        "new_study_instance_uids = studies[~studies['StudyInstanceUID'].isin(oldStudies['StudyInstanceUID'])]['StudyInstanceUID'].nunique()\n",
        "\n",
        "# Count of PatientID values in oldStudies but not in studies\n",
        "missing_patient_ids = oldStudies[~oldStudies['PatientID'].isin(studies['PatientID'])]['PatientID'].nunique()\n",
        "\n",
        "# Count of StudyInstanceUID values in oldStudies but not in studies\n",
        "missing_study_instance_uids = oldStudies[~oldStudies['StudyInstanceUID'].isin(studies['StudyInstanceUID'])]['StudyInstanceUID'].nunique()\n",
        "\n",
        "# Total count of unique PatientID and StudyInstanceUID values in studies\n",
        "total_unique_patient_ids = studies['PatientID'].nunique()\n",
        "total_unique_study_instance_uids = studies['StudyInstanceUID'].nunique()\n",
        "\n",
        "# Count of new PatientID values in series but not in oldSeries\n",
        "new_series_patient_ids = series[~series['PatientID'].isin(oldSeries['PatientID'])]['PatientID'].nunique()\n",
        "\n",
        "# Count of new SeriesInstanceUID values in series but not in oldSeries\n",
        "new_series_instance_uids = series[~series['SeriesInstanceUID'].isin(oldSeries['SeriesInstanceUID'])]['SeriesInstanceUID'].nunique()\n",
        "\n",
        "# Count of PatientID values in oldSeries but not in series\n",
        "missing_series_patient_ids = oldSeries[~oldSeries['PatientID'].isin(series['PatientID'])]['PatientID'].nunique()\n",
        "\n",
        "# Count of SeriesInstanceUID values in oldSeries but not in series\n",
        "missing_series_instance_uids = oldSeries[~oldSeries['SeriesInstanceUID'].isin(series['SeriesInstanceUID'])]['SeriesInstanceUID'].nunique()\n",
        "\n",
        "# Total file size/images in oldSeries\n",
        "old_image_count_sum = oldSeries['ImageCount'].sum()\n",
        "old_file_size_sum = oldSeries['FileSize'].sum()\n",
        "\n",
        "# Total count of values in series\n",
        "total_unique_series_patient_ids = series['PatientID'].nunique()\n",
        "total_unique_series_instance_uids = series['SeriesInstanceUID'].nunique()\n",
        "image_count_sum = series['ImageCount'].sum()\n",
        "file_size_sum = series['FileSize'].sum()\n",
        "disk_space = nbia.format_disk_space(file_size_sum)\n",
        "\n",
        "# Change in images/size\n",
        "image_diff = image_count_sum - old_image_count_sum\n",
        "size_diff = file_size_sum - old_file_size_sum\n",
        "disk_space_diff = nbia.format_disk_space(size_diff)\n",
        "\n",
        "# Print the summary statistics\n",
        "print(\"Summary Statistics:\")\n",
        "print(f\"New subjects added (in study report): {new_patient_ids}\")\n",
        "print(f\"Number of subjects removed (in study report): {missing_patient_ids}\")\n",
        "print(f\"New subjects added (in series report): {new_series_patient_ids}\")\n",
        "print(f\"Number of subjects removed (in series report): {missing_patient_ids}\")\n",
        "print(f\"New studies: {new_study_instance_uids}\")\n",
        "print(f\"Studies removed: {missing_study_instance_uids}\")\n",
        "print(f\"New series: {new_series_instance_uids}\")\n",
        "print(f\"Series removed: {missing_series_instance_uids}\")\n",
        "print(f\"Change in image count: {image_diff}\")\n",
        "print(f\"Change in disk size: {size_diff} or {disk_space_diff}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J6WDxHQP2TYc"
      },
      "source": [
        "# 5 National Lung Screening Trial (optional)\n",
        "If you want to include **all** TCIA datasets you must separately account for the National Lung Screening Trial collection.  This collection lives on a separate server due to its size.  Also, this is a completed collection and is very unlikely to change at this point.  However, the following steps can be used to ensure you have the latest inventory of what's available."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PKsCLGZS29-W"
      },
      "outputs": [],
      "source": [
        "# get inventory of studies\n",
        "collections = [\"NLST\"]\n",
        "nlstStudies = pd.DataFrame()\n",
        "\n",
        "for collection in collections:\n",
        "    studyDescription = nbia.getStudy(collection, api_url = \"nlst\")\n",
        "    nlstStudies = pd.concat([nlstStudies, pd.DataFrame(studyDescription)], ignore_index=True)\n",
        "\n",
        "nlstStudies.to_csv('nlst_study_metadata_{}.csv'.format(datetime.date.today()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cvBj7ca729-c"
      },
      "outputs": [],
      "source": [
        "# get inventory of series\n",
        "collections = [\"NLST\"]\n",
        "nlstSeries = pd.DataFrame()\n",
        "\n",
        "for collection in collections:\n",
        "    seriesDescription = nbia.getSeries(collection, api_url = \"nlst\")\n",
        "    nlstSeries = pd.concat([nlstSeries, pd.DataFrame(seriesDescription)], ignore_index=True)\n",
        "\n",
        "nlstSeries.to_csv('nlst_series_metadata_{}.csv'.format(datetime.date.today()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vcrHl9Af2KCd"
      },
      "outputs": [],
      "source": [
        "# Total count of unique NLST PatientID and StudyInstanceUID values\n",
        "total_unique_nlst_patient_ids = nlstStudies['PatientID'].nunique()\n",
        "total_unique_nlst_study_instance_uids = nlstStudies['StudyInstanceUID'].nunique()\n",
        "total_unique_nlst_series_instance_uids = nlstSeries['SeriesInstanceUID'].nunique()\n",
        "nlst_image_count_sum = nlstSeries['ImageCount'].sum()\n",
        "nlst_file_size_sum = nlstSeries['FileSize'].sum()\n",
        "nlst_disk_space = nbia.format_disk_space(nlst_file_size_sum)\n",
        "\n",
        "# Print the summary statistics\n",
        "print(\"NLST Summary Statistics:\")\n",
        "print(f\"Total subjects: {total_unique_nlst_patient_ids}\")\n",
        "print(f\"Total studies: {total_unique_nlst_study_instance_uids}\")\n",
        "print(f\"Total series: {total_unique_nlst_series_instance_uids}\")\n",
        "print(f\"Total images: {nlst_image_count_sum}\")\n",
        "print(f\"Total FileSize: {nlst_file_size_sum} bytes or {nlst_disk_space}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XvkFKr75lrw"
      },
      "source": [
        "Now let's add everything together again with NLST included."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KOSvHo4f6AZ2"
      },
      "outputs": [],
      "source": [
        "print(\"TCIA DICOM radiology collections include:\",\n",
        "      total_unique_patient_ids + total_unique_nlst_patient_ids, \"subjects,\",\n",
        "      total_unique_study_instance_uids + total_unique_nlst_study_instance_uids, \"studies,\",\n",
        "      total_unique_series_instance_uids + total_unique_nlst_series_instance_uids, \"series,\",\n",
        "      image_count_sum + nlst_image_count_sum, \"images, which requires\",\n",
        "      nbia.format_disk_space(file_size_sum + nlst_file_size_sum), \"of storage.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYXsNGcY93B8"
      },
      "source": [
        "# Acknowledgements\n",
        "TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/), and is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/).\n",
        "\n",
        "This notebook was created by [Justin Kirby](https://www.linkedin.com/in/justinkirby82/).  If you leverage this notebook or any TCIA datasets in your work, please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF). In particular, make sure to cite the DOI(s) for the specific TCIA datasets you used in addition to the following paper!\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}