The RSNA Pulmonary Embolism CT Dataset

Errol Colak; Felipe C Kitamura; Stephen B Hobbs; Carol C Wu; Matthew P Lungren; Luciano M Prevedello; Jayashree Kalpathy-Cramer; Robyn L Ball; George Shih; Anouk Stein; Safwan S Halabi; Emre Altinmakas; Meng Law; Parveen Kumar; Karam A Manzalawi; Dennis Charles Nelson Rubio; Jacob W Sechrist; Pauline Germaine; Eva Castro Lopez; Tomas Amerio; Pushpender Gupta; Manoj Jain; Fernando U Kay; Cheng Ting Lin; Saugata Sen; Jonathan Wesley Revels; Carola C Brussaard; John Mongan; For the RSNA-STR Annotators and Dataset Curation Contributors

doi:10.1148/ryai.2021200254

. 2021 Jan 20;3(2):e200254. doi: 10.1148/ryai.2021200254

The RSNA Pulmonary Embolism CT Dataset

Errol Colak

Errol Colak, MD

¹From the Department of Medical Imaging, Unity Health Toronto, University of Toronto, 30 Bond St, Toronto, ON, Canada M5B 1W8 (E.C.); Department of Diagnostic Imaging, Universidade Federal de São Paulo, São Paulo, Brazil (F.C.K.); Diagnósticos da América SA (Dasa) (F.C.K.); Department of Radiology, University of Kentucky, Lexington, Ky (S.B.H.); Department of Diagnostic Radiology, University of Texas MD Anderson Cancer Center, Houston, Tex (C.C.W.); Department of Radiology, Stanford University, Stanford, Calif (M.P.L., S.S.H.); Department of Radiology, The Ohio State University, Columbus, Ohio (L.M.P.); Department of Radiology and Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Mass (J.K.); The Jackson Laboratory, Bar Harbor, Maine (R.L.B.); Department of Radiology, Weill Cornell Medical College, New York, NY (G.S.); MD.ai, New York, NY (A.S.); Department of Radiology, Koc University School of Medicine, Istanbul, Turkey (E.A.); Department of Radiology and Nuclear Medicine, Alfred Health, Monash University, Melbourne, Australia (M.L.); Department of Radiodiagnosis, Fortis Escorts Heart Institute, New Delhi, India (P.K.); Department of Diagnostic Radiology and Nuclear Medicine, Faculty of Medicine, University of Jordan, Amman, Jordan (K.A.M.); Department of Departamento de Imagenología, Hospital Regional de Alta Especialidad de la Península de Yucatán, Mérida, Mexico (D.C.N.R.); Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, Pa (J.W.S.); Department of Radiology, Cooper University Hospital, Camden, NJ (P. Germaine); A Coruña University Hospital, A Coruña, Spain (E.C.L.); Swiss Medical Group, Buenos Aires, Argentina (T.A.); Inland Imaging, Spokane, Wash (P. Gupta); AMRI Hospitals, Kolkata, India (M.J.); Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (F.U.K.); Department of Radiology, Johns Hopkins University School of Medicine, Baltimore, Md (C.T.L.); Department of Radiology and Imaging Sciences, Tata Medical Center, Kolkata, India (S.S.); Department of Radiology, University of New Mexico, Albuquerque, NM (J.W.R.); Department of Radiology, Universitair Ziekenhuis Brussel, Jette, Belgium (C.C.B.); Department of Radiology and Biomedical Imaging, University of California–San Francisco, San Francisco, Calif (J.M).

Find articles by Errol Colak

^1,^✉, Felipe C Kitamura

Felipe C Kitamura, MD, PhD

Find articles by Felipe C Kitamura

¹, Stephen B Hobbs

Stephen B Hobbs, MD

Find articles by Stephen B Hobbs

¹, Carol C Wu

Carol C Wu, MD

Find articles by Carol C Wu

¹, Matthew P Lungren

Matthew P Lungren, MD, MPH

Find articles by Matthew P Lungren

¹, Luciano M Prevedello

Luciano M Prevedello, MD

Find articles by Luciano M Prevedello

¹, Jayashree Kalpathy-Cramer

Jayashree Kalpathy-Cramer, PhD

Find articles by Jayashree Kalpathy-Cramer

¹, Robyn L Ball

Robyn L Ball, PhD

Find articles by Robyn L Ball

¹, George Shih

George Shih, MD

Find articles by George Shih

¹, Anouk Stein

Anouk Stein, PhD

Find articles by Anouk Stein

¹, Safwan S Halabi

Safwan S Halabi, MD

Find articles by Safwan S Halabi

¹, Emre Altinmakas

Emre Altinmakas, MD

Find articles by Emre Altinmakas

¹, Meng Law

Meng Law, MD, MBBS

Find articles by Meng Law

¹, Parveen Kumar

Parveen Kumar, MD

Find articles by Parveen Kumar

¹, Karam A Manzalawi

Karam A Manzalawi, MD

Find articles by Karam A Manzalawi

¹, Dennis Charles Nelson Rubio

Dennis Charles Nelson Rubio, MD

Find articles by Dennis Charles Nelson Rubio

¹, Jacob W Sechrist

Jacob W Sechrist, MD

Find articles by Jacob W Sechrist

¹, Pauline Germaine

Pauline Germaine, DO

Find articles by Pauline Germaine

¹, Eva Castro Lopez

Eva Castro Lopez, MD

Find articles by Eva Castro Lopez

¹, Tomas Amerio

Tomas Amerio, MD

Find articles by Tomas Amerio

¹, Pushpender Gupta

Pushpender Gupta, MD

Find articles by Pushpender Gupta

¹, Manoj Jain

Manoj Jain, MD

Find articles by Manoj Jain

¹, Fernando U Kay

Fernando U Kay, MD

Find articles by Fernando U Kay

¹, Cheng Ting Lin

Cheng Ting Lin, MD

Find articles by Cheng Ting Lin

¹, Saugata Sen

Saugata Sen, MD

Find articles by Saugata Sen

¹, Jonathan Wesley Revels

Jonathan Wesley Revels, DO

Find articles by Jonathan Wesley Revels

¹, Carola C Brussaard

Carola C Brussaard, MD

Find articles by Carola C Brussaard

¹, John Mongan

John Mongan, MD, PhD

Find articles by John Mongan

¹; For the RSNA-STR Annotators and Dataset Curation Contributors¹

^✉

Address correspondence to E.C. (e-mail: Errol.Colak@unityhealth.to).

Author contributions: Guarantors of integrity of entire study, E.C., F.C.K., S.S.H., P.K., K.A.M., T.A., S.S., J.W.R., C.C.B., J.M.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, E.C., C.C.W., G.S., P.K., K.A.M.; clinical studies, E.C., F.C.K., S.B.H., L.M.P., S.S.H., E.A., M.L., K.A.M., D.C.N.R., J.W.S., P. Germaine, E.C.L., T.A., P. Gupta, M.J., F.U.K., C.T.L., S.S., J.W.R., C.C.B.; statistical analysis, E.C., J.K.C., R.L.B., A.S., K.A.M.; and manuscript editing, E.C., F.C.K., S.B.H., C.C.W., M.P.L., L.M.P., J.K.C., R.L.B., G.S., A.S., M.L., K.A.M., J.W.S., P. Germaine, E.C.L., T.A., F.U.K., S.S., J.W.R., J.M.

^✉

Corresponding author.

PMCID: PMC8043364 PMID: 33937862

Abstract

Supplemental material is available for this article.

Keywords: CT-Angiography, Diagnosis, Embolism/Thrombosis, Feature detection, Informatics, Pulmonary Arteries, Thorax

Summary

This dataset is composed of CT pulmonary angiograms and annotations related to pulmonary embolism. It is available at https://www.rsna.org/education/ai-resources-and-training/ai-image-challenge/rsna-pe-detection-challenge-2020.

Key Points

■ This multinational dataset is, to our knowledge, the largest publicly available pulmonary embolism (PE) CT dataset that includes expert annotations from a large group of subspecialist thoracic radiologists.
■ A subset of the dataset was used by the Radiological Society of North America and Society of Thoracic Radiology Pulmonary Embolism Detection competition in 2020, which was hosted by Kaggle.
■ This dataset is made freely available to the research community for noncommercial use and aims to enable the creation of high-quality machine learning models that can diagnose PE.

Introduction

Pulmonary embolism (PE) is a serious and potentially life-threatening condition that represents the third most common cause of cardiovascular-related deaths after myocardial infarction and stroke (1). Establishing a diagnosis of PE can be challenging, as patients often present with nonspecific symptoms. Classic presentations include acute onset of dyspnea, pleuritic chest pain, tachycardia, and signs of right heart strain. Prompt and accurate detection of PE and an assessment of its severity are critical to guiding patient treatment. In addition, it is important to avoid overdiagnosis, as the treatment of PE carries its own set of risks (2).

CT pulmonary angiography (CTPA) is the preferred and most frequently used imaging modality to evaluate patients suspected of having PE (3,4). CTPA studies consist of hundreds of images that require detailed radiologist review to identify filling defects within the pulmonary arterial vasculature. Given the number of images that require scrutiny and increased use of CTPA and imaging in general (5), the constraint on radiologists’ time may contribute to delays in diagnosis. Machine learning models have been proposed as a means of triaging medical imaging examinations with critical findings (6) and providing computer-assisted detection of abnormalities (7). Other potential use cases include prognostication and quantification of clot burden. The development and optimization of machine learning models typically require large amounts of data. A commonly encountered obstacle faced when developing machine learning models, especially those that analyze medical imaging, is the availability of large, well-annotated datasets. Unfortunately, medical imaging datasets are often only accessible to researchers working within the particular institutions that house the data, and even these single-institution datasets often lack the diversity necessary to construct broadly applicable, transferrable machine learning models.

In an effort to advance both medical imaging and machine learning research, we have curated what is to our knowledge the largest reported publicly available annotated CTPA dataset. Our hope is that this multinational dataset will be used to develop innovative machine learning models that will have a direct impact on patient care.

Materials and Methods

Dataset Curation

The curation of this dataset builds upon the experience of the past three data releases by the Radiological Society of North America (RSNA) (8–10) and represents our most complex release to date. Data were collected from institutions in five different countries, which provides diversity in patient populations, imaging equipment, and protocols. This dataset is our largest release in terms of the total number of images, images per study, anatomic coverage, and annotation labels.

The specifications of the dataset were determined by the 2020 RSNA AI Challenge committee. The dataset was to be composed of axial soft-tissue window images from chest CT scans performed using a pulmonary angiography protocol. Preference would be made for images with 2.5- or 3.0-mm section thickness, as it would facilitate a more efficient annotation process than thinner-section images. Imaging data were contributed by Stanford University (Palo Alto, Calif), Unity Health Toronto (Toronto, Canada), Federal University of São Paulo (São Paulo, Brazil), Alfred Health (Melbourne, Australia), and Koç University (Istanbul, Turkey). Each contributing site was responsible for obtaining institutional approval and adhering to local legal regulations and best practices. The specifics of scan identification, image extraction, and de-identification were left to the discretion of the contributing sites. Study identification and data extraction for each site is described in Appendix E1 (supplement). The de-identification processes are described in Appendix E2 (supplement).

Annotation Process

The annotation of this dataset was pursued as a collaborative effort between the RSNA and Society of Thoracic Radiology (STR). An open call to RSNA and STR members was made for volunteers to serve as expert annotators of the dataset. Potential volunteers were invited to complete a survey indicating their level of training and expertise in thoracic radiology. A total of 190 of 650 survey respondents were invited to participate in annotation training. These individuals indicated that a majority of their current practice was in thoracic imaging or, at a minimum, they were fellowship-trained in thoracic imaging. Each volunteer was sent a welcome e-mail that included login information for the commercial web-based annotation platform (MD.ai), a detailed instructions document, and an instructional video on how to use the annotation tool. Each participant was required to label 20 practice cases that were compared with the “ground truth” labels established by S.B.H. (7 years of experience in cardiothoracic radiology, fellowship trained). Comparison was done using Python notebooks. To identify labeling outliers, a weighted scoring system was developed where each practice case was assigned points based on the ground truth labels (Table E1 [supplement]). A minimum threshold score was set based on comparisons of this scoring system against labeler interobserver variability. Eighty-six volunteers met the minimum score of 65% and were invited to label the dataset.

We adopted an annotation strategy that would balance the richness of annotation labels with the effort required to label the dataset. Bounding boxes, regions of interest, and centroid markers were initially considered but dismissed due to task complexity, time required to label each scan, and concerns about potential poor interobserver reliability (11). Labels were defined both at the study and image level (Table 1, Fig 1). Examination-level labels (Fig 2) are as follows: Negative Exam for PE, Indeterminate, Central PE, Right-sided PE, Left-sided PE, Right ventricle (RV)/left ventricle (LV) ratio: <1, RV/LV ratio ≥ 1, Chronic PE, Acute and Chronic PE, True Filling Defect not PE, Flow Artifact, Quality Assurance (QA)–motion, and QA-contrast. The only image-level label was PE Present on Image. The training portion of the dataset was annotated by a single individual, while the test portion of the dataset was triple read with each reader blinded to the annotations of the rest. Annotations for both images and studies in the test set were determined by consensus across three annotators and adjudicated, if necessary, by C.C.W. and/or S.B.H. (12 and 7 years of experience in cardiothoracic radiology, respectively, and fellowship trained).

Table 1:

Label Definitions for Data Annotation

Open in a new tab

Examples of study- and image-level labels. (a) Central Pulmonary Embolism (PE): saddle embolus within the main, right, and left pulmonary arteries (arrow). (b) Right-sided PE and Left-sided PE: pulmonary emboli within the right interlobar, right lower lobe (arrow), and lingular pulmonary arteries (arrowhead). (c) Chronic PE: nonocclusive intraluminal web within the right lower lobe pulmonary artery (arrow). (d) True Filling Defect not PE: left lung malignancy invading the left main pulmonary artery. (e) Flow Artifact: an apparent filling defect within the left pulmonary artery, which is due to laminar flow of contrast media rather than PE (arrow). (f) RV/LV Ratio: < 1: normal RV (red line) to LV (blue line) ratio. (g) RV/LV Ratio: ≥ 1: evidence of right heart strain characterized by an elevated RV/LV ratio. (h) QA-motion: impaired image quality at the lung bases due to respiratory motion (arrows). (i) QA-contrast: insufficient opacification of the pulmonary arterial tree (arrow) to allow for the assessment of PE. LV = left ventricle, QA = quality assurance RV = right ventricle. — Examples of study- and image-level labels. **(a)** Central Pulmonary Embolism (PE): saddle embolus within the main, right, and left pulmonary arteries (arrow). **(b)** Right-sided PE and Left-sided PE: pulmonary emboli within the right interlobar, right lower lobe (arrow), and lingular pulmonary arteries (arrowhead). **(c)** Chronic PE: nonocclusive intraluminal web within the right lower lobe pulmonary artery (arrow). **(d)** True Filling Defect not PE: left lung malignancy invading the left main pulmonary artery. **(e)** Flow Artifact: an apparent filling defect within the left pulmonary artery, which is due to laminar flow of contrast media rather than PE (arrow). **(f)** RV/LV Ratio: < 1: normal RV (red line) to LV (blue line) ratio. **(g)** RV/LV Ratio: ≥ 1: evidence of right heart strain characterized by an elevated RV/LV ratio. **(h)** QA-motion: impaired image quality at the lung bases due to respiratory motion (arrows). **(i)** QA-contrast: insufficient opacification of the pulmonary arterial tree (arrow) to allow for the assessment of PE. LV = left ventricle, QA = quality assurance RV = right ventricle.

Study-level label schema. If a study had at least one image annotated as PE Present on Image, the study had additional labels for location (one or more), RV/LV ratio (only one), and type (only one, where Acute PE is assumed if neither Chronic PE nor Acute and Chronic PE is annotated). Dashed lines indicate implied study-level labels. Similarly, if the study was labeled as Indeterminate, it was annotated with one or more quality assurance (QA) labels. Note that True Filling Defect not PE and Flow Artifact are informational labels only and not included in this diagram. LV = left ventricle, PE = pulmonary embolism, RV = right ventricle.

Volunteers were assigned a batch of 50 studies for annotation. As several volunteers completed these first batches quickly, assignments were extended to as many as 150 studies per person. A subset of studies required reassignment due to volunteer dropout.

Adjudication Process and Technical Validation

The annotation of each study was evaluated for completeness and conflicting labels. If a study had an image-level label for PE, it required one or more labels for the location (eg, right, left, central) and one label for the RV/LV ratio. Conversely, if a study had a location label or a Chronic or Acute and Chronic label, it needed to have at least one image-level label for PE present. Studies labeled Negative for PE could not have image-level PE Present labels or labels indicating PE location or Chronic or Acute and Chronic labels. Conflicting RV/LV ratios for positive studies were flagged, and RV/LV ratios for negative studies were removed, as this was inconsistent with annotation guidelines. Similarly, studies labeled both Chronic (chronic only) and Acute and Chronic were flagged for review. Partially labeled studies, according to the aforementioned rules, were flagged for adjudication.

Test cases were additionally flagged if the global-level diagnostic labels did not achieve a majority. For example, flagging may have occurred for the following scenarios: no consensus reached on the presence of PE (neither Negative for PE nor PE Present), no location labels had at least two votes, or a positive for PE study did not have at least two votes for one of the RV/LV ratio labels. A total of 185 test cases and 217 training cases were adjudicated by C.C.W. and S.B.H., with each reviewing half of the flagged cases. Adjudicators had the choice to downvote existing labels and to add their own on a dedicated adjudication label group or upvote existing labels if the adjudicators agreed with an annotator’s label. The adjudicators were not blinded and had the final word on the study labels. For example, if an adjudicator read a study as Negative for PE, that superseded all other diagnostic labels for all other readers.

QA labels for motion and contrast were used to distinguish studies that were not diagnostic. Thus, any study that had QA issues such that a negative or positive diagnosis could not be ascertained was assigned the label “indeterminate,” and any other labels assigned to this study were removed. Some annotators labeled studies for QA-contrast or QA-motion while a negative or positive label was also entered. Because this was inconsistent with annotator guidelines, these QA labels were removed from the dataset.

Some studies were excluded from the final dataset. Studies that did not include sufficient lung coverage were marked with an incomplete label and were not included with the final image data. Additionally, there were some studies in the training data that had neither a positive nor negative label. If they also did not have any other labels indicating a possible positive study (eg, right, left, or central) and thus needed adjudication, these studies were considered unannotated and excluded from the final dataset. Further, a few studies that expert adjudicators could not classify were also excluded. Only one series per study was included, so additional series were excluded from the final dataset. Finally, the final dataset was evaluated to ensure only one study per patient from a site; if there were multiple studies per patient, any additional studies from the patient were excluded (n = 129). In total, 298 studies were excluded from the dataset.

Source Code

The Python scripts used to organize Digital Imaging and Communications in Medicine (DICOM) files by study and standardize DICOM metadata tag inclusion are available at https://github.com/dila-ai. The script that adjusts the section thickness of DICOM images is available at https://github.com/kitamura-felipe/thicken_dicom_volume. The source code and installer for RSNA Anonymizer are available at http://mirc.rsna.org/download/Anonymizer-installer.jar. The annotated dataset can be found at https://www.rsna.org/education/ai-resources-and-training/ai-image-challenge/rsna-pe-detection-challenge-2020.

Resulting Dataset

Dataset Overview

The dataset consists of 12 195 patients with 9385 (77.0%) in the training set and 2810 (23.0%) in the test set (Table 2). The dataset is composed of 2 995 147 images with 2 306 802 in the training and 688 345 (23.0%) in the test sets. The training set contains 96 540 images that were annotated as Positive for PE (4.2%).

Table 2:

Distribution of Dataset Labels

Open in a new tab

The dataset is provided as a collection of images in DICOM format and annotations in a comma-separated values file. Image files are stored in a single folder and named according to the value stored in the SOPInstanceUID DICOM metadata tag, which is a unique identifier for each image. The columns in the annotation file are StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID, and the annotation labels. Each row corresponds to a unique image where StudyInstanceUID (unique study identifier), SeriesInstanceUID (unique series identifier), and SOPInstanceUID (unique instance identifier) are DICOM metadata tags corresponding to the image data. If a label was annotated to the image or study, the value in the column corresponding to the label is 1; otherwise, the value is 0. Note that the only image-level label is PE Present on Image; all other labels correspond to the study. All the positive PE Present on Image labels are considered acute only unless the Chronic PE or Acute and Chronic PE study-level labels are positive.

RSNA/STR Pulmonary Embolism Detection Challenge on Kaggle

Due to size restrictions, only a subset of the data was made available for the RSNA/STR Pulmonary Embolism Detection Challenge on Kaggle (12). To decrease the overall size of the dataset for the challenge, 30% of the negative examinations in both the training and test sets were randomly chosen and excluded from the challenge dataset. In total, the challenge dataset comprised 2 322 685 images over 9446 examinations. There were 1 790 594 images over 7279 examinations in the challenge training set and 532 091 images over 2167 examinations in the challenge test set.

Discussion

We describe the curation and expert annotation of a high quality CTPA dataset from multiple international institutions with a variety of imaging equipment, protocols, radiology information system and picture archiving and communication system (PACS) configurations, and local regulations. On the basis of the experience of past RSNA data releases (8–10), it was clear from the outset that a single approach to study selection and image extraction was not feasible due to heterogeneous radiology information system and PACS configurations among the participating institutions. Instead, we opted to standardize DICOM image de-identification and metadata tag inclusion. The use of a dedicated de-identification tool followed by manual review ensured the elimination of “private” DICOM metadata tags and that “burned-in” or pixel-level patient identifiers were not present.

As in prior RSNA datasets (8–10), the amount of volunteer labor required to compile, curate, and annotate a large complex dataset of this type was substantial. The annotations were limited to PE and related findings, yet the relative size of each examination was, on average, two to three times larger than the examinations in prior RSNA datasets, requiring thousands of radiologist-hours to produce the final dataset. The size of this dataset, both in terms of the number of studies and images, as well as the narrow window of time to complete dataset annotation, ultimately limited the number of examinations that could be labeled by multiple annotators. Defining the labeling schema required a balance of effort with potential benefit toward ground truth and clinical applicability. Alternative annotation approaches were considered, including pixel-level segmentation, bounding boxes, and regions of interest. A marginal improvement in the information content of the ground truth labels would likely be realized and, given the increased workload on volunteer annotators, far fewer studies would be included in the final dataset with these more time-consuming approaches. Nonetheless, inclusion of labels that impact clinical decision making and offer value beyond the positive or negative option were included; in particular, PE disease–specific descriptive tags such as acute or chronic and RV/LV ratio and labels referring to artifacts and study quality all contribute substantial expert annotation value toward building clinically useful machine learning tools for the PE use case. The entire process of organizing this machine learning challenge took approximately 11 months (Figs 3, 4). The ideation and solicitation of datasets from sites started in early December after the 2019 RSNA annual meeting. The announcement of the Kaggle competition winners was recorded in mid-November 2020.

Workflow diagram for image data contributed by the five participating institutions. CTPA = CT pulmonary angiography, DICOM = Digital Imaging and Communications in Medicine, PACS = picture archiving and communication system, PE = pulmonary embolism, RSNA = Radiological Society of North America.

Workflow process diagram illustrating the steps involved in the creation of the dataset from initial solicitation to the final curated dataset.

As a publicly available dataset, further annotations can be performed by the global community to enhance the value of the data, including applying multiple annotations, pixel-level segmentation of pulmonary emboli, labeling the dataset for pathologic conditions other than PE, and application of a variety of automated and self-supervised machine learning approaches. In particular, as these are large feature-rich CT examinations of the chest and thoracic region in a diverse patient population, there are a variety of pathologic conditions present unrelated to PE, such as pneumonia, malignancy, and chronic lung diseases (eg, emphysema, interstitial lung disease, pulmonary edema), which are important and could be labeled to increase clinical value. As all of these CT scans were acquired prior to the coronavirus disease 2019 (COVID-19) pandemic, CT scans with pneumonia or ground-glass opacities within this dataset may serve as COVID-19–negative scans for the purposes of machine learning model development and evaluation.

There are several limitations of this dataset. While the web-based annotation tool shares much of the functionality of PACS environments, the annotation process cannot fully emulate real-world clinical practice, as annotators did not have access to patient medical information such as age, clinical presentation, radiology reports, or prior imaging studies. While this represents our most diverse medical imaging dataset assembled to date, further study will be needed toward understanding the generalization of supervised models trained with this dataset and application to new populations, protocols, or scanner manufacturers, as there are most certainly inherent qualities related to the patients and imaging examinations that could result in underlying biases and diminished performance.

In summary, the RSNA Pulmonary Embolism CT Dataset is, to our knowledge, the largest publicly available expert annotated dataset of CTPA studies. The intent of this dataset is to spur research and innovation in machine learning that will ultimately lead to improvements in the quality, efficiency, and availability of patient care worldwide. This dataset is made freely available to all researchers for noncommercial use.

Acknowledgments

The authors would like to thank and acknowledge the contributions of Chris Carr, Phil Culliton, Jamie Dulkowski, Julia Elliot, and John Perry.

Dataset Curation Contributors: Nitamar Abdala, MD, PhD, Benjamin Bearce, Henrique Carrete, Jr, MD, PhD, Hakan Dogan, MD, Shih-Cheng Huang, MS, Priscila Crivellaro, MD, Seval Dincler, Helen Kavnoudias, PhD, Robin Lee, Hui-Ming Lin, Hojjat Salehinejad, PhD, Oleksandra Samorodova, MD, Ernandez Rodrigues dos Santos, BIT, Jarrel Seah, MBBS, and Adil Zia, MSc

RSNA-STR Annotators: Veronica A. Arteaga, MD, Kiran Batra, MD, Augusto Castelli von Atzingen, MD, PhD, Anith Chacko, MBBCh, SA, Paul B. DiDomenico, MD, Ritu R. Gill, MD, Mona A. Hafez, MD, Susan John, MD, Robert L. Karl, MD, Jeffrey P. Kanne, MD, Rajesh V. Mathilakath Nair, MD, Shaunagh McDermott, DDR (RCSI), Pardeep K. Mittal, MD, Amy Mumbower, MD, Christopher Lee, MD, Paola J. Orausclio, Diana Palacio, MD, Chiara Pozzessere, MD, Prabhakar Rajiah, MD, FRCR, Oswaldo A. Ramos, MD, PhD, Sonia Rodriguez, MD, Mahmoud N. Shaaban, MSc, MBChB, Palmi N. Shah, MD, Hongju Son, MD, Sushilkumar K. Sonavane, MD, Bradley Spieler, MD, Emily Tsai, MD, Andrés Vásquez, MD, MSc, Deepti Vijayakumar, DMRD, MD, Praveen P. Wali, MBBS, DMRD, Austin Wand, MD, G. Elizabeth Zamora Endara, MD

Authors declared no funding for this work.

Disclosures of Conflicts of Interest: E.C. disclosed no relevant relationships. F.C.K. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: consultant for MD.ai; employed by DASA as Head of AI. Other relationships: disclosed no relevant relationships. S.B.H. disclosed no relevant relationships. C.C.W. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: institution received grant from NIBIB and has MIDRC grant pending; author receives royalties from Elsevier. Other relationships: disclosed no relevant relationships. M.P.L. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: board member of Nines Radiology, SegMed, and BunkerHill; author has stock/stock options in Nines Radiology, SegMed, and BunkerHill. Other relationships: disclosed no relevant relationships. L.M.P. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: editorial board member of Radiology: Artificial Intelligence. Other relationships: disclosed no relevant relationships J.K.C. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: institution has grants pending with GE and Genentech; author receives support for travel/accommodation/meeting expenses from IBM; deputy editor of Radiology: Artificial Intelligence. Other relationships: disclosed no relevant relationships. R.L.B. Activities related to the present article: author paid consulting fee or honorarium from RSNA. Activities not related to the present article: employed by Stanford University. Other relationships: disclosed no relevant relationships. G.S. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: board member of MD.ai (no compensation); consultant for MD.ai (no compensation); stock/stock options in MD.ai; editorial board member of Radiology: Artificial Intelligence. Other relationships: disclosed no relevant relationships. A.S. Activities related to the present article: employee of MD.ai. Activities not related to the present article: employed by MD.ai. Other relationships: disclosed no relevant relationships. S.S.H. disclosed no relevant relationships. E.A. disclosed no relevant relationships. M.L. disclosed no relevant relationships. P.K. disclosed no relevant relationships. K.A.M. disclosed no relevant relationships. D.C.N.R. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: speaker for Bayer de Mexico (author gave one lecture for the International Day of Radiology and received speaker training from Bayer). Other relationships: disclosed no relevant relationships. J.W.S. disclosed no relevant relationships. P. Germaine disclosed no relevant relationships. E.C.L. disclosed no relevant relationships. T.A. disclosed no relevant relationships. P. Gupta disclosed no relevant relationships. M.J. disclosed no relevant relationships. F.U.K. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: editorial board member of Radiology: Cardiothoracic Imaging; employed at UT Southwestern Medical Center. Other relationships: disclosed no relevant relationships. C.T.L. disclosed no relevant relationships. S.S. disclosed no relevant relationships. J.W.R. disclosed no relevant relationships. C.C.B. disclosed no relevant relationships. J.M. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: institution has grant from GE Healthcare; editorial board member of Radiology: Artificial Intelligence. Other relationships: disclosed no relevant relationships.

Abbreviations:

CTPA: CT pulmonary angiography
DICOM: Digital Imaging and Communications in Medicine
LV: left ventricle
PACS: picture archiving and communication system
PE: pulmonary embolism
QA: quality assurance
RSNA: Radiological Society of North America
RV: right ventricle
STR: Society of Thoracic Radiology

Contributor Information

Errol Colak, Email: Errol.Colak@unityhealth.to.

Collaborators: Nitamar Abdala, Benjamin Bearce, Henrique Carrete, Hakan Dogan, Shih-Cheng Huang, Priscila Crivellaro, Seval Dincler, Helen Kavnoudias, Robin Lee, Hui-Ming Lin, Hojjat Salehinejad, Oleksandra Samorodova, Ernandez Rodrigues dos Santos, Jarrel Seah, Adil Zia, Veronica A. Arteaga, Kiran Batra, Augusto Castelli von Atzingen, Anith Chacko, Paul B. DiDomenico, Ritu R. Gill, Mona A. Hafez, Susan John, Robert L. Karl, Jeffrey P. Kanne, Rajesh V. Mathilakath Nair, Shaunagh McDermott, Pardeep K. Mittal, Amy Mumbower, Christopher Lee, Paola J. Orausclio, Diana Palacio, Chiara Pozzessere, Prabhakar Rajiah, Oswaldo A. Ramos, Sonia Rodriguez, Mahmoud N. Shaaban, Palmi N. Shah, Hongju Son, Sushilkumar K. Sonavane, Bradley Spieler, Emily Tsai, Andrés Vásquez, Deepti Vijayakumar, Praveen P. Wali, Austin Wand, and G. Elizabeth Zamora Endara

References

1. Goldhaber SZ , Bounameaux H . Pulmonary embolism and deep vein thrombosis . Lancet 2012. ; 379 ( 9828 ): 1835 – 1846 . [DOI] [PubMed] [Google Scholar]
2. Becattini C , Agnelli G . Risk stratification and management of acute pulmonary embolism . Hematology (Am Soc Hematol Educ Program) 2016. ; 2016 ( 1 ): 404 – 412 . [DOI] [PMC free article] [PubMed] [Google Scholar]
3. Weiss CR , Scatarige JC , Diette GB , Haponik EF , Merriman B , Fishman EK . CT pulmonary angiography is the first-line imaging test for acute pulmonary embolism: a survey of US clinicians . Acad Radiol 2006. ; 13 ( 4 ): 434 – 446 . [DOI] [PubMed] [Google Scholar]
4. Lee J , Kirschner J , Pawa S , Wiener DE , Newman DH , Shah K . Computed tomography use in the adult emergency department of an academic urban hospital from 2001 to 2007 . Ann Emerg Med 2010. ; 56 ( 6 ): 591 – 596 [Published correction appears in Ann Emerg Med 2011;57(3):256.] . [DOI] [PubMed] [Google Scholar]
5. Donohoo JH , Mayo-Smith WW , Pezzullo JA , Egglin TK . Utilization patterns and diagnostic yield of 3421 consecutive multidetector row computed tomography pulmonary angiograms in a busy emergency department . J Comput Assist Tomogr 2008. ; 32 ( 3 ): 421 – 425 . [DOI] [PubMed] [Google Scholar]
6. Jha S . Value of triage by artificial intelligence . Acad Radiol 2020. ; 27 ( 1 ): 153 – 155 . [DOI] [PubMed] [Google Scholar]
7. Remy-Jardin M , Faivre JB , Kaergel R , et al. Machine learning and deep neural network applications in the thorax: pulmonary embolism, chronic thromboembolic pulmonary hypertension, aorta, and chronic obstructive pulmonary disease . J Thorac Imaging 2020. ; 35 ( Suppl 1 ): S40 – S48 . [DOI] [PubMed] [Google Scholar]
8. Halabi SS , Prevedello LM , Kalpathy-Cramer J , et al. The RSNA pediatric bone age machine learning challenge . Radiology 2019. ; 290 ( 2 ): 498 – 503 . [DOI] [PMC free article] [PubMed] [Google Scholar]
9. Shih G , Wu CC , Halabi SS , et al. Augmenting the National Institutes of Health chest radiograph dataset with expert annotations of possible pneumonia . Radiol Artif Intell 2019. ; 1 ( 1 ): e180041 . [DOI] [PMC free article] [PubMed] [Google Scholar]
10. Flanders AE , Prevedello LM , Shih G , et al. Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge . Radiol Artif Intell 2020. ; 2 ( 3 ): e190211 . [DOI] [PMC free article] [PubMed] [Google Scholar]
11. Zhou C , Chan HP , Chughtai A , et al. Variabilities in Reference Standard by Radiologists and Performance Assessment in Detection of Pulmonary Embolism in CT Pulmonary Angiography . J Digit Imaging 2019. ; 32 ( 6 ): 1089 – 1096 . [DOI] [PMC free article] [PubMed] [Google Scholar]
12. RSNA STR Pulmonary Embolism Detection . Kaggle; . https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/. Accessed September 30, 2020 . [Google Scholar]

[r1] 1. Goldhaber SZ , Bounameaux H . Pulmonary embolism and deep vein thrombosis . Lancet 2012. ; 379 ( 9828 ): 1835 – 1846 . [DOI] [PubMed] [Google Scholar]

[r2] 2. Becattini C , Agnelli G . Risk stratification and management of acute pulmonary embolism . Hematology (Am Soc Hematol Educ Program) 2016. ; 2016 ( 1 ): 404 – 412 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[r3] 3. Weiss CR , Scatarige JC , Diette GB , Haponik EF , Merriman B , Fishman EK . CT pulmonary angiography is the first-line imaging test for acute pulmonary embolism: a survey of US clinicians . Acad Radiol 2006. ; 13 ( 4 ): 434 – 446 . [DOI] [PubMed] [Google Scholar]

[r4] 4. Lee J , Kirschner J , Pawa S , Wiener DE , Newman DH , Shah K . Computed tomography use in the adult emergency department of an academic urban hospital from 2001 to 2007 . Ann Emerg Med 2010. ; 56 ( 6 ): 591 – 596 [Published correction appears in Ann Emerg Med 2011;57(3):256.] . [DOI] [PubMed] [Google Scholar]

[r5] 5. Donohoo JH , Mayo-Smith WW , Pezzullo JA , Egglin TK . Utilization patterns and diagnostic yield of 3421 consecutive multidetector row computed tomography pulmonary angiograms in a busy emergency department . J Comput Assist Tomogr 2008. ; 32 ( 3 ): 421 – 425 . [DOI] [PubMed] [Google Scholar]

[r6] 6. Jha S . Value of triage by artificial intelligence . Acad Radiol 2020. ; 27 ( 1 ): 153 – 155 . [DOI] [PubMed] [Google Scholar]

[r7] 7. Remy-Jardin M , Faivre JB , Kaergel R , et al. Machine learning and deep neural network applications in the thorax: pulmonary embolism, chronic thromboembolic pulmonary hypertension, aorta, and chronic obstructive pulmonary disease . J Thorac Imaging 2020. ; 35 ( Suppl 1 ): S40 – S48 . [DOI] [PubMed] [Google Scholar]

[r8] 8. Halabi SS , Prevedello LM , Kalpathy-Cramer J , et al. The RSNA pediatric bone age machine learning challenge . Radiology 2019. ; 290 ( 2 ): 498 – 503 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[r9] 9. Shih G , Wu CC , Halabi SS , et al. Augmenting the National Institutes of Health chest radiograph dataset with expert annotations of possible pneumonia . Radiol Artif Intell 2019. ; 1 ( 1 ): e180041 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[r10] 10. Flanders AE , Prevedello LM , Shih G , et al. Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge . Radiol Artif Intell 2020. ; 2 ( 3 ): e190211 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[r11] 11. Zhou C , Chan HP , Chughtai A , et al. Variabilities in Reference Standard by Radiologists and Performance Assessment in Detection of Pulmonary Embolism in CT Pulmonary Angiography . J Digit Imaging 2019. ; 32 ( 6 ): 1089 – 1096 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[r12] 12. RSNA STR Pulmonary Embolism Detection . Kaggle; . https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/. Accessed September 30, 2020 . [Google Scholar]

PERMALINK

The RSNA Pulmonary Embolism CT Dataset

Errol Colak, MD

Felipe C Kitamura, MD, PhD

Stephen B Hobbs, MD

Carol C Wu, MD

Matthew P Lungren, MD, MPH

Luciano M Prevedello, MD

Jayashree Kalpathy-Cramer, PhD

Robyn L Ball, PhD

George Shih, MD

Anouk Stein, PhD

Safwan S Halabi, MD

Emre Altinmakas, MD

Meng Law, MD, MBBS

Parveen Kumar, MD

Karam A Manzalawi, MD

Dennis Charles Nelson Rubio, MD

Jacob W Sechrist, MD

Pauline Germaine, DO

Eva Castro Lopez, MD

Tomas Amerio, MD

Pushpender Gupta, MD

Manoj Jain, MD

Fernando U Kay, MD

Cheng Ting Lin, MD

Saugata Sen, MD

Jonathan Wesley Revels, DO

Carola C Brussaard, MD

John Mongan, MD, PhD

Abstract

Summary

Key Points

Introduction

Materials and Methods

Dataset Curation

Annotation Process

Table 1:

Figure 1a:

Figure 2:

Figure 1b:

Figure 1c:

Figure 1d:

Figure 1e:

Figure 1f:

Figure 1g:

Figure 1h:

Figure 1i:

Adjudication Process and Technical Validation

Source Code

Resulting Dataset

Dataset Overview

Table 2:

RSNA/STR Pulmonary Embolism Detection Challenge on Kaggle

Discussion

Figure 3:

Figure 4:

Acknowledgments

Acknowledgments

Abbreviations:

Contributor Information

References

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases