Sharing Data Is Essential for the Future of AI in Medical Imaging

Abstract

If we want artificial intelligence to succeed in radiology, we must share data and learn how to share data.

Summary

If we want artificial intelligence to succeed in radiology, we must share data and learn how to share data.

Key Points

• ■ Data are a key component of AI model development. To succeed, we must learn to share data.

• ■ Sharing datasets is increasingly being required by institutions that provide publicly funded research. Let’s do it right.

• ■ Open science is science, and it’s in all of our best interest for accelerating patient care to embrace it.

In January, the U.S. federal government declared 2023 to be a “Year of Open Science” and set new requirements and guidelines from the National Institutes of Health on data sharing (1). Although we have a few exemplary open datasets in our radiology artificial intelligence (AI) community, we can and must do better as physicians and scientists for the benefit of our patients. Sharing data, increasingly important for open science, promotes nonmaleficence (a value of the Hippocratic oath) by reducing duplicate imaging, using data more efficiently, and ensuring clear communication to disseminate research and advance medical care. For example, a dataset shared online by one research group as part of a specific study can spark ideas in other researchers and stimulate new research directions. By embracing open science, we have the opportunity to stimulate innovative research while being time-conscious (think of the time it takes to collect and curate data and pretrain AI models!) during a time of reduced public funding; to recover public trust in science and medicine by improving reproducibility (2); and to support diverse and inclusive research environments.

Public sharing of imaging datasets is a crucial element of open science, which most notably has been associated with open-source code. Open science broadly supports the transparent and collaborative sharing of code, data, methodology, and education—tenets that allow for reproducibility within our research. The concept of open science is not new; scientists in the 17th century recognized the need to communicate results efficiently, and journals and conferences were conceived. However, in modern research where science and information technology meet, the tools for how we conduct research and communicate it have changed drastically. Consider, for instance, the recent COVID-19 pandemic, during which the AI community had a chance to demonstrate its value to the public by building predictive models based on information contained within chest radiographs and CT images. By the conclusion of the pandemic’s initial year, it was reported that out of the hundreds of AI models published, none were deemed suitable for clinical translation (3). From these initial publications and reports, data sharing was applauded, and large databases were built (4). Nevertheless, it was clear that we still had to learn how to share quality datasets and ensure close collaboration with radiologists and imaging scientists (5,6). If we want AI to succeed in radiology, we must share data and learn how to share data.

Sharing medical imaging datasets is an ongoing global effort; numerous datasets and repositories have been around for the last decade or so. For example, a major resource is The Cancer Imaging Archive, an expansive repository housing cancer-related medical imaging datasets with corresponding clinical data that aims at supporting the reproducibility of quantitative imaging metrics (7,8). These various datasets have allowed researchers to build and share postprocessing algorithms for various cancer applications. Another important resource is the UK Biobank Imaging Study (9), which provides an extensive collection of imaging data, paired with comprehensive clinical and genetic information. Furthermore, initiatives such as the Alzheimer’s Disease Neuroimaging Initiative (10), Open Access Series of Imaging Studies (11), and the Human Connectome Project (12) offer vast datasets that contribute substantially to neuroscience research and foster a deeper understanding of brain function. Stanford Medicine’s Artificial Intelligence in Medicine and Imaging datasets platform (13) offers a diverse range of medical imaging datasets. In MRI, datasets such as fastMRI (14), mridata.org (15), SKM-TEA (16), M4Raw (17), and others (18) offer raw, nonreconstructed MRI data (k-space measurements), hence enabling development of advanced MRI reconstruction techniques aimed at reducing scan times. Other databases are more focused on certain diseases. The Multimodal Brain Tumor Image Segmentation Benchmark (BraTS) dataset, for example, which is published and continuously updated as part of the BraTS challenges, is dedicated to automated segmentation of brain glioma (19–21). Additionally, the Open Source Imaging Consortium dataset focuses on pulmonary fibrosis (22), and the Mammographic Image Analysis Society dataset (23) and Digital Database for Screening Mammography (24) offer mammographic images for breast cancer detection.

The recent rise of large language models (LLMs) has drawn interest to datasets that contain medical text. LLMs are advanced AI systems capable of processing and generating humanlike text, which can be immensely valuable in radiology (eg, in assisting radiologists in generating reports). Recently developed models include RadBERT (25) and Radiology-GPT (26); additionally, databases such as MIMIC-CXR (27) that contain chest radiographs paired with free-text radiology reports hold much promise for development of further LLMs. In summary, publicly available databases are leveraged for research and development, such as through community challenges; these databases promote the development of reproducible code for clinical translation (eg, segmentation tools and diagnostic models for disease prediction). Nevertheless, open-access datasets are still needed for different applications (eg, dynamic MRI), and in many cases, articles are published without sharing data and code.

The common reluctance to share datasets at the time of publication is associated with the substantial challenges related to curation and publication of open-access databases. These challenges come in different forms. First is the issue of patient privacy, which is closely related to a patient’s well-being and confidentiality—core principles of ethical practice in medicine. Protecting privacy requires applying data anonymization steps, including removal of patient-specific metadata often stored in Digital Imaging and Communications in Medicine images and skull-stripping from three-dimensional brain scans where face recognition may be a concern. Furthermore, the risk of identification necessitates comprehensive informed consent from individuals, which further complicates database curation. Moreover, with the intention of patient privacy, certain legislative policies, such as the General Data Protection Regulation in Europe and the California Consumer Privacy Act in the United States, may potentially complicate data sharing (28). Second, acquiring sufficient data for training models poses a formidable challenge, particularly in modalities such as MRI, where costs can be prohibitively high, and CT, where radiation exposure is a concern. Third, the process of data annotation, a crucial step for training AI models, necessitates expertise in radiology. However, imaging experts have limited time to generate reference standard annotations and their time can be very expensive. An additional barrier stems from the lack of standardized data formats when it comes to raw MRI data (specifically k-space), with disparate vendors like GE, Siemens, and Philips adopting their own data saving conventions. Such variation not only complicates data sharing but also poses challenges for seamless integration and analysis. For instance, the use of various formats like Digital Imaging and Communications in Medicine, NIfTI, and raw MRI data hinders cross-vendor compatibility and requires extensive preprocessing, consuming valuable time and resources in the medical imaging workflow. Last (and anecdotally), we acknowledge that institutional policies may create barriers for various reasons when it comes to sharing datasets. A potential solution is agreeing on the appropriate license that clearly outlines rights and permissions associated when publishing a dataset. We encourage researchers to advocate for change within their institutions as we have done from personal experience.

Due to the many challenges in data curation, researchers often face scarcity or complete lack of data suitable for training AI models, which causes additional challenges and sometimes methodologic failures. For example, a database curated for one purpose might be used for a different purpose. A dataset published for brain tumor segmentation, for example, could be used mistakenly to develop MRI reconstruction algorithms. A recent study demonstrated that such “off-label” data use could lead to biased results and overly optimistic algorithm evaluation (29) due to subtle, task-specific preprocessing steps that are applied to different datasets. Although preprocessing often cannot be detected by the human eye, it can alter the performance of the trained AI algorithms and lead to biased results. Moreover, such workflows could lead to algorithmic failure in clinical settings, as algorithms developed using preprocessed data have very limited applicability to the clinical real world and might miss clinically important details (29).

Additionally, a lack of data may also lead to the improper construction of training and test datasets which can create a different source of bias. For example, AI algorithms trained on data from healthy individuals or those who have a limited range of conditions may not generalize well to a test dataset that includes a larger range of conditions, as typically seen in a hospital (6). Furthermore, studies have shown that AI algorithms are sensitive to sex and racial bias and could perform poorly in underserved populations, such as Black individuals or Hispanic women (30,31). Another source of bias is highly specific data; AI methods trained on data acquired using a medical imaging device from a single vendor or anatomic region might not generalize to data of other vendors or organ systems (32). Finally, another problem that arises from lack of training data is the shift of research efforts toward research problems where data are abundant. For example, the scarcity of open-access datasets in abdominal medical imaging poses a major obstacle to the advancement of algorithms for diagnosing inflammatory bowel disease and hinders the development of AI techniques for accelerated, motion-robust dynamic body imaging (33).

These substantial challenges necessitate the development of novel approaches to facilitate data sharing and open science. At present, the medical imaging community is dedicating efforts along several lines. First, centralized platforms have been developed that enable online storage of open-access data. A notable example is the Medical Imaging and Data Resource Center (MIDRC), established by the National Institutes of Health; MIDRC serves as a centralized platform for COVID-19–related medical imaging data, facilitating in-depth analysis of virus impact. Additionally, platforms like Zenodo, a well-known open-access repository, also contribute to the accessibility and sharing of medical imaging datasets, fostering collaboration and research endeavors across the field. Zenodo assigns a unique digital object identifier to each repository. This ensures a persistent and citable link, enhancing the discoverability and long-term accessibility of research datasets, and facilitates proper attribution in scholarly publications. Second, community-driven efforts focus on establishing standard data formats. For example, the International Society of Magnetic Resonance in Medicine (ISMRM) community has established a unified data format, known as ISMRM Raw Data format, which enables storing raw MRI data acquired with scanners of different vendors and facilitates easy sharing of data and code (34). Additionally, scientists from University Medical Center Utrecht recently published a blueprint describing a vendor-neutral AI platform that they developed and implemented that aims to overcome some of the barriers related to clinical deployment of AI methods (35).

Substantial efforts are also dedicated to promoting a culture of open code. GitHub, a platform for version control and collaborative software development, has emerged as a leading venue for this aim. Community-based toolboxes that have been shared on Github, such as BART (36), Gadgetron (37), and PulseSeq (38), have garnered extensive and widespread use in the MRI field. Additionally, initiatives such as Papers with Code (https://paperswithcode.com/) have also become popular for code sharing. Furthermore, competitions have emerged as major catalysts for promoting reproducible research and open code practices across various domains. For instance, challenges such as BraTS (19–21), which is organized as part of the Medical Image Computing and Computer-Assisted Intervention conference, and Kaggle competitions have spurred collaborative efforts in medical imaging tasks. Moreover, the fastMRI challenge (39) has pushed the boundaries of MRI reconstruction algorithms, encouraging the development of innovative solutions for faster and more accurate medical image reconstruction. Altogether, these platforms and initiatives promote transparency, collaboration, and accessibility, enabling researchers to share their work, facilitate peer review, and enhance the credibility and replicability of scientific findings.

In addition to the aforementioned initiatives, journals have a pivotal role in endorsing and advancing open science. This journal, Radiology: Artificial Intelligence, provides an opportunity for authors to submit a Data Resources manuscript. This manuscript category allows authors to publish a summary of a publicly available dataset intended specifically for AI research, along with appropriate links. The publication of a Data Resources manuscript allows for a centralized location to clearly describe the dataset and provide all potential corresponding links, such as the publicly available dataset link, GitHub code repository, and/or previous publications, that can easily be cited by the authors and potential data users. Furthermore, this journal’s Data Resources Collection page (https://pubs.rsna.org/page/ai/data_resources) provides a full list of all published Data Resources manuscripts to date and will be continually updated as new such manuscripts are published, further facilitating access and reference to these publicly available datasets. We understand that the perceived notion that there is more to lose when sharing data, but opportunities to submit publications such as Data Resources manuscripts offer a way to give credit to the researchers and the patients—and we believe this outweighs the concerns.

We understand the hesitancy to share data and code openly given the personal apprehension of criticism. However, from our experience, we found that sharing data and code has many benefits, such as facilitating collaborations and follow-up studies. We encourage you to start small. Choose one goal to focus on, whether it is actively participating in an open science initiative within your society (40), committing to publishing open code, reading guidelines for how to prepare medical imaging data for publication (41), releasing an open dataset from a previously published article, or allocating funds in your next grant to support open-science practices. Science has always been an iterative process, and in this dynamic field of AI for medical imaging, we must learn to embrace open science to accelerate the translation of our tools into clinical practice.

Keywords: Open Science, Data Sharing