Open access image repositories: high-quality data to enable machine learning research

Abstract

Originally motivated by the need for research reproducibility and data reuse, large-scale, open access information repositories have become key resources for training and testing of advanced machine learning applications in biomedical and clinical research. To be of value, such repositories must provide large, high-quality data sets, where quality is defined as minimising variance due to data collection protocols and data misrepresentations. Curation is the key to quality. We have constructed a large public access image repository, The Cancer Imaging Archive, dedicated to the promotion of open science to advance the global effort to diagnose and treat cancer. Drawing on this experience and our experience in applying machine learning techniques to the analysis of radiology and pathology image data, we will review the requirements placed on such information repositories by state-of-the-art machine learning applications and how these requirements can be met.

INTRODUCTION

Imaging data and, in particular, quantitative features extracted by image analysis have been identified as a critical source of information for classification (imaging phenotypes), cancer diagnosis and tracking response to therapy^1–4. Radiomics and pathomics, where quantitative features are algorithmically extracted from radiology and pathology imaging studies, provide valuable diagnostic and prognostic indicators of cancer ^{5–14,15, 16}. Machine learning algorithms have shown promising results in analysis of digitised tissue specimens and better performance than most traditional image analysis techniques^17–19. Deep learning based pipelines are also gaining increased acceptance and use in radiology^{14, 20, 21}. Identifying quantitative imaging phenotypes across scale through the use of machine learning is a rapidly evolving approach to improve our understanding of cancer biology ^1,2.

In 2010, the US National Cancer Institute realised that open access to radiological images and other supporting data such as demographics, outcomes, and clinical trial data were required to promote cancer research. A contract was issued to create The Cancer Imaging Archive (TCIA), which has supported open science and cancer research since it went live in 2011, by acquiring, curating, hosting, and distributing collections of multi-modal information^{22, 23}. TCIA adheres to the FAIR principle (Findable, Accessible, Interoperable, Reusable) ²⁴. To make data findable and accessible TCIA provides both a web user interface, a visual query based user interface,²⁵ and an application programming interface (API)²⁶ to identify and retrieve images and related data. TCIA also serves as a data publisher²⁷ and makes use of digital object identifiers (DOIs)²⁷ to reference data collections in the literature and enable direct retrieval of the referenced data. All data within TCIA have been fully de-identified using tools and procedures that have been fully validated to comply with US and international privacy laws²⁸. Although the number of on-line image repositories is growing rapidly (e.g., ^29–35) to the authors’ knowledge there is no data management service that can support the range of image data types, rich metadata, robust curation processes, and breadth of both human and computer access methods as TCIA.

The rapid growth of quantitative image analysis based on machine learning has extended the mission of TCIA to include providing data for training and testing of new algorithms³⁶. This has allowed us to better understand the limitations imposed on new algorithm development by existing data resources.

QUANTITATIVE IMAGE ANALYSIS BY MACHINE LEARNING

The use of computers to aid in the detection of regions of clinical interest in images and diagnosis was introduced in the 1960s³⁷. The field drew heavily on work in computer vision and became known as CAD (computer-aided diagnosis³⁸, computer-aided detection³⁹). Since the first system was approved by the FDA in 1998⁴⁰, CAD has been part of radiology with many approved systems in clinical practice^{40, 41}. CAD research generated many of the machine learning tools we use today. CAD also extended to pathology image analysis^{42, 43}. In spite of years of research and development, the number of clinically successful CAD products with US Food and Drug Agency (FDA) approval are rather limited⁴⁴.

The new emphasis in medical imaging research is radiomics^{10, 11} where features are extracted from images either by quantitative measurements of objects of interest in the image^{12, 16} or by deep learning algorithms that learn features to support classification and risk assessment^{14, 21}. Image-derived features are considered more useful for cross-scale analyses, e.g., radiogenomics^{20, 45, 46}. Machine learning is also being used to remove image artefacts and enhance image quality⁴⁴.

Machine learning algorithms, in particular deep learning methods, have shown promising results in analysis of digitised tissue specimens and better performance than most traditional image analysis techniques^17–19. Deep learning models are increasingly employed in imaging based quantitative analysis pipelines. Developed as a specific case of machine learning, deep learning algorithms teach themselves the informative representations, unlike classic machine learning approaches where human experts perform the feature engineering¹⁹. In pathology applications, the models are employed to segment tumours or identify niches in tissue such as tumour infiltrating lymphocytes¹³, or to segment epithelial versus stromal tissue regions⁴⁷. In other cases, the models are directly employed to predict patient outcome or response to treatment^{48, 49}.

Machine learning and other branches of artificial intelligence have found numerous applications in diagnostic decision support beyond radiomics and pathomics⁵⁰. These applications include patient similarity analysis⁵¹ and cognitive assistants⁵⁰.

DATA REQUIREMENTS FOR MACHINE LEARNING

Deep learning methods require large, representative and accurately annotated training datasets to train robust models and achieve acceptable performance⁴⁰. Goodfellow et al.⁵² proposed the following rule of thumb: “a supervised deep learning algorithm will generally achieve acceptable performance with around 5,000 labelled examples per category and will match or exceed human performance when trained with a dataset containing at least 10 million labelled examples.” Why are such large data sets required? For any machine learning algorithm to perform sufficiently well as to be clinically useful it must be trained on data that appropriately represent the variance in the human population, the presentation of disease (target and comorbidities), and in data collection systems themselves^{10, 40, 53–55}. Such large data requirements pose a challenge to the medical imaging research community and form the basis for the call for large scale data sharing^{10, 56–58}.

Given the massive clinical image collections available in healthcare systems around the world, acquiring large data sets would not seem to pose a problem. Although some argue that clinical data must be used so that algorithms explicitly deal with the huge inconsistency in acquisition protocols inherent in clinical data,¹⁰ it is widely held that imaging has to be of sufficient quality and acquired with uniform parameters as in a clinical trial to make certain that conclusions drawn from artificial intelligence can be validated⁵⁵. Whether the data comes from clinical practice or clinical trials, large scale data sharing is essential^{12, 40, 56} and the problem that collections do not truly represent the population, i.e., the lack of healthy controls, remains.

Supervised learning techniques, commonly used in radiomic/pathomic studies, are hampered by the lack of labelled data for training and testing⁴⁴. In general, training data must be created manually by human experts resulting in high cost and limited volume of high-quality training (and testing) datasets^{44, 56}. It is well established that there is poor agreement among human experts both for identification of objects of interest and performing analysis tasks such as segmentation^{10, 40}. Manual segmentations are also often imprecise^{10, 56}.

Reproducibility of radiomics/pathomics analyses are difficult to assess due to a lack of standards for validating results^{12, 58}. Fundamentally there is no reference standard of truth against which to evaluate algorithms^{44, 59}. One approach widely used in computer science is the establishment of open access benchmark databases to compare algorithm performance⁴³. On the other hand, the FDA has questioned if testing data should be sequestered (i.e., retained as private, proprietary information) particularly for evaluating algorithms that require FDA approval⁵⁶. Thus, intellectual property concerns limit access to valuable data sets. Several authors have discussed the problems of defining ground truth standards in medical imaging.^{60, 61} Approaches to modelling truth by combining observations from multiple observers (machine and human) attempt to create standards that do not penalise algorithms that outperform the human observer.^{62, 63}

There are some unique challenges in developing high-quality training datasets in digital pathology. First, tissue images capture much denser information than many other imaging methods. State-of-the-art digitising microscopes can scan a whole slide tissue at resolutions ranging from 30,000×30,000 pixels to over 100,000×100,000 pixels. A whole slide tissue image can contain more than a million cells, nuclei, and other cellular-level structures. The sheer number of objects in a whole slide tissue image makes it impossible to manually segment and annotate each and every object. Second, there will be heterogeneity across tissue specimens and even within a tissue specimen. A whole slide tissue specimen may have multiple types of regions: normal tissue, tumour, stroma, etc. The development of representative datasets for machine/deep learning has to take into account heterogeneity. Third, tissue specimens go through a tissue processing phase before they are imaged. Variations in tissue preparation lead to artefacts in images, such as tissue folds, poor staining, and pen markings. In addition, there will be variations in image quality and characteristics, such as sharpness and colour profiles from different digitising microscopes. Even if images are obtained by the same digitising microscope, there can be artefacts such as out-of-focus or blurred regions. The image capture performance of a digitising microscope can degrade overtime and may need to be tuned at intervals. All these factors may lead to batch effects, i.e., images of the same tissue type coming from different source sites may have significant variations in colour, sharpness, etc. Consistent and unified protocols are necessary to reduce artefacts in image datasets used to generate training data. Lastly, there are no standard image file formats in digital pathology that are universally accepted, despite ongoing efforts^64–66. Almost every vendor in the digital pathology imaging domain has its own file format; there can even be variations between the file formats of different digitising microscopes from the same vendor. Libraries have been developed that can parse certain vendor formats^{67, 68}. Tools and applications also have been developed to view and annotate pathology image data^68–71. Nevertheless, lack of a standard data format remains a major challenge in curating training datasets.

CAREFUL IMAGE CURATION PERMITS DATA AGGREGATION

Careful curation and strict quality-control processes are two key activities essential to the success of any information resource. When data adheres to a standard (e.g., Digital Imaging and Communications in Medicine [DICOM]⁷²), de-identification may be unambiguously and completely performed in compliance to a standard (i.e. DICOM Standard PS 3.15-2011, Part 15: Security and System Management Profiles) providing the ability to generate data at a publicly shareable level of compliance and on a large scale. Following industry best practices, TCIA uses a standards-based approach to de-identification of DICOM objects to ensure that all managed objects are free of protected health information (PHI). The TCIA de-identification process ensures that the HIPAA de-identification standard is met by following the Safe Harbor Method as defined in section 164.514(b)(2) of the Health Insurance Portability and Accountability Act of 1996 (HIPAA) Privacy Rule. TCIA redacts PHI while retaining as much data as possible to ensure scientific usability of the data⁷³.

TCIA and other research projects⁷⁴ use the Posda open source framework^{75, 76} to implement curation workflows for all DICOM defined objects. Posda incorporates DICOM validation rules and maintains an up-to-date DICOM private tag dictionary. These underpin DICOM validation and guide de-identification processes. Posda supports redaction of PHI, validation and correction of linkages to referenced objects, correction of DICOM series, study, and patient level inconsistencies, analysis and correction of DICOM encoding errors, provenance tracking, and prioritisation of multiple data streams.

During curation it is important to ensure that data objects are usable and properly linked to supporting data, such as DICOM structure sets properly linked to the associated imaging. It is also important to ensure the same imaging data are not represented within an archive (or across archives) under different subject IDs. TCIA has had submissions of the same data by different research groups with different IDs assigned. This duplication was discovered using digests of the pixel data and the data were harmonised.

One mechanism for creating large datasets from smaller ones is through a collection of digital object identifiers (DOIs). If all accessible datasets have been published with DOIs, one can search within resources such as DataCite or EZID, to find and retrieve large datasets from wherever they hosted. When publishing results from machine learning algorithms, DOIs should be referenced to facilitate validation of results.

Not all image data are acquired in DICOM format. Metadata play a significant role in ensuring open and easy access to the image data. Metadata enables access and sharing of image data such as multi-dimensional microscopy image data that consists of several proprietary file formats⁷⁷.

CAPTURING LABELLED DATA

In addition to collecting large volumes of image data and cross-linking compatible data from multiple collections, it is essential to collect and publicly distribute labelled data. The generation of labelled data remains a roadblock. Two approaches have shown promising results: crowdsourcing and the use of augmentation and synthetic data generation. Crowdsourcing employs large populations of experts (e.g., at national meetings) or citizen scientists to perform data labelling tasks including segmentation^{78, 79}. The Crowds Cure Cancer project⁸⁰ conducted at RSNA 2017 and 2018 uses the experts who attend this professional conference to label image data provided by TCIA. With potentially hundreds of participants labelling each data set, the variance in label estimates can be adequately modelled. Challenge competitions are another source of data labelled by consensus.³⁶. Augmentation and synthetic data generation are potential approaches to alleviate the cost of generating training and test datasets⁸¹. Nevertheless, tools and methods are needed to review and interactively refine results from a machine/deep learning algorithm applied to datasets consisting of hundreds to thousands of images⁷⁰.

FUTURE DEVELOPMENTS

Management of labelled training and test sets and the resulting image derived features raise special curation and validation issues. Currently, standards and standard operating processes for data representation, curation, evaluation, and sharing of high-quality labelled datasets are in early stages of development and leverage the curation pipelines developed for imaging data⁸². Basic processes, such as checking if all required metadata elements are included and whether the dataset can be parsed correctly, can be adopted from the image curation pipelines.

Centralised open access information repositories provide a single point for search and retrieval but are hampered by variations in international regulations and policies governing patient privacy and data sharing, and of course potentially limit access due to the need to physically move large quantities of data over the internet. Augmenting centralised repositories with distributed, loosely federated databases allows more local access and control. Such a federated environment should still provide access to data through a minimum set of ad hoc, community accepted interfaces and a minimum set of metadata elements so the data of interest can be searched, retrieved, and interpreted. Enabling open access to data is also often limited by organisational policies on data access and sharing. Research has proposed hybrid on-demand data integration from distributed data sources and service-based data sharing without actually replicating the content for biomedical research data⁸³. We posit that such approaches will complement the open access image repositories in further empowering the machine learning research.

CONCLUSION

Machine learning algorithms require access to large amounts of data for training and testing. Machine learning will require access to data from multiple cross-linked multi-modal archives containing data such as radiology, pathology, genomics, radiomics, and other metadata such as survivability data. These data must be easy to query and retrieve the correct information. If the results from machine learning are to be generalisable the data on which they are based must be of sufficient quality or acquired with uniform parameters, as in a clinical trial, and publicly available so the algorithms, models, and conclusions can be tested and validated by the research community. No reference standard of truth against which to validate new machine learning based quantitative analyses exists. Validation sets with estimates of the variance of the labels and synthetic data are the best approximations currently available. Systems such as The Cancer Imaging Archive are capable of managing these data and making them freely available to the research community.

Highlights.

• Machine learning algorithms, in particular Deep learning methods, have shown promising results in both radiology and pathology image analysis, but the number of clinically successful AI products with FDA approval is limited.

• Access to appropriate data for training, testing and evaluation is a key limitation to the field.

• Open access information repositories such as The Cancer Imaging Archive support the collection and curation of both large data sets and labeled data needed for training and testing machine learning algorithms.