Big Data Requirements for Artificial Intelligence

Abstract

Purpose of review:

To summarize how big data and artificial intelligence (AI) technologies have evolved, their current state, and next steps to enable future generations of AI for ophthalmology.

Recent findings:

Big data in healthcare is ever increasing in volume and variety, enabled by the widespread adoption of electronic health records (EHR) and standards for health data information exchange, such as Digital Imaging and Communications in Medicine (DICOM) and Fast Healthcare Interoperability Resources (FHIR). Simultaneously, the development of powerful cloud-based storage and computing architectures supports a fertile environment for big data and AI in healthcare. The high volume and velocity of imaging and structured data in ophthalmology and is one of the reasons why ophthalmology is at the forefront of AI research. Still needed are consensus labeling conventions for performing supervised learning on big data, promotion of data sharing and reuse, standards for sharing AI model architectures, and access to AI models through open application program interfaces (APIs).

Summary:

Future requirements for big data and AI include fostering reproducible science, continuing open innovation, and supporting the clinical use of AI by promoting standards for data labels, data sharing, AI model architecture sharing, and accessible code and APIs.

Introduction

“Big Data” is a term often informally used in healthcare to describe any dataset bigger than a few thousand data points. However, the definition of big data is not guided by sample size cutoffs; rather, the scale and the challenge of “big data” in the modern age is that datasets are now so large that storage or processing of the data strains the computational capacity of a single computer and require more specialized computing solutions. In healthcare, big datasets may have many rows comprising the records of millions of patients, and/or have many columns corresponding to the multiple features captured for each patient. Furthermore, besides the sheer volume of data, big data is often characterized by velocity (speed of acquisition) and by variety, which in healthcare often implies multimodal data: a combination of images, text, and structured fields.¹

The volume, velocity, and variety of big data in healthcare combined with the recent leaps in data storage and processing technologies have created a fertile environment for the development of artificial intelligence (AI) models to perform a wide variety of classification and prediction tasks. Ophthalmology in particular has many quantitative structured data fields, such as visual acuity and intraocular pressure, meshed with structural imaging and functional measurements, leading to an era where ophthalmology is at the forefront of AI in medicine.² In the present review, we summarize how big data and associated technologies have evolved to this critical juncture, describe the current frontier of standards and technologies for big data in AI, and discuss what is needed for big data to enable the next generation of AI.

Current Capabilities Enabled by Data Standards and AI Hardware Technologies

Previously, some of the biggest data in ophthalmology research came from national government-sponsored studies,³ insurance claims,⁴ large clinical trials,^5,6 cohort studies,^7,8 and individual institutional registries.⁹ However, largely scalable methods of analysis and data aggregation were not widespread. The increasing adoption of electronic health records (EHRs) since 2008,¹⁰ driven by the Meaningful Use program of the HITECH Act,¹¹ and the simultaneous development and widespread adoption of standards for health data exchange have enabled the aggregation of health data from multiple sources, creating a new rich environment for big data and AI.

Standards for health data exchange define formats and fields for the storage of health data from different health systems and sources, therefore enabling the syntactic interoperability required for machines to read data from different sources using common conventions. Digital Imaging and Communications in Medicine (DICOM), established in 1993, is the premier standard for radiology and ophthalmology imaging.¹² Standards for EHR data have undergone significant evolution through different versions of the Health Level-7 guidance and the development of the new Fast Healthcare Interoperability Resources (FHIR): the next-generation standards framework for EHR data which supports the latest web service technologies.¹³ FHIR stores data into modular components called resources designed to be simultaneously flexible, interoperable, and easy to implement. These standard data exchange formats enable data to be aggregated from different sources into ever larger datasets, which, in turn, enable the development of powerful AI predictive algorithms that can be validated across different settings, eg. predicting hospitalization mortality, length-of-stay, and readmissions across different healthcare systems using a common deep learning approach.¹⁴

Training an AI model on such large scale data requires intensive computing resources. Options now extend beyond utilizing local on-premise supercomputing hardware to utilizing cloud-based computing and storage infrastructures, many of which include HIPAA-compliant security measures enabling the storage and processing of protected health information.¹⁵ These cloud-based solutions offer the ability to house and analyze data on a large scale, enabling the ability to train powerful AI models. Once a model is trained, the process of using that model to generate predictions is called inference. Inference hardware is relatively inexpensive and becoming ubiquitous in some places: smartphones’ voice-activated assistants use inference to recognize commands, as do image search, spam filtering, and product recommendation applications. However, in hospitals and healthcare settings, this type of infrastructure needs to be optimized, yet will be needed to realize the future benefits of deploying AI models to support and enhance clinical treatment decisions.

Requirements for Reproducible Science in Big Data for AI

To maximize the potential of big data and AI in ophthalmology, there are several critical areas of need going forward. An important minimum standard for scientific research is producing reproducible results. In training AI models on big data, reproducibility is critically reliant on the following: 1) how the data used to train the AI model is labeled, 2) the underlying structure and characteristics of the data, and 3) the details of the model architecture. Each of these components are vital areas to consider when developing and evaluating AI models using big data.

Common Labels.

Supervised AI models are trained for prediction or classification tasks based on data that are labeled with the “ground truth” or “gold standard” classification; thus, there is a critical need for ophthalmologists to develop standardized definitions and classification approaches. For example, when diagnosing glaucoma or determining glaucoma-related outcomes, many glaucoma specialists rely upon visual field (VF) testing, disc morphology, or optical coherence tomography (OCT) findings to varying degrees. Therefore, for research studies, adjudication of glaucoma outcomes is often carried out by consensus of a panel of glaucoma specialists, with work continually ongoing to develop standardized research definitions.¹⁶ Similarly, for grading retinal images, some AI studies also use a panel of graders,¹⁷ similar to in clinical trials. However, utilizing a consensus or panel to grade one particular dataset does not harmonize potentially idiosyncratic definitions across different datasets.

While reference standards for diseases across different studies would be ideal, they are challenging to develop and deploy in any research setting, and especially challenging in big data for AI. Ideally, reference standards are based on clinical outcome, which can be challenging in the case of chronic diseases.¹⁸ Applying strict and complex grading criteria that require the consensus of ophthalmologic specialists is very expensive in time, effort, and dollars, even for the relatively small scale of clinical trials, and often infeasible on the scale of modern big data. Complex diagnostic criteria often require many disparate data components, such as history, examination, and imaging elements. Observational data sources inevitably include patients who are missing data for some of these elements, and those missing data are unlikely to be missing at random. For example, patients may be missing OCT’s, VF’s, or lab measurements in a pattern which is systematically related to their suspected disease.¹⁹ EHRs also introduce unique challenges, since data fields may differ across EHR systems, both for data input and storage, and providers may use different terminology even within a single EHR system (e.g., corneal SPK/superficial punctate keratitis vs. corneal PEE/punctate epithelial erosions to represent the same clinical finding). These variations often necessitate custom disease definitions for each study based on the availability and content of the data fields. Thus, there is also a need to standardize how definitions are developed to label big data and the criteria for judging whether the definitions are acceptable for big data that are used to train AI algorithms. These criteria are likely to differ from those used to judge clinical trials because they must balance feasibility with quality and consensus. Successful scaling of big data AI will require leadership to (a) standardize, where possible, the consensus definitions for disease diagnoses so they can be used to generate high quality, “ground truth” labels, and (b) standardize how consensus definitions are developed for big data studies.

To sidestep the issue of using complex “ground truth” disease definitions as labels, some studies use different types of proxy or surrogate outcomes which are more readily available in the dataset. Examples include using one type of imaging modality to predict the parameters of another modality: predicting various retinal nerve fiber layer OCT parameters from fundus photos^20,21 and using OCT to predict other types of imaging outcomes,^22,23 or predicting future results from past results, as when predicting future visual fields from prior visual fields, or other biomarkers validated against true outcome.^24–26 Prediction labels may also be a proxy based on cost or utilization, as when predicting high-cost or high-utilization patients, which is convenient when many sources of big data in healthcare include only such administrative data and not clinical outcomes. However, caution must be exercised before deploying such algorithms for decision support, as there may be underlying societal biases that affect healthcare access, utilization, and cost.²⁷ This speaks to the larger concern that predicting health outcomes requires a careful understanding of those outcomes in the context of the system being studied and recognition that predicting health outcomes in one system may not generalize to other systems.

Common Data.

Data sharing is an important mechanism to ensure that, after the expensive process of data curation and producing high-quality labels is completed, results from AI models can be reliably reproduced and the data can be used by other researchers, therefore having the widest possible impact by vastly multiplying the number of studies that can be performed. The FAIR guiding principles for data stewardship and management provide a framework for ensuring that data are Findable, Accessible, Interoperable, and Reusable.²⁸

Although there has been progress in development of data interoperability standards, many challenges remain in the actual process of data sharing and data reuse. Big data today is often observational, raw, and uncurated or minimally curated. Using data exchange standards does not guarantee semantic interoperability: data from different sources may be stored in the same format, but its meaning may differ depending on the context of how and from whom it was collected. Discrepancies may exist in data formatting and/or units of measure. In ophthalmology, for example, intraocular pressure may be collected by a wide variety of methods, by different types of providers, and under different conditions, all of which may affect the recorded data. Visual acuity is even more widely variable, with the possibility to measure using different methods, under different lighting conditions, with different types of correction, and with varying notation (“20/400” vs “400”). For data to be reusable, methods for data cleaning, harmonization, and validation should be standardized and fully transparent. Careful annotation with metadata describing the provenance of different data fields remains an important component of data sharing. A reasonable balance must be struck between validating the underlying data, which is expected to contain some errors, and manually validating the entire dataset, which is usually infeasible and would defeat the advantages of data analysis at scale.

Finally, data sharing is subject to privacy concerns. While imaging may be more easily de-identifiable and shareable, EHR data are relatively difficult to de-identify²⁹ and suffer from the risk of possible re-identification.³⁰ Furthermore, storing data in HIPAA-compliant environments requires thoughtful management, including safeguards against data unmasking as investigators potentially use different platforms for data joining or extract and snapshot data from one system to another. Despite the challenges in data sharing, it is absolutely crucial to support and incentivize mechanisms for secure data sharing in order to enable continued innovation in big data and AI for ophthalmology in the academic sector. If the largest and highest-quality datasets become proprietary or exclusive, it will become exceedingly challenging to support continued innovation outside of those groups. In the ideal world, all those with access to clinical data could become data stewards, ensuring the safety of the data and the use of the data for the public good, enabling the development of knowledge and tools that benefit future patients as widely as possible.³¹

Common Code and Application Programming Interfaces (APIs).

When data cannot be shared due to privacy concerns, the code that specifies the AI models and the innumerable underlying data pre-processing decisions should be made public in order to best enable reproducible research. There are several different levels of sharing of AI models, enabling different degrees of reproduction. A bare minimum requires sharing the model architecture as code, without the trained weights or parameters. The next level includes sharing the model architecture and the training code itself, so that other researchers can train their own weights using different data. Sharing the training code would include sharing the many hidden parameters that can affect model performance,³² such as how the data are split into batches, initialization of terms, and rates of learning, thus enabling other researchers to more closely mimic the original research.The highest level of sharing, short of including the full original data, would include the model architecture, training code, and the trained weights with inference code, which would allow other researchers to plug in their own data to make predictions based on the original AI model. While standards for how to share AI models are nascent, Open Neural Network Exchange (ONNX), a community project that specifies an open format to represent machine learning and deep learning models,³³ provides one approach to model-sharing. Beyond enabling reproducible research, code that is made publicly available can have additional uses in contributing to research endeavors beyond its original purpose, becoming a resource for further collaborative innovation.

Many complex AI models also depend upon specific underlying hardware configurations to execute. Standards for application program interfaces (APIs) to access AI models would be enormously helpful in enabling researchers at different sites to access common AI models while retaining full control over their own data. Standardization of AI APIs would enable validation of AI models across different sites and allow the training of better models based on more diverse data. Realizing the promise of AI models also requires addressing the challenges of implementing and integrating the use of AI models into clinical workflow. Here, too, standards for interfacing with APIs are critically needed in order for locally generated data to be able to interface with AI models. APIs for access to AI models could and should take advantage of the standardized data formats now available (such as FHIR or DICOM) in specifying the format of data inputs for widest possible use. For example, the Substitutable Medical Applications, Reusable Technologies (SMART) on FHIR API is an open, free, and standards-based API that aims to enable the creation of an ecosystem of apps built on access to EHR data.^34,35 While most SMART on FHIR apps are not specifically in the AI space, AI-based applications can use a similar approach for access to their AI models. As with data-sharing, the challenge in model-sharing is to continue to support academic innovation by promoting openness, interoperability, and standardized APIs, thereby accruing the widest possible benefits of AI and preventing AI models from becoming “locked down” into separate inaccessible silos as we have seen with EHRs. In ophthalmology, in particular, efficient real-time inference workflows are important as the time between image acquisition and need for AI inference results are comparatively short compared to other fields of medicine such as radiology or pathology where interpretation traditionally happens asynchronously.

Conclusion

The environment for big data and AI is evolving rapidly, supported by the adoption of electronic health records, standards for health data exchange, and modern technologies for storing and processing large datasets. Future requirements for big data and AI include fostering reproducible science, continuing open innovation, and supporting the clinical use of AI. Researchers, academic institutions, healthcare companies, and regulatory agencies should work together to achieve these goals by promoting data label standards, data-sharing, standards for sharing AI model architecture, and accessible code and APIs.

KEY POINTS.

• Big data is often characterized by volume (size), velocity (speed of acquisition) and variety, which, in healthcare, often implies data which are multimodal: a combination of images, text, and structured fields.

• Increasing use of electronic health records (EHRs) and the simultaneous development and widespread adoption of standards for health data exchange, such as Digital Imaging and Communications in Medicine (DICOM) for imaging and Fast Healthcare Interoperability Resources (FHIR) for EHR, have enabled the aggregation of health data from multiple sources, creating a rich new environment for big data and AI.

• Conducting reproducible research in big data for AI is critically reliant on 1) the labels on which AI models are trained, 2) the underlying structure and characteristics of the data, and 3) the details of the AI model architecture.

• Consensus is needed on definitions for labeling data, standards for sharing and reusing data, the sharing of code specifying AI models, and adoption of open application program interfaces to AI models.

• To foster reproducible science, common data labels, data sharing, and openness in AI models are also critically important to promote the widest possible benefits from AI research.