Unmasking biases and navigating pitfalls in the ophthalmic artificial intelligence lifecycle: A narrative review

Abstract

Over the past 2 decades, exponential growth in data availability, computational power, and newly available modeling techniques has led to an expansion in interest, investment, and research in Artificial Intelligence (AI) applications. Ophthalmology is one of many fields that seek to benefit from AI given the advent of telemedicine screening programs and the use of ancillary imaging. However, before AI can be widely deployed, further work must be done to avoid the pitfalls within the AI lifecycle. This review article breaks down the AI lifecycle into seven steps—data collection; defining the model task; data preprocessing and labeling; model development; model evaluation and validation; deployment; and finally, post-deployment evaluation, monitoring, and system recalibration—and delves into the risks for harm at each step and strategies for mitigating them.

Author summary

In recent years, the surge in data availability, computational power, and AI techniques has sparked interest in using AI in fields like ophthalmology. However, before widespread AI deployment can happen, we must carefully navigate its lifecycle, comprising 7 key steps: data collection, task definition, data preparation, model development, evaluation, deployment, and post-deployment monitoring. This review article stands out by identifying potential pitfalls at each stage and offering actionable strategies to address them. Our article serves as a guide for harnessing AI effectively and safely in ophthalmology and related fields.

Introduction

The use of computers simulating humans in performing cognitive tasks was first described in 1943 [1], but the term “Artificial Intelligence” (AI) would not be coined until 1956. At that time, the fundamental premise was that “every aspect of learning or any other feature of intelligence can, in principle, be so precisely described that a machine can be made to simulate it” [2]. Over the last 2 decades, the exponential growth of data availability and computational power has led to an expansion in interest, investment, and research in AI applications [3].

Machine Learning (ML) is a subfield of AI that focuses on “teaching” computers how to make predictions, classify or optimize some function from data sets. While traditional ML requires a fair amount of work with feature engineering [4], in Deep Learning (DL) computers utilize multiple processing layers and multiple levels of abstraction to produce predictions, classifications, or optimal policies [5,6]. DL models have been used in several healthcare processes, including electronic medical records analysis, predicting patient outcomes, and image analysis [7].

In ophthalmology, the advent of telemedicine screening programs and the widespread use of ancillary imaging examinations have created large volumes of ophthalmic data, enabling the development of AI algorithms [8]. The ophthalmic community has since witnessed the application of AI to a wide range of diagnostic examinations with specialist-level performance [9–12]. Previous studies report the ability of AI-based algorithms to detect diabetic retinopathy [13–15], diabetic macular edema [16], retinopathy of prematurity [17,18], age-related macular degeneration [19–22], glaucoma [23–26], and uveitis [27], among others.

However, biased AI models can perpetuate existing health disparities. For example, algorithms designed to predict COVID-19 outcomes have been found to harbor implicit biases stemming from their reliance on non-representative data sets [28]. Similarly, facial recognition systems employed in criminal detection have demonstrated poorer accuracy rates when identifying females and individuals of black descent [29,30]. Additionally, language models have been shown to inadvertently perpetuate racism and stereotypes through their word choices [31,32]. Furthermore, commercial diabetic retinopathy screening algorithms have exhibited notable discrepancies in performance when applied across different populations within the same country, which can lead to delayed diagnosis and treatment in groups not represented in the training data set [33].

One huge concern is the unintentional encryption and propagation of biases at every stage of the AI lifecycle from data collection to post-deployment and recalibration [34]. In addition to data and algorithmic bias, other issues in the life cycle include reliance on irrelevant evaluation metrics, and a lack of post-deployment studies [28,35,36].

Biased predictions in artificial intelligence are a growing concern; however, the challenge is multifaceted. Bias can arise from every step of the AI lifecycle. This manuscript is the first to explore the biases that can arise in ophthalmological AI systems from data collection to deployment. Identifying pitfalls at every step of the AI pipeline is fundamental to breaking the cycle of health inequities.

The AI lifecycle in ophthalmology

Previous articles have explored the biases and pitfalls inherent in the AI development lifecycle, particularly in domains such as medical imaging [37–39]. However, this is the first to explore the ophthalmology specialty.

Ophthalmology relies on multiple imaging modalities to furnish supplementary data for clinical decision-making. Within the field, AI holds promise across various domains. For example, in diabetic retinopathy screening, AI-enabled analysis of retinal fundus photos can effectively discern patients who require specialist evaluation referrals, with deployed systems that provide autonomous recommendations based on retinal fundus analysis [40]. Moreover, in diagnosing and monitoring age-related macular degeneration, the utilization of retinal fundus photos, fundus autofluorescence, and optical coherence tomography are fundamental. Similarly, in the context of glaucoma, integrating fundus photos, visual field exams, and optical coherence tomography facilitates efficient screening and patient follow-up.

In this narrative review, we divided the ophthalmic AI lifecycle into 7 steps consisting of data collection; defining the model task; data preprocessing and labeling; model development; model evaluation and validation; deployment; and finally, post-deployment evaluation, monitoring, and system recalibration. In every step, we identify pitfalls and challenges (Fig 1).

Fig 1. The 7 steps of the Ophthalmologic AI lifecycle.

Diagram designed using icons from Flaticon.com.

Data collection

Clinical and imaging data needs to be collected to train AI algorithms. During this process, it is important to understand and document the population characteristics, data modalities, and included features and variables. In ophthalmology, images are collected during clinical practice and in screening programs (Fig 2).

Fig 2. Ophthalmological data modalities, subtypes, and diagnostic aims.

Assembling new data sets is a laborious and expensive process. As a result, the ML community has mostly relied on publicly available data sets [41,42]. When employing these data sets, it is necessary to consider the data generation process, including but not limited to the demographic diversity of the patients, the examination criteria, and the disease distribution within the cohort. It is vital that the target population is adequately represented in the data during the development of ophthalmological models that detect ocular abnormalities in real-world settings [9].

Pitfalls

Unbalanced data sources

Representation is an important attribute to consider when assessing data to train algorithms. The use of overly restrictive inclusion criteria can result in the exclusion of several subgroups, while applying too broad criteria can lead to an increased number of “false positive” patients within a cohort [43]. Moreover, underrepresented populations traditionally are not included in data sets, and even in perfectly representative data, historical biases can lead to misrepresentation [44].

To date, ophthalmic data sets used in AI research are significantly imbalanced with no retinal imaging data sets from middle-low or low-income countries [42]. This imbalance has significant implications for the development of generalizable models.

In ophthalmology, including only images from screening programs and patients from a prespecified population, demographic, or health system will introduce bias and lead to a disparity in ML performance across subpopulations [45]. While it may be impossible to fully represent all subpopulations, efforts to understand the social patterning of the data generation process and engage with marginalized populations to develop strategies for inclusive data collection and analysis are required. At the minimum, awareness of the limitations of data sets should be reported.

When faced with imbalanced data sets that cannot be improved with more data, machine learning techniques offer strategies to mitigate biases. One approach involves adjusting class weights in the loss function, giving more weight to underrepresented classes and vice versa for the majority class. Oversampling and undersampling techniques can also be employed to balance the data set, although they come with their own set of challenges. Specifically, oversampling, which creates (equal or similar) copies of the minority class data points, can make the model more prone to overfit and computationally expensive to train; on the other hand, undersampling, whereby samples from the majority class are discarded, can result in a model less robust that does not leverage the data set to its fullest. When applying these techniques, it is fundamental to be cognizant of their limitations [46].

Image acquisition, protocols, and image quality

The image acquisition process involves the implementation of an imaging protocol, which can vary depending on the specific disease being investigated. In the case of diabetic retinopathy screening, it initially entailed capturing seven 30-degree fundus photos per eye, but evolved to either two 45-degree photos or a single image, depending on the screening program [47].

Image quality standards for data sets were established by experts from a handful of high-income countries, with little, if any, contribution from low- and middle-income countries. Such data standards may discourage creators of small or “low-quality” data sets from contributing to data repositories. Additionally, the same rich countries have the infrastructure and resources to create, curate, and maintain data warehouses and AI pipelines. Setting high standards for image quality in ophthalmological examinations may lead to better model performance but trades off real-world applicability [48]. It is important to ensure that initiatives to establish dataset standards involve more inclusive teams that consider the realities in most countries.

Spectrum bias

Spectrum bias occurs when the diseases studied within the data set are not fully representative of the spectrum of disease severity in the target population, leading to poor model performance [49]. Ophthalmological data sets skewed toward mild cases of diabetic retinopathy may lead to algorithms that are better at detecting mild cases while missing severe cases and vice versa [42,49].

Therefore, it is crucial to consider the model’s performance across the full spectrum of the disease. The AI model should be trained and validated on diverse patient data, including different ages, ethnicities, disease stages, and comorbidities, to ensure applicability and reliability in ophthalmology.

Data standardization

A standardized data format is required across different systems to enable AI and should occur during the initial development phase [50,51]. Electronic healthcare record platforms are often fragmented and have limited interoperability [51]. In ophthalmology, data standardization and interoperability are challenging and require integrating multimodal data, including imaging, tabular, and clinical notes.

Defining the model task

In ophthalmology, understanding the target populations’ clinical context, examination modalities, and disease prevalence and distribution is essential. Moreover, a comprehensive understanding of the socioeconomic context, the healthcare delivery system, and its resources are important for identifying ocular conditions to target and suitable tasks for training algorithms [52].

Pitfalls

Defining the appropriate disease target

The prevalence of ocular diseases varies significantly across geography and is shaped by several demographic and socioeconomic factors. Defining the disease target must be tailored according to the specific population and the healthcare context. While AI has seen applications in various ophthalmological diseases, diabetic retinopathy screening has been the most extensively explored to date [53]. While diabetic retinopathy is a global epidemic [2], there are many other causes of preventable blindness such as unaddressed refractive errors, affecting 914 million people worldwide; cataracts, affecting 94 million individuals; age-related macular degeneration, affecting 8 million; and glaucoma, affecting 7.7 million people [54]. Several other critical causes of blindness, such as keratoconus, infectious keratitis, and deficiency neuropathies, remain uncharted territories for AI technologies. These conditions are especially prevalent in economically disadvantaged regions where access to diagnostic laboratory services is limited [6]. Resources must be allocated based on the specific burden of ocular diseases in different regions.

Data sparsity bias

Data sparsity bias can manifest when a data set lacks sufficient case numbers for specific combinations of exposure and outcomes [55]. This limitation often occurs despite the use of large databases, resulting in a restricted sample of cases, risk factors, variables, and outcomes.

Traditionally, 2 approaches are employed to mitigate this: adjustment and penalization. While these methods may be effective, a deep understanding of the clinical context and underlying social determinants is necessary [55]. Failing to incorporate this information in the models will exacerbate existing biases. Researchers should actively seek to enhance data collection efforts, particularly in underrepresented or vulnerable populations.

Data preprocessing and labeling

Data preprocessing is an important step in building AI algorithms. Computer vision models, especially with traditional image analysis techniques, require preprocessing, consisting of image enhancement, filtering, and normalization. Additionally, data augmentation can enhance the size and quality of training data sets. This improves the robustness of models, particularly when dealing with limited data.

In ophthalmology, the preprocessing stage is even more critical, given the number of distinct capturing processes and retinal fundus photography devices. Ensuring accurate and reliable results necessitates accounting for the inherent technical differences among images.

Pitfalls

Bias in the handling of missing data

Missing data are due to many reasons, including human and machine limitations, errors during data collection, limited accessibility to clinical screening and examinations, and respondents’ refusal to participate in research [56]. In ophthalmology data sets, missing information on comorbidities, demographics, and/or ophthalmological examination data is common [42]. Several methods exist for dealing with missingness, such as deleting instances with missing values or imputing the missing data using estimated values. However, the choice of imputation method should be made carefully, as different methods may introduce additional layers of bias.

Handling missing data with data imputation requires careful consideration of the context in which it is applied. For example, when dealing with a data set that disproportionately represents a certain group, such as the male population, employing the average weight or height from the entire data set to fill in missing values for the female population can lead to skewed results.

A similar issue arises in other domains, like ophthalmological imagery, where the use of generative AI techniques to augment data sets can inadvertently perpetuate biases if the training data is not representative of all groups. To mitigate such biases, employing stratified data imputation methods—where missing values are imputed based on the subgroup they belong to (e.g., using male data to impute missing values for males)—can be a more accurate and equitable approach. This strategy helps ensure that the imputed values are more representative of the specific subgroups, thereby reducing the introduction of bias during the imputation process.

To enhance the quality of data sets during development and reduce potential biases, it is essential to engage a diverse team of ophthalmologists, statisticians, data scientists, and even patient representatives as multi-stakeholder discussions will lead to more informed decisions [56,57].

Labeling

In supervised machine learning, accurate and reliable labels are necessary for training algorithms to predict diagnoses and outcomes. The labeling process in ophthalmology is complex and challenging, involving varying application of grading criteria and standards. Errors and inconsistencies in the labeling process can lead to biases in the models, ultimately impacting the performance and reliability of algorithms [58].

To address this issue, alternatives such as graders’ consensus and expert adjudication can be employed to enhance the reliability of ophthalmological labeling [58]. The use of weakly supervised learning techniques offers a promising approach to improve the annotation process, making labeling more efficient and effective [59,60].

Model development

The model development phase consists of data curation and representation, and knowledge creation and validation. Avoiding biases during modeling is an arduous task [61,62] and requires an in-depth understanding of data disparities, clinical confounders, and the socioeconomic context [62].

Pitfalls

Flawed feature engineering or selection

The relevance of features when predicting or optimizing outcomes may vary across patient subgroups. When dealing with tabular data or traditional imaging techniques that require feature engineering and selection, it is essential to acknowledge the risk of both under- and overreliance on certain features. Not all features that can be extracted from EHRs or imaging exams are relevant and their inclusion in modeling may worsen task performance. The input of domain experts in this complex process is paramount.

Diagnostic suspicion bias

The knowledge of a patient’s prior exposure or preexisting conditions can introduce diagnostic suspicion bias in data sets [63,64]. For example, when a patient has multiple known comorbidities, it may lead to a more extensive work-up, affecting the data collected for model development. Careful consideration is necessary when incorporating variables related to the outcome, such as the frequency of optical coherence tomography (OCT) and retinal fundus photos, as it could inadvertently bias the model towards specific diseases and risk factors.

Data leakage

Data leakage occurs when information from the training data is also captured in the validation cohort, or information about the outcome is reflected in the features, resulting in a falsely accurate model performance [65]. Notable examples from previously published models include using an antibiotic prescription to predict a sepsis diagnosis, or the blessing of a hospital chaplain to predict mortality. However, data leakage may be subtle and more difficult to detect [66].

In the context of ophthalmology, data leakage can occur when 3D volumetric OCT images are split on a per-image basis [65]. In this scenario, valuable spatial information from the volumetric data may inadvertently influence the model’s predictions in ways that compromise the model’s true ability to generalize to new data.

Shortcuts

Model shortcuts pertain to features that are learned during model development that are not clinically related to a prediction or classification, such as the hospital where a patient is seen or the medical equipment used for imaging [67]. Studies have shown that algorithms can perform seemingly impossible tasks, such as determining a patient’s sex from a retinal fundus photography [68] or identifying race from a chest X-ray [69]. Such findings raise concerns about the use of such “invisible” features for diagnosis and treatment recommendation, rather than those pertaining to the clinical features of the disease.

In ophthalmology, small data sets, image details, and class imbalance can contribute to shortcut learning and measurement bias [34,70]. A thorough data analysis, careful interpretation of results, investigation of model explainability, and generalizability of a test model are needed to assess the impact of shortcut features [67].

Model evaluation and validation

AI algorithms are usually tested on external data sets to evaluate their real-world performance. Typical metrics include accuracy, sensitivity, specificity, the receiver operating characteristic (ROC) curve, and F1-score. It is crucial to choose the appropriate metrics and to perform a thorough analysis of downstream consequences of errors.

Pitfalls

Wrong evaluation metric

Using inappropriate evaluation metrics can hide the underperformance of models in certain patient subgroups and widen outcome disparities [34]. To ensure algorithmic fairness, it is imperative to assess the performance across marginalized cohorts [71]. It is also essential to acknowledge that all evaluation metrics are only estimations of a construct and, therefore, may not fully capture the true extent of algorithmic fairness [67]. Metrics relying solely on accuracy, or other characteristics based on historical data, run the risk of perpetuating health inequities present in the data [71].

Wrong evaluation method

Evaluation bias arises when the methods used to assess model performance themselves are biased. Inadequate external and benchmark data sets that fail to represent the population accurately can result in a biased evaluation of the model’s performance. To improve the robustness and generalizability of algorithms, and avoid shortcut learning, it is important to use external validation data sets that are representative of the target population [34,67].

Model deployment

The deployment of AI systems in real-world settings is the culmination of understanding the data characteristics and the social patterning of the data generation process, model task definition, data preprocessing, and model evaluation. Deployment and post-deployment monitoring and recalibration may introduce additional bias and require as much risk mitigation.

Pitfalls

Defining the model threshold

Defining the decision thresholds depends on the model’s purpose (e.g., screening, diagnosis, triage) and the health system it is to be deployed in.

For example, in diabetic retinopathy screening, setting lower thresholds may increase the number of false positive results and unnecessary referral cases and workup. While this could lead to a higher sensitivity for detecting potential cases, it will also burden the healthcare system. Conversely, setting higher thresholds might reduce false positives but could lead to missed cases, compromising the sensitivity of the screening program and potentially delaying necessary interventions.

Comprehending the setting in which the model will be deployed is pivotal to its impact [49]. Deploying AI models, typically developed on data from well-resourced health systems, in limited-resource settings necessitates careful consideration of the available resources within the healthcare system [15]. To strike a balance between sensitivity and specificity, it is essential to consider factors such as the prevalence of the condition, the healthcare resources available, and the anticipated impact on patient outcomes. Multi-stakeholder engagement is crucial in this process.

Covariate and data set shift

Covariate shifts can arise when there are changes in the distribution of the input variables between the training and deployment phase. In the field of ophthalmology, the generalization capabilities of AI models can be compromised due to shifts that occur when transitioning from the training data set to the actual population they are being deployed to. These shifts can be attributed to various factors, including changes in patient demographics, changes in disease prevalence, differences in ophthalmological assessment timing, or variations in equipment use. Furthermore, population drift may occur as a result of a practice pattern change or enactment of new health policies [72,73].

Post-deployment evaluation, monitoring, and system recalibration

It is required to conduct post-deployment monitoring of algorithms and analyze their impact on clinical outcomes with special attention to historically disenfranchised groups within the health system. Continuous measurement of the impact disaggregated across patient subgroups and recalibration of the model as necessary will ensure equitable benefit of AI systems [74,75]. Moreover, it is important to preempt unintended consequences, including heightening social stigma when outcomes among certain groups are highlighted, before integrating the algorithm into clinical practice.

Pitfalls

Model updates

The decision of when and how to update AI systems can be a potential source of bias. The procedures for updating AI systems may differ across different settings, which can lead to differential performance and impact on patient outcomes. Consequently, it becomes essential to thoughtfully assess the potential biases that may arise during the updating process and implement appropriate measures to mitigate them. Timely updates to AI systems are essential for preserving their relevance as new clinical insights and practice patterns emerge.

Conclusion

The implementation of AI systems in healthcare holds great promise for enhancing clinical decision-making. Ophthalmology, in particular, employs imaging ancillary examinations for clinical practice, and telemedicine screening programs enable the development of automated systems. Future directions include incorporating autonomous systems into ophthalmological practice, thereby improving accessibility to ocular care. These initiatives include autonomous screening programs for diabetic retinopathy, age-related macular degeneration, and retinopathy of prematurity among others. Additionally, it can improve the decision-making on ocular treatments such as intravitreous injections and during ophthalmological surgeries.

However, it is important to be aware that bias can be introduced at every step of the AI lifecycle. To address this concern, we propose a seven-step process that encompasses data collection; defining the model task; data preprocessing and labeling; model development; model evaluation and validation; deployment; and finally, post-deployment evaluation, monitoring, and system recalibration. Understanding the impact of data disparities, modeling decisions, and evaluation metrics is vital to developing models that are accurate, equitable, and clinically relevant.

Collaboration between ophthalmologists, data scientists, social scientists, community representatives, and other stakeholders is crucial to this endeavor. In resource-limited settings, the tailoring of AI systems to address practical challenges becomes essential for successful real-world deployment. Moreover, the continuous monitoring of AI systems post-deployment is paramount to identify any biases that may emerge over time and ensure ongoing fairness and transparency. Vigilance in identifying and addressing biases, combined with preemptive measures to prevent unintended consequences, enables AI systems to be leveraged responsibly and ethically in clinical practice.

Funding Statement

The authors received no specific funding for this work.