Evaluation of an OCT-AI–Based Telemedicine Platform for Retinal Disease Screening and Referral in a Primary Care Setting

Abstract

Purpose

To evaluate the performance of a telemedicine platform integrated with optical coherence tomography (OCT) and artificial intelligence (AI) techniques for retinal disease screening and referral.

Methods

We constructed an OCT-AI–based telemedicine platform and deployed it at four primary care stations located in Jing'an district, Shanghai, to detect retinal disease cases among aged groups and refer them to Shanghai Tenth People's Hospital (TENTH Hospital). Two ophthalmologists jointly graded the data set collected from this pilot application, and then the performance of this platform was analyzed from multiple aspects.

Results

This study included 1257 participants between July 2020 and September 2020, of whom 394 had retinal pathologies and 146 were even considered urgent cases by the ophthalmologists. The OCT-AI models achieved a sensitivity of 96.6% (95% confidence interval [CI], 91.8%–98.7%) and specificity of 98.8% (95% CI, 98.0%–99.3%) for detecting urgent cases and a sensitivity of 98.5% (95% CI, 96.5%–99.4%) and specificity of 96.2% (95% CI, 94.6%–97.3%) for detecting both urgent and routine cases. Coupled with AI, our platform reduced the workload of human consultation by 96.2% for massive normal cases. The detected disease cases received online medical suggestions at an average time of 21.4 hours via this platform.

Conclusions

This platform can automatically identify patients with retinal disease with high sensitivity and specificity, support timely human consultation, and bring necessary referrals.

Translational Relevance

The OCT-AI–based telemedicine platform shows great practical value for retinal disease screening and referral in a real-world primary care setting.

Introduction

Retinal diseases, including diabetic retinopathy (DR), age-related macular degeneration (AMD), epiretinal membranes (ERMs), macular hole, and so on, are the major causes of vision impairment and affect hundreds of millions of people worldwide.^1–5 The rising prevalence of various retinal diseases that mainly exist in aging populations over 50 years old poses a heavy burden on eye care services.^6,7 Early detection and referral are necessary for preventing vision loss. Thus, it is beneficial for aged groups to undergo regular retinal screening. However, several surveys of the global ophthalmology workforce show that there is a significant shortfall of ophthalmologists, especially in developing countries.^8,9 The reality that most eye care services in many countries are concentrated in superior hospitals further adds to the difficulty in access. This highlights the importance of strengthening the integration of eye care services within primary health care to realize convenient screening of retinal diseases and ensuring effective referral to superior hospitals.¹⁰

With advancements in telecommunication networks, some telemedicine-based applications for the screening of DR and some other retinal findings in the primary care setting have been proven to be clinically effective and cost-effective.^11–14 Coupled with automated artificial intelligence (AI) analysis, it may help to identify patients with sight-threatening retinal diseases and reduce a large portion of unnecessary consultations and referrals.¹⁵ Several studies focused on applying AI in fundus photograph (FP)–based DR screening systems in real-world clinical use.^16–19 For instance, He et al.¹⁶ reported a sensitivity of 90.79% and a specificity of 98.5% via an AI software for DR screening among 889 diabetic patients from a community hospital in Shanghai, China, and Xie et al.¹⁹ observed that applying a semiautomated telemedicine platform combining a deep learning system with human assessment achieved the best economic return for DR screening in Singapore, which could save roughly 20% of the annual cost.

Compared with FP, optical coherence tomography (OCT) is a more powerful imaging technique for the diagnosis of retinal diseases because it provides high-resolution images of cross-sectional retinal structures and reveals subtle macular changes.^20–22 In recent years, many studies have obtained promising performances in the automated identification of various retinal diseases via deep learning algorithms from OCT images.^23–25 However, to the best of our knowledge, there is no relevant research on the evaluation of applying OCT-AI for real-world retinal disease screening.

In our previous study, an intelligent OCT-based approach containing retinal pathology detection and urgent referral decision was developed and validated using retrospective clinical data sets collected from two hospitals.²⁶ On this basis, we built an OCT-AI–based telemedicine platform and implemented it for the screening and referral of common retinal diseases among older adults, not only DR. The objective of this study was to evaluate the AI analysis performance and practical effectiveness of this platform in a real-world primary care setting.

Methods

The study was approved by the ethical committee of Shanghai Tenth People's Hospital (TENTH Hospital) and conducted according to the tenets of the Declaration of Helsinki. All participants voluntarily signed informed consent forms. This study was conducted to evaluate the performance of an OCT-AI–based telemedicine platform in real-world application, which consisted of four stages: (1) building the platform for the identification and referral of retinal disease cases, (2) implementing it in a primary care setting and gathering study participants, (3) inviting ophthalmologists to grade all participants’ data independently, and (4) evaluating the performance of this platform from multiple aspects. The details are described in the following.

OCT-AI–Based Telemedicine Platform

The OCT-AI–based telemedicine platform featuring cloud-based services, web-interface connections, and AI analysis is developed for automated screening and remote assessment of patients with common retinal diseases. A brief workflow of this platform is depicted in Figure 1A. The community inhabitants underwent retinal examination using OCT devices operated by the trained nurses in nearby primary care stations. OCT scans of both eyes captured from each participant are uploaded to the cloud platform automatically and analyzed by AI functions (see Appendix AI Function Details) running on the cloud server in real time. The examination record of each participant, including age, gender, original OCT scans, and corresponding AI analysis report, can be easily viewed on a website by logging into the station's account. When the AI analysis report indicates that the imaging quality of at least one eye is poor, the corresponding participant is required to be rescanned. If the report displays detected retinal pathologies and urgent or routine referral suggestions, then this possible retinal disease case is delivered immediately for human consultation. On the other hand, an ophthalmologist reviews the examination records of those cases delivered from primary care stations on the website via his or her account. Then, the ophthalmologist offers professional medical suggestions and pays especially more attention to cases accompanied by AI suggestions for urgent referral. Immediately, the stations as well as the corresponding participants receive medical suggestions. For more convenient operation, offering and receiving medical suggestions can also be completed using an application on mobile phones. Those urgent cases remotely confirmed by the ophthalmologist are instructed to visit the superior hospital as soon as possible for further diagnosis.

Figure 1.

Schematic of the OCT-AI–based telemedicine platform. (A) The workflow of our OCT-AI–based telemedicine platform that connects primary care stations and superior hospitals together. It consists of six steps (a–f) to complete retinal disease screening and referral. (B) The locations of the four primary care stations and the TENTH Hospital marked on the map of Jing'an district, Shanghai.

Implementation of the Platform in a Primary Care Setting

From July 2020 to September 2020, two OCT devices (iScan; Optovue, Inc., Fremont, CA, USA) integrated with our OCT-AI–based telemedicine platform were deployed in four primary care stations within different periods. These primary care stations are in four large residential communities, including PengPu Town, GongHe Road, DaNing Road, and BaoShan Road, located in the urban district of Jing'an in central Shanghai. During the 3-month trial period, the community inhabitants with an age of no less than 50 years living around the abovementioned stations were recruited prospectively to participate in this study. As indicated by the process depicted in Figure 1A, they accepted retinal examinations by the OCT devices, and the captured OCT scans were uploaded to the OCT-AI–based platform to undergo AI analysis. In the meantime, a senior ophthalmologist (XL) from the TENTH Hospital specializing in retina cases provided online medical suggestions for suspected retinal disease identified via the platform, and consequently those confirmed urgent participants were suggested to visit the hospital for further diagnosis. TENTH Hospital is also located in Jing'an district, and the distances to the four primary care stations range from 1.3 to 5.1 km (Fig. 1B).

Grading Procedure

Two ophthalmologists (XL and CZ) jointly carried out the grading task for all study participants in late October 2020. Both come from the TENTH Hospital and have over 15 years of clinical experience specializing in retinal diseases. The grading ophthalmologists categorized each participant into three levels for referral, including basically normal for the retinal structure of both eyes (normal), follow-up inspection (routine), and timely visit to the superior hospital (urgent), based on both eyes’ OCT scans.²⁵ If at least one eye of one participant was treated as urgent, then he or she was graded for urgent referral. At first, approximately 10% of the participants were randomly selected to evaluate the grading consistency of two experienced ophthalmologists, and the test result indicated a high agreement (κ score ≥0.85) between them. Then, all involved participants were randomly divided between these two ophthalmologists. Each reviewed the OCT scans of half of the assigned participants and made the corresponding referral suggestions, which were considered the gold standard of this study. In the meantime, retinal pathologies observed in each participant were simultaneously recorded by the ophthalmologists.

Performance Evaluation

We evaluated the performance of the OCT-AI–based telemedicine platform in this practical application from two aspects. First, the AI analysis performance on referral decisions was evaluated by several common evaluation metrics, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and κ score (see Appendix Evaluation Metrics), compared with the gold standard determined by ophthalmologists. Sensitivity, specificity, PPV, and NPV were measured with 95% confidence intervals (CIs). Moreover, the case-based sensitivity and specificity in terms of detecting each category of retinal pathology were also calculated. Then, the working effectiveness of this platform was also assessed with regard to the reduction of burden on eye care resource, response efficiency for the potential retinal disease cases, and actual referrals to the superior hospital.

Results

A total of 1311 community inhabitants 50 years and older who visited the primary care stations within the 3-month platform trial period accepted OCT examinations. Of these, 54 (4.1%) inhabitants were excluded from the analysis due to the poor quality of the captured OCT images, which were judged by the imaging quality evaluation model (Supplementary Fig. S1).²⁷ The remaining 1257 inhabitants with average age of 64.2 years were all involved in the further evaluation of the platform, and 657, 397, 135, and 68 of them came from the PengPu, GongHe, DaNing, and BaoShan stations, respectively (Table 1). Of these involved participants, 19 (1.5%) claimed medical histories of retinal diseases such as DR, AMD, and within the past 10 years when they accepted retinal examinations.

Table 1.

Participant Characteristics

Characteristic	PengPu	GongHe	DaNing	BaoShan	Overall
Participant features
No. of imagesa	10,512	6352	2160	1088	20,112
No. of eyes (scans)b	1314	794	270	136	2514
No. of individuals	657	397	135	68	1257
Age, mean (SD), y	65.6 (10.5)	61.8 (13.3)	63.0 (10.5)	66.3 (8.4)	64.2 (11.0)
Female No./total (%)	327 (49.8)	191 (48.1)	79 (58.5)	46 (67.7)	643 (51.2)
Grading distribution
No. (%) of urgent individuals	81 (12.3)	38 (9.6)	15 (11.1)	12 (17.7)	146 (11.6)
No. (%) of routine individuals	115 (17.5)	77 (19.4)	40 (29.6)	16 (23.5)	248 (19.7)
No. (%) of normal individuals	461 (70.2)	282 (71.0)	80 (59.3)	40 (58.8)	863 (68.7)

^aEach OCT scan consists of eight-line cross-sectional images with a resolution of 1024 × 640 pixels.

^bThe number of OCT scans is equal to that of eyes, and each OCT scan corresponds to one unique eye.

The platform provides AI analysis reports for all 1257 participants. Each report shows the detected retinal pathologies and the predicted urgent levels for both eyes of each participant. According to the ophthalmologists’ grading results, the numbers of urgent, routine, and normal groups were 146, 248, and 863, respectively (Table 1). As shown in Figure 2A, the overall referral accuracy of AI was 95.5%, and the κ score between AI and ophthalmologists was 0.907. For detecting urgent cases, AI achieved a sensitivity of 96.6% (95% CI, 91.8%–98.7%), specificity of 98.8% (95% CI, 98.0%–99.3%), PPV of 91.6% (95% CI, 85.7%–95.2%), and NPV of 99.5% (95% CI, 98.9%–99.8%). For detecting both urgent and routine cases, AI achieved a sensitivity of 98.5% (95% CI, 96.5%–99.4%), specificity of 96.2% (95% CI, 94.6%–97.3%), PPV of 92.2% (95% CI, 89.1%–94.5%), and NPV of 99.3% (95% CI, 98.4%–99.7%). In Figure 2B, 154 (12.3%) urgent cases and 267 (21.2%) routine cases were identified by AI among all participants, of which 146 (11.6%) and 248 (19.7%) were graded by ophthalmologists, respectively. The confusion matrix (Fig. 2C) indicates that 57 incorrect cases (46 due to overreferral and 11 due to underreferral) appear in the top-right and bottom-left triangles, respectively.

Figure 2.

Results of AI analysis for referral decisions. (A) Statistical metrics of AI referral of retinal diseases in all participants. (B) Proportion of participants with different referral levels divided by AI and ophthalmologists. (C) Confusion matrix with participant numbers that represent the combination of gold-standard referrals and predicted referrals. The number of correct referral decisions is found on the diagonal.

Of these 57 cases with incorrect predicted referral decisions, 33 (57.9%) normal cases predicted as routine were caused by false detection of RPE irregularity, including drusen (11, 19.3%), ERM (8, 14.0%), RPE atrophy (7, 12.3%), hyperreflective foci (4, 7.0%), vitreomacular traction (2, 3.5%), and intraretinal fluid (1, 1.8%). There were 6 (10.5%) routine cases predicted as normal due to the absence of several minor pathologies, such as RPE irregularity including drusen (2, 3.5%), ERM (2, 3.5%), RPE atrophy (1, 1.8%), and choroid curvature abnormality (1, 1.8%). The remaining 18 (31.6%) incorrect cases occurring between urgent and routine resulted from 16 misclassifications of urgent levels and 2 false detections of intraretinal fluid and RPE atrophy. Some typical examples of correct and incorrect AI analysis cases are shown in Figure 3 and Figure 4, respectively.

Figure 3.

Correct OCT-AI analysis reports of four participants. (A) Routine case in which RPE irregularity including drusen was found in both eyes. (B) Urgent case as macular hole and intraretinal fluid are detected in the left eye. (C) Another urgent case with fibrovascular PED detected in the right eye. (D) A case also predicted as urgent: RPE atrophy is found in both eyes and the right eye is considered an urgent level. Note that among eight OCT images per eye, one image containing typical pathologies or passing through the fovea is selected for display in the report.

Figure 4.

Incorrect OCT-AI analysis reports of four participants. (A, B) Two false-positive cases caused by wrongly detecting ERM and intraretinal fluid, respectively. (C) A false-negative case caused by misdetection of choroid curvature abnormality in the right eye. (D) Case graded as routine but misclassified as urgent by the referral decision model.

Based on the two grading ophthalmologists’ observations, the numbers of participants in the urgent and overall pathologic groups that contained each retinal pathology category, together with the percentiles among all study participants, are provided in Table 2. ERM (150, 11.9%), RPE irregularity including drusen (104, 8.3%), and RPE atrophy (33, 2.6%) were the top three retinal pathologies observed in all study participants. In addition, some sight-threatening pathologies such as intraretinal fluid (25, 2.0%), fibrovascular pigment epithelium detachment (PED) (15, 1.2%), macular hole (14, 1.1%), retinoschisis (13, 1.0%), and subretinal fluid (11, 0.9%) were also relatively prevalent and mainly distributed among the urgent group. As listed in Table 2, the overall case-based sensitivities and specificities for detecting different pathologic categories by the pathology detection model ranged from 85.7% to 100.0% and 98.9% to 100.0%, respectively.

Table 2.

Incidence of Retinal Pathologies Observed in Study Participants and Corresponding AI Detection Performance

	Urgent				Urgent + Routine
Pathology	No. of Individuals	Percent/Total Individuals (%)	Sensitivity (%)	Specificity (%)	No. of Individuals	Percent/Total Individuals (%)	Sensitivity (%)	Specificity (%)
ERM	33	2.6	97.0	99.9	150	11.9	98.0	99.2
RPE-IID	6	0.5	83.3	99.8	104	8.3	97.1	98.9
RPE atrophy	18	1.4	94.4	99.8	33	2.6	93.9	99.3
IRF	20	1.6	100.0	99.9	25	2.0	100.0	99.8
Fibrovascular PED	15	1.2	100.0	100.0	15	1.2	100.0	100.0
Hyperreflective foci	7	0.6	100.0	99.9	15	1.2	100.0	99.6
Macular hole	14	1.1	100.0	100.0	14	1.1	100.0	100.0
Retinoschisis	11	0.8	100.0	100.0	13	1.0	100.0	100.0
SRF	11	0.9	100.0	100.0	11	0.9	100.0	100.0
Double layer sign	8	0.6	100.0	100.0	8	0.6	100.0	100.0
Dome-shaped PED	8	0.6	100.0	100.0	8	0.6	100.0	100.0
VMT	5	0.4	100.0	100.0	7	0.6	100.0	99.8
Othersa	5	0.4	100.0	100.0	7	0.6	85.7	99.9

IRF, intraretinal fluid; RPE, retinal pigment epithelium; RPE-IID, retinal pigment epithelium irregularity including drusen; SRF, subretinal fluid; VMT, vitreomacular traction.

^aOthers: choroid curvature abnormality, diffuse hyperreflective material, and staphyloma.

Both the urgent and routine retinal pathologic cases detected by OCT-AI were sent to an ophthalmologist for online consultation. As shown in Figure 2, a total of 421 (33.5%) participants were identified as urgent or routine cases and subsequently delivered for human consultation via our platform within the implementation period. According to the gold standard, 6 mildly retinal pathologic cases were treated as normal and excluded from the process of online consultation, while 33 normal cases wrongly detected with retinal pathologies by AI were unnecessarily sent for human consultation. Combined with the automated OCT-AI analysis, this telemedicine platform saved the workload of 830 normal participants for online consultation performed by an ophthalmologist at the implementation stage, which accounted for 96.2% of all normal participants. Based on the platform records, the average response time from sending the 421 detected cases to getting a reply by the ophthalmologist was 21.4 hours, and 91.7% received medical suggestions within 48 hours. Furthermore, according to our follow-up investigation, 191 participants visited the TENTH Hospital for further diagnosis after they received medical suggestions via this platform by the end of October 2020, which included 132 urgent and 59 routine cases.

Discussion

Population aging leads to sharp increases in the number of people with common retinal diseases globally.^1–6,10 Timely detection and referral are essential for preventing permanent vision loss. However, most people have difficulty obtaining access to professional eye care due to the shortage and unbalanced distribution of ophthalmologists, especially in low- and middle-income countries. In this context, telemedicine offers solutions by providing remote eye care services. Especially during the current health care crisis caused by COVID-19, unnecessary visits to superior hospitals with high patient volumes need to be avoided, which has created a sudden surge in telemedicine demand.^28,29

Several prior studies^11–19 reported real-world applications of telemedicine platforms for retinal screening. Some of them adopted FP-AI technologies and evaluated the effectiveness of FP-AI–based telemedicine platforms for DR screening, either reporting high accuracies of AI analysis or achieving the best economic return.^16–19 However, these AI-based platforms are unable to handle many other common retinal diseases that exist widely in communities. Recently, developing AI algorithms for automated detection of several specific retinal diseases from OCT images has drawn more attention and demonstrated promising achievements.^23–25 In our former study, we developed the models of retinal pathology detection and referral decision, which were capable of recognizing 15 categories of pathologic signs occurring in the OCT scans and providing referral suggestions to comprehensively identify patients with various common retinal diseases.²⁶ Here, these AI models were integrated into our newly established telemedicine platform for the screening and referral of retinal disease cases among the aged groups.

Statistics show that human resources for primary eye care in Shanghai, one of the richest cities in China, are still extremely lacking.³⁰ Meanwhile, the population aging problem is prominent; for instance, more than 40% of the community inhabitants were over 50 years old in Jing'an district, Shanghai.³¹ We selected four primary care stations located in this district to deploy and implement the OCT-AI–based telemedicine platform. Our study aimed to prove the usefulness of this platform and was the first to our knowledge to apply OCT-AI techniques in a real-world screening setting that included a large portion of the aged population and a lack of ophthalmologists. In addition, the OCT device adopted in our study required only simple training for the operators due to its high automatization.³²

During the 3-month trial period, a total of 1257 participants who were over 50 years old and ordinary inhabitants living in communities around the primary care stations were prospectively involved. Most were unaware of their own retinal conditions. However, 394 participants had different degrees of pathologic changes in the retina from the captured OCT images, of whom 146 were considered urgent cases with sight-threatening pathologic retinal changes who needed timely referrals to the superior hospital for more accurate diagnosis and treatment to avoid potential vision impairments. The prevalence of several kinds of retinal pathologies, such as ERM, RPE irregularity including drusen, RPE atrophy, and intraretinal fluid, was moderately high among the study participants. These findings further demonstrate the importance and necessity of retinal disease screening among the high-risk aged groups.

Compared with the gold standard set by two retinal specialists, the AI models achieved an overall accuracy of 95.5% and a κ score of 0.907 for referral decisions. The sensitivity, specificity, PPV, and NPV of the AI models to detect urgent patients reached 96.6% (95% CI, 91.8%–98.7%), 98.8% (95% CI, 98.0%–99.3%), 91.6% (95% CI, 85.7%–95.2%), and 99.5% (95% CI, 98.9%–99.8%), respectively. Those of the AI models to detect routine patients were 98.5% (95% CI, 96.5%–99.4%), 96.2% (95% CI, 94.6%–97.3%), 92.2% (95% CI, 89.1%–94.5%), and 99.3% (95% CI, 98.4%–99.7%), respectively. In general, the OCT-AI models integrated with our platform performed reliably for automated identification of retinal disease cases in this pilot trial. Missed and false detection of retinal pathologies mainly occurred in several categories, such as ERM, RPE irregularity including drusen, and RPE atrophy. The missed pathologies were usually small and slight, whereas some normal tissues (e.g., interface reflection, retinal vascular acoustic shadow, and RPE wrinkle), which are similar to the mentioned pathologic categories, were falsely detected out.

In total, 421 (33.5%) study participants were identified as pathologic cases by the AI models and delivered for online medical consultation via the platform. Of these 421 delivered participants, only 6 mildly pathologic cases were not covered due to the missed detection. On the other side, 830 (96.2%) of the normal participants were filtered out by AI and did not trigger the human consultation procedure during platform implementation period. These statistical results demonstrated that our OCT-AI–based platform can help to not only screen patients with retinal diseases but also reduce the workload of remote eye care resource personnel for handling massive normal cases.

In addition to the performance of AI models in the primary care setting, we emphasized the importance of observing the actual effectiveness of our platform in the implementation process. This platform supports convenient and rapid online consultation by using the advantages of mobile telecommunication. The ophthalmologist preferred to review the delivered cases and give medical advice on a mobile phone to maximize the use of fragmented time. The efficiency of online consultation was promoted with these actions, and the average response time was less than 1 day. In the course of this project, the inhabitants gained more consciousness and attention to their own retinal conditions. Most of the detected urgent patients had visited the superior hospital for further diagnosis according to the received medical suggestions through the end of October 2020.

Limitations

There are several major limitations of this study. First, the number of participants involved was relatively small and distributed unevenly among different retinal pathologies. Further studies need to extend the reach of this platform, covering diverse communities (e.g., low-, middle-, and high-income areas or in a central city and surrounding rural areas) to serve a larger scale of community inhabitants. Second, certain clinical data such as visual acuity, duration of diabetes, and history of hypertension of the participants were not recorded in this study. As a result, we could not assess the impact of these factors on the screening results. Third, we did not follow the diagnosis and treatment of referred patients in the superior hospital and could not examine the concrete cost-effectiveness produced by this platform in this pilot application. Fourth, the OCT-AI functions still need to solve some incorrect analysis problems encountered in this pilot study and consequently obtain more reliable performance in future applications.

Conclusions

The characteristics of the study participants reveal that a considerable portion of aged community inhabitants has different degrees of retinal pathologic changes and is unable to be identified in a timely manner due to difficult access to eye care resources, even in a high-income region of China. Our study demonstrates that the OCT-AI–based telemedicine platform can automatically identify patients with retinal disease with high sensitivity and specificity among aged populations, support timely online medical consultation, and bring necessary referrals in the pilot primary care setting. This platform shows great practical value to perform effective screening and referral of retinal disease cases for aged community inhabitants, whereas further research is needed to evaluate the clinical and cost-effectiveness of our telemedicine platform in larger-scale applications.

Appendix

AI Function Details

The AI functions adopted in our OCT-AI–based telemedicine platform comprise a range of deep learning and rule decision models. First, a deep and shallow feature fusion network is applied for OCT image quality evaluation, which can quickly recognize poor-quality OCT scans.²⁷ It plays an important role in the regulation of data collection in the primary care setting. Then, a lesion detection network is responsible for identifying 15 categories of retinal pathologies, including epiretinal membrane, macular hole, vitreomacular traction, intraretinal fluid, subretinal fluid, hyperreflective foci, diffuse hyperreflective material, RPE irregularity including drusen, RPE atrophy, dome-shaped PED, fibrovascular PED, double layer sign, retinoschisis, staphyloma, and choroid curvature abnormality occurring in various retinal diseases from OCT images. The following is the referral level (urgent, routine, or normal) of each eye predicted by a random forest decision model using fused features of detected pathologies and retinal thickness measurement as input. The detailed definitions of 15 categories of retinal pathologies and referral levels are described in our previous study.²⁶

Evaluation Metrics

Several common evaluation metrics were adopted to measure the performance of AI analysis, including accuracy, κ score, sensitivity, specificity, PPV, and NPV. Accuracy reflects the overall correct predictions among all test samples. The κ score is frequently used to test interrater reliability and measures the consistency between the predicted results and the given labels. Sensitivity is the proportion of true positives out of all positive samples; specificity is the proportion of true negatives out of all negative samples. PPV determines how many true positives come from all positive findings; NPV determines how many true negatives come from all negative findings. The specific calculations of these metrics are illustrated below.

Illustration of True Positive, False Positive, True Negative, and False Negative

 .

 

		Label Class
		0 (Negative)	1 (Positive)
Predicted Class	0 (Negative)	TN (True Negative)	FN (False Negative)
	1 (Positive)	FP (False Positive)	TP (True Positive)

$$\begin{array}{c}\hfill Accuracy=\frac{TP+TN}{TP+FP+TN+FN}\end{array}$$

$$\begin{array}{c}\hfill \kappa =\frac{N\times \left(TP+TN\right)-\left(TN+FN\right)\times \left(TN+FP\right)-\left(FN+TP\right)\times \left(FP+TP\right)}{N^2-\left(TN+FN\right)\times \left(TN+FP\right)-\left(FN+TP\right)\times \left(FP+TP\right)},\end{array}$$

$$\begin{array}{c}\hfill N=TN+FN+TP+FP\end{array}$$

$$\begin{array}{c}\hfill Sensitivity=\frac{TP}{TP+FN}\end{array}$$

$$\begin{array}{c}\hfill Specificity=\frac{TN}{TN+FP}\end{array}$$

$$\begin{array}{c}\hfill PPV=\frac{TP}{TP+FP}\end{array}$$

$$\begin{array}{c}\hfill NPV=\frac{TN}{TN+FN}\end{array}$$