Evaluating the accuracy of the Ophthalmologist Robot for multiple blindness-causing eye diseases: a multicentre, prospective study protocol

Abstract

Introduction

Early eye screening and treatment can reduce the incidence of blindness by detecting and addressing eye diseases at an early stage. The Ophthalmologist Robot is an automated device that can simultaneously capture ocular surface and fundus images without the need for ophthalmologists, making it highly suitable for primary application. However, the accuracy of the device’s screening capabilities requires further validation. This study aims to evaluate and compare the screening accuracies of ophthalmologists and deep learning models using images captured by the Ophthalmologist Robot, in order to identify a screening method that is both highly accurate and cost-effective. Our findings may provide valuable insights into the potential applications of remote eye screening.

Methods and analysis

This is a multicentre, prospective study that will recruit approximately 1578 participants from 3 hospitals. All participants will undergo ocular surface and fundus images taken by the Ophthalmologist Robot. Additionally, 695 participants will have their ocular surface imaged with a slit lamp. Relevant information from outpatient medical records will be collected. The primary objective is to evaluate the accuracy of ophthalmologists’ screening for multiple blindness-causing eye diseases using device images through receiver operating characteristic curve analysis. The targeted diseases include keratitis, corneal scar, cataract, diabetic retinopathy, age-related macular degeneration, glaucomatous optic neuropathy and pathological myopia. The secondary objective is to assess the accuracy of deep learning models in disease screening. Furthermore, the study aims to compare the consistency between the Ophthalmologist Robot and the slit lamp in screening for keratitis and corneal scar using the Kappa test. Additionally, the cost-effectiveness of three eye screening methods, based on non-telemedicine screening, ophthalmologist-telemedicine screening and artificial intelligence-telemedicine screening, will be assessed by constructing Markov models.

Ethics and dissemination

The study has obtained approval from the ethics committee of the Ophthalmology and Optometry Hospital of Wenzhou Medical University (reference: 2023-026 K-21-01). This work will be disseminated by peer-review publications, abstract presentations at national and international conferences and data sharing with other researchers.

Trial registration number

ChiCTR2300070082.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• This study focuses on the Ophthalmologist Robot, an automated device that enables non-dilated fundus and ocular surface imaging, making eye screening possible in remote areas without specialist ophthalmologists.

• This study is a multicentre, prospective study which will recruit participants from three eye hospitals or clinical eye centres with experienced ophthalmologists and a large number of outpatients.

• This study aims to develop an intelligent screening system for multiple eye diseases using ophthalmologists and artificial intelligence models, covering both the fundus and ocular surface.

• This device has yet to be further upgraded, such as improving the algorithm for automatic eye position recognition and enhancing image clarity.

• These hospital-based results may not fully represent screening results in remote areas due to potential variations in types of eye diseases encountered.

Introduction

Early detection of eye diseases plays a crucial role in improving visual health and preventing blindness. According to the WHO, there are over 2.2 billion cases of visual impairment, with approximately half of them being preventable or curable through various treatments.¹ These include conditions such as cataract, glaucoma, corneal blindness (mainly due to keratitis and corneal scar), diabetic retinopathy (DR), age-related macular degeneration (AMD) and refractive error. Collectively, these conditions are known as blindness-causing eye diseases. However, maintaining effective blindness prevention initiatives through early screening and timely treatment faces challenges due to a shortage of ophthalmologists, inadequate coverage of eye care services and disparities in the quality of care.²

Advancements in artificial intelligence (AI), particularly deep learning models, have shown potential in ophthalmological screening and cost reduction.³ A number of studies have been conducted to develop AI models for identifying blindness-causing eye diseases using images captured by different devices. For instance, Li et al ⁴ developed a deep learning system for keratitis using slit lamp images. Elsawy et al ⁵ created a multidisease deep learning diagnostic network that uses anterior segment optical coherence tomography (OCT) images to diagnose corneal diseases. Tham et al ⁶ developed deep learning algorithms for cataract screening based on fundus images. Lin et al ⁷ constructed the Comprehensive Artificial intelligence Retinal Expert system, capable of identifying 14 retinal abnormalities, including DR, AMD, glaucomatous optic neuropathy (GON) and pathological myopia (PM). Additionally, several studies have conducted cost-effectiveness analyses of community screening for eye diseases such as cataract, glaucoma, DR, AMD and PM.^8–12 These AI models and systems have demonstrated promise as low-cost diagnostic tools that could aid in triage situations where access to eye care is limited.

However, AI models used for remote screening have certain limitations. First, these models rely on the collection of images, which requires the assistance of ophthalmology professionals. Unfortunately, the scarcity of such professionals in remote areas restricts the availability of images for AI screening. Second, most AI models are designed to screen either ocular surface or fundus diseases, and the effective integration and validation of AI systems capable of screening both types of diseases remains limited. Third, there is a lack of studies evaluating the cost-effectiveness of ocular surface disease screening using AI models.

To address these limitations, our team has developed the Ophthalmologist Robot, a novel device that enables automatic and simple capture of ocular surface and fundus images without the need for pupil dilation or ophthalmologist intervention. The intelligent automated eye screening robot technology (Patent No. CN202011441476.0) integrates ocular surface and fundus examinations into a single device. This device offers a wide framing range, covering from the cornea to the retina. With the help of software settings, the device can detect the overall eye contour, automatically focus and capture fundus images. It then adjusts the focus distance to capture ocular surface images by moving the focus point towards the centre of the cornea. Once the imaging of one eye’s ocular surface and fundus is completed, the device’s imaging system seamlessly moves on to capture images of the other eye. The entire examination process for both eyes usually takes less than 1 min.

In this study, our primary objectives are to evaluate the accuracy of ophthalmologists in screening for multiple blindness-causing eye diseases using images captured by the Ophthalmologist Robot. Our secondary objectives include: (1) assessing the screening accuracy of deep learning models based on these images, (2) comparing the consistency between Ophthalmologist Robot images and slit lamp anterior segment images as interpreted by ophthalmologists, in screening for keratitis and corneal scar and (3) conducting cost-effectiveness analyses of three eye screening methods: non-telemedicine screening, ophthalmologist-telemedicine screening and AI-telemedicine screening. If proven accurate and cost-effective, the Ophthalmologist Robot has the potential to reduce screening costs and aid in large-scale primary care settings where diagnosing blinding ocular diseases is challenging.

Methods and analysis

Study design

This multicentre, prospective study aims to assess the screening accuracy for multiple blindness-causing eye diseases in China. The study will involve three hospitals: Zhejiang Eye Hospital, Ningbo Eye Hospital and Hangzhou Branch of Zhejiang Eye Hospital. These hospitals are tertiary care ophthalmology centres or specialist ophthalmology hospitals, equipped with advanced examination tools, experienced ophthalmologists and a high volume of outpatient cases. Newly diagnosed patients will be recruited from these hospitals starting from April 2023, with an anticipated recruitment period of 12 months. Participants will be followed up until they complete one clinical consultation, which may involve ophthalmic examinations, prescriptions or necessary surgeries. Figure 1 illustrates the entire research process.

Figure 1.

An overview of the study’s workflow. AI, artificial intelligence.

Study participants

Patients who are newly diagnosed in the convenience series will undergo eligibility screening conducted by outpatient ophthalmologists with a minimum of 5 years of clinical experience.

Inclusion criteria

1. Participants included newly diagnosed individuals with blindness-causing eye diseases (keratitis, corneal scar, cataract, DR, AMD, GON and PM), as well as relatively healthy individuals (not reporting any of the afore-mentioned blindness-causing eye diseases at baseline).

2. Aged 18 years or older.

3. Provide informed consent.

Exclusion criteria

1. Inability to cooperate with the Ophthalmologist Robot examination, such as due to a serious mental illness.

2. Missing outpatient data with key information, such as diagnoses.

3. Poor quality ocular images.

4. Significant changes in the ocular images since diagnosis due to factors such as postdiagnostic treatment or cataract surgery.

Gold standard diagnosis

An outpatient ophthalmologist, with a minimum of 5 years of clinical experience, will interview patients and conduct slit lamp examinations on both eyes to determine the outpatient diagnosis. Additionally, if required, the clinical consultation may include tests such as best-corrected visual acuity, intraocular pressure measurements, OCT, fluorescein staining and other tests to ensure the most accurate diagnoses are made.

The Ophthalmologist Robot screening process

The predefined screening process of the device is depicted in figure 2. In a darkroom setting, the subject’s information is entered into the software, initiating the image acquisition interface (figure 2 B & C). The subject’s lower jaw is placed on the machine’s jaw rest, and the device’s image acquisition system automatically aligns the subject’s right eye, adjusting the height to ensure the eye is centred within the image (figure 2 D). The fixation lamp assists the subject in maintaining fixation on the centre. Subsequently, the camera system automatically focuses and captures colour images of the fundus’s posterior pole, eliminating the need for pupil dilation (figure 2 E-a). The machine then adjusts the focus automatically to capture images of the ocular surface, targeting the cornea’s centre (figure 2 E-b). After completing the imaging process for the right eye, the camera system moves both vertically and horizontally to focus on the subject’s left eye, repeating the same process to capture fundus and ocular surface images.

Figure 2.

The process of ocular image collection through the Ophthalmologist Robot. OD, oculus dexter. OS, oculus sinister. SP, small pupils. (A) Prototype construction of the Ophthalmologist Robot. (B) Brief interface of the software. (C) Entering patient information. (D) Automatic picture taking. (E) Pictures of the ocular fundus (a) and surface (b).

Collection of data and images

Table 1 provides an overview of the data and images that will be collected throughout the study, including basic information, medical history, gold standard diagnosis, ophthalmic examination results, images and medical expenses. Furthermore, table 1 incorporates additional parameters such as prevalence, transition probabilities, compliance, utility, discount rates, mortality rates and screening costs. These data, sourced from the literature, hospitals and official reports, will be integral to the subsequent construction and analysis of the Markov models.

Table 1.

Overview of the data and images

Data and images	Description	Collectors
Basic information	Desensitised ID code, gender, date of birth	Ophthalmologists
Medical history	Chief complaint, present medical history, past medical history, etc	Ophthalmologists
Gold standard diagnosis	Outpatient ophthalmologists, with a minimum of 5 years of clinical experience, will use the slit lamp to examine both eyes and make outpatient diagnoses. Additional tests may be employed, if necessary, to further confirm the accuracy of the diagnoses	Ophthalmologists
Ophthalmic examination	BCVA test, IOP, OCT and fluorescein staining, etc	Specialist ophthalmic examiners
Images
The Ophthalmologist Robot images	Ocular surface and fundus	Non-ophthalmic examiners
The slit lamp images	Ocular surface	Specialist ophthalmic examiners
Medical expenses	Drug expenses, surgery expenses, examination expenses, etc	Ophthalmologists
Additional parameters	Prevalence, transition probabilities, compliance, utility, discount rates, mortality rates, screening cost, etc	Ophthalmologists

BCVA test, best-corrected visual acuity test; IOP, intraocular pressure; OCT, optical coherence tomography.

Quality control

Ocular images captured by the Ophthalmologist Robot will be obtained by non-ophthalmic examiners at each study centre, such as nurses. Brief instructions on operating the Ophthalmologist Robot will be provided prior to the study.

The study centres will upload outpatient data and images, which will be reviewed by trained research personnel to ensure data completeness, accuracy and equipment stability. Images that exhibit extreme blurring, skewed eye position, underexposed anatomical structures or poor lesion visibility will be considered of low quality, excluding the effects of eye diseases. If more than 10% of the total images are of low quality, indicating equipment instability, the study may be prematurely terminated.

Participant information that has already been collected will be analysed at a later stage.

Data management and confidentiality

The data from each centre will be uploaded to the responsible centre after undergoing desensitisation. Professional engineers will manage and protect the data. Research data will be stored using a unique study identification code for each participant. The key to the identification code list will only be accessible to the research team during the study. After the completion of the study, the principal investigator will document and safeguard the key according to research guidelines. No participant identification details will be disclosed in publications.

Images screening

Screening the Ophthalmologist Robot images and the slit lamp Images by ophthalmologists

The study will involve recruiting at least three junior, three intermediate and three senior ophthalmologists. Each image will be independently diagnosed by two ophthalmologists with the same qualification. If there is a difference in opinion, a third ophthalmologist with the same qualification will arbitrate and provide the final diagnosis. The main diseases to be diagnosed include keratitis, corneal scar, cataract, DR, AMD, GON and PM, which will be further specified based on the actual outpatient diagnoses.

Screening the Ophthalmologist Robot images by AI models

Using the images from the Ophthalmologist Robot, we will assess the accuracy of established corneal recognition models, cataract recognition models and fundus recognition models that have demonstrated high accuracy in previous studies and within our team.4 6 7 13 Additionally, we will incorporate recently published AI models for training and evaluation. This evaluation process will involve debugging AI models and converting images. The results will be shared in a forthcoming research publication.

After evaluating and debugging previous models, along with the latest iteration of AI models, our priority will be to deploy the most accurate AI models on the cloud platform instead of the device itself. This decision is influenced by the computing power and memory requirements of the AI software. Images will be transferred from the device to the cloud platform for screening by the AI model. Additionally, ophthalmologists and engineers will review, maintain and manage the images as necessary to ensure the highest level of quality and accuracy.

Cost-effectiveness of the Ophthalmologist Robot

Comparator strategies

From a healthcare perspective, we will evaluate the cost-effectiveness of the following three strategies:

1. Non-telemedicine screening: this approach involves conducting on-site screenings by ophthalmologists using specialised ophthalmic equipment such as slit lamp microscopes, optometers, tonometers, etc.

2. Ophthalmologist-telemedicine screening: in this approach, a trained non-ophthalmology professional (eg, volunteer, caregiver, etc) captures images and relevant information of the individual on-site using the Ophthalmologist Robot. These data are then transmitted to the cloud via the Internet, where an online ophthalmologist performs the screening for eye diseases.

3. AI-telemedicine screening: similar to the ophthalmologist-telemedicine screening, a trained non-ophthalmology professional captures images and necessary information on-site using the Ophthalmologist Robot. These data are synchronised to the cloud through the internet, where an AI model conducts the screening for eye diseases.

Model structure

Based on the accuracy of screening by ophthalmologists and AI, as well as the cost of examination and treatment, along with additional parameters obtained from the authoritative literature and official sources, we will construct a decision-analytic Markov model. This model will enable the examination of costs and benefits associated with three different eye screening strategies for multiple blinding eye diseases. To depict these strategies, a decision tree will be developed (see online supplemental figure 1). The classification of health states for each blinding eye disease and the transitions between these states will be established based on relevant literature on health economic analyses of eye disease screening from reputable journals.^{9 11 12}

Participant timeline

We will follow the study steps outlined in figure3, which encompass enrolment, data collection, screening and statistical analyses, to conduct this study.

Figure 3.

The timeline of this study. t₀=enrolment, t₁=after enrolment, t₂=25% of the images collection completed, t₃=50% of the images collection completed, t₄=75% of the images collection completed, t₅=100% of the images collection completed, t₆=the whole study is completed. AI, artificial intelligence.

Participant and public involvement

None.

Sample size

As the primary outcomes is to assess the accuracy of ophthalmologists’ online screening, the study calculated sample size based on the diagnostic accuracy test formula: $n={\left(\frac{{\mu }_{\alpha }}{\delta }\right)}^{2}p\left(1-p\right).$ ¹⁴ With a tolerance error (δ) of 0.05, a type I error (α) of 0.05 and an attrition rate of 10%, the required sample size is 1578 participants, including 1425 patients with multiple blindness-causing diseases and 153 controls with other eye conditions. The online supplemental material provides more details of sample size calculation for sensitivity and specificity (see online supplemental table 1 and details of sample size calculation).

Outcomes

The primary focus of this study is to assess the accuracy of ophthalmologists in screening for multiple eye diseases that can cause blindness, such as keratitis, corneal scar, cataract, DR, AMD, GON and PM. The screenings will be performed using images captured by the Ophthalmologist Robot, and the accuracy will be assessed using receiver operating characteristic (ROC) curve.

The secondary outcomes of the study are as follows:

1. Assessing the accuracy of deep learning models in screening for multiple blindness-causing eye diseases using ROC curve analysis.

2. The Kappa test will be used to evaluate the consistency between the images captured by the Ophthalmologist Robot and the slit lamp when the ophthalmologist conducted screenings for keratitis and corneal scarring.

3. Evaluating the cost-effectiveness of three screening methods: non-telemedicine screening, ophthalmologist-telemedicine screening and AI-telemedicine screening. We will assess the effectiveness using the years of blindness avoided. Incremental cost-effectiveness ratios (ICERs) will be calculated as:

$${\displaystyle ICERs=\frac{incrementalcost}{incrementaleffectiveness}}$$

Statistical analysis

We will collect and analyse basic information such as gender, age, medical history and types of diagnosed diseases in various study centres. The performance of ophthalmologists and AI models in screening for multiple eye diseases using the Ophthalmologist Robot will be evaluated. Sensitivity, specificity, positive predictive value, negative predictive value and likelihood ratio will be calculated using the one-versus-rest method. ROC curves will be plotted, and the area under the curve will be calculated.

The consistency of screening results will be assessed using the Kappa test. Values below 0.20 indicate poor consistency, 0.21–0.40 indicate fair consistency, 0.41–0.60 indicate moderate consistency, 0.61–0.80 indicate relatively strong consistency and 0.81–1.00 indicate extremely strong consistency. The Kappa test will be used to assess consistency between slit lamp ocular surface images and Ophthalmologist Robot images.

The Markov models will be used to analyze a population aged 50 years and older for a period of 30 1-year Markov cycles. Deterministic one-way and simulated probabilistic sensitivity analyses will be conducted to account for uncertainty around the incremental cost-effectiveness ratio.

All statistical tests will be two sided, and results will be deemed statistically significant at a p value of <0.05. The analysis software used will be R language V.4.1.3 and TreeAgePro 2022 R V.1.2.

Ethics and dissemination

The study has obtained approval from the ethics committee of the Ophthalmology and Optometry Hospital of Wenzhou Medical University (reference: 2023-026 K-21-01). The results of this research will be disclosed completely in international peer-reviewed journals.

Discussion

The Ophthalmologist Robot is primarily designed for easy and automated collection of ocular surface and fundus images to facilitate primary eye screening in community and remote areas. This study aims to assess the accuracy of ophthalmologists and deep learning models in screening for multiple eye diseases that can cause blindness using the Ophthalmologist Robot, as well as the cost-effectiveness of these screening approaches.

The study will evaluate the screening accuracy of ophthalmologists using the Ophthalmologist Robot by comparing their results with the gold standard outpatient diagnosis. Furthermore, the cost-effectiveness of the Ophthalmologist Robot will be assessed. In addition, the study will compare the screening results of ophthalmologists for corneal diseases, specifically keratitis, corneal scar and negative controls, based on the images captured by the Ophthalmologist Robot, with slit lamp ocular surface images (considered as gold standard images).

While this study focuses on multiple blindness-causing eye diseases, subsequent studies may include other eye diseases such as conjunctivitis, pterygium and eyelid masses. It should be noted that early glaucoma without optic neuropathy, myopia, hyperopia, presbyopia and amblyopia without fundus pathology are not covered in this study. However, as equipment improves, screening for these diseases may be added in the future. The study will address issues related to patients who cannot be located automatically and analyse the reasons behind it, aiming to resolve these issues through device upgrades and algorithm iterations. It is important to acknowledge that when screening in remote areas, the results may be biased due to differences in disease types and proportions compared with hospitals where the data are collected.

The goal is to establish a multidisease screening system for multiple blindness-causing eye diseases based on the Ophthalmologist Robot. This will include evaluating its social cost-effectiveness and providing theoretical guidance for the implementation of eye disease screening in the community and remote areas. The successful implementation of this system would enable early detection and treatment of blindness-causing eye diseases, slow down disease progression, reduce blindness incidence and promote visual health maintenance.

In the future, our intention is to deploy this technology in areas where there is a shortage of specialised ophthalmologists to conduct local screening programmes. In addition to image analysis, we plan to incorporate qualitative tasks such as filling out ophthalmic and systemic disease questionnaires, lifestyle habit questionnaires, etc. These additional tasks will assist doctors in gaining a better understanding of users and patients, facilitating the development of personalised treatment plans and establishing correlations between eye images and systemic diseases.

Ethics statements

Patient consent for publication

Not applicable.