Abstract
Purpose:
A deep learning system (DLS) using artificial intelligence (AI) is emerging as a very promising technology in the future of healthcare diagnostics. While the concept of telehealth is emerging in every field of medicine, AI assistance in diagnosis can become a great tool for successful screening in telemedicine and teleophthalmology. The aim of our study was to assess the acceptability of AI-based retina screening.
Methods:
This was a prospective non-randomized study performed in the outpatient department of a tertiary eye care hospital. Patients older than 18 years who came for a regular eye check-up or a routine retina screening were recruited in the study. Fundus images of the posterior pole were captured on fundus on a phone camera (REMIDIO™, India) with a built-in AI software (Netra.AI) that can identify normal versus abnormal retina. The patients were then given an 8-point questionnaire to assess their acceptance and willingness toward AI-based screening. We recruited 104 participants.
Results:
We found that 90.4% were willing for an AI-based fundus screening; 96.2% were satisfied with AI-based screening. Patients with diabetes (P = 0.03) and the male population (P = 0.029) were more satisfied with the AI-based screening. The majority (i.e., 97.1%) felt that AI-based screening gave them a better understanding of their eye condition and 37.5% felt that AI-based retina screening prior to a doctor’s visit can help in routine screening
Conclusion:
Considering the current COVID-19 pandemic situation across the globe, this study highlights the importance of AI-based telescreening and positive patient approach toward this technology.
Keywords: Acceptance, artificial intelligence, deep learning in retina, retina screening
A deep learning system (DLS) using artificial intelligence (AI) is emerging as a very promising technology in the future of healthcare diagnostics. Currently, it is being extensively trained in various medical fields such as ophthalmology,[1,2] dermatology,[3] oncology,[4] pathology,[5] and radiology.[6,7] While the concept of telehealth is emerging in every field of medicine, AI assistance in diagnosis can become a great tool for successful screening in telemedicine.
The need for this technology is basically to support the healthcare system across the world. Problems with the existing healthcare system include lack of specialized health services in rural places, economic burden due to regular screening, low doctor-to-patient ratio in most countries, increased working hours for doctors to provide health care services for the increasing aging population, and insufficient public health expenditure. A DLS would aid doctors by effective screening, reducing the burden of screening normal subjects, and reducing the overall economic burden on the healthcare system.
From a practical use point, any new technology has to satisfy two parameters: a) performance and b) acceptability by the people in the society. There is a lot of evidence in the literature about the excellent performance of AI in the field of ophthalmology, especially to detect diabetic retinopathy (DR),[8,9,10,11] age-related macular degeneration (AMD),[12,13] and glaucoma.[14,15] However, the second question remains unanswered. Whether people would be ready to get screened by AI? The aim of our study was to assess the acceptability of AI-based retina screening in our patients.
Methods
This was a prospective nonrandomized cohort study performed in the outpatient department (OPD) of a tertiary eye care center from 2019 to 2020. Institutional scientific and ethical committee board approval was obtained, and the study adhered to the tenets of the Declaration of Helsinki.
Inclusion criteria was patients older than 18 years who attended the general ophthalmology department for a regular eye checkup, or referred for retina screening by a general ophthalmologist, or patients who visited retina clinic of our hospital for the first time, and patients who were able to read and understand the English language. Exclusion criteria were patients with previous history of treatment for retinal disorders and media opacity precluding fundus imaging. Informed consent was obtained from all the patients before recruiting.
After preliminary vision assessment and anterior segment evaluation, patients underwent fundus imaging and AI-based retina screening. Fundus images of posterior pole (45° field of view) were captured using fundus on phone camera (REMIDIO™, India) with an in-built AI software (Netra.AI developed by Lebencare Technologies Pvt Ltd, Singapore) that can identify normal versus abnormal retina (in the posterior pole). Patients were shown their fundus images, and the results of Netra.AI were demonstrated to them. They were then given an 8-point questionnaire. Patients were later seen by a retina specialist, and a detailed evaluation using slit-lamp biomicroscopy and indirect ophthalmoscopy was performed.
Based on the study by Keel et al.,[16] we adopted a similar sample size. We recruited 104 patients in our study. The patient recruitment flowchart is shown in Fig. 1.
Figure 1.
Flowchart depicting patient recruitment
Questionnaire
A preliminary literature review was done to meet the study objective. As there were no previous studies performed to address the main objective, a new questionnaire was formed. Thus, external validation was omitted. The questionnaire focussed on three main aspects: to understand patients’ awareness about AI, to understand their acceptance and attitude, and to understand whether they were willing to replace AI with a doctor.
The questionnaire consisted of the following questions:
Whether the patient underwent prior retina screening
Patient’s awareness about artificial intelligence
Awareness of application of AI in the diagnosis of eye diseases
Willingness to be screened by AI-based algorithm
Satisfaction with AI-based retina screening
Whether AI-based screening would save their time
Whether AI would help improve their understanding of the disease
Whether AI-based retina screening can replace a doctor visit for routine screening
In question 5, patients’ satisfaction level with AI-based retina screening was measured on a 5-point Likert scale ranging from 1 (“highly unsatisfied”) to 5 (“highly satisfied”). The remaining questions were graded based on a binary decision of yes or no.
Statistical analysis
All the data were entered into Microsoft Excel version 16.46 and analyzed on the same. The mean, median, and range of the population age were calculated. The sensitivity, specificity, and positive and negative predictive values of AI in identifying abnormal retina were calculated. Fisher’s exact test was performed to test the correlation between categorical variables by using SPSS software.
Results
Of the 104 patients recruited, there were 63 (60.6%) males and 41 (39.4%) females. The mean age of the participants screened was 53.15 ± 16.41 and 59 years, respectively (range: 18–85 years); 37.5% of the population was below 50 years of age. Of these 104, 56 (53.8%) patients were diabetics and 45 (43.3%) were hypertensive patients.
Out of the 104 patients screened, 30.8% were aware about artificial intelligence in general but only 1.9% were aware of AI-based eye screening. Of these, 36.5% of the participants had prior retina screening. Out of the total subjects who participated, 90.4% of the patients were willing for an AI-based retina screening and 97.1% felt that AI-based screening gave them a better understanding of their eye condition/disease as the fundus images and AI annotations gives them a better knowledge of their disease condition [Table 1]. Moreover, 99% of them felt AI-based screening, especially if made available in outreach centers/optical shops/rural areas, can help save their time, especially for those who were traveling several hundreds of kilometers to consult retina specialists particularly if it is only for screening.
Table 1.
Results based on response to the questionnaire
| Questionnaire | Frequency (%) |
|---|---|
| Did you have prior retina screening (Yes) | 38 (36.5%) |
| Are you aware of artificiaI intelligence (AI) (Yes) | 32 (30.8%) |
| Are you aware of AI application in diagnosing eye diseases (Yes) | 2 (1.9%) |
| Are you willing for AI-based retina Screening (Yes) | 94 (90.4%) |
| Are you satisfied with AI-based retina screening | |
| Not Satisfied | 4 (3.8%) |
| Slightly Satisfied | 26 (25.0%) |
| Satisfied | 67 (64.4%) |
| Highly Satisfied | 7 (6.7%) |
| Do you think AI-based screening can save patient’s time (Yes) | 103 (99.0%) |
| Do you think AI-based retina screening can replace a doctor visit for routine screening (Yes) | 39 (37.5%) |
| Did AI-based screening help you in understanding your disease better (Yes) | 101 (97.1%) |
Further, 96.2% of these patients were satisfied with the AI-based screening, with 25% being slightly satisfied, 64.4% being satisfied, and 6.7% being highly satisfied. The remaining 3.8% were not satisfied with AI-based screening. There were no patients who were highly unsatisfied.
When asked whether AI can replace a doctor, 62.5% of the study population felt that it still cannot replace a doctor. These patients felt that there is a lack of human touch and moral support when evaluated by AI alone or that the technology is not yet developed to that extent to rely on it completely.
There was a positive association between younger age and awareness of AI, Q. No. 2 (Fisher’s exact test- P < 0.001), and Q. No. 8 “Do you think AI-based screening can replace a doctor’s visit for routine screening?” (Fisher’s exact test- P = 0.015). Gender-wise, we found a significant association between men and women in Q. No. 5 and 8. Men were more satisfied with AI-based screening (P = 0.029) than women and felt that AI-based screening can replace a doctor’s visit for routine screening (P = 0.026). Patients with diabetes were more satisfied with AI-based screening than the non-diabetics (P = 0.03). There was no significant difference in responses between patients with normal versus abnormal retinal findings.
Of the 104 patients (208 eyes) screened, 113 eyes had no retinal pathology while 95 eyes had abnormal findings in the posterior pole on clinical examination. Posterior pole analysis by AI revealed 123 eyes as normal and 85 as abnormal (with 3 false positives and 13 false negatives). The overall sensitivity and specificity of AI to identify abnormal retina was 86.3% and 96.3%, respectively. The positive predictive value of AI to identify abnormal fundus was 96.5%, while the negative predictive value was 89.4%. Abnormal peripheral lesions found on clinical evaluation were excluded from analysis as AI was tested only in posterior pole 45° fundus photographs.
Diseases that were identified in these patients included glaucoma (6 patients), diabetic retinopathy (24) [Fig. 2], retinal vein occlusion (4), age-related macular degeneration (4), central serous chorioretinopathy (2), and epiretinal membrane (4). There were two cases of mild nonproliferative diabetic retinopathy (NPDR), which were initially classified as no diabetic retinopathy (DR) by the clinician but reclassified as mild NPDR after the AI results, after reviewing the fundus photographs.
Figure 2.
Lesion identification in a case of diabetic retinopathy by AI platform
Discussion
Very few studies have so far studied the willingness and acceptability of AI-based screening in healthcare.[16,17,18] We conducted this study to understand the patient psychology, especially in a healthcare system like India where facilities are distributed in a highly disproportionate manner, with most of them concentrated in urban cities, while 70% of the rural population are deprived of tertiary care services.
From the questionnaire, it was very interesting to know that most of the patients had a positive perception on AI-based retina screening. Our study shows a positive attitude and acceptance of AI-based screening among patients who visit the hospital for routine eye screening; 90.4% of patients were willing for AI-based screening. Patients with diabetes and men were more satisfied with AI-based screening. Men and younger participants felt that AI-based retina screening can replace a doctor’s visit for routine screening. Of the rest 9.6% patients who were not willing for AI-based screening, the reasons given by them were moral support, human touch, and trust in the doctor. Patients were satisfied with the idea of being screened by optometrists and AI before visiting a retina specialist. They felt that it would prepare them mentally and save time and money for traveling.
Keel et al.[16] studied the feasibility and patient acceptability of AI-based DR screening in an endocrinology outpatient clinic in Australia. In their study, 96% of participants reported that they were satisfied with the AI screening model and 78% reported that they preferred the automated model over the manual telescreening model. Our study adds the comparison of the AI screening model to a hospital-based screening model. Also, the AI that we used can distinguish normal from any abnormal fundus, along with the identification of DR and glaucoma. Thus, it has a wider utility on a screening basis.
Gao et al.[17] published a study to explore the public perception of AI in medical care through a content analysis of social media data. They studied the attitude of the public toward AI in medical care and whether people believe that medical AI can replace human doctors, through a social media platform (Sina Weibo). They found that 59.4% expressed a positive attitude, 34.4% conveyed a neutral attitude, and 6.2% expressed a negative attitude. The main reason for the negative attitude in their study was the immaturity of AI technology was the leading reason for doubt.
Lennartz et al.[18] studied patient perspectives on application of AI-based diagnosis in CT and MRI imaging. They found that acceptance of AI was lower for more severe diseases than for less severe diseases. In addition, patients were significantly more comfortable with the use of AI under the physician’s supervision than without such supervision.
Limitations of our study are limited sample size and inclusion of only patients who visited the hospital, which may not simulate the general population. In addition, with the current imaging technique and existing AI models, there is inability to screen the periphery with AI. We had 8 patients with peripheral retinal degenerations on clinical examination, which was missed by AI as only posterior pole images were taken. However, this will not be reflected in the general population as this study was done in a retina clinic of a tertiary care hospital, where one may find more patients with peripheral retinal degenerations than in the general population where AI may be deployed.
Considering the paradigm shift in AI and retinal imaging, it would be of great benefit to understand the patient’s mindset and acceptability toward the newer automated screening techniques that we tried to evaluate in our study. With the improving ability to capture undilated fundus images with newer cameras, this AI integration model can be introduced as kiosk set-ups in various public places for eye screening and even help health workers for early identification and referral from the grass-root level. Our study has shown better acceptance and more positive attitude from the patients compared to the previous studies.[16,17,18] Considering the current COVID-19 pandemic across the globe, this study highlights the importance of AI-based telescreening and positive patient approach toward this technology.
Conclusion
Along with developing newer AI algorithms that would help in screening in telemedicine, it is of great value to understand the patient acceptability and willingness toward the approach.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
References
- 1.Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;304:649–56. doi: 10.1001/jama.2016.17216. [DOI] [PubMed] [Google Scholar]
- 2.Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci. 2016;57:5200–6. doi: 10.1167/iovs.16-19964. [DOI] [PubMed] [Google Scholar]
- 3.Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–8. doi: 10.1038/nature21056. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ehteshami Bejnordi B, Veta M, Johannes van Diest P, van Ginneken B, Karssemeijer N, Litjens G, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA. 2017;318:2199–210. doi: 10.1001/jama.2017.14585. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Rashidi HH, Tran NK, Betts EV, Howell LP, Green R. Artificial intelligence and machine learning in pathology: The present landscape of supervised methods. Acad Pathol. 2019;6 doi: 10.1177/2374289519873088. 2374289519873088. doi: 10.1177/2374289519873088. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Lakhani P, Sundaram B. Deep learning at chest radiography: Automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 2017;284:574–82. doi: 10.1148/radiol.2017162326. [DOI] [PubMed] [Google Scholar]
- 7.Ting DSW, Yi PH, Hui F. Clinical applicability of deep learning system in detecting tuberculosis with chest radiography. Radiology. 2018;286:729–31. doi: 10.1148/radiol.2017172407. [DOI] [PubMed] [Google Scholar]
- 8.Ting DSW, Cheung CY, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318:2211–23. doi: 10.1001/jama.2017.18152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Li Z, Keel S, Liu C, He M. Can artificial intelligence make screening faster, more accurate, and more accessible? Asia Pac J Ophthalmol (Phila) 2018;7:436–41. doi: 10.22608/APO.2018438. [DOI] [PubMed] [Google Scholar]
- 10.Ting DSW, Peng L, Varadarajan AV, Keane PA, Burlina PM, Chiang MF, et al. Deep learning in ophthalmology: The technical and clinical considerations. Prog Retin Eye Res. 2019;72:100759. doi: 10.1016/j.preteyeres.2019.04.003. doi: 10.1016/j.preteyeres.201904.003. [DOI] [PubMed] [Google Scholar]
- 11.Shah P, Mishra DK, Shanmugam MP, Doshi B, Jayaraj H, Ramanjulu R. Validation of deep convolutional neural network-based algorithm for detection of diabetic retinopathy –Artificial intelligence versus clinician for screening. Indian J Ophthalmol. 2020;68:398–405. doi: 10.4103/ijo.IJO_966_19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Burlina P, Pacheco KD, Joshi N, Freund DE, Bressler NM. Comparing humans and deep learning performance for grading AMD: A study in using universal deep features and transfer learning for automated AMD analysis. Comput Biol Med. 2017;82:80–6. doi: 10.1016/j.compbiomed.2017.01.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Grassmann F, Mengelkamp J, Brandl C, Harsch S, Zimmermann ME, Linkohr B, et al. A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology. 2018;125:1410–20. doi: 10.1016/j.ophtha.2018.02.037. [DOI] [PubMed] [Google Scholar]
- 14.Liu H, Li L, Wormstone IM, Qiao C, Zhang C, Liu P, et al. Development and validation of a deep learning system to detect glaucomatous optic neuropathy using fundus photographs. JAMA Ophthalmol. 2019;137:1353–60. doi: 10.1001/jamaophthalmol.2019.3501. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Li Z, He Y, Keel S, Meng W, Chang RT, He M. Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology. 2018;125:1199–206. doi: 10.1016/j.ophtha.2018.01.023. [DOI] [PubMed] [Google Scholar]
- 16.Keel S, Lee PY, Scheetz J, Li Z, Kotowicz MA, MacIsaac RJ, et al. Feasibility and patient acceptability of a novel artificial intelligence-based screening model for diabetic retinopathy at endocrinology outpatient services: A pilot study. Sci Rep. 2018;8:4330. doi: 10.1038/s41598-018-22612-2. doi: 10.1038/s41598-018-22612-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Gao S, He L, Chen Y, Li D, Lai K. Public perception of artificial intelligence in medical care: Content analysis of social media. J Med Internet Res. 2020;22:e16649. doi: 10.2196/16649. doi: 10.2196/16649. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Lennartz S, Dratsch T, Zopfs D, Persigehl T, Maintz D, Große Hokamp N, et al. Use and control of artificial intelligence in patients across the medical workflow: Single-center questionnaire study of patient perspectives. J Med Internet Res. 2021;23:e24221. doi: 10.2196/24221. doi: 10.2196/24221. [DOI] [PMC free article] [PubMed] [Google Scholar]