Abstract
Background
Home monitoring of hyperacuity allows early detection of progression in exudative neovascular age-related macular degeneration (nvAMD) and diabetic macular oedema (DMO). However, false alarms may pose a significant burden to both patients and healthcare professionals alike.
Purpose
To assess the false alarm rate and positive predictive value of smartphone-based home monitoring of nvAMD and DMO.
Methods
Patients treated with anti-angiogenic therapy in a pro re nata scheme for nvAMD or DMO at the Medical Retina service (Lucerne, Switzerland) between March and June 2016 were included in this prospective cohort study. The home monitoring test Alleye (Oculocare Ltd, Switzerland) provided a session score from 0–100 in addition to a traffic-light system feedback via the smartphone application. Three consecutive “red” scores were considered as a positive test or alarm signal. Specificity, 1-specificity (false alarm rate) and the predictive value for optical coherence tomography-based disease progression were analysed.
Results
73 eyes of 56 patients performed 2258 tests in 222 “follow-up periods”. Progression was observed in 141 periods (63.5%). The specificity of the test was 93.8% (95% CI: 86.2–98.0%), the false alarm rate 6.1% (95% CI: 2.0–13.8%), and the positive predictive value 80.0% (95% CI: 59.3–93.2%) for the detection of progression.
Conclusion
False alarm rates for the detection of progression in macular disease via home monitoring is low. These findings suggest that home monitoring may be a useful adjunct for remote management of nvAMD and DMO.
Subject terms: Diagnosis, Health services
Background
The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic of 2020 has fuelled a discussion regarding the provision of comprehensive care for macular disease patients whilst limiting face-to-face visits. Globally, most health systems are not yet in a position to conduct a post mortem into the consequences of postponed and cancelled appointments, affecting up to 80% of patients who attend eye clinics [1]. Eye care professionals worldwide have been challenged to risk stratify patients using information available from health records of a patients’ last visit. The key limitation of this forward triage strategy is that the clinical information used for decision making may be incomplete or out-of-date. The use of telemedicine as a strategy to collect more contemporaneous clinical data has seen a sharp rise across all specialties in healthcare with uptake of schemes such as video-consultations and remote monitoring [2–5]. Although this addresses the priority in which patients should be seen whilst conserving resources in the time of crisis, there remains the question- what is a safe follow-up interval between hospital visits?
In Ophthalmology, the lack of clinic capacity for the monitoring and treatment of chronic eye diseases such as glaucoma, diabetic macular oedema (DMO) and age-related macular degeneration (AMD) is a long-standing issue, particularly with the reliance on imaging assessments requiring specialist equipment such as visual fields analysers and optical coherence tomography (OCT). In recent years, the introduction of Virtual Clinics and new therapeutic approaches, aimed at extending the intervals between treatments looks to address this. Home-monitoring as an adjunct could provide “in-between visit” clinical data that may be used to personalise a patient’s follow-up interval according to need.
Today, valid home monitoring of visual function could be achieved by software with high usability running on patients’ mobile devices [6]. Our current offering to patients for home monitoring of their eye condition consists of the paper-based Amsler grid, despite its well documented limitations [6–8]. Namely, technical obstacles such as difficulties with fixation in home monitoring experiments have demonstrated its limited validity in the hands of our patients, leading to a highly variable diagnostic accuracy [9].
Two Food and Drug Administration approved mobile medical apps are currently available for the home monitoring of hyperacuity: myVisiontrack (mVT) [10] and Alleye [11, 12], testing a 3 and 12.7 degree visual field, respectively. It has been shown that the assessment of hyperacuity allows for the early detection of progression in patients with exudative neovascular age-related macular degeneration (nvAMD) and DMO [12]. However, the burden generated through false alarms in the long-term monitoring situation must not be underestimated, for both healthcare professionals and patients.
In this study, we addressed the extent of this false alarm rate and the positive predictive value of regular home monitoring with Alleye in prospective follow-up study of patients currently treated with anti-angiogenic therapy in a pro re nata scheme.
Methods
Ethics, study design and setting
This prospective cohort study was performed with consecutive patient enrolment. All patients attending the injection clinic between March and June 2016 at the Medical Retina Service of the Eye Clinic of the Cantonal Hospital Lucerne were eligible to be involved. The study was approved by the local Ethics Committee. Participating patients were provided with written information, and written informed consent was obtained.
Patient recruitment and enrolment
Patients with exudative nvAMD or DMO in a pro re nata (PRN) treatment regime were provided with written information in advance of their appointment. Subsequently, the clinic research fellow assessed the selection criteria on an individual case by case basis prior to the clinic consultation. In eligible cases, patients were provided with more detailed verbal and written information regarding the study and were asked as to whether they were willing to participate.
Selection criteria
Patients with exudative nvAMD or DMO under treatment with anti-angiogenic therapy (ranibizumab (Lucentis®) or aflibercept (Eylea®)) who had completed treatment induction qualified for inclusion if they presented with a best-corrected visual acuity (BCVA) of at least 60 Early Treatment Diabetic Retinopathy Study (ETDRS) letters. If both eyes were affected by nvAMD or DMO, both eyes were included in the study. Patients with a neurological or physical condition that impeded them from using a mobile device to adequately perform the home monitoring task were excluded.
Examination setting
Relevant clinical characteristics of each patient were recorded. Following ophthalmological routine assessments, standardised and protocol-based instructions for the training of hyperacuity testing were provided by healthcare professionals. After a training period involving repeated testing, patients were asked to perform a baseline assessment in the clinic and home monitoring. Patients performed the test whilst wearing their own glasses with each eye tested individually. Home monitoring was either conducted on an iPod Touch® (Apple Inc.) that was provided by the hospital or on the patients’ mobile device once the application was downloaded from Apple’s App or the Android Google Play store. In addition, all patients received an instruction manual for the application’s usage, and healthcare professionals provided technical assistance during clinical follow-up visits if needed. Patients were asked to conduct home monitoring twice weekly and to return the device at each follow-up visit.
The home monitoring test
The home monitoring test is based on a computerised version of a hyperacuity alignment task (Fig. 1) and has been described in detail elsewhere [12]. Hyper-, or Vernier acuity is the ability to detect a misalignment that deteriorates due to morphological retinal changes causing metamorphopsia [13, 14]. Therefore, measurement of hyperacuity can provide information regarding retinal changes and even enable the quantification of metamorphopsia by presenting stimuli with artificial distortions of different amplitudes [15].
Fig. 1. The task of the Alleye test is to place the middle of three points on the invisible connecting line between the outer points.
A total of 12 points must be placed. The Alleye test is performed with one eye. You cover the eye that you are not testing. To test, place both elbows on a table and hold your mobile device approx. 30 cm in front of your face with both hands.
Several studies have demonstrated the superiority of active over passive hyperacuity testing (Amsler grid) in the detection of metamorphopsia [13, 16–18].
In this active alignment task, three black dots with a one millimetre diameter were presented on a white screen. Alignments were made on four axes; including vertical, horizontal, left and right oblique axes. Patients are tasked with aligning the randomly misaligned dot between two fixed dots to form one straight axis precisely. This is achieved through moving the central dot only. Patients are able to control the middle dot via arrow-buttons on the screen of the touch-based user interface. Once patients were satisfied with the placement of a dot, they confirmed their choice and entered the result. This task was repeated three times for each of the four axes, resulting in a total of 12 inputs per test. The test was performed monocularly after having covered the non-tested eye. Technical specifications are available elsewhere (https://alleye.io/#/product-alleye).
Assessment of disease progression
All study participants were managed in a PRN protocol and routinely returned for follow-up visits between 28 and 35 days. At this visit, patients received ETDRS BCVA testing, biomicroscopy and imaging in the form of an OCT scan. A spectral domain OCT (Spectralis® Heidelberg Engineering GmbH, Heidelberg) was used to obtain four b-scans with a length of six millimetres each. A masked, board-certified senior retinal specialist (MKS) assessed the OCT scan for progression of sub- or intraretinal fluid, and serous or fibrovascular pigment epithelial detachment. Based on this assessment, OCTs were classified as either “progression”; “stable” or “improved” compared to the previous scan. Treatment decisions and date of the intravitreal, anti-angiogenic injection were also recorded.
Statistical analysis
We defined a “monitoring period” as the time interval between two visits at the injection clinic. We excluded measurements with extreme values (changes in the score of >±50 score points from previous measurement), incomplete periods (i.e. periods that did not end with a follow-up visit) and eyes with less than one monitoring period or periods in which patients performed less than one measurement per week (8 periods (3.5%). The mean age and BCVA of patients excluded due to this was similar to the included patient groups. We also excluded measurements with a recorded length of all 12 tasks <40 s as this indicated inappropriate testing. Using these selection criteria, we removed <20% of all data points stored in the database. Continuous variables were summarised with means and standard deviations (SD) or medians and 25th–75th interquartile ranges (IQR). Dichotomous variables were described as rate and percentages.
We defined the outcome progression as “present” versus “absent” by collapsing the “stable” and “improved” category. As patients contributed a number of tests during the “follow up periods” (the time between two follow-up visits), we created a panel dataset in which we specified a user identification per eye and patient, the corresponding period, and the day within the period. The unit of observation was the period per eye. Per period and eye we obtained the score and recorded the occurrence of three sequential red results – defined as a sustained relevant drop of the score as provided by the test, which defined a positive test result or “alarm”.
We correlated the occurrence of a positive test result during a monitoring period with the outcome “progression” at the next follow-up visit. The specificity, defined as the percentage of eyes without progression in the next follow-up visit who had a negative result, the false alarm rate defined as 1-specificity and the positive predictive value, defined as the percentage of positive tests with the outcome progression were calculated. We performed subgroup analyses for patients older than 85 years and patients with DMO. Analyses were performed using the Stata 16.1 statistical software package (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LP.).
Results
Sixty-two patients were enroled in the study. Six patients (10%) were excluded over the course of the study as they either failed to provide home monitoring results or terminated the study early (within the one month minimal observation and measurement period). We analysed 73 eyes of 56 patients. The group of DMO included 13 eyes (17.8%). The mean age was 73.2 years (SD 10.7, range 47–88) and the median number of previous intravitreal injections was 13 (IQR: 5–22). The mean BCVA at study entry was 77.5 letters (SD 7.4, range 62–93) on the ETDRS chart. The DMO group was younger (−15.9 years (95% CI: −21.8 to −10.0; p < 0.001)), but BCVA at study entry was similar (−1.3 letters (95% CI: −6.3 to 3.8; p = 0.619)).
Patients contributed 2258 tests in 222 monitoring periods. Progression was observed in 141 monitoring periods (63.5%).
The average score value across all periods was 58.0 (SD 18.1) and ranged from 3 to 100 points. On average, each patient performed 6.4 tests per period and contributed 2.5 (SD 1.4) observation periods. Per period, patients performed an average of 10.8 (SD 4.8) tests. Surprisingly, 70% of patients continued with home monitoring of their macular function after the official termination of the study.
Specificity, false alarms and positive predictive value
The specificity was 93.8% (95% CI: 86.2–98.0%), the corresponding false alarm rate was 6.1% (95% CI: 2.0–13.8%). The positive predictive value (the probability that a patient with a positive result had disease progression in the next clinical follow-up visit) was 80.0% (95% CI: 59.3–93.2%).
Subgroup analyses
Elderly subjects (age >85 years at study entry) providing 16 eyes (38 observation periods) performed slightly worse. The specificity was 85.7% (95% CI: 57.2–98.2%) and the corresponding false alarm rate was 14.3% (95% CI: 1.8–42.8%). The predicted probability for a progression dropped to 60.0% (95% CI: 14.7–94.7%).
Patients performing >12 tests/period were more likely to show a false alarm (16.7% (95% CI: 3.6–41.4%). The diagnostic performance of eyes with DMO providing 51 observation periods was similar to eyes with AMD (specificity 94.4% (95% CI: 72.7–99.9%), predicted probability of progression 75.0% (95% CI: 19.4–99.4%).
Discussion
Main findings
To the best of our knowledge, this is the first study assessing the performance of smartphone-based hyperacuity assessment through home monitoring in patients with active macular disease requiring intravitreal injections. With the low threshold set of three consecutive red results deemed as a positive test result or “alarm”, home monitoring of hyperacuity showed a reassuringly low false alarm rate. A mere 6 of 100 patients with nvAMD or DMO had a false alarm during a defined monitoring period. If the test was positive, 80% of patients showed signs of anatomical progression at the next follow-up visit. In a screening environment however, a drop of 25 score points or more had a high accuracy for discriminating between disease and age-matched controls [12, 19]. Test performance characteristics were lower in older patients and those that tested more than recommended.
Strengths and limitations
The pragmatic set-up of this study has both strengths and limitations. Given that it is embedded in a real-world clinical setting, the results obtained in this study are more likely to reflect the realistic supplementary nature that home monitoring provides as auxiliary tool for patient management. Conversely, the lack of a strict protocol and the absence of other measures to improve the quality of the testing may have introduced substantial confounders to the data. There is also potential that this study may underestimate the value of home monitoring in patients with less advanced disease through imposing relatively few exclusion criteria. However, by including patients with a BCVA more than 60 letters, we selected a group that usually has a higher performance than patients with a lower visual acuity than this.
Moreover, OCT scans have not been conducted on the day of the alarm, but during the next routine visit. Therefore, we cannot exclude that the alarm signal may have preceded an actionable progression of disease (i.e. SRF or IRF).
Furthermore, OCT scan gradings were not verified by a second assessor. Gradings were performed by a masked retinal specialist as per current guidelines. However, the lack of standardisation and verification may have introduced further uncertainty into the data. These limitations may have distorted the results of this study and there is potential that they may have introduced bias.
The engagement of this elderly patient cohort with mobile technology to perform home testing, coupled with the fact a high proportion of these patients continued with home monitoring following the official cessation of the study indicates that elderly patients with impaired vision are not only highly capable of technological participation in a way that is not always acknowledged, but that they are also motivated to perform self-measurement tests of their macular function independently and to take ownership of their own eye health [9, 10]. As suggested by Thomas et al., a qualification test within the ophthalmic practice could help in selecting patients that qualify for home monitoring [20].
Implications for research
It is imperative that further evaluation of this application occurs by individuals with no affiliation to Alleye working in a multitude of differing clinical settings. In view of the results obtained here, further studies should be designed to quantify clinical efficacy when using home monitoring in the management of patients with macular disease both with and without exudative neovascularisation. Future work involving the evaluation of Alleye and home monitoring in general, particularly in the optimisation of treatment outcomes in a treat and extend management regimen and in the context of drugs with prolonged effects would be invaluable, with the promise of great potential to both patients and healthcare services. It is important to prioritise the evaluation of patient reported outcomes for those receiving care with the adjunct of home monitoring compared to standard care in health service research settings.
Implications for practice
We chose this cohort of patients under anti-angiogenic treatment in a PRN scheme due to the availability of frequent clinical follow-up data. However, in clinical practice, home monitoring may be of particular value in a treat and extend therapy protocol, where it could be used to provide immediate feedback and reassurance to patients regarding their level of macular disease, in addition to the early detection of visual deterioration, if needed.
It is important that home monitoring is viewed as an aid during the intervals between visits as it is not intended to replace clinical judgement; diagnosis and the management of clinical decision-making remain the responsibility of the eye care professional. Clinicians may decide to access individual patients’ home monitoring data via a web application as needed only, or -in high risk situations- they may indeed choose to receive notification alerts when patients generate alarms. Remote access to a patient’s home test values may therefore allow eye care professionals to remain connected with their patients from afar, enabling truly individual patient care plans to be crafted with scheduling and management decisions fine-tuned through a symbiotic relationship between doctor and patient. Furthermore, home monitoring is ideally suited for chronic care models by adequately informing and empowering patients with macular disease to self-care [21]. Improved patient education and awareness of the disease could have a major impact on the prevention of vision impairment [22].
Conclusions
The false alarm rate for the detection of disease progression in macular disease by home monitoring is low. These findings indicate that home monitoring is a useful adjunct to telemedicine in the remote management of patients with nvAMD and DMO.
Summary
What was known before
The lack of clinic capacity for the monitoring and treatment of chronic macular diseases such as diabetic macular oedema and age-related macular degeneration is a long-standing issue.
It has been shown that the assessment of hyperacuity allows for the early detection of progression in patients with macular disease.
Furthermore, this visual function can be monitored at home through software running on patients’ mobile devices.
What this study adds
The burden generated through false positives (or false alarms) in the long-term monitoring situation for both healthcare professionals and patients must not be underestimated.
This paper therefore reports on the false alarm rate and the positive predictive value of smartphone-based home monitoring in macular disease.
We found that a mere 6 of 100 patients had a false alarm during a defined monitoring period and if the test was positive, 80% of patients showed signs of anatomical progression at the next follow-up visit.
These findings indicate that home monitoring is a useful adjunct to telemedicine in the remote management of patients with macular disease featuring a minimal burden of false alarms.
Compliance with ethical standards
Conflict of interest
LMB, KL, and MKS are founders of Oculocare medical Inc., which develops innovative products in eye care, such as the self-monitoring test described in this paper. The other authors declare that they have no conflict of interest.
Footnotes
The original online version of this article was revised due to a retrospective Open Access cancellation.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Change history
3/31/2021
A Correction to this paper has been published: 10.1038/s41433-021-01512-2
References
- 1.Mehrotra A, Chernew M, Linetsky D, Hatch H, Cutler D. The impact of the COVID-19 pandemic on outpatient visits: a rebound emerges,” To the Point (blog). Commonwealth Fund. 2020; 10.26099/ds9e-jm36.
- 2.Faes L, Bachmann LM, Sim DA. Home monitoring as a useful extension of modern tele-ophthalmology. Eye (Lond). 2020;34:1950–3. [DOI] [PMC free article] [PubMed]
- 3.Kang S, Thomas PBM, Sim DA, Parker RT, Daniel C, Uddin JM. Oculoplastic video-based telemedicine consultations: Covid-19 and beyond. Eye (Lond). 2020;34:1193–5. [DOI] [PMC free article] [PubMed]
- 4.Kern C, Fu DJ, Kortuem K, Huemer J, Barker D, Davis A, et al. Implementation of a cloud-based referral platform in ophthalmology: making telemedicine services a reality in eye care. Br J Ophthalmol. 2020;104:312–7. doi: 10.1136/bjophthalmol-2019-314161. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Kortuem K, Fasler K, Charnley A, Khambati H, Fasolo S, Katz M, et al. Implementation of medical retina virtual clinics in a tertiary eye care referral centre. Br J Ophthalmol. 2018;102:1391–5. doi: 10.1136/bjophthalmol-2017-311494. [DOI] [PubMed] [Google Scholar]
- 6.Faes L, Bodmer NS, Bachmann LM, Thiel MA, Schmid MK. Diagnostic accuracy of the Amsler grid and the preferential hyperacuity perimetry in the screening of patients with age-related macular degeneration: systematic review and meta-analysis. Eye. 2014;28:788–96. doi: 10.1038/eye.2014.104. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Schmidt-Erfurth U, Chong V, Loewenstein A, Larsen M, Souied E, Schlingemann R, et al. Guidelines for the management of neovascular age-related macular degeneration by the European Society of Retina Specialists (EURETINA) Br J Ophthalmol. 2014;98:1144–67. doi: 10.1136/bjophthalmol-2014-305702. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Simunovic MP. Metamorphopsia and its quantification. Retina. 2015;35:1285–91. doi: 10.1097/IAE.0000000000000581. [DOI] [PubMed] [Google Scholar]
- 9.Crossland M, Rubin G. The Amsler chart: absence of evidence is not evidence of absence. Br J Ophthalmol. 2007;91:391–3. doi: 10.1136/bjo.2006.095315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kaiser PK, Wang YZ, He YG, Weisberger A, Wolf S, Smith CH. Feasibility of a novel remote daily monitoring system for age-related macular degeneration using mobile handheld devices: results of a pilot study. Retina. 2013;33:1863–70. doi: 10.1097/IAE.0b013e3182899258. [DOI] [PubMed] [Google Scholar]
- 11.Schmid MK, Faes L, Bachmann LM, Thiel MA. Accuracy of a self-monitoring test for identification and monitoring of age-related macular degeneration: a diagnostic case-control study. Open Ophthalmol J. 2018;12:19–28. doi: 10.2174/1874364101812010019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Schmid MK, Thiel MA, Lienhard K, Schlingemann RO, Faes L, Bachmann LM. Reliability and diagnostic performance of a novel mobile app for hyperacuity self-monitoring in patients with age-related macular degeneration. Eye. 2019;33:1584–9. doi: 10.1038/s41433-019-0455-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Goldstein M, Loewenstein A, Barak A, Pollack A, Bukelman A, Katz H, et al. Results of a multicenter clinical trial to evaluate the preferential hyperacuity perimeter for detection of age-related macular degeneration. Retina. 2005;25:296–303. doi: 10.1097/00006982-200504000-00008. [DOI] [PubMed] [Google Scholar]
- 14.Midena E, Vujosevic S. Metamorphopsia: an overlooked visual symptom. Ophthalmic Res. 2015;55:26–36. doi: 10.1159/000441033. [DOI] [PubMed] [Google Scholar]
- 15.Schwartz R, Loewenstein A. Early detection of age related macular degeneration: current status. Int J Retin Vitreous. 2015;1:20. doi: 10.1186/s40942-015-0022-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Alster Y, Bressler NM, Bressler SB, Brimacombe JA, Crompton RM, Duh YJ, et al. Preferential hyperacuity perimeter (PreView PHP) for detecting choroidal neovascularization study. Ophthalmology. 2005;112:1758–65. doi: 10.1016/j.ophtha.2005.06.008. [DOI] [PubMed] [Google Scholar]
- 17.Isaac DL, Avila MP, Cialdini AP. Comparison of the original Amsler grid with the preferential hyperacuity perimeter for detecting choroidal neovascularization in age-related macular degeneration. Arquivos brasileiros de oftalmologia. 2007;70:771–6. doi: 10.1590/s0004-27492007000500009. [DOI] [PubMed] [Google Scholar]
- 18.Loewenstein A, Malach R, Goldstein M, Leibovitch I, Barak A, Baruch E, et al. Replacing the Amsler grid: a new method for monitoring patients with age-related macular degeneration. Ophthalmology. 2003;110:966–70. doi: 10.1016/S0161-6420(03)00074-5. [DOI] [PubMed] [Google Scholar]
- 19.Faes L, Bachmann LM, Schmid MK, Thiel MA. The diagnostic accuracy of the patient self-test Alleye to detect diabetic macular edema. ePoster EURETINA. 2019: 117.
- 20.Thomas M, Wolfson Y, Zayit-Soudry S, Bressler SB, Bressler NM. Qualifying to use a home monitoring device for detection of neovascular age-related macular degeneration. JAMA Ophthalmol. 2015;133:1425–30. doi: 10.1001/jamaophthalmol.2015.3684. [DOI] [PubMed] [Google Scholar]
- 21.Markun S, Dishy A, Neuner-Jehle S, Rosemann T, Frei A. The chronic care for wet age related macular degeneration (CHARMED) study: a randomized controlled trial. PLoS One. 2015;10:e0143085. doi: 10.1371/journal.pone.0143085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Keane PA, de Salvo G, Sim DA, Goverdhan S, Agrawal R, Tufail A. Strategies for improving early detection and diagnosis of neovascular age-related macular degeneration. Clin Ophthalmol. 2015;9:353–66. doi: 10.2147/OPTH.S59012. [DOI] [PMC free article] [PubMed] [Google Scholar]