Multi-grader validation of the telemedicine retinopathy of prematurity severity score

Abstract

We aimed to statefully validate Retinopathy of Prematurity (ROP) grading by evaluating intergrader variability over age-matched simulated exams. In a prospective retinal photographic grading cohort study, both eyes of twenty patients with five images per eye per visit were graded by twelve expert graders across six weekly visits. For each eye visit, ROP experts graded the Zone, Stage, and Plus, from which the Telemedicine ROP Severity Score (tROP-SS) was calculated. Graders retained knowledge of prior assessments for a given eye and followed International Classification of ROP rules. Intergrader coefficient of variation (CoV) for Zone, Stage, Plus, and tROP-SS remained below 20% across six weeks of age-matched visits. Zone showed the most changes in CoV (p < 0.001) with a minimum at week three of 6.0% (95% CI: 3.0-8.9%). Stage (9.2–13.1%) did not significantly vary over time. CoV of Plus gradually increased (p = 0.002) from week 1 (1.2%, 95% CI: 0.3-2.0%) to week 6 (3.3%, 95% CI: 1.7–4.9%). Agreement for tROP-SS varied over time (p < 0.001) and reached a local minimum at week three (10.5%, 95% CI: 7.3–13.8%). High intergrader agreement in a stateful data set validates the use of tROP-SS as a scoring system to capture week-to-week ROP progression.

Introduction

Retinopathy of prematurity (ROP) remains a leading cause of preventable childhood blindness,¹ but the rapid expansion of telemedicine-based screening has transformed how at-risk infants are monitored and managed worldwide. As telemedicine for ROP management continues to grow, especially in low- and middle- income countries.^2,3 there is an increasing need for robust scoring systems that reflect disease status and trajectory to a multidisciplinary care team and patients’ families.

The Telemedicine ROP Severity Score (tROP-SS) was developed to integrate the three variables that initiate treatment–Zone, Stage, Plus – into a single value that could be tracked over time and reflect risk for intervention from either Type 1 or Aggressive ROP (AROP) disease. In the tROP-SS system, a score between 55 and 85 is consistent with Treatment Warranted ROP (TW-ROP) and consideration for intervention. Scores above 89 represent screening failures since retinal detachment has already occurred. The tROP-SS is based upon two central tenets: independent weighting of ROP features and longitudinal tracking.⁴.

The first tenet – independent weighting – is predicated on the asymmetric role certain ROP features bear on treatment indications. In the natural history arm of the CRYOROP study, Zone I vasculature, irrespective of the presence of Stage or Plus, conferred approximately 35% risk of developing Threshold disease.⁵ Therefore, Zone I is allotted 30 points in tROP-SS. For similar reasons, Stage 3 and Plus disease are weighted at 25- and 30-points in tROP-SS, respectively. AROP is characterized by rapid progression, Plus disease and vaso-obliteration.⁶ Given considerable risk associated with AROP and heterogeneity in grader agreement, the score for AROP is determined by the cumulative of Zone I, Stage 3, and Plus disease—or 85-points.

The second tenet of tROP-SS – longitudinal tracking – is made possible by integrating independent features into one tROP-SS that can dynamically change per eye over subsequent visits. This progression can be illustrated via a plot which displays both the proximity of tROP-SS to the intervention point (≥ 55 points) and the slope of relative score changes illustrating disease “Tempo” over time. Tempo is defined as rate of change of ROP, with horizontal slopes of tROP-SS over time reflecting stabilization, positive slopes reflecting worsening, and negative slopes reflecting improvement. The steepness of the slope reflects how rapidly the Tempo is changing (Fig. 1).

Fig. 1.

The Telemedicine Retinopathy of Prematurity Severity Score (t-ROP-SS) is graphically represented as a function of time in both the right eye (left panel) and left eye (right panel in this theoretical example. Tempo is manifested as disease worsening with a steadily increasing upward slope beginning on 1/19/24 reaching its apex on 1/24/24 when it crossed the TW-ROP inflection of 55 points (dashed red line), and vascular endothelial growth factor inhibitor was given. The left eye responded to treatment more rapidly than the right eye, as manifested by steeply downward sloping lines from 1/24/24 to 1/29/24, and then stabilized.

Retrospective analyses in large cohorts of at-risk infants, such as the SUNDROP and TELEROP databases, demonstrate that tROP-SS can return a score for all patients and more accurately identifies treatment-warranted ROP compared to the modified ROP Activity Score (mROP-ActS).⁴ In addition, tROP-SS provides a more granular assessment of disease by incorporating features such as Posterior Zone II involvement, pre-Plus disease, and regression.

There is a significant need to validate scoring systems like tROP-SS in clinical practice. Most validation methodologies are stateless in that they treat each image as a fresh start, not remembering any previous grades for the given image. However, this does not accurately simulate clinical practice, where graders retain knowledge of prior images and grades that in turn affect subsequent judgements in a ‘stateful’ fashion. Moreover, teleophthalmology-based scoring systems should reflect the totality of ROP in the eye by including all standard retinal images. To assess only one and ignore the other four standard retinal images might inaccurately reflect the totality of ROP in an eye. Grading in accordance with the International Classification of ROP, Third Edition (ICROP3) rules is done per eye, as Early Treatment ROP (ETROP) found only 82% symmetry in ROP disease severity between eyes.⁷.

Herein we report the results of the tROP-SS multi-grader prospective validation within a simulated cohort of 20 age matched patients graded prospectively in real time over a six week period by ROP experts. Our primary outcome of interest was intergrader agreement of tROP-SS, Zone, Stage, and Plus, with follow-up analysis to examine if the observed agreement was robust to changes over simulated weekly consecutive visits.

Methods

The study described adheres to the Declaration of Helsinki with approval from the Stanford Institutional Review Board (IRB 8752). Legal guardians of all infants whose photos were included in the study provided informed consent for all retinal imaging, as well as the use of images for research and publication.

Study setting and design

This prospective study used a longitudinal validation set of retinal images from the Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP) database.⁸ The validation dataset consisted of deidentified wide-field retinal images from 40 eyes of 20 infants eligible for ROP screening and imaged over six consecutive weeks in each eye, matched for PMA from 31 weeks to 37 weeks. The image sets per eye consisted of the following six images: anterior (n = 1) and fundus (n = 5, optic nerve centered, nasal, superior, inferior, and temporal).

Twelve pediatric retinal specialists prospectively analyzed twenty patients weekly for six consecutive weeks. Two eyes were evaluated per subject and for each eye, graders observed one anterior image and five retinal images (optic nerve centered, superior, inferior, temporal, nasal) per each eye. This resulted in a total of 1440 images per cohort (20 subjects, 2 eyes, 6 images per eye, 6 sessions). Graders were required to denote Zone, Stage and Plus per International Classification of ROP, Third Edition (ICROP3) guidelines. Namely, ICROP3 stipulates that the lowest Zone (e.g., most posterior extent of vascularization anywhere in the eye determines the Zone), the highest Stage (e.g., highest Stage anywhere in the eye determines the Stage), and greatest Plus (e.g., highest level of Plus anywhere in the eye determines the Plus status) per eye should be recorded at each evaluation.⁶.

After findings were recorded, a tROP-SS was generated for each eye/visit/grader. Graders observed photos of the same set of forty eyes, capturing the weekly progression and development, instead of grading unrelated images in isolation. Furthermore, graders were allowed to look back at previous photos and grades of a given eye to assess disease trajectory.

Since less than 2% of infants with ROP generally require treatment, the study was enriched to enable analysis of whether experts were able to determine presence of disease. Hence, the validated dataset was designed with 30% of patients eventually developing treatment-warranted ROP.

Statistical analysis

For each eye in each visit, intergrader variability was assessed via the calculation of a coefficient of variation (CoV), which represents variability as a function of standard deviation with respect to the mean score. Higher CoV is associated with greater variability. A score of 0 for CoV implies no variability from the mean for the graders. Zone (Zone I, Posterior Zone II, Zone II, Zone III) and Stage (Incomplete Vascularization, 1, 2, 3) were assigned values ranging from 1 to 4 per category. Plus (No Plus Disease, Pre-Plus Disease, Plus Disease) was assigned values ranging from 1 to 3 per category. The tROP-SS was scored on a scale of 0-100. CoV was calculated by taking the standard deviation for the category scores for each eye at each visit (Zone, Stage, Plus and tROP-SS), dividing each by the mean of all the grader’s scores, and multiplying by 100 to represent the percentage CoV.

Mean and 95% confidence intervals (CIs) for the CoVs for left and right eyes in each category (Zone, Stage, Plus, tROP-SS) across the six screening weeks were visualized to understand the relationship between grader variability and screening PMA. A repeated measures ANOVA was conducted for all categories to evaluate if changes in the percent CoV across time points (weeks 1 through 6) for all eyes combined (n = 40 eyes) was significant. Mauchly’s test was used to assess if the assumption of sphericity was violated. If sphericity was violated, the Greenhouse-Geisser corrected p-values were used. While the repeated measures ANOVA assess if grader agreement significantly varies overall across the six weeks, it cannot specify exactly when those shifts occur. Pairwise comparisons are necessary to pinpoint these specific week-to-week differences. Follow up statistical analysis for week-to-week changes in CoV were done with ad-hoc pairwise T-tests with the Bonferroni correction. P values less than 0.05 were deemed significant. Analyses and plots were performed with Python version 3.11.8 software (Python Software Foundation).

Results

All 4 variables–Zone, Stage, Plus, and the calculated tROP-SS–demonstrated mean CoV < 20% at all timepoints for both eyes. Zone demonstrated the greatest CoV for the three independent variables over the six weeks with a minimum value at the third week of screening: 6.0% (95% CI: 3.0% − 8.9%) (Fig. 2a). Plus and Stage CoV were relatively stable. Plus CoV gradually increased from week 1 at 1.2% (0.3% − 2.0%) to week 6 at 3.3% (1.7% − 4.9%). Stage CoV was relatively stable during all time points ranging from 9.2 to 13.1% (Fig. 2b, c). The calculated tROP-SS were below 20% at all timepoints and mimicked the Zone curve somewhat (Fig. 2d). The tROP-SS CoV reached a local minimum at week 3: 10.5% (7.3% − 13.8%) (Fig. 2d).

Fig. 2.

Mean percent coefficient of variation between graders in relation to exam week for left (blue) and right (red) eyes with 95% Confidence Intervals for (a) Zone, (b) Plus, (c) Stage, and (d) Telemedicine Retinopathy of Prematurity Severity Score.

A repeated measures ANOVA revealed a statistically significant effect of time of screening (week) on CoV between graders for Zone (p < 0.001, η² = 0.20), Plus (p = 0.002, η² = 0.05) and tROP-SS (p < 0.001, η² = 0.08). For all three, Mauchly’s test indicated that the assumption of sphericity was violated, and Greenhouse-Geisser corrected p-values were reported.

Post-hoc pairwise comparisons with Bonferroni correction revealed significant differences between several time points (Table 1). The Zone CoV for week 1 was significantly higher versus all subsequent weeks (p < 0.001). Pairwise comparisons of Plus CoV showed significantly lower values at week 2 versus week 6 (p = 0.019). tROP-SS CoV significantly decreased from week 1 to week 2, 3, and 4 (all p < 0.01). Additionally, tROP-SS CoV significantly increased from week 2, 3, 4, and 5 to week 6 (all p < 0.02). No other pairwise comparisons reached statistical significance after correction (all p > 0.05).

Table 1.

Post-hoc pairwise comparisons of week-to-week variation in coefficient of variation for Zone, Plus, Stage, and Telemedicine Retinopathy of Prematurity Severity Score (tROP-SS).

		Zone		Plus		Stage		tROP-SS
Week Comparison		Paired T-Statistic	P Value	Paired T-Statistic	P Value	Paired T-Statistic	P Value	Paired T-Statistic	P Value
1	2	6.17	< 0.001*	0.51	1.000	-0.95	1.000	4.28	0.002*
1	3	7.81	< 0.001*	-1.21	1.000	-1.13	1.000	4.46	0.001*
1	4	7.63	< 0.001*	-2.71	0.150	0.00	1.000	3.87	0.006*
1	5	7.02	< 0.001*	-2.41	0.310	-1.87	1.000	2.78	0.125
1	6	5.07	< 0.001*	-2.81	0.117	-0.12	1.000	-0.16	1.000
2	3	2.50	0.248	-2.11	0.625	-0.44	1.000	0.93	1.000
2	4	1.99	0.801	-2.90	0.091	0.86	1.000	0.20	1.000
2	5	1.77	1.000	-2.99	0.073	-1.18	1.000	-1.07	1.000
2	6	0.15	1.000	-3.48	0.019*	0.69	1.000	-3.53	0.016*
3	4	-0.85	1.000	-1.39	1.000	1.64	1.000	-0.50	1.000
3	5	-2.00	0.792	-1.19	1.000	-0.96	1.000	-1.65	1.000
3	6	-2.41	0.309	-1.94	0.892	1.14	1.000	-4.65	0.001*
4	5	-0.61	1.000	0.20	1.000	-2.51	0.243	-2.06	0.690
4	6	-1.91	0.948	-0.89	1.000	-0.16	1.000	-4.55	0.001*
5	6	-1.65	1.000	-2.00	0.789	1.88	1.000	-3.56	0.015*

All p values shown are adjusted with the Bonferroni correction. *P values < 0.05 were deemed significant.

Discussion

This study prospectively validates tROP-SS, in which a set of stateful images were matched on PMA over time. Masked grading by twelve expert pediatric retina specialists demonstrated excellent tROP-SS agreement by CoV that was robust to repeated eye assessments over time. The same held true for Zone, Stage, and Plus – the three ROP variables which dictate TW-ROP. In addition to use of twelve expert graders and masking, the four unique strengths of this study are its weekly assessment interval, image series comparison, matching on PMA, and stateful image assessment of each eye by retaining knowledge of previous data.

This study mimicked real world screening conditions where each of 40 eyes was separately assessed at 1-week intervals over a period of 6 weeks. This is the first known study to demonstrate intergrader variability of Zone, Stage and Plus as a function of time for a PMA-matched dataset. This contrasts with traditional validation schemas leveraging stateless and isolated images with ideal representations of feature severity. Often, the most ideal image occurs in the older infant, which biases validation studies to images of later PMAs.

Essentially, the graders validated the tROP-SS in 2 eyes of 20 patients on 6 separate occasions, each separated by 1 week with PMA-matched subjects in a ROP population enriched to evaluate disease heterogeneity assessment. This longitudinal validation has never been performed before and is significant in that it reflects the graders’ ability to determine disease status at different timepoints of the disease lifecycle, in the same population. These 6 separate validations demonstrated that with expert graders, tROP-SS CoV is remarkably stable for the graders over all 6 time periods, ranging from 10.5% − 18.2%.

This study also avoids the uncertainty of solo image analysis as one image may not be representative of the ROP severity within the eye. The grading of the entire image series per visit as opposed to a single representative image from each eye forces the grader to be purposeful in determining per ICROP3 rules the most posterior extent (Zone), greatest disease (Stage), and most vascular activity (Plus), all of which are ultimately integrated to calculate tROP-SS.

Notably, CoV for tROP-SS and each of the features was determined on a per week basis over 31.5–36.5 weeks PMA; this time window is also when images are often lower in quality yet risk of rapid progression in the infants who will develop TW-ROP is high. Image quality improves innately over time, independent of disease status, due to developmental physiology. Beyond the typical confounders of image quality (e.g., focus, illumination, artifact), physiologic limitations on image quality spontaneously improve over time (e.g., vitreous haze lessens, axial lengths increase up to 50% between 25 and 50 weeks PMA, the cornea and lens flatten, astigmatism improves, and the anterior chamber deepens).^9–14 PMA-matching controls for disparities in physiologic image quality. This physiological maturation explains our findings regarding Zone agreement; the significantly higher CoV for Zone observed in week 1 reflects the anatomical and imaging limitations of the highly premature eye. As the eye develops and these physiological barriers lessen, intergrader agreement predictably stabilizes by week 3.

It remains unknown how ROP experts develop a clinical intuition for ROP Tempo and risk of blindness in the face of three disparate disease features. However, analysis of the LONGROP grading standard as well as the prospective LONGROP study indicate that individual graders prioritize some features over others in their assessment of Tempo.^8,15 The tROP-SS acts as a smoothing function by integrating the variations in Zone, Stage, and Plus into a single score and forcing a binary output of TW vs. No TW-ROP. In this validation study, the coefficient of variation (CoV) for tROP-SS remained well below 20% at all timepoints and was robust to repeat assessments of eyes.

Study limitations include a relatively small sample size of forty eyes. To account for this limitation while still determining grader ability to assess ROP status, the chosen image dataset was enriched to have approximately equal numbers of TW-ROP, spontaneously regressing ROP, and no ROP. However, due to increased disease presence, this dataset may not reflect the value of tROP-SS in the general screening populations. Similarly, five retinal images do not technically represent the totality of retina when using 130° field of view images. However, this limitation likely carries minimal impact from a screening perspective; the ROP screening function is to determine an inflection point—in this case TW-ROP, and all treatment indications in ROP are guided by disease in Zone I or II, with 130° images capturing all of Zone I and II. Therefore, any retina that is not imaged due to field of view limitations should not impact the essential question of ROP screening. Additionally, due to the stateful design, there is an inherently possibility of graders developing anchoring bias to their prior grades. This may manifest as a smoothing function between an individual grader’s repeated assessments on the same patient, although this bias is equally present in analogous real-world telemedicine grading practices. Given these constraints, future studies should focus on reproducing similar stateful methodologies on a larger scale across diverse, unenriched screening populations to further assess the broad applicability of the tROP-SS.

In conclusion, in this study, stateful validation in a prospective, longitudinal dataset confirmed that tROP-SS provides reliable, longitudinal disease assessment optimized for telemedicine workflows. Furthermore, this study confirmed that feature assessment by expert graders is also a function of PMA. By capturing grader consistency across the screening continuum-particularly during acute phase screening of ROP, this study countered the limitations of stateless validation paradigms. The analysis of temporal agreement patterns establishes a new benchmark for validating ROP scoring systems in dynamic clinical environments, and a similar model can be considered for future validation studies.

Author contributions

H.W.P, S.V.S, and D.M.M. wrote the main manuscript text. S.V.S, C.S.Z, A.R.B, and J.K performed the statistical analysis. C.R.B, C.G.B., A.M.B., K.A.D., A.L.E., C.A.H., G.B.H., E.D.N., P.Q., I.T, E.W., Y.Y, and D.M.M. prospectively graded the images used in the analysis. Study design and methodology conceptualization was carried out by D.M.M, S.V.S, A.R.B, J.K, and T.R. All authors reviewed the manuscript.

Funding

This work was supported by a grant from Research to Prevent Blindness, Inc. and the National Eye Institute (NEI P30-EY026877).

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Competing interests

D.M.M.– GigCapital7 (BOD), RetinaBio, Inc (C, E), Vision Labs, Inc (E), Visunex (E, defunct), AinslyLimited (E), Akebia (SAB), Affamed (C), Alexion (C), Aspire Pharma (C), Boehringer-Ingelheim (C), Clinical Trials Resource Group (SAB), dSentz Inc. (F, E, BOD), Feliqs Inc. (C), Genentech (G), Icon (C, DSMB), Insite DME (C), Ocular Therapeutix (C, E), Osanni Bio (C, E), Plenoptika Inc. (E, CAB), Pr3vent Inc (F, E, BOD), Promisight, Inc. (F, E, BOD, defunct), Pykus (E, SAB), Regeneron (SC, PI), SLACK, Inc (C), University of Miami (H), Vanotech Chengdu (C), RESEARCH TO PREVENT BLINDNESS, INC. (G), NEI P30- EY026877. The other authors indicate no financial support or conflicts of interest.