The prediction capability of a cataract surgery risk stratification model based on a large electronic medical record dataset

Abstract

Purpose:

The aim of this study was to develop a risk stratification system that predicts visual outcomes (uncorrected corrected visual acuity at one week and five weeks postoperative) in patients undergoing cataract surgery.

Methods:

This was a retrospective analysis in a multitier ophthalmology network. Data from all patients who underwent phacoemulsification or manual small-incision cataract surgery between January 2018 and December 2019 were retrieved from an electronic medical record system. There were 122,911 records; 114,172 (92.9%) had complete data included. Logistic regression analyzed unsatisfactory postoperative outcomes using a main effects model only. The final model was cross-checked using forward stepwise selection. The Hosmer–Lemeshow goodness of fit test, the Bayesian information criterion, and Nagelkerke’s R² assessed model fit. Dispersion was calculated from deviance and degrees of freedom and C-stat from receiving operating characteristics analysis.

Results:

The final phacoemulsification model (n = 48,169) had a dispersion of 1.08 with a Hosmer–Lemeshow goodness of fit of 0.20, a Nagelkerke R² of 0.19, and a C-stat of 0.72. The final manual small-incision cataract surgery model (n = 66,003) had a dispersion of 1.05 with a Hosmer–Lemeshow goodness of fit of 0.00015, a Nagelkerke R² of 0.14, and a C-stat of 0.68.

Conclusion:

The phacoemulsification model had reasonable model fit; the manual small-incision cataract surgery model had poor fit and was likely missing variables. The predictive capability of these models based on a large, real-world cataract surgical dataset was suboptimal to determine which patients could benefit most from sight-restoring surgery. Appropriate patient selection for cataract surgery in developing settings should still rely on clinician thought processes, intuition, and experience, with more complex cases allocated to more experienced surgeons.

Cataract surgery has long been recognized among the most commonly performed surgical procedures, and its indications (and demands) are evolving to include younger patients with better visual acuities (Vas), who expect good postoperative visual function (presenting VA ≥6/12).[1,2] The expected visual outcome following cataract surgery is an important factor to guide the patient’s and clinician’s decision-making process.[1] The 73^rd and 74^th World Health Assembly recently proposed and endorsed the effective cataract surgery coverage (eCSC) indicator as a proxy indicator to track changes in surgical uptake and quality of eye care services.[2] This indicator is defined as the proportion of adults aged ≥50 years who are in need of and received cataract surgery and have a resultant good-quality distance VA outcome. The updated Rapid Assessment of Avoidable Blindness (RAAB7) standardized survey methodology now reports eCSC.[3] However, RAAB does not report ocular comorbidities,[3] which are a major factor that could lead to poorer postoperative visual outcomes.[1]

Cataract surgical risk stratification models have been proposed as a potential preoperative tool to help decide who could benefit most from surgery. Since 1988, there have been seven models reported by ten publications that predict good visual outcomes [Table 1]. The models were predominantly for phacoemulsification and excluded manual small-incision cataract surgery (MSICS), which is often used in resource-limited settings. Similarly, there was no risk stratification for developing settings, where patients are more likely to present with poorer VA and more advanced cataracts. Previous systems were developed using data obtained from manually-filled datasheets and were based on perceived risk scores, empirical scores, and mathematical regression models. They varied considerably on how many factors they integrated into the models, but most models considered anatomic factors, comorbidities, and demographic factors that could potentially influence postoperative visual acuity [Table 1]. Statistical analysis and complexity varied widely. The lack of statistical validation and/or external validation were limitations of most systems. In spite of cataract surgery being a common procedure, most systems were developed using a paltry sample size, and there does not appear to be widespread clinical use of risk stratification systems. We therefore aimed to create a statistical risk stratification model populated by standardized electronic medical record (EMR) data based on the significant predictive variables previously identified [Table 1] and on potential socioeconomic and geographical variables. The objective of this study was to develop a comprehensive risk stratification system that could predict visual outcomes (postoperative uncorrected corrected visual acuity [UCVA] at one week and five weeks postoperatively) in Indian patients undergoing MSICS or phacoemulsification cataract surgery at a high-volume surgical center. We hoped that such a model could be used with EMR by ophthalmologists to guide the patient selection process for cataract surgery in clinical practice.

Table 1.

Summary of cataract surgical risk stratification models previously published in the literature

Authors, year	Sample Size	Targeted Outcome	Type of Risk Stratification System	Model Variables
Graney et al., 1988[4]	293	Postoperative VA	Clinical index formula	Age, preoperative Snellen VA, number of current prescription medications, and newspaper reading (based on frequency scale of daily to never=1.0-5.0)
Mangione et al., 1995[5]	426	Postoperative VA	Prediction rule based on ADVS score (0-100 points) improving, defined as a 12-month change >2 SDs in the test-retest variability	Age <78 years (P<0.001), a poorer preoperative ADVS scores (P<0.001), posterior subcapsular cataract (P=0.09), absence of AMD (P=0.07), and/or diabetes (P=0.006).
Olsen, 1993[6] and 1996[7]	333	Postoperative VA after phacoemulsification	Regression formula	VAsc=Uncorrected Snellen acuity (1.0=20/20), S=Spherical equivalent of manifest refraction (absolute value), C=Postoperative keratometric cylinder
Perea-Milla et al., 2011[8]	5512	Probability of postoperative VA and visual function (VF-14) improvement after phacoemulsification	Validated multivariate logistic regression score based on a reference patient aged≥80 years old, without uncomplicated cataract, no vision loss, contralateral VA≥0.5, equal VA in both eyes, unilateral cataract, not complex, baseline VA≥0.5, baseline VF-14 ≥70 points	Baseline VA, age <75 years, and uncomplicated cataract
Butler, 2012[9]and Kim et al., 2016[10]	None	Difficulty and general risk of cases to assign cases to trainees	Perceived risk score system (20 points) with scores <3 for junior trainees, 3-5 for more senior trainees, >5 for consultants	Age, y: 80-90, >90; DR; brunescent/white cataract/no fundus view; pseudoexfoliation/phacodonesis; small pupil; shallow anterior chamber (<2.5 mm, 1 point); axial length: <21.5 mm, >26.0 mm), >30.0 mm; on alpha blocker doxazosin, on alpha blocker, Fuchs’ dystrophy, only eye
Hahn et al., 2014[11]	14,924	Refractive accuracy and postoperative VA 2-5 weeks after phacoemulsification	Risk stratification based on statistical significance and relevance of the relationship between the external risk factor and outcome	Age≥80 years, female, baseline BCVA≤0.1, myopia/axial eye length≥25 mm, hyperopia/axial eye length <22 mm, ≥1 preexisting condition potentially reducing VA, ≥1 previous ocular surgery, ≥1 surgically relevant risk factor
Sorrentino et al., 2016[12] and 2017[13]	50	Discrimination between excellent and good BCVA after phacoemulsification or torsional phacoemulsification	Prediction model based on Likert-type harm scale (score of 1-5), based on level of damage against corneal endothelium after phacoemulsification/torsional phacoemulsification	LOCS cataract hardness (1, 1-2, 2-3, 3-4, 5), surgical sculpting time, minutes (<10, <16, <22, <28, ≤28) and cumulative dissipative energy, minutes (<2.5, <5, <7.5, <10, ≤10), quadrant removal time, minutes (<15, <30, <45, <60, ≤60) and cumulative dissipative energy, minutes (<5, <10, <15, <20, ≤20)

ADVS=Activities of daily vision scale; ASC/PSC=anterior subcapsule opacities/posterior subcapsule opacities; AUC=Area under the curve; BCVA=Best-corrected visual acuity; DR=Diabetic retinopathy; LOCS=Lens opacities classification system; NS=Nuclear sclerosis; VA=Visual acuity; VF=Visual function

Methods

Study design

This retrospective, hospital-based study included all patients who underwent phacoemulsification or MSICS between January 2018 and December 2019 in a multitier ophthalmology network, Author 3 Institute, in City, Country (multitier ophthalmology network in India). The patients or their parents/guardians provided written informed consent. Data were de-identified for analysis. The study adhered to the Declaration of Helsinki and was approved by the Institutional Ethics Committee (IRB approval number).

Data collection and analysis

During a comprehensive ophthalmic examination, trained ophthalmic personnel entered patient data using a standardized, structured template in a browser-based EMR system (Author 3 Institute EMR, eyeSmart EMR) under supervision by an ophthalmologist.[14] All data columns available that corresponded to the patient demographics, clinical presentation, ocular diagnosis, and lens status predictive variables previously identified by the literature [Table 1], in addition to socioeconomic status, district status (rural vs urban), operation performed, and operation notes were exported to a Microsoft Excel (Microsoft Corporation, 2018, Redmond, USA) database for analysis.

There were 122,911 surgical records, of which 4818 were missing postoperative UCVA at one week and five weeks, resulting in 117,909 valid records. A total of 29,899 records with available UCVA data at one week were used to impute results at five weeks due to missing data (25.4%). For preoperative VA (advised logMAR), 3737 records were missing data (3.2%), and for the type of cataract 522 records (0.4%) were missing data and coded as unknown. The final number of available records for analysis was 114,172 (92.9%).

In terms of patients and eyes, the percentage of patients with two eyes in the dataset versus one eye in the dataset was 7.2%. Consequently, it was decided to ignore the correlation between eye pairs. The dependent variable was unsatisfactory outcome at five weeks (UCVA < 6/12 or equivalent). Patient age at surgery was divided into half-decade function, while high surgical experience (for the surgeon) was set at 1000.

A logistic regression was carried out to analyze unsatisfactory postoperative outcomes at five weeks using PASW 28 (SPSS, IBM, Armonk, NY) with a main effects model only. Variables were entered as one block with stepwise elimination for nonsignificant covariates. The final model was cross-checked using forward stepwise selection. In some instances, notably center and socioeconomic status, multicollinearity was checked using variance inflation and a decision was made to use one or two available variables when it was high (2.5). Model fit (phacoemulsification and MSICS models were created separately) was assessed using the Hosmer–Lemeshow goodness of fit test, the Bayesian information criterion, and Nagelkerke’s R². Dispersion was calculated from deviance and degrees of freedom and C-statistic (C-stat) from receiving operating characteristics analysis.

Results

The final phacoemulsification model (n = 48,169) had a dispersion of 1.08 with a Hosmer–Lemeshow goodness of fit value of 0.20, a Nagelkerke’s R² value of 0.19, and a C-stat of 0.72. Preoperative VA was the most important predictor with an odds ratio (OR) of 1.67 per unit of logMAR. The highest ORs for unsatisfactory outcomes were associated with prior vitreoretinal surgery (11.29), high hypermetropia (6.42), cystoid macular edema (CME; 5.40), iris anomalies (4.67), retinal vein occlusion (4.57), complicated cataract type (3.85), and subluxated lens (3.04) [Table 2]. Surgical experience, where the surgery was performed, and socioeconomic status had relatively minor effect. Only the eldest patients (≥86 years old) had a greater than three-fold risk (OR: 3.04) of unsatisfactory outcomes compared to the reference group.

Table 2.

Logistic regression for unsatisfactory outcomes using phacoemulsification. Variables are arranged by patient demographics, comorbidities, cataract characteristics, and surgical center

Variable	B	P	Odds Ratio	95% CI
				Lower	Upper
Surgical age (years)*
≤39	−0.351	9.0-9	0.70	0.63	0.79
40-45	−0.376	1.3x10-13	0.69	0.62	0.76
46-50	−0.419	5.9x10-21	0.66	0.60	0.72
51-55	−0.321	2.0x10-15	0.73	0.67	0.79
56-60	−0.122	0.001	0.89	0.82	0.95
61-65	0.250	0.000026	1.18	1.14	1.44
71-75	0.332	6.9x10-14	1.39	1.28	1.52
76-80	0.471	2.0x10-15	1.60	1.43	1.80
81-85	0.764	1.6x10-15	2.15	1.78	2.59
≥86	1.111	4.0x10-10	3.04	2.14	4.30
Patient socioeconomic status†
Lower class	0.342	4.4x10-32	1.41	1.33	1.49
Upper-middle class	−0.056	0.33	0.95	0.85	1.06
Upper class	−0.203	0.016	0.82	0.69	0.96
Advised logMAR	0.511	7.7x10-213	1.67	1.62	1.72
Right eye‡	−0.065	0.005	0.94	0.90	0.98
One-eye vision§	0.586	2.3x10-16	1.80	1.56	2.07
AMD||	0.923	1.5x10-17	2.52	2.04	3.11
Diabetic retinopathy||	1.048	1.6x10-94	2.85	2.58	3.15
Glaucoma||	0.661	1.1x10-54	1.94	1.78	2.11
Uveitis||	0.617	0.000032	1.85	1.39	2.48
Fuch’s dystrophy||	0.739	8.1x10-11	2.09	1.68	2.62
Cystoid macular edema||	1.686	6.0x10-41	5.40	4.22	6.91
Retinal vein occlusion||	1.519	1.3x10-21	4.57	3.34	6.24
Corneal scar||	0.981	8.8x10-71	2.67	2.39	2.97
Corneal edema decompensation||	1.435	8.2x10-37	4.20	3.37	5.25
Iris anomalies||	1.541	6.1x10-22	4.67	3.41	6.39
Subluxated lens||	1.110	0.00092	3.04	1.58	5.85
High myopia||	0.346	4.0x10-6	1.41	1.22	1.64
High hypermetropia||	1.859	1.4x10-8	6.42	3.38	12.21
Previous vitreoretinal surgery||	2.424	1.2x10-211	11.29	9.69	13.16
Rural center¶	−0.277	9.8 × 10-17	0.76	0.71	0.81
District status**
Urban	0.204	1.8x10-8	1.23	1.14	1.32
Rural	0.228	3.4x10-10	1.26	1.17	1.35
Cataract type†
Cortical	0.391	0.00075	1.48	1.18	1.86
Nuclear	0.576	1.0x10-21	1.78	1.58	2.00
Posterior plaque cataract	0.545	0.000097	1.72	1.31	2.27
Posterior subcapsular cataract	0.143	0.05	1.15	1.00	1.33
Complicated	1.348	7.3x10-11	3.85	2.57	5.78
Traumatic	1.075	2.0x10-10	2.93	2.10	4.08
Surgical experience‡‡	0.201	1.3x10-13	1.22	1.16	1.29
Constant	6.844	1.0x10-112

AMD=Age-related macular degeneration; CI=Confidence interval. Reference groups: *66-70 years; ^†lower-middle class; ^‡left eye; ^§bilateral eye vision; ^||no disease or medical issue; ^¶urban; **metropolitan; ^†total; ^‡‡≥1000 surgeries breakpoint

The final MSICS model (n = 66,003) had a dispersion of 1.05 with a Hosmer–Lemeshow goodness of fit value of 0.00015, a Nagelkerke R² of 0.14, and a C-stat of 0.68. In terms of effect size, preoperative VA was not the most important predictor with an OR of 1.30 per unit of logMAR; rather, rural centers eclipsed this variable with an OR of 0.51 using urban centers as the reference group [Table 3]. Previous vitreoretinal surgery was still associated with bad outcomes (OR: 12.51), followed by complicated cataract (5.75), retinal vein occlusion (4.87), corneal edema decompensation (4.61), iris anomalies (4.39), and CME (3.68). Interestingly, patient age was less important compared to phacoemulsification.

Table 3.

Logistic regression for unsatisfactory outcomes using MSICS. Variables are arranged by patient demographics, comorbidities, cataract characteristics, and surgical center

Variable	B	P	Odds Ratio	95% CI
				Lower	Upper
Sex*	0.059	0.00086	1.06	1.03	1.10
Surgical age (years)†					0.98
≤39	−0.600	1.1x10−28	0.55	0.70	0.58
40-45	−0.623	1.0x10−48	0.54	0.49	0.60
46-50	−0.582	3.9x10−64	0.56	0.52	0.68
51-55	−0.446	8.0x10−54	0.64	0.61	0.81
56-60	−0.259	1.9x10−23	0.77	0.73	0.78
61-65	−0.410	1.3x10−8	0.66	0.58	1.24
71-75	0.156	2.1x10−7	1.17	1.10	1.58
76-80	0.380	6.0x10−21	1.46	1.35	2.23
81-85	0.678	7.0x 10−27	1.97	1.74	2.80
≥86	0.820	2.7x10−14	2.27	1.84
Patient socioeconomic status‡
Lower class	0.342	4.4x10-32	1.41	1.33	1.49
Upper-middle class	−0.056	0.33	0.95	0.85	1.06
Upper class	−0.203	0.016	0.82	0.69	0.96
Advised logMAR	0.265	2.4x10−156	1.30	1.28	1.33
One-eye vision§	0.589	3.1x10−17	1.80	1.57	2.07
AMD||	0.937	9.0x10−18	2.55	2.06	3.16
Diabetic retinopathy||	0.917	6.4x10−35	2.50	2.16	2.89
Glaucoma||	0.762	4.4x10−58	2.14	1.95	2.35
Fuchs’ dystrophy||	0.732	0.004	2.08	1.26	3.43
Cystoid macular edema||	1.304	1.6x10−16	3.68	2.70	5.02
Retinal vein occlusion||	1.583	1.1x10−13	4.87	3.21	7.39
Pseudoexfoliation||	0.248	0.000039	1.28	1.14	1.44
Corneal scar||	0.615	7.9x10−46	1.85	1.70	2.01
Corneal edema decompensation||	1.528	6.1x10−48	4.61	3.76	5.67
Iris anomalies||	1.479	5.7x10−22	4.39	3.25	5.93
Subluxated lens||	0.877	0.00095	2.40	1.43	4.04
Hearing impairment||	0.459	0.002	1.58	1.18	2.12
Previous vitreoretinal surgery||	2.527	1.1x10−87	12.51	9.75	16.06
Previous laser surgery||	0.998	0.035	2.71	1.07	6.86
Rural center¶	−0.668	3.4x10−208	0.51	0.49	0.54
District status**
Urban	−0.141	0.019	0.87	0.77	0.98
Rural	0.249	0.000039	1.28	1.14	1.45
Cataract type††
Cortical	−0.081	0.50	0.92	0.73	1.17
Nuclear	0.238	3.1x10−20	1.27	1.21	1.33
Posterior plaque cataract	0.305	0.11	1.36	0.93	1.97
Posterior subcapsular cataract	−0.181	0.003	0.83	0.74	0.94
Complicated	1.749	2.3x 108	5.75	3.11	10.62
Traumatic	0.822	5.7x10−7	2.27	1.64	3.14
Surgical experience‡‡	0.097	6.9x10−7	1.10	1.06	1.14
Constant	7.036	1.4x10−78

AMD=Age-related macular degeneration; CI=Confidence interval Reference groups: *female; ^†66-70 years; ^‡lower-middle class; ^§bilateral eye vision; ^||no disease or surgery; ^¶urban; **metropolitan; ^††total; ^‡‡≥1000 surgeries breakpoint

Discussion

We aimed to create a comprehensive cataract surgical risk stratification model that could mine EMR data to determine which patients are most likely to benefit from cataract surgery and have good outcomes. We analyzed the largest cataract surgical dataset to date (n = 114,172). We considered all previously significant variables that were possible to collect from our EMR and potential variables that would be important to surgical delivery among vulnerable populations in resource-limited settings. We also included MSICS for the first time in this type of analysis. In our models, we attempted to keep the computations straightforward, as we were aware that other published models used variables that were not collected in our study [Table 1].

The predictive capability of our models was lower than optimal, with C-stat values of 0.72 for phacoemulsification and 0.68 for MSICS. When analyzing model goodness of fit in statistics, C-stat values range from 0.5 to 1.0. A C-stat of 0.5 means the model is no better than “chance” at predicting visual outcomes, while a 1.0 indicates the model is perfectly accurate in predicting outcomes. A C-stat greater than 0.7 suggests that the model is reasonable (and has good fit), while a value of >0.8 is a model with strong predictive capabilities.[15] Although the phacoemulsification model had reasonable model fit with a C-stat >0.7, the MSICS model had poor fit with a C-stat <0.7 and was an incomplete model that was likely missing variables that are not captured by the EMR, despite being the most comprehensive model to date [Table 1]. Therefore, the phacoemulsification model by itself could be used to predict outcomes, reiterating that it has a good but not strong fit. The MSICs model should not be used to predict outcomes. Importantly, the majority of surgeries done at our institution in this study were MSICS (56%, 66,003/117,909). Therefore, the combined model would not be useful to the majority of patients in clinical practice.

The major issue exposed by our models are that the logistic regressions of large, real-world populations hit a barrier in main effects model. The accuracy of predictive models, such as logistic regression, is dependent on what variables are collected and how complete the values are for each variable. If key variables are missing, then the model fit may be poor. Higher level effects can also be added (e.g., full factorial models), although there is a danger of overfitting with results that may not be easy to understand. Machine-learning algorithmic approaches can also penalize variables in different ways to achieve more parsimonious results by eliminating redundancy in training sets and corresponding better prediction in test sets.

Furthermore, our study design strength—a very large sample size—was also its weakness. Statistical considerations are moving beyond the notion that the use of big data in ophthalmology will minimize potential biases, as missing data, confounding, and oversimplification can lead to statistical correlations to support a hypothesis that is based on inconsequential variables.[16] The use of big, real-world data in mathematical modelling cannot reveal the clinician’s subjective thought processes involved or substitute clinician intuition when assessing the complexity of a case, particularly when using a structured EMR system that might not capture the full written record.[16,17]

There has been an increase in the number of studies in recent years that risk stratify cataract patients using predictive mathematical models [Table 1]. These models have their limitations, especially with limited external validity on real-world populations, and they have only been developed for phacoemulsification cataract surgery in high-income settings. We aimed to develop a combined model that stratified postoperative visual outcomes for both phacoemulsification and MSCIS in a developing setting. In conclusion, the predictive capability of the logistic regressions model based on a large, real-world cataract surgical dataset was suboptimal to determine which patients would benefit most from sight-restoring surgery, particularly for MSICS. Although we developed the most comprehensive model to date that included all potential factors captured by an EMR that could influence postoperative visual outcomes, the MSICS model was still incomplete, suggesting that there are external factors beyond those captured by EMRs. MSICS is increasingly the predominant type of cataract surgery done in many low- and middle-income countries, and the lack of an adequate statistical model to perform risk stratification means that in these developing settings, appropriate patient selection for cataract surgery should still rely on clinician thought processes, intuition, and experience, with more complex cases allocated to more experienced surgeons.

Conclusion

Predictive statistical models should not be used to risk stratify cataract surgeries in developing settings. Measuring effective cataract surgery is a very complex study of coverage and equity, and there is more than meets the eye to achieving good postoperative visual outcomes.

Financial support and sponsorship

Nil.

Conflicts of interest

Kristen A. Eckert was a paid consultant of Strategic Solutions for her work in this study; the remaining authors report there are no competing interest to disclose.