A Prediction Model to Discriminate Small Choroidal Melanoma from Choroidal Nevus

Abstract

Objective

This study aimed to develop a validated machine learning model to diagnose small choroidal melanoma.

Design

This is a cohort study.

Subjects, Participants, and/or Controls

The training data included 123 patients diagnosed as small choroidal melanocytic tumor (5.0–16.0 mm in largest basal diameter and 1.0 mm–2.5 mm in height; Collaborative Ocular Melanoma Study criteria). Those diagnosed as melanoma (n = 61) had either documented growth or pathologic confirmation. Sixty-two patients with stable lesions classified as choroidal nevus were used as negative controls. The external validation dataset included 240 patients managed at a different tertiary clinic, also with small choroidal melanocytic tumor, observed for malignant growth.

Methods

In the training data, lasso logistic regression was used to select variables for inclusion in the final model for the association with melanoma versus choroidal nevus. Internal and external validation was performed to assess model performance.

Main Outcome Measures

The main outcome measure is the predicted probability of small choroidal melanoma.

Results

Distance to optic disc ≥3 mm and drusen were associated with decreased odds of melanoma, whereas male versus female sex, increased height, subretinal fluid, and orange pigment were associated with increased odds of choroidal melanoma. The area under the receiver operating characteristic “discrimination value” for this model was 0.880. The top four variables that were most frequently selected for inclusion in the model on internal validation, implying their importance as predictors of melanoma, were subretinal fluid, height, distance to optic disc, and orange pigment. When tested against the validation data, the prediction model could distinguish between choroidal nevus and melanoma with a high discrimination of 0.861. The final prediction model was converted into an online calculator to generate predicted probability of melanoma.

Conclusions

To minimize diagnostic uncertainty, a machine learning-based diagnostic prediction calculator can be readily applied for decision-making and counseling patients with small choroidal melanoma.

Introduction

Controversy regarding management of small choroidal melanoma stems from the diagnostic uncertainty [1, 2, 3, 4]. The majority of tumors labeled as small choroidal melanoma (5.0–16.0 mm in largest basal diameter and 1.0 mm–2.5 mm in height) within the Collaborative Ocular Melanoma Study (COMS) observation arm remained stable during observation with clinical behavior compatible with a diagnosis of choroidal nevus rather than a melanoma [5, 6]. As the tumors within the small size criteria may include nevi and melanoma, these small choroidal melanocytic tumors are best referred to as indeterminate choroidal melanocytic tumor (IMT) emphasizing the need to differentiate choroidal nevus from melanoma [7, 8].

Over the years, there has been a distinct shift away from the size-only-based classification system [5] of IMTs to include extrinsic and intrinsic features as “risk factors” predictive of growth (Table 1) [1, 7, 9, 10, 11, 12, 13, 14]. The presence of orange pigment and subretinal fluid (SRF) favors a diagnosis of a small choroidal melanoma, whereas drusen and retinal pigment epithelium (RPE) changes are likely to indicate a benign lesion (such as nevus). Although these “risk factors” have consistently been identified as significant predictors of growth [15], it is worth emphasizing that the growth “risk factors” carry externally unvalidated probabilities that have limited their clinical application in prediction of small choroidal melanoma [8].

Table 1.

Published reports on the direction of correlation between clinical features and growth of a small choroidal melanocytic lesion

Author	Quantitative parameters		Qualitative data
	base diameter	height	orange pigment	Subretinal fluid	drusen	RPE atrophy	juxtapapillary location	symptoms
Gass et al. [1]	Positive	Positive	Positive	Positive	Negative	Negative	NA	Positive
Augsburger et al. [11]	Positive	Positive	Positive	Positive	NS	NS	Positive	Positive
Butler et al. [7]	NS	Positive	Positive	Positive	NS	NS	NS	Positive
COMS [5]	Positive	Positive	Positive	NS	Negative	Negative	NS	NA
Singh et al. [9]	NS	Positive	Positive	NS	NS	NS	Positive	Positive
Shields et al. [3]	NS	Positive	Positive	Positive	NS	NS	Positive	Positive
Shields et al. [14]	Positive	Positive	Positive	Positive	NS	NS	NS	Positive

RPE, retinal pigment epithelium; NA, not assessed; NS, not significant.

In absence of diagnostic biopsy or documented growth [16, 17], the diagnosis is generally relied upon presence of “risk factors” predictive of growth in the future [7, 8, 9, 10, 11, 12, 15]. What is pertinent however is the presence or absence of malignant growth at initial presentation. In other words, the diagnosis of choroidal melanoma relies upon “risk factors” that may or may not be present to predict growth that may or may not be observed for several years. Of note, the “risk factors” have not been validated on an independent dataset (i.e., without external validation) [7, 8, 9, 10, 11, 12, 15]. To address these limitations and provide clinicians with a diagnostic tool that can be readily applied at the time of initial presentation, we developed and validated a prediction model to diagnose small choroidal melanoma and provided an associated online calculator that is accessible and easy to use.

Methods

Training Data

Institutional review board approval was obtained. The study adhered to the Declaration of Helsinki. Since this is a retrospective case series study without identification of the patient, informed consent from individual patients was not required. The training data, used to select variables for inclusion in the final model, were derived from 123 patients with small choroidal melanocytic tumor (5.0–16.0 mm in largest basal diameter and 1.0 mm–2.5 mm in height; Collaborative Ocular Melanoma Study criteria) seen at a tertiary ophthalmology clinic between 2010 and 2018. They were classified as small choroidal melanoma (n = 61) either by growth (growth confirmed group, n = 30) or pathology (pathology confirmed group, n = 19) or both (combined group, n = 12). Sixty-two patients were classified as choroidal nevus after at least 24 months of documented stability under observation.

All patients were evaluated using a standard slit-lamp and fundus examination to make a clinical diagnosis of SCM. Detailed fundus drawing depicting the entire extent of the lesion along with color fundus photography was performed for all the patients. The clinical records were reviewed for the following variables at the initial examination: patient age and sex, laterality, visual symptoms, presenting best-corrected visual acuity as measured by logMar chart, quadratic distribution (supero-temporal quadrant, supero-nasal quadrant, infero-temporal quadrant, infero-nasal quadrant, juxtapapillary, or macular), posterior tumor margin in relation to optic disc and foveola (<3 mm or ≥3 mm), and tumor dimensions. The largest tumor base diameter was estimated in millimeters by ophthalmoscopy, and the greatest tumor height in millimeters was measured by ultrasonography. Specific tumor features, such as the presence of SRF, surface orange pigment, drusen, and RPE atrophy, were also assessed by 90D ophthalmoscopic examination and supplemented by ancillary studies such as optical coherence tomography and autofluorescence. The record of each patient was reviewed to establish if there was documented evidence of growth at any time during follow-up [18]. Growth was judged by an increase in basal dimension of at least 0.5 mm by meticulous comparison of serial fundus photographs or by an increase in thickness of 0.3 mm by serial ultrasonograms. The time interval between the initial examination and the documentation of tumor growth was recorded (months).

Validation Data

The validation data, used to test the model that was constructed in training data, included 240 patients seen at a different tertiary ophthalmology clinic between January 1, 1997, and December 31, 2001. The cohort has been described in detail previously [9]. In brief, patients with small choroidal melanocytic tumors (5.0–16.0 mm in largest basal diameter and 1.0 mm–2.5 mm in height; Collaborative Ocular Melanoma Study criteria) were observed for growth. During the observation period, 11 patients had growth of at least 0.3 mm in any dimension consistent with the diagnosis of choroidal melanoma. The remaining 229 patients were classified as having nevus at the time they were last evaluated.

Statistical Analysis

Patient and disease characteristics of the training data were summarized using the median and interquartile range for continuous variables and the number and percentage for categorical variables. Fisher's exact test and the Wilcoxon rank-sum test were used to test for univariable differences according to melanoma status for categorical and continuous variables, respectively.

In the training data, lasso logistic regression was used to select variables for inclusion in the final model for the association with melanoma versus choroidal nevus. Variables of interest were determined a priori and included age at diagnosis, sex, laterality, presence of symptoms at presentation, best-corrected visual acuity at presentation, distance to optic nerve, distance to fovea, largest basal diameter, thickness, SRF, orange pigment, Drusen, and RPE atrophy. Lasso logistic regression conducts variable selection by applying a penalty term that shrinks some model coefficients to zero, thus removing the associated variable from the model. The result is a model with a smaller set of variables representing a subset of the initial variables that are most strongly associated with the outcome of interest. We used 10-fold cross-validation to choose the tuning parameter, which controls the number of nonzero coefficients that identified the subset of variables that have the maximum area under the receiver operating characteristic curve (AUC). We limited consideration to subsets containing a maximum of five variables to avoid overfitting. We evaluated the performance of the model using discrimination and calibration. Discrimination was assessed with the AUC, a measure of how accurately patients were classified as having melanoma or not. Calibration was assessed with a calibration plot, a measure of agreement between the predictions and the observed data. To create the calibration plot, the predicted probabilities were divided into 15 equal bins. Then, in each bin, the mean predicted probability was calculated and plotted on the x-axis, and the mean observed probability was calculated and plotted on the y-axis. Locally weighted scatterplot smoothing was used to add a smoothed trend line to plot.

The apparent AUC was calculated based on predictions obtained for the same data that were used to create the model. When we use the same data to create a model and to calculate measures of model performance, an overly optimistic measure of performance is obtained. To correct for this effect, we performed internal validation using bootstrap methods. We drew 500 bootstrap samples, which are random samples from the original data drawn with replacement, and thus each sample representing a unique dataset. On each bootstrap sample, we fit the lasso model with 10-fold cross-validation to choose the tuning parameter, and then we generated the predicted probabilities of melanoma and calculated the AUC. Next, we applied the model that was built on the bootstrap sample to the original dataset, generated the predicted probabilities of melanoma, and calculated the AUC. We subtracted the AUC calculated on the original data from the AUC calculated on the bootstrap data to obtain an estimate of the optimism. The average optimism across the 500 bootstrap samples was then subtracted from the apparent AUC to obtain an optimism-corrected estimate of the AUC. We additionally summarized the frequency with which each variable was selected across the bootstrap samples as a measure of variable importance, where the variables selected most frequently can be considered the most influential for predicting melanoma in this context.

Finally, we performed an external validation using the validation dataset. External validation is considered the gold standard for assessing the performance of any prediction model, as it represents how the model performs on a dataset that was not used in any way to develop the model. We generated a predicted probability of melanoma for each patient in the validation data based on the lasso regression model developed in the training data. The final prediction model was converted into an easy-to-use online calculator that returns a predicted probability of melanoma given a set of specified patient and disease characteristics. A p value <0.05 was considered statistically significant. All statistical analyses were conducted in R software version 4.0.3 [19], including the glmnet [20], and ROC [21] packages.

Results

Patients with melanoma were significantly younger, more frequently male, more frequently presented with symptoms, more frequently located closer to the optic disc and fovea, had a larger largest basal diameter and height, more frequently had SRF and orange pigment, and less frequently had drusen (all p < 0.05; Table 2). We observed that the distance to optic disc ≥3 mm versus <3 mm and drusen were associated with decreased odds of melanoma, whereas male versus female gender, increased height, SRF, and orange pigment were associated with increased odds of melanoma (Table 3). The apparent AUC for this model was 0.880. We noted that below a predicted probability of about 0.225, the model tends to overestimate the predicted probability, whereas above a predicted probability of about 0.325, the model tends to underestimate the predicted probability (Fig. 1).

Table 2.

Patient and disease characteristics by melanoma status in the training data

Characteristic	Nonmelanoma (N = 62)1	Melanoma (N = 61)1	p value2
Age at diagnosis	64 (56, 71)	61 (55, 67)	0.2
Gender
Female	44 (71)	25 (41)	0.002
Male	18 (29)	36 (59)
Laterality
Left eye	30 (48)	31 (51)	>0.9
Right eye	32 (52)	30 (49)
Symptoms	9 (15)	24 (39)	0.004
Distance from optic disc
<3 mm	6 (9.7)	33 (54)	<0.001
≥3 mm	56 (90)	28 (46)
Distance from fovea
<3 mm	16 (26)	30 (49)	0.013
≥3 mm	46 (74)	31 (51)
Largest basal diameter, mm	7.50 (6.50, 9.00)	9.00 (7.50, 10.50)	0.003
Height, mm	1.50 (1.30, 1.80)	2.10 (1.70, 2.30)	<0.001
SRF	9 (15)	43 (70)	<0.001
Orange pigment	19 (31)	47 (77)	<0.001
Drusen	51 (82)	26 (43)	<0.001
RPE atrophy	27 (44)	17 (28)	0.10

SRF, subretinal fluid; RPE, retinal pigment epithelium.

¹Statistics presented: median (IQR); n (%).

²Statistical tests performed: Wilcoxon rank-sum test; χ² test of independence.

Table 3.

Odds ratios of nonzero coefficients in the lasso regression model for association with melanoma in the training data

Characteristic	Estimate	Odds ratio
Intercept	−1.413	0.24
Male	0.047	1.05
Distance to optic disc ≥3 mm versus <3 mm	−0.776	0.46
Height, mm	0.814	2.26
Subretinal fluid	1.089	2.97
Orange pigment	0.190	1.21
Drusen	−0.148	0.86

Fig. 1.

Calibration plot in the training data. The diagonal dashed black line represents perfect calibration. The points represent the average predicted and observed probability of melanoma within each quintile. The smooth line, surrounded by a 95% confidence interval, is the LOWESS.

After conducting bootstrap validation, we found that the optimism-corrected AUC was 0.849. The top four variables that most frequently had a nonzero coefficient in the lasso models fit to the 500 bootstrap samples also had a nonzero coefficient in our original lasso model: SRF, height, distance to optic nerve, and orange pigment (Fig. 2). They all had nonzero coefficients in over 50% of the bootstrap samples. Drusen was ranked fifth with a nonzero coefficient in 49% of bootstrap samples, and male versus female gender was ranked sixth with a nonzero coefficient in 33% of bootstrap samples.

Fig. 2.

Proportion of bootstrap samples in which each variable had a nonzero coefficient in the lasso model in the training data.

Not all variables analyzed in the training data were available in the validation data, but importantly, all variables with nonzero coefficients in the lasso model were available, so it was still possible to use these data for validation purposes. The available patient and disease characteristics of the training data were presented by melanoma status (Table 4). There were no significant differences according to gender, distance from optic disc, largest basal diameter, SRF, or drusen. The frequency of presentation with symptoms was lower for both groups in the validation data (6.1% and 27% in the nevus and melanoma groups) compared to the training data (15% and 39% in the nevus and melanoma groups). The nevus group more frequently had <3-mm distance from the optic disc (27%) compared to the training data (9.7%), and the melanoma group more frequently had <3-mm distance from the fovea (73%) compared to the training data (54%). Median largest basal diameter was 6.0 mm in both groups compared to 7.5 mm and 9.0 mm in the nevus and melanoma groups, respectively, in the training data. In the validation data, no patients in the melanoma group had SRF compared to 70% of melanoma patients in the training data.

Table 4.

Patient and disease characteristics by melanoma status in the validation data

Characteristic	Nonmelanoma (N = 229)1	Melanoma (N = 11)1	p value2
Gender
Female	151 (66)	4 (36)	0.056
Male	78 (34)	7 (64)
Symptoms	14 (6.1)	3 (27)	0.034
Distance from optic disc
<3 mm	62 (27)	5 (45)	0.2
≥3 mm	167 (73)	6 (55)
Distance from fovea
<3 mm	62 (27)	8 (73)	0.003
≥3 mm	167 (73)	3 (27)
Largest basal diameter, mm	6.00 (4.00, 8.00)	6.00 (5.25, 7.00)	0.3
Height, mm	1.28 (1.00, 1.50)	2.00 (1.94, 3.00)	<0.001
SRF	11 (4.8)	0 (0)	>0.9
Orange pigment	6 (2.6)	3 (27)	0.005
Drusen	176 (77)	7 (64)	0.3
RPE atrophy	140 (62)	8 (73)	0.8
Unknown	5	0

SRF, subretinal fluid; RPE, retinal pigment epithelium.

¹Statistics presented: n (%); median (IQR).

²Statistical tests performed: Fisher's exact test; Wilcoxon rank-sum test.

The AUC in the validation data was 0.861, which represents a high level of discrimination. However, the calibration was suboptimal (Fig. 3) with the model overpredicting the risk of melanoma across the entire range of prediction. The online calculator to generate a predicted probability of melanoma at present is available at https://riskcalc.org/SCMprediction/ (Fig. 4).

Fig. 3.

Calibration plot in the validation data. The diagonal dashed black line represents perfect calibration. The points represent the average predicted and observed probability of melanoma within each quintile. The smooth line, surrounded by a 95% confidence interval, is the LOWESS.

Fig. 4.

Online calculator to obtain a personalized predicted diagnostic probability of small choroidal melanoma.

Discussion

A choroidal nevus could be safely observed, whereas therapeutic intervention for small choroidal melanoma would be advisable with consequent risk of vision loss [22]. Therefore, an accurate classifier is needed to differentiate choroidal nevus from choroidal melanoma within the IMT group of tumors [23]. Here, we developed a prediction model to diagnose small choroidal melanoma. Our machine learning approach using lasso regression allows for automated selection of the most important variables for inclusion in the final model, by shrinking the coefficients of less important variables to zero. The model was internally validated using bootstrapping and externally validated on an independent dataset. Internal and external validation both help ensure that the model is not overfit, and that the performance is generalizable to new and different data from that on which the model was developed.

The model was based upon the training data that included positive cases classified as small choroidal melanoma (n = 61) either by growth (growth confirmed group, n = 30) or pathology (pathology confirmed group, n = 19) or both (combined group, n = 12). Diagnostic equivalence between the growth confirmed group and pathology confirmed has been previously established [18]. The negative controls (choroidal nevus, 62 patients) were observed with small choroidal melanocytic lesion that was stable with a minimum documented follow-up of 24 months. As malignant growth is detectable by ophthalmoscopy and/or ancillary studies such as photography and ultrasonography within 12- to 24-month period of observation revealing the lesion to be melanoma, stability beyond 24 months has virtually excluded the lesion being a choroidal melanoma until the last observation [18], although possibility of later growth due to malignant transformation (transformational growth) persists. Such distinction in growth patterns [24] and estimation of growth, quantified as growth rates, are therefore important prerequisites for establishing the diagnostic criteria for small choroidal melanoma [8].

Our statistical methods are robust. In the training data, lasso logistic regression was used to select variables for inclusion in the final model for the association with melanoma versus nevus cases. We evaluated the performance of the model using discrimination and calibration. Discrimination is a measure of how accurately patients are classified as having melanoma or not, as measured by the AUC, which can take values between 0.5 and 1 where 0.5 represents prediction as good as a flip of the coin and 1 represents perfect prediction. The optimism-corrected AUC of 0.849 in the training data and the AUC in the external validation data which was 0.861 represent high level of discrimination.

The calibration was reasonable in the training data; however, the calibration in the validation data was suboptimal with the model overpredicting the risk of melanoma across the entire range of predictions. This was likely caused by the differences in the characteristics of the two populations and the discrepancy in timing of patients in the two cohorts. Larger and more contemporary validation cohorts are needed to further confirm these findings. The online calculator to generate a predicted probability of melanoma at present is available at https://riskcalc.org/SCMprediction/ (Fig. 4).

The predictive model reported herein provides direct diagnostic prediction of the lesion being small choroidal melanoma expressed as probability (%). Such an estimate can facilitate decision-making regarding therapy based upon the data-driven model. In a setting with limited resources, the predictive model can be used for triaging patients for referral to a specialized center. The predictive model can also assist in generating inclusion criteria for intervention trials exploring novel therapies.

Limitations

Of note, the independent dataset used for external validation was unbalanced with only 11 melanoma cases of the total 240 cases. Access to balanced datasets is limited by the current practice patterns, given that many suspicious tumors likely to be melanoma are treated rather than observed to document growth. We are hoping to use additional datasets as and when they become available. The predictive value can be enhanced through training on a larger number of cases, considering a greater number of intrinsic and extrinsic variables, and validation on multiple independent datasets. Such approach may lead to identification of as yet undiscovered predictive factors. To this end, the dataset used in this study is available upon request for collaborative efforts. Moreover, the data input could be automated in the future to include images such as fundus photography, autofluorescence, optical coherence tomography, and ultrasonography forming a basis for artificial intelligence-based diagnostic approach.

In conclusion, to minimize diagnostic uncertainty surrounding small choroidal melanoma, use of a machine learning-based model provides a diagnostic prediction calculator that can be readily applied for decision-making and patient counseling. Further refinements can be undertaken with additional datasets, forming the basis for automated diagnosis.

Statement of Ethics

The study protocol (17-397) was approved by the Cleveland Clinic Institutional Review Board. Patient consent was not required as the study was based on review of medical records. The research complied with the guidelines for human studies, and the research was conducted ethically in accordance with the World Medical Association Declaration of Helsinki.

Conflict of Interest Statement

Arun D. Singh: Editor-in-Chief of Ocular Oncology and Pathology. Reported financial activities outside the submitted work: Aura Biosciences (stock options), IsoAid LLC (consultancy), Immunocore (consultancy), Isoaid (consultancy), and Eckert and Zeigler (consultancy). Other authors have no conflicts of interest to declare.

Funding Sources

This work was supported in part by an unrestricted grant from Research to Prevent Blindness to the Cole Eye Institute. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Author Contributions

All authors contributed to writing and editing of the manuscript.

Data Availability Statement

All data generated or analyzed during this study are available upon request from the corresponding author.