Machine learning classification of multiple sclerosis in children using optical coherence tomography

Abstract

Background:

In children, multiple sclerosis (MS) is the ultimate diagnosis in only 1/5 to 1/3 of cases after a first episode of central nervous system (CNS) demyelination. As the visual pathway is frequently affected in MS and other CNS demyelinating disorders (DDs), structural retinal imaging such as optical coherence tomography (OCT) can be used to differentiate MS.

Objective:

This study aimed to investigate the utility of machine learning (ML) based on OCT features to identify distinct structural retinal features in children with DDs.

Methods:

This study included 512 eyes from 187 (n_eyes = 374) children with demyelinating diseases and 69 (n_eyes = 138) controls. Input features of the analysis comprised of 24 auto-segmented OCT features.

Results:

Random Forest classifier with recursive feature elimination yielded the highest predictive values and identified DDs with 75% and MS with 80% accuracy, while multiclass distinction between MS and monophasic DD was performed with 64% accuracy. A set of eight retinal features were identified as the most important features in this classification.

Conclusion:

This study demonstrates that ML based on OCT features can be used to support a diagnosis of MS in children.

Introduction

Multiple sclerosis (MS) is the ultimate diagnosis after an episode of first-time demyelination in only one-fifth to one-third of children manifesting with acute central nervous system (CNS) demyelination and other monophasic or relapsing disorders can be seen in others. While magnetic resonance imaging (MRI) lesion patterns and serum markers such as myelin oligodendrocyte glycoprotein (MOG) IgG antibodies and aquaporin 4 (AQP4)-IgG antibodies assist in diagnosis, limitations exist. Given these considerations, a tool that is sensitive to finding common abnormalities in children with demyelinating disorders can be of value.

To this point, the visual pathway is affected in many or most children with MS, MOG antibody-associated disease (MOGAD) and AQP4-IgG seropositive neuromyelitis optica spectrum disorder (NMOSD) in the form of clinically apparent or clinically silent optic neuritis. Optical coherence tomography (OCT), a high-resolution tool that allows for a histological-level delineation of retinal structures, can be used to assess visual outcomes in children with demyelinating diseases. Studies performed with OCT in MS have shown changes in the average peripapillary retinal nerve fiber layer (RNFL) and average macular ganglion cell inner plexiform layer (GCIPL) thicknesses.^1–4 While overall thinning of the retinal nerve fiber layer and ganglion cell inner plexiform layer thicknesses have been shown in various demyelinating disorders,^1,3,5–8 it is yet unclear whether signature retinal profiles exist among these. Machine learning (ML) algorithms offer a suitable analysis method for this multi-dimensional OCT output obtained from various layers, sectors, and quadrants of retina, as they handle high-dimensional data well without the need for adjustments or making a priori assumptions.

In this study, we hypothesized that ML classifiers based on retinal OCT measures can be used to identify and to differentiate between MS, MOGAD, neuromyelitis optica spectrum disorders (NMOSDs) and monophasic acquired demyelinating syndromes (monoADS) in children. We explored the utility of supervised ML classifiers based on auto-segmented OCT features to identify which OCT features are the most distinctive features. Our secondary objective was to determine (1) which ML classifiers yielded the highest predictive accuracies and (2) which particular retinal features were the specific retinal features that were most likely to distinguish MS from other children.

Material and methods

Study design

This was an observational and cross-sectional study. The study was performed in compliance with the ethical codes of the Declaration of Helsinki. Institutional review board approval was obtained, and all participants and/or their guardians provided written informed consent. A flowchart of study participants is shown in Figure 1.

Figure 1.

Study flow chart per STROBE statement, summarizing recruitment into the study.

DD: demyelinating diseases; monoADS: monophasic acquired demyelinating syndrome; MS: multiple sclerosis; OCT: optical coherence tomography; RNFL: retinal nerve fiber layer. Refer to Reporting Guidelines Checklist for STROBE Statement, checklist of items. “n” denotes number of participants; “n_eyes” denotes number of eyes included in the analysis.

Participants

Healthy individuals and children with CNS demyelinating diseases were screened for eligibility. Consecutive participants were recruited through the Neuro-inflammatory Disorders Program at The Hospital for Sick Children (Toronto, Ontario, between the years 2010–2020) and The Alberta Children’s Hospital (Calgary, Alberta, between the years 2006–2016). Inclusion criteria were as follows: <18 years of age at the time of diagnosis, having at least one OCT scan available, and having an acquired demyelinating syndrome as per the international consensus definitions.^9–11

Children were classified as having MS following 2017 McDonald criteria,¹² NMOSD as defined by the 2015 Wingerchuk criteria,¹³ MOGAD per the international recommendations by Jarius et al. in 2018,¹⁴ and monoADS if they only had an incident demyelinating attack, such as optic neuritis, transverse myelitis, or other monofocal or polyfocal syndrome.¹¹ In participants who had more than one OCT scan available, the scan that was obtained at a date closest to the date of first attack were included for the purpose of this cross-sectional analysis. Both eyes of each participant, that is, clinically affected and unaffected eyes, were included in the analysis. Participants were excluded if they had other CNS conditions, had concomitant systemic inflammatory, genetic, neurodevelopmental or other diseases, had severe refractive errors (±6 diopters), or if they had acute optic neuritis to prevent the confounding effect of acute retinal swelling (i.e. visual testing performed within 3 months of a clinical episode of optic neuritis). Poor quality OCT scans, as indicated by a signal strength less than 7/10 or those that were artifactual were also excluded from the analysis. Healthy controls between the ages of 8–18 years were recruited using flyers and word of mouth, and the same exclusion criteria were applied.

Data collection

Demographic and clinical information, including age at disease onset, age at outcome measurement, sex, and cerebrospinal fluid oligoclonal bands status, was recorded using a standardized case report form.

OCT scans were acquired using two Cirrus Spectral Domain OCT Scanners (Model 4000 and 5000, Carl Zeiss Meditec, software version v7.0.3.19). All measurements were performed via the automated segmentation software pre-installed on the device. Trained technicians performed the scans in all participants in a dark room, without eye tracking. All scans were manually corrected if needed. From the eligible OCT scans, optic disk (200 × 200 cube) and macular cube (512 × 128 cube) protocols were included in the analysis for each eye. From the 6 × 6 mm³ cube captured by the optic disk cube scan, peripapillary RNFL thicknesses (µm) were measured at a 1.7 mm radius from the optic nerve head from all four quadrants. From the 6 mm × 6 mm area captured by the macular cube scans, macular sector thicknesses (µm), macular central subfield thickness (µm), macular cube volume (µm³), macular cube average thickness (µm), and combined ganglion cell inner plexiform layer thickness (µm) were measured. Macular sector thicknesses were measured as per the Early Treatment Diabetic Retinopathy Study grid system.¹⁵ GCIPL (µm) thickness was measured in an elliptical annulus centered at the fovea from the macular cube scans in six sectors. Average and minimum GCIPL, as well as the macular cube volume were also recorded. In total, 24 auto-segmented OCT features, all of which appear on conventional OCT printouts were included in the analysis (Figure 2).

Figure 2.

Schematic diagram of the optic disk cube quadrants and macular cube sectors as measured by the automated OCT scanning software.

GCIPL: ganglion cell inner plexiform layer; INL: inner nuclear layer; RNFL: retinal nerve fiber layer thickness. The RNFL thicknesses (microns) were measured globally and from all four quadrants in the optic disk cube scan (on the right; 2r = 2.4 mm): the superior, nasal, inferior, and temporal quadrants. The average of the measurements from these quadrants are given as the average RNFL thickness, not shown here. Two sets of measurements are obtained from the macular cube scans (on the left): the thickness of the GCIPL that lies between the retinal nerve fiber and inner nuclear layers, and the thickness of the macular sectors lying between the vitreoretinal and choriocapillary interfaces. The GCIPL thickness is measured off of six sectors in the macular cube scan (on the left, shown in black color; (2.0 × 2.4 mm)): the superior, superonasal, inferonasal, inferior, inferotemporal, and superotemporal sectors. The average of these is given as average GCIPL thickness, not shown here. The macular sector thicknesses (on the left, shown in white color) are measured off of three main regions: fovea at the center (2r = 1.5 mm), parafovea (2r = 2.5 mm, four subsectors: inner superior, inner nasal, inner inferior, inner temporal), and perifovea (2r = 5.5 mm, four subsectors: outer superior, outer nasal, outer inferior, and outer temporal). Fundus photo by Häggström.¹⁶ Public Domain. This file is licensed under the Creative Commons Attribution 2.0 Generic license. This fundus photo represents a schematic adaptation intended to demonstrate the optic disk cube and macular cube OCT features and is not intended to be completely anatomically accurate.

Statistical analysis

Statistical analysis was performed using commercially available software R (RStudio Inc., version 1.1.463, R project version 4.0.4) and JASP (JASP Team (2020) version 0.13.1). Demographic and clinical characteristics of the participants were compared using analysis of variance (ANOVA) and independent samples t-test for parametric variables, and Kruskal–Wallis and Mann–Whitney U-tests for non-parametric continuous variables, as appropriate.

Normality of sample distribution was evaluated using the Shapiro–Wilk’s test. Categorical variables were expressed as absolute (n) and percentage values and compared using chi-square or Fisher’s exact test, where appropriate. Continuous variables were expressed as mean ± standard deviation (SD). Imputation was not performed in the analysis where data were missing. A p value of <0.05 was considered to infer statistical significance.

ML analysis

For each analysis, we created a distinct balanced subset of the main data set, where data were split at eye level and each class had the same number of data points. After the subset generation, we performed random under-sampling of the larger classes to meet the count in the smallest class included in the specific analysis. With these pre-processing steps, we are able to achieve balanced subsets of the master data set for each of the four sets of analysis.¹⁷

In each analysis, the data set was split in an 80:20 fashion whereby the classifier was “trained” using 80% of the samples, and the subsequently trained model was tested on the remaining 20% of the samples to test the predictive accuracies. Five-fold cross-validation was performed in each analysis for random resampling of the initial 80/20 split of the data set, in order to evaluate the generalizability of the results. Performance of ML classifiers was evaluated using statistical measures of accuracy, sensitivity, and specificity as described in Supplementary Table 1.

The prediction performance of several supervised learning algorithms was evaluated and compared (Figure 3). Implementation-specific details can be found in the documentation of the Python package Scikit-learn: Machine Learning, Version 0.22.1.^18,19

Figure 3.

Heat map presenting the cross-validated predictive accuracies of all machine learning classifiers in four different sets of analyses.

HC: healthy controls; DD: demyelinating diseases; MS: multiple sclerosis; monoADS: monophasic acquired demyelinating syndrome; NMOSD: neuromyelitis optica spectrum disease; SVM: support vector machine; KNN: K-nearest neighbors; SGD: stochastic gradient descent.

For feature selection, recursive feature elimination (RFE), a wrapper method, designed around the baseline inferences drawn from the initial Random Forest model, again with five-fold cross-validation was implemented.¹⁸

Results

A total of 512 eyes from 69 (n_eyes = 138) controls and 187 (n_eyes = 374) children with demyelinating diseases were included in this analysis. Participants were children with MOGAD (n = 27, n_eyes = 54), MS (n = 57, n_eyes = 114), monoADS (n = 92, n_eyes = 184), and NMOSD (n = 11, n_eyes = 22). The basic demographic and clinical characteristics of the study population are summarized in Table 1.

Table 1.

Demographic and clinical characteristics of study participants by disease classes.

	Healthy controls, n = 69	MOGAD, n = 27	MS, n = 57	MonoADS, n = 92	NMOSDa, n = 11	P value
Females, n (%)	41 (59%)	17 (63%)	40 (70%)	37 (40%)	8 (73%)
Age at disease onset (y)	n/a	7.5 (4.3)	14.7 (3.4)	8.0 (6.7)	12.9 (2.1)	<0.001
Disease duration at the time of OCT scan (y)	n/a	2.0 (3.9)	0.6 (1.5)	1.9 (4.0)	1.7 (2.5)	<0.001
Oligoclonal bands positive: negative	n/a	1:4	19:0	5:19	1:1
Age at OCT scan (y)	15.1 (2.0)	9.9 (5.7)	15.7 (2.2)	11.7 (6.3)	15.2 (4.2)	<0.001

n/a: not applicable; MOGAD: myelin oligodendrocyte glycoprotein associated disease; MS: multiple sclerosis; MonoADS: monophasic acquired demyelinating syndrome; NMOSD: neuromyelitis optica spectrum disease; OCT: optical coherence tomography.

Results are reported as median (IQR) unless denoted otherwise.

^a2 (18%) children were seropositive for AQP4 antibodies.

The input features included 24 auto-segmented OCT features. The complete list of all input features is shown in Supplementary Table 2.

Baseline analysis and comparison of classifiers

In the baseline analysis, 10 different classifiers were used in four separate binary classification analyses. The details of these four analyses are given in the subsections to follow. Below, we summarize the comparison of ML classifiers. In each analysis, classes were balanced at the lowest number of eyes in either class with random under-sampling of the larger class. Results of five-fold cross-validated predictive accuracies of ML classifiers are summarized in Table 2. Overall, tree and boosted tree algorithms such as Random Forest, Decision Tree, AdaBoost, and XGBoost yielded higher predictive accuracies (Figure 3). Accordingly, the results of Random Forest algorithm have been chosen for reporting in the rest of this manuscript (Figure 4).

Table 2.

Accuracy, sensitivity, specificity, and AUC of the Random Forest Classifier before (baseline) and after the application of recursive feature elimination.

		HC vs. DD	HC vs. MS	MS vs. other DD	MS vs. MonoADS
Baseline Random Forest	Accuracy	0.72	0.79	0.68	0.63
	Sensitivity (SD)	0.62 (0.12)	0.76 (0.06)	0.69 (0.09)	0.58 (0.09)
	Specificity (SD)	0.83 (0.02)	0.82 (0.10)	0.67 (0.08)	0.69 (0.09)
	AUC (SD)	0.82 (0.04)	0.87 (0.04)	0.73 (0.04)	0.68 (0.07)
Recursive feature elimination	Accuracy	0.75	0.80	0.61	0.64
	Sensitivity (SD)	0.65 (0.12)	0.80 (0.09)	0.57 (0.10)	0.58 (0.10)
	Specificity (SD)	0.85 (0.05)	0.80 (0.06)	0.66 (0.08)	0.70 (0.8)
	AUC (SD)	0.84 (0.06)	0.85 (0.61)	0.65 (0.06)	0.68 (0.13)

DD: demyelinating diseases; HC: healthy controls; MonoADS: monophasic acquired demyelinating syndromes; MS: multiple sclerosis; RFE: recursive feature elimination; SD: standard deviation.

Figure 4.

ROC curves showing the five-fold cross-validation results of the binary classification between children with demyelinating diseases and controls.

Differentiating between healthy versus demyelinating eyes

To differentiate children with demyelinating diseases from controls, all demyelinating syndrome children were clustered in one class (n_eyes = 374 eyes of 187 children collectively with MS, monoADS, MOGAD, and NMOSD) and controls were clustered in another class (n = 138 eyes of 69 controls). The age at the time of OCT scan was significantly different between controls and collective group of children with demyelinating diseases (p < 0.001; mean ± SD: 14.7 ± 2.3 years and 12.8 ± 3.6 years, respectively). Classes were balanced at the lowest n of 138 eyes. Input features comprised of the 24 auto-segmented OCT features as listed in Supplementary Table 2. Random Forest classified controls versus children with demyelinating diseases with 0.72 five-fold cross-validated accuracy. Feature importance ranking indicated macular cube features to be important. RFE identified the optimal number of features as 15 (Table 2, Figure 5(a-b)).

Figure 5.

Feature importance plots per recursive feature elimination wrapped around the Random Forest classifier in identifying demyelinating diseases (a-b) and multiple sclerosis (c-d). (Right) Plot showing the number of OCT features required for input to achieve the highest accuracy; and (Left) Feature importance ranking of the best set of input features.

In a separate binary analysis, we evaluated control children (n_eyes = 138) and children with MS (n_eyes = 114). Classes were balanced at the lowest n_e of 114. Random Forest differentiated MS from controls with 0.79 accuracy with primarily the macular cube features and the temporal quadrant RNFL being the most important features. After RFE, eight features were identified as the optimal set yielding highest accuracy, increased from 0.79 to 0.80 (Table 2, Figure 5 (c-d)).

Differentiating MS from other CNS demyelinating disorders

In this analysis, 114 eyes of 57 MS children versus children with other demyelinating diseases (MOGAD, n_eyes = 54; monoADS, n_eyes = 184; and NMOSD, n_eyes = 22) were compared in a binary fashion to differentiate MS from other demyelinating diseases. In pre-processing, under-sampling was made to the lowest n of 114 eyes. Random Forest predicted children with MS from those with other demyelinating diseases with 0.68 accuracy (Table 2, Figure 6 (a-b)).

Figure 6.

Feature importance plots per recursive feature elimination wrapped around the Random Forest classifier in differentiating multiple sclerosis from other demyelinating diseases (a-b) and differentiating multiple sclerosis from monophasic acquired demyelinating syndromes (c-d). (Right) Plot showing the number of OCT features required for input to achieve the highest accuracy; and (Left) feature importance ranking of the best set of input features.

In a smaller subgroup analysis, we explored the distinction between children with MS (n_eyes = 114) and monoADS (n_eyes = 184). Classes were balanced at the lowest n_eyes of 114. Random Forest performed with a predictive accuracy of 0.63 with the optic cube followed by outer macular features as the most important. After RFE, 18 features were identified as the optimal set of features, with accuracy increased from 0.63 to 0.64 (Table 2, Figure 6 (c-d)).

Discussion

Our results demonstrate that ML classifiers based on OCT measures can be used to predict MS and other pediatric CNS demyelinating disorders in children with 75% and 80% accuracy, respectively. We found that the ganglion cell inner plexiform layer thickness in the superotemporal sector and retinal nerve fiber layer thickness in the temporal quadrant were among the most distinguishing features in identifying MS from controls. Among the supervised learning classifiers explored in this analysis, boosted tree algorithms provided the best fit to this model of OCT data analysis.

In this analysis, use of the Random Forest classifier identified MS as compared to controls with 80% accuracy (80% sensitivity and specificity). This sensitivity and specificity surpass VEP, where the sensitivity for predicting MS diagnosis in adults ranges between 20% and 50% in children with clinically isolated syndromes, and 25% and 50% in children with clinically definite MS.^20,21 Importantly, the sensitivity and specificity of OCT in diagnosing MS comes close to that of the current diagnostic gold standard, MRI. Studies looking at binary classification between MS and controls^22,23 or MS and NMOSD²⁴ using artificial intelligence have reported high sensitivity at 96% and 87%, respectively. Our results suggest that auto-segmented OCT features may be as useful and readily clinically applicable in diagnosing MS in children without any a priori criteria, although MRI may have greater sensitivity and specificity when screening for other pathologies. This should be the subject of future studies.

The multiclass distinction results yielded 63%–68% accuracy in identifying MS from monoADS or from other demyelinating diseases, respectively. The distinction between different demyelinating diseases is important for its therapeutic and prognostic implications, and several studies in children have suggested that distinct retinal patterns may exist in these disease entities.^6,8 However, there are no comparative studies on the identification of signature retinal profiles in CNS demyelinating diseases based on the full spectrum of OCT features in children. Future studies are needed to evaluate the sensitivity in differentiating different demyelinating diseases and to prune the set of distinctive retinal features.

Next, we found that the retinal ganglion cell inner plexiform layer thicknesses, primarily in the superior and superior temporal sectors could be used to distinguish children with demyelinating diseases from controls. Notably, we identified the specific feature of the RNFL thickness, particularly in the temporal quadrant, to favor a diagnosis of MS over other demyelinating diseases, in line with the studies reporting that the thinning of the temporal quadrant RNFL thickness could be a feature of MS.^3,25–27 For instance, Costello et al. showed that the temporal quadrant RNFL thickness was lower in children with relapsing remitting MS compared to those with a clinically isolated syndrome of an optic neuritis.²⁷ Lower temporal quadrant RNFL values were also confirmed in both optic neuritis and non-optic neuritis MS eyes in comparison to controls.^3,28 Furthermore, a previous study evaluating the utility of OCT in predicting MS using standard statistical methods showed that combined macular ganglion cell complex thickness and peripapillary retinal nerve fiber layer thickness measures differentiated MS from controls with an area under the receiver operating characteristics curve (AUC) of 0.85 and a sensitivity of 72% at 80% specificity,²⁹ suggesting a similar yield to our ML analysis.

We found that tree and boosted tree algorithms were of greatest utility for this heterogenous disease cohort with high-dimensional OCT data, that is, multi-dimensional OCT output obtained from various layers, sectors, and quadrants of the retina. Given that different ML classifiers utilize different mathematical and theoretical approaches in tackling the classification queries, there is no fixed answer as to which classifier should be used to obtain the most accurate results in a given data set. Factors such as the ability to handle sparse data with some incomplete datapoints versus a requirement for complete dense data, size of the training data, available computational power, and number of features in a classification query might impact algorithm selection at the onset. Ultimately, this selection of classifiers is best refined by actually running the algorithms and comparing their predictive performances.^30,31 With this perspective, we explored the performance of 10 different supervised learning algorithms. Of these, the tree and boosted tree algorithms appeared provide better fit to this model, not only in terms of their higher predictive performances but also for their potential for interpretability.

This study has limitations including data sparsity in less than 5% of all the datapoints for any single feature, unequal number of participants that was addressed by the use of random under-sampling, and differences in age and disease duration between groups. While the focus of this analysis was to identify specific structural qualities that can predict a diagnosis regardless of age, it would be an interesting future project to investigate differences between MS and other demyelinating disorders in cohorts matched for age. Another limitation is the exclusion of the eye status (i.e. history of optic neuritis, “eye affection status”). Each eye was included in the analysis independently, that is, data were split at eye level, as a data augmentation approach and also to minimize a priori subject characterizations. The ability to make direct predictions based on unadjusted data is one of the main advantages of ML over traditional statistical approaches and mimics the real-life practice as closely as possible. While ours was not a study of inter-eye differences in demyelinating diseases with and without optic neuritis and a subject-level data split would not be possible in this data set, addressing the eye affection status in future studies would be valuable to reproduce these results. Finally, we set the exclusion threshold for acute optic neuritis as 3 months, which may be considered early by some. Studies on the longitudinal trajectory of structural retinal changes after an episode of optic neuritis have reported ongoing changes lasting up to 12 months.^32–34 However, our decision is based on data that show that the most marked decline occurs in the first 3–4 months after an episode of optic neuritis,^32,35 with minimal changes thereafter.

Conclusion

This study demonstrates that ML classifiers based on structural OCT measures can be used to support a diagnosis of MS in children with high accuracy, sensitivity, and specificity. Macular cube features can help identify CNS demyelinating diseases, while retinal nerve fiber layer thickness measures from optic disk cube are more relevant in identifying a diagnosis of MS in children. While the predictive value is similar to other studies have used standard regression, this ML technique allows for the use of all features in the OCT and has highlighted specific anatomic areas that may be of high utility in differentiating MS from other disorders. This may be used to identify specific anatomic structures that have pathological significance in demyelinating disorders in children in future studies.