Development and validation of a multi-stage self-supervised learning model for optical coherence tomography image classification

Abstract

Objective

This study aimed to develop a novel multi-stage self-supervised learning model tailored for the accurate classification of optical coherence tomography (OCT) images in ophthalmology reducing reliance on costly labeled datasets while maintaining high diagnostic accuracy.

Materials and Methods

A private dataset of 2719 OCT images from 493 patients was employed, along with 3 public datasets comprising 84 484 images from 4686 patients, 3231 images from 45 patients, and 572 images. Extensive internal, external, and clinical validation were performed to assess model performance. Grad-CAM was employed for qualitative analysis to interpret the model’s decisions by highlighting relevant areas. Subsampling analyses evaluated the model’s robustness with varying labeled data availability.

Results

The proposed model outperformed conventional supervised or self-supervised learning-based models, achieving state-of-the-art results across 3 public datasets. In a clinical validation, the model exhibited up to 17.50% higher accuracy and 17.53% higher macro F-1 score than a supervised learning-based model under limited training data.

Discussion

The model’s robustness in OCT image classification underscores the potential of the multi-stage self-supervised learning to address challenges associated with limited labeled data. The availability of source codes and pre-trained models promotes the use of this model in a variety of clinical settings, facilitating broader adoption.

Conclusion

This model offers a promising solution for advancing OCT image classification, achieving high accuracy while reducing the cost of extensive expert annotation and potentially streamlining clinical workflows, thereby supporting more efficient patient management.

Introduction

Optical coherence tomography (OCT) plays a pivotal role in medical diagnostics, particularly in ophthalmology, for obtaining high-resolution, cross-sectional images of eye tissues. This noninvasive imaging technique enables ophthalmologists to visualize and assess the retina and optic nerve,¹ thus aiding in the early detection and accurate diagnosis of ocular diseases, such as age-related macular degeneration (AMD), drusen, and diabetic macular edema (DME). Early detection is of paramount importance for the prevention of vision loss and the preservation of visual acuity.^2,3 As of 2020, approximately 196 million individuals globally were affected by AMD, with a projected increase to 288 million by 2040. This highlights the significant global burden of disease and the necessity for early detection.^4–6

OCT enables clinicians to identify subtle disease progression signs. However, the intricate nature of OCT images necessitates manual assessment by skilled ophthalmologists, which can result in significant levels of fatigue and subjectivity. Consequently, developing an automated, precise, and efficient OCT image classification system is imperative. The integration of artificial intelligence in medical imaging has enhanced efficiency, as evidenced by reductions in diagnostic time, clinician workload, and healthcare costs.^7–9

Deep learning models have demonstrated efficacy in image classification problems, autonomously extracting features and analyzing intricate relationships from high-resolution images.^10–14 Numerous deep learning models for OCT image classification have been introduced to seize its outstanding performance.^15–17 Lee et al¹⁵ achieved 87.63% accuracy by training the model based on the VGG16 using 101 002 OCT images with 2 labels (normal and AMD). Li et al¹⁶ achieved 97.30% accuracy by training the model based on the ResNet-50 using 21 357 OCT images with 4 labels (normal, choroidal neovascularization [CNV], DME, and drusen). Kamran et al¹⁷ achieved 99.34% accuracy by training the model based on the ResNet-50 using 84 484 OCT images. However, these methods necessitate a substantial number of OCT images, incurring high labeling costs. Consequently, the effectiveness of these models depends on the data quality and quantity.

Self-supervised learning techniques have been proposed for the effective training of deep learning models with limited data.^18–21 Among the various self-supervised learning methods, the contrastive self-supervised learning approaches, such as SimCLR,²¹ MOCO,²⁰ and SwAV²² are widely utilized. This approach involves learning representations by contrasting positive data pairs against negative data pairs, which drives the model to differentiate between them in the embedding space. In numerous studies, self-supervised contrastive learning has consistently demonstrated superior model performance, even in medical fields with limited labeling.^23–26 For instance, Soni et al²³ employed contrastive learning to diagnose heart and lung diseases from digital stethoscope data. Similarly, Zhang et al,²⁴ Han et al,²⁵ and Azizi et al²⁶ utilized contrastive learning to diagnose diseases from chest radiography data. Most relevantly, Fang et al²⁷ achieved superior performance in OCT image classification compared with conventional deep learning models by utilizing a self-supervised learning scheme. However, the model’s generalizability is limited because it was only trained and evaluated on a single specific OCT dataset.

This study proposes a novel method that employs a self-supervised learning approach, augmented by multi-staging techniques, to achieve high performance that can be widely applicable in practical clinical environments, even when using a limited amount of annotated data. Our model incorporates a multi-staging approach comprising 2 phases: a self-supervised learning phase derived from pre-trained datasets, and a fine-tuning phase for use with a private or target dataset. The model was trained on 2 large public datasets and subsequently evaluated on another public dataset or a private dataset collected from a tertiary general university hospital. Furthermore, the pre-trained model and associated code are made available to the public as a user-accessible training framework.

Our major contributions are summarized as follows:

• We achieved state-of-the-art performance in OCT image classification task by leveraging our proposed multi-stage self-supervised learning scheme.

• The proposed method can decrease the costs of annotating OCT images while retaining the superior performance of the deep learning model.

• We validated the effectiveness of the model which was trained using our proposed method, on both publicly available datasets and a dataset collected from a real clinical setting.

• To the best of our knowledge, this is the first publicly available pre-trained OCT image classification model that provides a training framework that allows the use of personal clinical data, thus making it accessible to everyone.

Methods

This study was planned in accordance with the “Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research” statements.²⁸

Data collection, preprocessing, and ethical approval

Two publicly available OCT image datasets were utilized to train our model: OCT2017 and Srinivasan2014.^29,30 The publicly available datasets were employed for self-supervised learning, with each dataset corresponding to a specific stage of the process. It is notable that the training dataset labels were excluded during self-supervised learning training, given that only the image data itself was being considered. In addition, sets of predefined test images were reserved for internal validation and comparison of performance with other studies (see Figure S1).

Furthermore, 2 additional OCT image datasets were employed to validate our approach: a public dataset and a private dataset. For external validation of the trained model, this study utilizes another small open dataset³¹ known as OCTID. To validate the clinical utility, we constructed a private OCT dataset, abbreviated as KNUH-OCT in this study, from a clinical tertiary hospital. The OCT images were acquired using a Heidelberg Spectralis OCT machine at Kyungpook National University Hospital (KNUH). Three skilled ophthalmologists (7, 13, and 15 years of experience) performed OCT image labeling. Two independent ophthalmologists (C.G.Y. and J.R.D.) labeled the OCT images into 4 categories; in case of inconsistencies, a senior masked retinal specialist (Y.K.K.) independently re-evaluated all images and adjudicated the discrepancies. For clinical validation, 200 OCT images were randomly selected, while the remaining 2519 images were used for training. The acquisition and analysis of the private KNUH-OCT dataset have been approved by the Institutional Review Board of KNUH (IRB-2019-07-022). Table 1 and Method S1 present detailed information about the class labels and statistics included in each dataset.

Table 1.

OCT dataset statistics.

Dataset	Class type	OCT2017	Srinivasan-2014	OCTID	KNUH-OCT
Train	Normal	26 315	1302	186	525
	CNVa	37 205	–	–	1028
	DMEb	11 348	996	–	458
	AMDc	–	618	35	–
	Drusen	8616	–	–	508
	MHd	–	–	82	–
	CSRe	–	–	82	–
	DRf	–	–	87	–
Test	Normal	250	105	20	50
	CNVa	250	–	–	50
	DMEb	250	105	–	50
	AMDc	–	105	20	–
	Drusen	250	–	–	50
	MHd	–	–	20	–
	CSRe	–	–	20	–
	DRf	–	–	20	–
Total images		84 484	3231	572	2719

^aCNV: choroidal neovascularization

^bDME: diabetic macular edema

^cAMD: age-related macular degeneration

^dMH: macular hole

^eCSR: central serous retinopathy

^fDR: diabetic retinopathy

The OCT images from the public datasets (OCT2017, Srinivasan2014, and OCTID) were resized to 512 × 512 pixels and normalized by their respective means and standard deviations. Conversely, images from the private KNUH-OCT dataset were cropped from 1520 × 596 pixels to 1024 × 496 pixels and normalized in a manner analogous to that described above. To ensure robust training and enhance the generalizability of our model, we employed image augmentation techniques such as random resized cropping, random horizontal flipping, and Gaussian blurring. A detailed description of the image preprocessing is provided in Method S1.

Multi-stage self-supervised learning and model training

We propose a novel model-building strategy that considers the consistency of OCT image characteristics and the variability of OCT image quality resulting from different measurement instruments. Since OCT images consistently present the patient’s condition in a similar way, we anticipate that a self-supervised learning scheme can enhance the model’s performance by retrieving inherent feature information from multiple sources of data and learning from the images themselves. During deep learning model training, subtle differences or variances in equipment from various manufacturers and settings could lead to training inefficiencies. In consideration of these issues, we developed a multi-stage self-supervised learning approach and trained a novel deep learning model.

In each stage of our self-supervised learning phase, we employed SimCLR according to our previous experiments.³² The stages were divided according to the dataset used. SimCLR learns similarities by maximizing the similarity between positive pairs, generated from different views of the same image, and the dissimilarity between negative pairs, generated from different images. It was selected due to its demonstrated efficacy in addressing the distinctive attributes of OCT images, including the low variance in lesion distribution and fixed image orientation, by employing a contrastive learning scheme. SimCLR has demonstrated superior performance in capturing discriminative features essential for differentiating between diseased and non-diseased images. The ResNet-50¹⁴ architecture was employed as the baseline convolutional neural network (CNN) model in this study.

Figure 1 depicts the proposed modeling framework. In Stage 1, self-supervised learning was conducted on the OCT2017²⁹ open dataset, which constituted a large-scale OCT image dataset. Subsequently, the CNN model that had been trained in Stage 1 was transferred to the CNN model that was to be trained in Stage 2. In Stage 2, self-supervised learning occurred on the Srinivasan2014³⁰ open dataset using a CNN model that was initialized with weights transferred from Stage 1. Then, the CNN model trained in Stage 2 was transferred to the CNN model in Stage 3. Stage 3 is designed to be optional, providing users with the flexibility to train the CNN model with their own downstream OCT image dataset if they so desire. This customization allows the model to be tailored to specific clinical environments, thus enhancing its suitability for specific clinical applications. Stage 3-1 follows, during which the CNN model was fine-tuned using a supervised learning scheme with labels. A small subset (20%) of labeled images from the training datasets was used for fine-tuning in Stage 3-1. Further details regarding the data usage in our method are provided in Figure S1.

Figure 1.

Multi-stage self-supervised learning model. The model consists of 3 self-supervised learning stages and 1 fine-tuning stage. In the self-supervised learning stage, the model is trained with only the training set images themselves without labels to learn robust feature representations. In the fine-tuning stage, the model is fine-tuned with a small subset (20%) of the training dataset with labels to adapt the model to specific tasks. The CNN encoder trained in the previous stage is transferred to the CNN encoder of the next stage for continuous learning. Abbreviations: CNN = convolutional neural network.

In all experiments, training was performed on the entire training images, and evaluation was performed using isolated testing images from each dataset (see Figure S1). The model training for all experiments consisted of 100 epochs. When transferring the CNN model, we implemented full-weight transfer without freezing any layers. Further details about the model training can be found in Method S2.

Performance evaluation

To ensure a comprehensive assessment of the models, several performance metrics were employed for evaluation. These include precision, recall, and F-1 score for each class, in addition to accuracy, macro precision, macro recall, and macro F-1 score, which are employed to evaluate the models’ performance across all classes. Moreover, confusion matrices were constructed to assess the model’s strengths and weaknesses in each class.

To assess the effects of the multi-stage self-supervised learning, we compared model performance through an ablation study involving 4 datasets: 2 open datasets for internal validation (OCT2017, Srinivasan2014), 1 open dataset for external validation (OCTID), and a private dataset for clinical validation (KNUH-OCT). First, the model was trained through a supervised learning scheme. Second, the model was trained via self-supervised learning without staging and only trained on a single dataset. Finally, the model was trained using our proposed multi-stage self-supervised learning method. For internal validation, Stage 3 was omitted and proceeded directly to Stage 3-1 (fine-tuning) with only 20% of the labeled images before evaluation. To comprehensively assess the effectiveness of our method, we conducted 2 evaluations. A predefined dataset evaluation was conducted on discrete test sets that had been previously partitioned for evaluation purposes to facilitate comparison with existing literature. In order to ensure the robust assessment of model stability, a 5-fold cross-validation (5-CV) was performed. In addition, to evaluate the statistical difference between models trained on different methods, the 5 $\times$ 2 cv test³³ was employed. A P-value of <.001 was deemed to be statistically significant.

To evaluate the efficacy of multi-staging, we compared the model’s performance with that of baseline and single-stage models trained by combining all the data used for self-supervised learning into one, without stage distinction. The training data were shuffled in this experiment for baseline and single-stage model training to ensure robust training.

Furthermore, the performance of the model was also compared based on the order of the datasets used in multi-stage training, where each dataset was trained on separate stages in ascending and descending order by data size.

Subsampling analyses were conducted to evaluate the efficiency of our framework in a small dataset. For the OCTID dataset, we trained our models by reducing the number of randomly sampled images from the data from 472 to 120. Similarly, for the KNUH-OCT dataset, we trained the models by decreasing the number of randomly selected images from 2519 to 500. These experiments were designed to assess how well our proposed model would perform in a small hospital with limited data.

Grad-CAM analysis

The qualitative analysis of Gradient-weighted Class Activation Mapping (Grad-CAM)³⁴ visually examines how effectively our model, trained using multi-stage self-supervised learning, focuses on disease-relevant areas compared to the supervised learning-based baseline model. We analyzed and compared the Grad-CAM results from both the baseline model and our model using the diagnosis results of ophthalmologists. To facilitate comparison with the Grad-CAM heatmaps, 3 ophthalmologists delineated the disease regions in red circles. The annotation of the disease regions was subjected to a second review and confirmation by 3 ophthalmology specialists, who applied the same labeling rules utilized in annotating our private dataset. This clinical analysis enabled us to assess whether the regions highlighted by our model correspond to those that are clinically important for diagnosing conditions like CNV and DME.

Data and code availability

All data, code, and the pre-trained model are publicly available except for the KNUH-OCT dataset. Further details regarding data availability can be found in the “Data Availability” section below.

Results

OCT dataset statistics

Table 1 presents the number of images in each class for the training set and test set of the 4 OCT datasets. The OCT2017²⁹ dataset comprises 84 484 OCT images labeled as normal, CNV, DME, and drusen. The Srinivasan2014³⁰ dataset comprises 3231 OCT images labeled as normal, DME, and AMD, obtained from 45 patients. The OCTID³¹ dataset consists of 572 OCT images labeled as normal, AMD, macular hole (MH), central serous retinopathy (CSR), and diabetic retinopathy (DR). All the aforementioned publicly available datasets include each predefined testing dataset. This testing dataset comprises 1000 images from 633 patients, 315 images from 3 patients, and 100 images for OCT2017, Srinivasan2014, and OCTID, respectively. The KNUH-OCT dataset comprises 2719 OCT images labeled as normal, CNV, DME, and drusen, from 493 patients. The training set comprises 2519 images from 457 patients and the test set comprises 200 images from 36 patients. The demographic information of the KNUH-OCT dataset can be found in Table S1.

Performance evaluation

Table 2 presents the results of the predefined dataset evaluation comparing the performance of the proposed model with that of previous works, the baseline model, and the single-stage self-supervised learning model for each dataset. These results demonstrate that our method outperforms the baseline model across all OCT datasets, achieving higher accuracy, macro precision, macro recall, and macro F-1 score. Furthermore, our approach demonstrates superior performance to single-stage self-supervised learning, except for the OCT2017 dataset. Notable performance enhancements were identified, particularly for small-scale data such as the OCTID with an improvement of 22.00% from supervised learning. Moreover, our model achieved state-of-the-art performance compared to the base performance of previous works.

Table 2.

Performance of the multi-stage self-supervised learning (predefined dataset evaluation).

Dataset (# classes)	Models	Classes	Precision	Recall	F-1 score	Accuracy	Macro precision	Macro recall	Macro F-1 score
OCT2017 (4)	Best performance of previous worksa11	Normal CNV DME Drusen	100.00 99.20 98.81 99.20	99.60 98.80 99.60 99.80	99.80 99.00 99.20 99.20	99.30	99.30	99.30	99.30
	Baseline model (Supervised learning)	Normal CNV DME Drusen	100.00 98.80 98.41 98.81	99.20 98.40 99.20 99.60	99.60 98.80 98.80 99.20	99.10	99.10	99.10	99.10
	Single-stage self-supervised learning model	Normal CNV DME Drusen	100.00 100.00 99.60 100.00	100.00 99.60 100.00 100.00	100.00 99.80 99.80 100.00	99.90	99.90	99.90	99.90
	Multi-stage self-supervised learning model (our model)	Normal CNV DME Drusen	100.00 99.60 99.60 100.00	100.00 99.60 99.60 100.00	100.00 99.60 99.60 100.00	99.80	99.80	99.80	99.80
Srinivasan2014 (3)	Best performance of previous worksa11	Normal DME AMD	95.33 96.17 94.29	97.14 94.29 94.29	96.23 95.19 94.29	95.24	95.24	95.24	95.23
	Baseline model (Supervised learning)	Normal DME AMD	95.33 97.12 96.15	97.14 96.20 95.24	96.23 96.65 95.69	96.19	96.20	96.19	96.19
	Single-stage self-supervised learning model	Normal DME AMD	90.74 92.23 90.38	93.33 90.48 89.52	92.02 91.35 89.95	91.11	91.12	91.11	91.11
	Multi-stage self-supervised learning model (our model)	Normal DME AMD	100.00 100.00 100.00	100.00 100.00 100.00	100.00 100.00 100.00	100.00	100.00	100.00	100.00
OCTID (5)	Baseline model (Supervised learning)	Normal AMD MH CSR DR	65.38 57.89 77.78 68.42 77.78	85.00 55.00 70.00 65.00 70.00	73.91 56.41 73.68 66.67 73.68	69.00	69.45	69.00	68.87
	Single-stage self-supervised learning model	Normal AMD MH CSR DR	58.33 40.91 61.11 58.82 63.16	70.00 45.00 55.00 50.00 60.00	63.64 42.86 57.89 54.05 61.54	56.00	56.47	56.00	56.00
	Multi-stage self-supervised learning model (our model)	Normal AMD MH CSR DR	90.91 84.21 94.74 90.48 94.74	100.00 80.00 90.00 95.00 90.00	95.24 82.05 92.31 92.68 92.31	91.00	91.01	91.00	90.92
KNUH-OCT (4)	Baseline model (Supervised learning)	Normal CNV DME Drusen	94.12 90.38 93.75 93.88	96.00 94.00 90.00 92.00	95.05 92.16 91.84 92.93	93.00	93.03	93.00	92.99
	Single-stage self-supervised learning model	Normal CNV DME Drusen	86.54 81.13 82.98 85.42	90.00 86.00 78.00 82.00	88.24 83.50 80.41 83.67	84.00	84.02	84.00	83.95
	Multi-stage self-supervised learning model (our model)	Normal CNV DME Drusen	100.00 96.08 95.92 96.00	100.00 98.00 94.00 96.00	100.00 97.03 94.95 96.00	97.00	97.00	97.00	96.99

^aThe model was retrained using the author’s source code to prevent dataset leakage concerns. See the “Discussion” section for the details.

Abbreviations: CNV = choroidal neovascularization; DME = diabetic macular edema; AMD = age-related macular degeneration; MH = macular hole; CSR = central serous retinopathy; DR = diabetic retinopathy.

Figure 2 shows each model’s confusion matrices for the KNUH-OCT dataset classification results from predefined dataset evaluation. The matrices highlight the superior performance of our model compared to the baseline and single-stage self-supervised learning models. Confusion matrices for the other dataset classification results are provided in Figure S2.

Figure 2.

Confusion matrices for the KNUH-OCT dataset classification results. The matrices show the classification performance across 4 classes: Normal, choroidal neovascularization (CNV), diabetic macular edema (DME), and drusen. (A) Confusion matrix of the baseline model (conventional supervised learning model) on the KNUH-OCT dataset. (B) Confusion matrix of the single-stage self-supervised learning model on the KNUH-OCT dataset. (C) Confusion matrix of the multi-stage self-supervised learning model on the KNUH-OCT dataset. Abbreviations: OCT = optical coherence tomography.

Table 3 provides the performance from the 5-CV evaluation. Our method consistently achieved superior performance over both the baseline and single-stage self-supervised learning models, excelling in all performance metrics. P-values for comparisons with both the baseline and single-stage self-supervised learning models were below .001, indicating statistically significant improvements, except for the OCT2017 dataset, where our model showed a statistically superior performance to the baseline model. These results highlight the robust performance improvement of our method, especially on smaller datasets. Detailed results of the statistical test can be found in Table S4.

Table 3.

Performance of the multi-stage self-supervised learning (5-CV evaluation).

Dataset (# classes)	Models	Classes	Precision	Recall	F-1 score	Accuracy	Macro precision	Macro recall	Macro F-1 score	P-valuea
OCT2017 (4)	Baseline model (Supervised learning)	Normal CNV DME Drusen	98.67$\pm$0.42 99.14$\pm$0.39 97.39$\pm$0.86 96.16$\pm$1.31	98.86$\pm$0.40 99.24$\pm$0.16 97.13$\pm$1.38 95.49$\pm$1.81	98.76$\pm$0.41 99.19$\pm$0.26 97.26$\pm$1.11 95.82$\pm$1.55	98.44$\pm$0.56	97.84$\pm$0.74	97.68$\pm$0.92	97.76$\pm$0.83	0.0012
	Single-stage self-supervised learning model	Normal CNV DME Drusen	99.35$\pm$0.21 99.58$\pm$0.18 98.39$\pm$0.89 98.07$\pm$1.09	99.32$\pm$0.38 99.54$\pm$0.35 98.95$\pm$0.83 97.55$\pm$0.33	99.34$\pm$0.28 99.56$\pm$0.27 98.67$\pm$0.83 97.81$\pm$0.60	99.18$\pm$0.38	98.85$\pm$0.59	98.84$\pm$0.39	98.84$\pm$0.49	0.5725
	Multi-stage self-supervised learning model (our model)	Normal CNV DME Drusen	99.32$\pm$0.25 99.57$\pm$0.17 98.51$\pm$0.66 98.34$\pm$0.53	99.40$\pm$0.28 99.40$\pm$0.28 98.80$\pm$0.52 98.41$\pm$0.29	99.36$\pm$0.26 99.48$\pm$0.22 98.66$\pm$0.58 98.38$\pm$0.36	99.21$\pm$0.29	98.94$\pm$0.38	99.00$\pm$0.31	98.97$\pm$0.35	N/A
Srinivasan2014 (3)	Baseline model (Supervised learning)	Normal DME AMD	95.54$\pm$1.18 94.83$\pm$1.38 92.49$\pm$1.75	95.81$\pm$1.19 94.92$\pm$1.51 91.84$\pm$1.57	95.67$\pm$1.14 94.87$\pm$1.44 92.16$\pm$1.44	94.62$\pm$1.30	94.28$\pm$1.35	94.19$\pm$1.33	94.23$\pm$1.33	<0.001
	Single-stage self-supervised learning model	Normal DME AMD	93.50$\pm$0.61 90.49$\pm$1.13 85.30$\pm$1.53	90.97$\pm$1.11 91.55$\pm$0.88 88.24$\pm$0.97	92.22$\pm$0.78 91.02$\pm$0.95 86.75$\pm$1.24	90.56$\pm$0.93	89.76$\pm$1.01	90.26$\pm$0.92	89.99$\pm$0.97	<0.001
	Multi-stage self-supervised learning model (our model)	Normal DME AMD	99.29$\pm$0.50 99.28$\pm$0.75 99.44$\pm$0.59	99.50$\pm$0.60 99.64$\pm$0.38 98.48$\pm$1.13	99.40$\pm$0.51 99.46$\pm$0.52 98.96$\pm$0.82	99.32$\pm$0.56	99.34$\pm$0.56	99.21$\pm$0.63	99.27$\pm$0.59	N/A
OCTID (5)	Baseline model (Supervised learning)	Normal AMD MH CSR DR	82.23$\pm$2.49 36.82$\pm$9.44 67.42$\pm$5.70 66.65$\pm$6.55 69.75$\pm$4.83	74.75$\pm$7.26 47.27$\pm$7.61 67.71$\pm$3.55 65.71$\pm$2.95 71.00$\pm$6.27	78.23$\pm$5.15 41.29$\pm$8.66 67.39$\pm$2.93 66.13$\pm$4.58 70.30$\pm$4.95	68.52$\pm$4.82	64.57$\pm$4.84	65.29$\pm$4.61	64.67$\pm$4.83	<0.001
	Single-stage self-supervised learning model	Normal AMD MH CSR DR	79.30$\pm$1.75 19.50$\pm$2.51 54.97$\pm$5.51 54.33$\pm$3.12 60.20$\pm$3.68	57.79$\pm$4.03 38.18$\pm$4.07 56.90$\pm$3.29 55.90$\pm$2.84 57.92$\pm$3.53	66.82$\pm$3.18 25.80$\pm$3.10 55.87$\pm$4.17 55.09$\pm$2.77 59.02$\pm$3.88	55.43$\pm$3.20	53.66$\pm$2.87	53.34$\pm$3.05	52.52$\pm$3.05	<0.001
	Multi-stage self-supervised learning model (our model)	Normal AMD MH CSR DR	99.01$\pm$1.35 59.62$\pm$8.21 92.78$\pm$2.80 90.29$\pm$5.84 91.03$\pm$3.01	95.63$\pm$2.67 69.09$\pm$8.13 88.19$\pm$2.92 89.29$\pm$3.85 94.42$\pm$1.96	97.28$\pm$1.61 63.95$\pm$8.03 90.40$\pm$2.34 89.75$\pm$4.55 92.66$\pm$1.88	90.38$\pm$2.80	86.54$\pm$3.49	87.32$\pm$3.36	86.81$\pm$3.45	N/A
KNUH-OCT (4)	Baseline model (Supervised learning)	Normal CNV DME Drusen	88.44$\pm$1.84 95.96$\pm$1.01 85.49$\pm$2.73 87.73$\pm$1.44	89.04$\pm$1.91 92.67$\pm$1.70 88.98$\pm$1.58 89.61$\pm$1.34	88.73$\pm$1.74 94.28$\pm$1.23 87.19$\pm$2.13 88.66$\pm$1.29	90.58$\pm$1.48	89.40$\pm$1.58	90.08$\pm$1.48	89.72$\pm$1.54	<0.001
	Single-stage self-supervised learning model	Normal CNV DME Drusen	77.83$\pm$1.89 91.73$\pm$0.57 75.61$\pm$1.92 78.23$\pm$1.67	79.30$\pm$1.56 88.50$\pm$1.29 79.93$\pm$2.09 77.95$\pm$1.80	78.56$\pm$1.62 90.08$\pm$0.74 77.71$\pm$1.95 78.09$\pm$1.71	82.79$\pm$1.26	80.85$\pm$1.43	81.42$\pm$1.43	81.11$\pm$1.42	<0.001
	Multi-stage self-supervised learning model (our model)	Normal CNV DME Drusen	94.31$\pm$1.13 98.05$\pm$0.62 94.51$\pm$1.59 95.47$\pm$0.61	95.13$\pm$0.78 98.14$\pm$1.18 94.69$\pm$1.11 94.27$\pm$0.47	94.72$\pm$0.93 98.10$\pm$0.90 94.60$\pm$1.32 94.86$\pm$0.49	96.06$\pm$0.88	95.59$\pm$0.94	95.56$\pm$0.81	95.57$\pm$0.88	N/A

^a P-value was calculated using 5 $\times$ 2 cv³³ test between each model and multi-stage self-supervised learning (our) model.

Abbreviations: CNV = choroidal neovascularization; DME = diabetic macular edema; AMD = age-related macular degeneration; MH = macular hole; CSR = central serous retinopathy; DR = diabetic retinopathy.

Table S2 illustrates the effect of stage distinction in our training process using 5-CV evaluation. Note that the baseline and single-stage self-supervised learning models were trained on a merged dataset comprising all available datasets, our multi-stage self-supervised learning model utilized distinct datasets at different stages of the training process. The multi-stage model consistently achieved the highest performance across all datasets, demonstrating its effectiveness. Table S3 compares the performance of the models based on the order of the datasets used at each stage in the multi-stage training. The results show that the multi-stage training with descending order by data size outperforms that of the ascending order across all OCT datasets.

Subsampling analysis

Table 4 presents the results of the subsampling analysis conducted on the OCTID dataset and the KNUH-OCT dataset. The accuracy of the model increased steadily as the data size grew, from 84.00% to 91.00% for OCTID and 92.00% to 97.00% for KNUH-OCT. Our model outperformed the supervised learning-based baseline model at all training data sizes. In the analysis of 120 images from the OCTID dataset, our model demonstrated an accuracy of 84.00%, whereas the supervised learning-based model exhibited an accuracy of only 50.00%. In the analysis of 500 images from the KNUH-OCT dataset, our model achieved an accuracy of 92.00%, whereas the supervised learning-based model achieved only 74.50%. Subsampling analysis for OCT2017 and Srinivasan2014 datasets are provided in Table S5.

Table 4.

Subsampling analysis.

Dataset (Training data size)	Models	Accuracy	Macro precision	Macro recall	Macro F-1 score
OCTID (472)	Baseline model (Supervised learning)	69.00	69.45	69.00	68.87
	Single-stage self-supervised learning model	56.00	56.47	56.00	56.00
	Multi-stage self-supervised learning model (Our model)	91.00	91.01	91.00	90.92
OCTID (237)	Baseline model (Supervised learning)	55.00	53.87	55.00	54.60
	Single-stage self-supervised learning model	51.00	51.48	51.00	51.10
	Multi-stage self-supervised learning model (Our model)	87.00	87.22	87.00	87.07
OCTID (120)	Baseline model (Supervised learning)	50.00	49.88	50.00	49.82
	Single-stage self-supervised learning model	48.00	48.49	48.00	48.06
	Multi-stage self-supervised learning model (Our model)	84.00	84.12	84.00	84.02
KNUH-OCT (2519)	Baseline model (Supervised learning)	93.00	93.03	93.00	92.99
	Single-stage self-supervised learning model	84.00	84.02	84.00	83.95
	Multi-stage self-supervised learning model (Our model)	97.00	97.00	97.00	96.99
KNUH-OCT (1500)	Baseline model (Supervised learning)	90.00	90.11	90.00	89.99
	Single-stage self-supervised learning model	82.00	81.96	82.00	81.96
	Multi-stage self-supervised learning model (Our model)	95.50	95.62	95.50	95.53
KNUH-OCT (500)	Baseline model (Supervised learning)	74.50	74.50	74.50	74.48
	Single-stage self-supervised learning model	78.50	78.47	78.50	78.48
	Multi-stage self-supervised learning model (Our model)	92.00	92.07	92.00	92.01

Grad-CAM analysis

Figure 3 displays the Grad-CAM results obtained from the supervised learning-based baseline model and our multi-stage self-supervised learning model for 4 different types of OCT images. The first column shows the original OCT images. The second and third columns present the Grad-CAM results obtained from the baseline and our proposed models, respectively. In these results, the red-marked heatmap area indicates the model’s focus.

Figure 3.

An example of Grad-CAM results in the KNUH-OCT test set. (A) Original OCT images from the test set, showing different ophthalmic conditions. For comparison with the Grad-CAM heatmaps, 3 ophthalmologists have circled the disease areas in red. (B) Grad-CAM heatmaps generated by the baseline model, highlighting the regions used for predictions. (C) Grad-CAM heatmaps generated by our model, showing the regions of interest that the model focused on for making predictions. Abbreviations: OCT = optical coherence tomography; Grad-CAM = gradient-weighted class activation mapping.

In Figure 3(2), the baseline model was unable to focus on CNV lesions. Conversely, our model effectively identified CNV lesions in both the outer retina and the retinal pigment epithelium, as well as the thickened choroidal layer. This figure illustrates the characteristics of polypoidal choroidal vasculopathy, a type of neovascular AMD associated with choroidal thickening.³⁵ In Figure 3(3), our model distinctly targets the outer retina and choroidal layer in DME lesions. DR is marked by choroidal thinning due to ischemia,³⁶ and our model accurately reflects this feature, whereas the baseline model focused on completely different regions. Furthermore, Figure 3(4) confirms that the baseline model focused on other regions other than drusen lesions. In contrast, our model identified the exact location of drusen bodies, which are well-defined hyperreflective deposits located between the basal membrane of the retinal pigment epithelium and Bruch’s membrane. Grad-CAM results of OCT2017, Srinivasan2014, and OCTID datasets are provided in Figure S3.

Discussion

In this study, we propose a multi-stage self-supervised learning approach that accurately classifies OCT images with limited data. We trained a model using the proposed method and compared its performance with both external and clinical validation. Our model outperforms traditional supervised learning-based models and the best-performing models in the literature. Across the 4 datasets used for evaluation, our model demonstrated an improvement in accuracy ranging from 0.7% to 22% compared with the conventional supervised learning-based model. Our model still showed superior performance in the 5-CV evaluation, highlighting its robustness. Remarkably, our models exhibited acceptable performance even with extremely small-scale OCT datasets. When we trained our model using only 20% of the data collected in a clinical setting, the supervised learning-based model underperformed by 18.5% compared to the full data training, whereas our model underperformed by only 5%. These findings support the efficacy of the proposed multi-stage learning approach. Moreover, our proposed model demonstrated superior performance compared to models trained on pooled datasets across all datasets. This underscores the efficacy of our multi-stage self-supervised learning strategy, which leverages the distinctive attributes of each dataset at different stages, rather than relying on a single merged dataset. The findings highlight that the enhanced performance of our model is a direct consequence of the stage distinction, rather than merely the quantity of data utilized for training. We also found that the performance is influenced by the order in which data are used during multi-stage training. When initially trained on smaller datasets, the performance decreased up to 3%. This finding suggests that transferring the latent features of OCT images from large OCT datasets to the next stage improves accuracy.

Clinical image data, such as OCT, possess unique characteristics that differ from those of general images. First, labeling incurs high costs. As clinical experts must manually inspect and label each image, obtaining and maintaining the large amounts of labeled data required for high-performance deep-learning model training are challenging. Second, variability exists in the clinical environment, such as regarding type of machine, settings, and measurer. Finally, owing to privacy concerns, hospitals seldom release sensitive clinical data, impeding the retention of large data. Consequently, most hospitals or users interested in building an OCT classifier typically have access to only small-scale clinical datasets, making it difficult to generate a robust deep-learning model.

We developed a method that considers relevant clinical data characteristics. Initially, we decreased labeling costs and the demand for a large dataset using self-supervised learning. The model acquires knowledge of the OCT images through contrastive learning by utilizing 2 open OCT datasets of significant sizes. Subsequently, we address the inefficiency of training a model owing to OCT image quality variability by adopting multi-stage learning. By ensuring that the model learns only from a single dataset in a single stage, learning inefficiencies that may arise from mixing images of varying qualities are avoided. Additionally, we released a pre-trained model with large-scale public data and provided a framework for users to fine-tune the model according to our methodology. This enables each hospital or user to own a superior OCT classifier that uses only their small data with minimal privacy concerns.

Most previous studies on classifying OCT images used supervised learning models that require abundant data and labels such as over 10 000 OCT images.^15,16,37 Additionally, most current studies that trained and assessed deep learning models used an undisclosed OCT dataset created by the researchers themselves. To ensure a reliable performance evaluation of deep learning models, it is essential to test their performance using various datasets, such as external or clinical validation datasets.

When evaluating model performance, the testing data should be entirely separate from other data, such as training or validation data. Unfortunately, we found that the Optic-Net,¹⁷ the current best-performing model for the OCT2017 and Srinivasan2014 datasets, had data leakage concerns. Specifically, the source code they provided utilized the same dataset for both validation and testing. Consequently, although the model may perform well in a given test dataset, its real-world clinical utility can be called into question. An accuracy of 99.80% in the OCT2017 dataset and a perfect score of 100.00% in the Srinivasan2014 dataset were reported; however, after re-training with separate testing data, our reproduced results showed a decline in performance to 99.30% and 95.24% for each respective dataset.

Our model’s performance was evaluated both reliably and objectively. We analyzed the performance internally, externally, and clinically and confirmed the superiority of the model trained using our proposed method. Furthermore, we publicly released the codes for transparent replication of our results.

The model has significant clinical potential. It enhances diagnostic accuracy even with limited data, which is critical for precision in resource-constrained environments. This automated, reliable classification model facilitates faster diagnostics and timely medical interventions, providing valuable second opinions and assisting clinicians with challenging conditions. By making our pre-trained models publicly available, we enable hospitals to enhance their diagnostic capabilities with minimal additional data and training, democratizing advanced tools and ensuring broader access.

Our study has some limitations. First, the CNN model performance can be enhanced by optimizing the model architecture and hyperparameters, which were not rigorously explored in this study. However, we made all the training and testing source codes of our proposed method available, enabling users to update the model architecture as desired. Second, the scope of our experiments was confined to OCT image classification. However, the proposed method has the potential to lay the groundwork for expanding its application to various modalities and tasks. Third, although we evaluated our model externally and clinically, this was a retrospective study that requires prospective validation for real-world clinical usage.

Conclusion

We developed and validated a multi-stage self-supervised learning model to classify OCT images. This approach allows the model to learn inherent features without high labeling costs and train efficiently without confusion over image quality. We believe that our contributions will facilitate the adoption of high-performing OCT classifiers in clinical practice, improving patient outcomes.

Author contributions

Sungho Shim, Min-Soo Kim, Hyun-Lim Yang (Conceptualization), Sungho Shim, Che Gyem Yae, Yong Koo Kang, Jae Rock Do, Hong Kyun Kim (Data curation), Sungho Shim, Hyun-Lim Yang (Formal analysis), Hong Kyun Kim (Funding acquisition), Sungho Shim, Min-Soo Kim, Hyun-Lim Yang (Investigation), Sungho Shim, Hyun-Lim Yang (Methodology), Min-Soo Kim, Che Gyem Yae, Yong Koo Kang, Jae Rock Do, Hong Kyun Kim (Resources), Sungho Shim (Software), Min-Soo Kim, Hyun-Lim Yang (Supervision), Sungho Shim, Hyun-Lim Yang (Validation), Sungho Shim (Visualization), Sungho Shim, Hyun-Lim Yang (Writing—original draft), Sungho Shim, Min-Soo Kim, Hong Kyun Kim, Hyun-Lim Yang (Writing—review and editing).

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was supported by a Biomedical Research Institute Grant, Kyungpook National University Hospital (2018); and the National Research Foundation of Korea (NRF) grant funded by the Ministry of Science and ICT (MSIT), Korea [grant numbers NRF-2018R1A5A1060031 and NRF-2021R1A2C2013939].

Conflicts of interest

The authors have declared that no competing interest exists.

Data availability

The data used to develop and validate this study are publicly available, except for the KNUH-OCT dataset. Due to the ethical restrictions imposed by the IRB, the private dataset is only available upon request. The source code, including data preprocessing, model building, model training, performance evaluation, weights of the pre-trained CNN model, and a user-customizable training framework, is available for use with personal clinical data through a public repository (https://github.com/Mercury22/multi-stage-self-supervised-learning-model-for-OCT-classification). The pre-trained CNN model has undergone training until Stage 2. The framework provided can execute both Stage 3 and Stage 3-1 consecutively when the user follows the prescribed folder structure to store their own OCT images.