Artificial intelligence and machine learning in ocular oncology: Retinoblastoma

Abstract

Purpose:

This study was done to explore the utility of artificial intelligence (AI) and machine learning in the diagnosis and grouping of intraocular retinoblastoma (iRB).

Methods:

It was a retrospective observational study using AI and Machine learning, Computer Vision (OpenCV).

Results:

Of 771 fundus images of 109 eyes, 181 images had no tumor and 590 images displayed iRB based on review by two independent ocular oncologists (with an interobserver variability of <1%). The sensitivity, specificity, positive predictive value, and negative predictive value of the trained AI model were 85%, 99%, 99.6%, and 67%, respectively. Of 109 eyes, the sensitivity, specificity, positive predictive value, and negative predictive value for detection of RB by AI model were 96%, 94%, 97%, and 91%, respectively. Of these, the eyes were normal (n = 31) or belonged to groupA (n=1), B (n=22), C (n=8), D (n=23),and E (n=24) RB based on review by two independent ocular oncologists (with an interobserver variability of 0%). The sensitivity, specificity, positive predictive value, and negative predictive value of the trained AI model were 100%, 100%, 100%, and 100% for group A; 82%, 20 21 98%, 90%, and 96% for group B; 63%, 99%, 83%, and 97% for group C; 78%, 98%, 90%, and 94% for group D, and 92%, 91%, 73%, and 98% for group E, respectively.

Conclusion:

Based on our study, we conclude that the AI model for iRB is highly sensitive in the detection of RB with high specificity for the classification of iRB.

Though the concept of artificial intelligence (AI) dates back to 1950 and is based on Alan Turing’s experiment that gave rise to The Imitation Games,[1] the term AI was coined by John McCarthy in 1956 at the Dartmouth Summer Research Project on AI. The use of AI in medicine has gained importance in the past few decades. AI in medicine is either in virtual form via machine learning (ML) or deep learning aiding in the development of complex treatment algorithms, predictive models via electronic medical records and big data, or softbots[2,3]; or physical form via sophisticated medical devices and robots or carebots.[2,4]

In the virtual form, there have been various applications of AI and ML in the field of medicine mainly based on numerical data and digital images. The various applications in medicine include the use of epidemiology informatics in public health,[5] improvement of decision-making in anesthesiology,[6] improvement of precision medicine in the fields of reproductive medicine[7] and cardiovascular medicine,[8] improvement of diagnostic accuracy and objectivity in pathology with the advent of digital pathology,[9] improvement of technological aspects, image quality and image reading in nuclear medicine,[10] and also aiding in diagnosis, evaluation of prognosis, prediction of the epidemic, and drug discovery in the recent coronavirus disease 2019 (COVID-19) pandemic.[11] In the field of cancer, AI and ML have contributed to image analysis, improving diagnosis, multilayered data analysis with the integration of cancer genomics, development of treatment algorithms, estimating prognosis, drug discovery, and ultimately improving cancer healthcare to precision oncology.[12]

In recent years, there has been increasing interest in the use of AI and ML in the field of ophthalmology. Ophthalmology is a branch of medicine that uses various imaging modalities such as fundus photographs, optical coherence tomography, computed tomography, and magnetic resonance imaging; and is based on numerical data such as visual acuity, intraocular pressure, corneal thickness, refractive error, and cup to disc ratio for diagnosis and prognosis of various ophthalmic diseases. Numerical data and digital images have been used to develop AI and ML models for various diseases in ophthalmology. Till date, there have been no AI models for the diagnosis of retinoblastoma (RB) and herein, we explore the utility of AI and ML for the diagnosis and classification of intraocular RB.

Methods

This was a retrospective study and an approval from the Institutional Review Board was obtained. The study adhered to the tenets of the Declaration of Helsinki. Fundus photographs of children diagnosed with treatment-naïve unilateral or bilateral RB from January 2017 to June 2021 were retrieved from the institutional database. All images were captured on Retcam II (Clarity Medical Systems, Inc., Pleasanton, CA, USA). Poor-quality images with reflex/artifacts that impaired assessment were excluded. All images were read by two independent ocular oncologists (VSV and NG) and labeled as normal or RB. In images with tumors, a group label was assigned to every image according to the International Classification of Intraocular Retinoblastoma (ICIoR) as Group A, B, C, D, or E.[13] After reviewing all images of a single eye, a cumulative label was assigned as Normal, Group A, B, C, D, or E. The images in the “normal” group belonged to unaffected eyes of patients with unilateral RB. The discrepancy between labels assigned by VSV and NG was considered for interobserver variability. In cases of discrepancy of interpretation between the two ocular oncologists, the label assigned by VSV was taken as the gold standard for comparison with the results from the AI model.

The images that were manually labeled were also used to develop and test the AI model for RB by feature extraction and the use of an RB classifier. These are described in detail below.

Feature extraction

The features needed for the classification of RB according to ICIoR were extracted using a series of algorithms that involved deep learning object detection, computer vision, and other geometry and intensity gradation algorithms. Open Computer Vision (OpenCV) techniques and deep learning (DL) models were employed to extract, identify, or calculate the following features from every image: hough circle (i.e. entire area of fundus captured within an image), optic disc (detection), tumor (detection), maximum tumor size (mm), number of tumors, distance between optic disc and the closest tumor, vitreous seeds, subretinal seeds, number of seeds, maximum distance between seeds and the nearest tumor, blood vessels, and intraocular hemorrhage. Feature vectors representing the image were used to train the RB classifier of the AI model [Fig. 1]. Eighty percent of the images in the dataset were used to train the classifier and 20% were used to test the RB classifier.

Figure 1.

Overview of methodology employed in training and testing of retinoblastoma classifier

For the detection of optic disc and tumor, MultiLabel Classification Model of Deep Learning (DL) was utilized on training and test images to predict a bounding box and class with a confidence score for each prediction [Fig. 2a and b]. Transfer learning[14] was used for training the AI model in the identification of optic disc and tumors.

Figure 2.

Algorithms used for identification features from fundus images. (a) and (b) Bounding boxes with confidence score for optic disc and tumor. (c) and (d) Calculation of tumor size with reference to optic disc and hough circle, respectively. (e) and (f) Identification of regions of interest (tumor seeds) and vitreous seeds within mapped areas of interest by OpenCV algorithm and Deep Learning model. (g) and (h) Mapping of hemorrhage overlying the tumor using combination of OpenCV and Deep Learning methods. (i) to (j) Mapping of blood vessels overlying the tumor by combination of OpenCV and Deep Learning methods

Transfer learning implementation used an efficient Mobile-Net v2 SSD-based deep learning model[15] as a base and removed the last layer with a combined optic disc and tumor detection. To train this model, we manually labeled optic disc and tumors in fundus images with rectangular boxes. A total of 327 images were labeled where 285 images were used for training and 42 images were used for validation. The modified deep learning network was trained on the labeled optic disc and tumor fundus images.

After the optic disc and tumors were identified with a degree of confidence, the relative distance between them and the size of the tumor was estimated. Estimation of tumor size was performed with a reference diameter of 1.5 mm for optic disc [Fig. 2c] when the latter was detected in an image. In the absence of an optic disc within the image frame, the calculation was based on the hough circular diameter of 12 mm, which was averaged from all images containing an optic disc [Fig. 2d].

Detection of tumor seeds was complex and the tumor seeds were identified by a two-step approach. The region of interest was identified by an OpenCV algorithm, which was further processed by DL model to identify the seeds [Fig. 2e and 2f]. Identification of regions of interest was done in eight directions from the center of the tumor to the edge of the hough circle diameter of the image by calculating the mean of the pixel intensity. A threshold was determined using the minimum of these means. Once a threshold was established, another pass through the eight directions was performed to determine the areas of interest where there could be tumor seeds that have pixel intensity above the threshold and away from the tumor. Depending on the direction, a directional straight line or a diagonal, either a rectangle or a square area of interest was selected. A trained deep learning object classifier that is trained specifically for seeds detection was used to identify the presence of tumor seeds and when present, the maximum distance of the seeds from the nearest tumor was computed. This model was trained and validated on 132 images.

Localization of intraocular hemorrhage [Fig. 2g and h] and blood vessels [Fig. 2i and 2j] were mapped by OpenCV and combination of OpenCV & DL models, respectively.[16] Calculations of distance of tumor from optic disc and size of the tumor were based on OpenCV algorithms and geometry, respectively.

Training, testing, and validation of images

Images were first split into 80% train-validation and 20% test sets. The 80% train-validation set is further split it 80% train and 20% validation sets. Within this overall train-validation set, cross-validation allowed models to train and test on multiple data sets (randomly chosen in each iteration of the training) [Table 1].

Table 1.

Distribution of images and eyes used for training, validation, and test

Learning model	Number of images labeled for training + validation + test of RB classifier (n)	Number of images used for training (n)	Number of images used for cross validation (n)	Number of images used for test (n)
Deep learning (MultiLabel Classification - Tumor + OD)	327	228	57	42
Deep Learning (Seeds)	132*	84	21	27
Final RB Classifier	771	493	123	155

*The Deep Learning (seeds) Model was automatically run on the 327 Fundus Images that were identified as having a tumor (by the Deep learning [MultiLabel Classification - OD + Tumour] Model) and are flagged as possible Regions of Interest (ROI) w.r.t seeds (by an OpenCV Algorithm run on the image)

Assigning group labels with RB classifier

Each of the fundus images was run through all the algorithms to obtain a feature vector that consisted of features including detection of the optic disc, tumor detection, determination of tumor size (≤3 mm or >3 mm), total number of tumors, detection of intraocular hemorrhage, localization of overlying retinal blood vessels, the distance between optic disc and tumor, presence of subretinal/vitreous seeds, number of tumor seeds, and maximum distance between seeds and the nearest tumor (≤3 mm or >3 mm).

Using the group classification labels by experts, a decision tree machine learning model[17] was trained. This RB classifier model was trained on 616 images and tested on 155 images.

The final RB classification was based on an ensemble of the RB classifier results of all the images of a given eye of the patient, i.e., the interpretation for an “eye” was cumulative from individual images. Following analysis of all images of one eye, the highest label was assigned as the group label for the eye.

Performance metrics

Performance metrics were calculated for the entire data set that included both “training” and “test” images. Parameters assessed included overall accuracy, misclassification as RB, underclassification rates, and overclassification rates. Sensitivity, specificity, positive predictive value, negative predictive value for detection of RB, and classification of RB were calculated. Performance metrics were calculated for every image and also for every eye since each eye had multiple images ranging between 4 and 20.

Results

Seven hundred and seventy-one fundus photographs from 109 eyes of 62 patients diagnosed with treatment naïve unilateral or bilateral retinoblastoma between January 2017 and June 2021 were retrieved from the institutional database. Of 771 fundus images, 181 images had no tumor and 590 images had iRB based on review by two ocular oncologists. The eyes were normal (n = 31) or harbored group A (n = 1), B (n = 22), C (n = 8), D (n = 23), and E (n = 24) RB. Interobserver variability in detection of RB or classification of RB was 0.98% for individual images and 0% based on cumulative results for each eye [Table 2].

Table 2.

Group labels assigned to of 771 images from 109 eyes by observers*

ICIoR Group	Images	Eyes
Normal	181	31
A	6	1
B	204	22
C	70	8
D	182	23
E	128	24
Total	771	109

*Interobserver variability between the two observers was 0.98% for images and 0% for eyes; ICIoR: International Classification for Intraocular Retinoblastoma

Performance metrics

Labels predicted by the RB classifier against true labels assigned by the observers are depicted in Fig. 3. Of 590 images with RB, the accuracy for detection of RB was 85%, with RB being detected in 500 images. Although 90 images (15%) were misclassified as normal, RB was detected but underclassified in 45 images (8%). Thus, a total of 135 images (23%) were underclassified. Of 181 images with no tumor, RB was detected in two (1%). Metrics for grouping of eyes yielded higher accuracy: of 78 eyes with RB, RB was identified in 75 (96%) with underclassification in one (1%) eye, and misclassified as normal in three (4%). Thus, a total of four eyes (5%) were underclassified. Of 31 normal eyes, 29 (94%) were classified as normal and two (6%) were misclassified as RB. Training accuracy including cross-validation of the RB classifier was 80.3% and testing accuracy was 74.2%.

Figure 3.

Performance metrics of retinoblastoma (RB) classifier for detection and classification of retinoblastoma. (a) and (b) Grids depicting labels assigned by observers versus labels assigned by RB classifier for 771 images and 109 eyes, respectively. (c) and (d) Graphical summary of accurate, overclassification, and underclassification by RB classifier in 771 images and 109 eyes, respectively

Test characteristics

The sensitivity and specificity of the trained AI model for the detection of RB in an image was 85% and 99%, respectively. Of 109 eyes, the sensitivity and specificity for detection of RB by AI model were 96% and 94%, respectively. The sensitivity and specificity of RB classification were 100% and 100% for group A, 82% and 98% for group B, 63% and 99% for group C; 78% and 98% for group D, and 92% and 91% for group E, respectively. Test characteristics for eyes in each group are summarized in Table 3.

Table 3.

Performance metrics of RB classifier

	Detection of RB (n=109 eyes; 771 images)	Normal (n=31 eyes; 181 images)	A (n=1 eye; 6 images)	B (n=22 eyes; 204 images)	C (n=8 eyes; 70 images)	D (n=23 eyes; 182 images)	E (n=24 eyes; 128 images)
For individual images
Sensitivity	85%	99%	17%	79%	84%	73%	60%
Specificity	99%	85%	100%	96%	97%	97%	99%
PPV	100%	67%	100%	87%	73%	89%	90%
NPV	67%	100%	95%	93%	98%	92%	100%
For all images of an eye
Sensitivity	96%	94%	100%	82%	63%	78%	92%
Specificity	94%	96%	100%	98%	99%	98%	91%
PPV	97%	91%	100%	90%	83%	90%	73%
NPV	91%	98%	100%	96%	97%	94%	98%

RB: retinoblastoma; PPV: positive predictive value; NPV: negative predictive value

Discussion

Although there has been increasing use of AI in daily life with Apple Inc.’s Siri and Amazon’s Alexa, there has also been increasing applications of AI and ML in healthcare. Despite increasing discussions about AI and ML in healthcare in the literature, it was estimated that 100% of US healthcare was delivered without any use of AI in 2017.[18] However, it is predicted that there will be increased use of AI and ML in healthcare in the future, with certain tasks being overpowered by AI-enabled technology.

In the field of ophthalmology, AI and ML have been capable of analyzing quantifiable data and images, thus aiding in the diagnosis and staging of various ocular pathologies including strabismus, refractive error, keratitis, keratoconus, cataracts, glaucoma, papilledema, optic disc abnormalities, diabetic retinopathy, age-related macular degeneration, retinopathy of prematurity, thyroid eye disease, choroidal nevus, and transformation to melanoma.[19-23] AI has also been used in the development of treatment algorithms, assessing treatment needs, and prognostication of ocular pathologies.[19-21]

Retinoblastoma is the most common malignant intraocular tumor in children with most patients presenting with advanced disease at presentation in lower and lower-middle-income countries.[24-26] Advanced disease at presentation results in poor chances of vision, globe, and life salvage. Locally advanced disease is mainly related to delayed presentation and delayed diagnosis.[24] The diagnosis of RB is mainly dependent on symptoms like leukocoria and strabismus and red reflex testing, whereas most countries do not have routine fundus screening protocols in children.[27] In this study, there were more number of group D and E (n = 47; 60%) eyes during the study period compared with Groups A to C (n = 31; 40%) eyes suggestive of a higher proportion of patients with advanced RB at presentation.

Fundus photographs have been the basis for the development of AI models for various retinal pathologies including diabetic retinopathy, age-related macular degeneration, and retinopathy of prematurity. AI and ML for diabetic retinopathy screening using fundus photographs have shown a sensitivity of 88% to 100% and specificity of 91% to 99% in detecting moderate diabetic retinopathy or worse.[28-30] Fundus image analysis has assisted in AI-related screening for age-related macular degeneration with an accuracy of 88% to 92%.[31] Similarly, AI models have been developed for retinopathy of prematurity for the detection of plus disease based on fundus photographs with 93% to 100% sensitivity and 78% to 94% specificity.[32,33] In this study, we used fundus photographs of children with treatment naïve RB to develop an AI model for the diagnosis and classification of RB. The most common DL method used for the analysis of medical images is a convoluted neural network (CNN).[34] In our study, we developed multiple CNN-based DL models and developed OpenCV-based computer vision algorithms for extracting features and machine learning classification algorithm for RB classification based on the extracted features. With the use of DL and OpenCV algorithm, the AI model achieved a sensitivity of 96% and specificity of 94% for the detection of RB.

ICIoR classification is based on quantifiable data such as the size of tumor (≤ or >3 mm), distance of tumor from optic disc and fovea, distance of tumor seeds from the tumor, and number of seeds.[13] Using the DL models, this data could be extracted from the fundus images. This allowed classification of intraocular RB with the AI model, which was based on fundus images and quantitative data. The AI model achieved sensitivity of 63% to 100% and specificity of 91% to 100% for the classification of RB based on ICIoR.

The limitations of the study include relatively smaller sample size and unequal distribution of RB in different groups based on ICIoR. Since the study was purely aimed at training RB classifier to assess fundus images, anterior segment features predictive of RB group could not be incorporated in the present stage. Also, we have not tested this model on fundus images of different races or on fundus images captured on different cameras. Thus the applicability of this AI model in these above circumstances is unknown. Furthermore, to expand the scope of utility on par with AI for other retinal disorders such as diabetic retinopathy,[28-30] age-related macular degeneration[31], and retinopathy of prematurity,[32,33] the RB classifier needs more work, which is underway. Nevertheless, this is the first AI model developed for the detection and grouping of RB and there is scope for improving the accuracy of the model with the use of more DL algorithms. The authors do acknowledge that although AI cannot replace a trained ocular oncologist, it has the potential to greatly aid in the screening and early detection of RB. Assessment of the accuracy of RB classifier by location of tumor, and identification of mimickers of RB is underway through multicenter collaboration. We believe this would expand the scope and utility of AI in early detection of RB.

Conclusion

In conclusion, this AI tool is a promising screening tool for RB. It has shown high sensitivity and specificity for the detection of RB, though the sensitivity and specificity are variable for grouping of intraocular RB. Based on our study, this AI model performs well on images captured on a wide-field fundus camera. If this AI model can be used on fundus images captured on an inexpensive nonmydriatic fundus camera, it can facilitate mass screening of children in the community, so as to diagnose RB early before the onset of leukocoria or strabismus.

Financial support and sponsorship

This study was supported by The Operation Eyesight Universal Institute for Eye Cancer (SK) and Hyderabad Eye Research Foundation (SK), Hyderabad, India. The funders had no role in the preparation, review, or approval of the manuscript.

Conflicts of interest

There are no conflicts of interest.