Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: A proof-of-concept study

Abstract

Background

Intrapapillary capillary loops (IPCLs) represent an endoscopically visible feature of early squamous cell neoplasia (ESCN) which correlate with invasion depth – an important factor in the success of curative endoscopic therapy. IPCLs visualised on magnification endoscopy with Narrow Band Imaging (ME-NBI) can be used to train convolutional neural networks (CNNs) to detect the presence and classify staging of ESCN lesions.

Methods

A total of 7046 sequential high-definition ME-NBI images from 17 patients (10 ESCN, 7 normal) were used to train a CNN. IPCL patterns were classified by three expert endoscopists according to the Japanese Endoscopic Society classification. Normal IPCLs were defined as type A, abnormal as B1–3. Matched histology was obtained for all imaged areas.

Results

This CNN differentiates abnormal from normal IPCL patterns with 93.7% accuracy (86.2% to 98.3%) and sensitivity and specificity for classifying abnormal IPCL patterns of 89.3% (78.1% to 100%) and 98% (92% to 99.7%), respectively. Our CNN operates in real time with diagnostic prediction times between 26.17 ms and 37.48 ms.

Conclusion

Our novel and proof-of-concept application of computer-aided endoscopic diagnosis shows that a CNN can accurately classify IPCL patterns as normal or abnormal. This system could be used as an in vivo, real-time clinical decision support tool for endoscopists assessing and directing local therapy of ESCN.

Key summary

Established knowledge on the subject

• The Japanese Endoscopic Society classification provides a clinically useful and simplified system by which to predict histology in early squamous lesions based on intrapapillary capillary loop (IPCL) patterns.

• Artificial intelligence (AI) may have a role in improving clinicians endoscopic recognition of IPCL patterns – no previous study has yet been reported in the literature.

Significant new findings in this study

• We introduce a novel AI system capable of classifying IPCL patterns as neoplastic or non-neoplastic in real time, using magnification endoscopy with Narrow Band Imaging images acquired during endoscopy.

• Our system demonstrates an impressive accuracy and could potentially assist clinicians in the recognition of neoplastic tissue, as well as providing a clinical decision support tool to guide endoscopists considering whether lesions are amenable to endoscopic resection.

• Our system provides clinically interpretable images of which IPCLs are abnormal to assist clinicians delineating lesions.

• Our system provides a performance benchmark for the assessment of other AI systems designed for this purpose in the future.

Introduction

Oesophageal squamous cell carcinoma is the eighth most common cause of cancer worldwide and the sixth most common cause of cancer deaths,¹ with the highest incidence across a ‘cancer belt’ extending from East Africa and across the Middle East to China and Japan.^1–3 Gastroscopy remains the investigation of choice for the diagnosis of early squamous cell neoplasia (ESCN). The endoscopic features of these early lesions are subtle and easily missed during endoscopy – studies show a significant miss rate for upper gastrointestinal tract cancers on endoscopies undertaken within three years of diagnosis.⁴ Early detection and accurate characterisation of ESCN lesions are vital to predicting histology and guiding intervention. Lesions confined to the mucosa have low rates of local lymph node metastasis (<2%) compared to lesions invading the submucosa (SM) (8% to 45.9%) and so are amenable to endoscopic therapy.^5–8 Endoscopic therapy, through endoscopic mucosal resection (EMR) or submucosal dissection (ESD), is associated with excellent rates of five-year survival⁹ and spares patients the attendant morbidity and mortality of oesophagectomy after a late diagnosis.^10,11

Intrapapillary capillary loops (IPCLs) are microvessels that were first characterised using magnification endoscopy (ME).^12,13 They are now an established marker of ESCN, and changes in their morphology correlate with the invasion depth of ESCN.^14,15 Normal IPCLs arise from the submucosal vessels, run adjacent to the basal layer of the oesophageal epithelium and are seen as fine-calibre looped structures on ME. As an ESCN progresses there is stepwise destruction of the oesophageal wall architecture, which manifests as morphologic changes in the ICPL pattern. Initially IPCLs become more tortuous and dilated. As the ESCN progresses and interrupts the mucosal layer, the IPCLs lose their looped structure and form linear dilated vessels. Avascular areas form between these non-looped vessels which also correspond to invasion depths deep in the mucosal layer. As the ESCN invades the deeper submucosal layer, the IPCLs are almost completely obliterated. In their place neovascularisation occurs with the formation of tortuous, grossly dilated and non-looped vessels.^16–18

Advanced endoscopic-imaging modalities, such as Narrow Band Imaging (NBI), in combination with ME, afford improved visualisation of subtle microvascular patterns in the oesophageal mucosa of patients with ESCN.¹⁹ Several classifications have been proposed to define the abnormal ICPL morphologies that correlate with histological invasion depth of ESCN lesions. Classifications described by Inoue et al.^12,13 and Arima and colleagues²⁰ associate progressive morphological abnormalities in IPCLs with deeper invasion of the ESCN.¹⁴ These classifications are complex and require a high degree of interpretation by endoscopists – as such their utility in a clinical setting is debatable.

The recently published Japanese Endoscopic Society (JES) IPCL classification is a comparably simplified system, allowing easy recognition of ESCN by endoscopists.^15,21 The JES classification has become widely accepted in areas of high prevalence such as China and Japan. Importantly, each ICPL subgroup corresponds with high accuracy to a given histological grade and invasion depth of ESCN – with increasing irregularity of the IPCL patterns representing more advanced, invasive disease¹⁵ (Figure 1(a) and (b)). The accuracy of the JES classification as reported by Oyama et al is high compared to other classifications – with the overall accuracy for histology prediction 90.5% across type B1–3. Overall accuracies for histology prediction were 91.9%, 93.4% and 95.9% for type B1, B2 and B3 IPCL patterns, respectively.¹⁵ Kim and colleagues also report excellent interobserver agreement using the JES classification.²²

Figure 1.

(a) Representative examples of lesions containing each of the JES IPCL subtypes, with a description of key features and the associated histological invasion depth that correlates with each type. (b) Summary of the JES magnification endoscopy IPCL classification system; typical invasion depths of ESCN compared with observed IPCL patterns. EET: endoscopic eradication therapy; ESCN: early squamous cell neoplasia; HGD: high-grade dysplasia; HGIN: high-grade intraepithelial neoplasia; IPCL: intrapapillary capillary loop; JES: Japanese Endoscopic Society; LGD: low-grade dysplasia; LGIN: low-grade intraepithelial neoplasia; LP: lamina propria; MM: muscularis mucosa; SM: submucosa.

Computer-aided endoscopic diagnosis (CAED) with artificial intelligence (AI) could provide a useful adjunct to endoscopists assessing ESCN lesions; a system capable of recognising and highlighting the presence of ESCN could improve detection rates. CAED using convolutional neural networks (CNNs) has shown great potential in a range of medical specialties, but its use in the recognition of endoscopically detected neoplasia is in its infancy.²³ We propose that the distinct patterns observed in IPCL morphology could provide the input data to train a CNN to classify such microvasculature patterns as normal or abnormal. Such a system could improve and automate the characterisation and prediction of ESCN invasion depth, which would be particularly useful in settings in which endoscopists are unfamiliar with the endoscopic appearance of early squamous lesions. Furthermore, a downstream application of such a validated system, trained to differentiate IPCL subtypes, has a role as a support tool for endoscopists to triage lesions as either amenable or unamenable to endoscopic eradication therapy (EET) – based on the predicted invasion depth deduced from the IPCL patterns, reducing procedure time and preventing endoscopic resection in patients for whom it would be either inappropriate or futile.

Study aims

The aims of this study are (1) to develop a novel AI system that can classify IPCL patterns as normal or abnormal in endoscopically resectable lesions (<SM1 invasion) in real time on videos acquired during ME, and (2) to develop a methodology that can be used to develop further AI systems, for use in vivo, that accurately predicts the grade of a lesion's histology and invasion depth based on IPCL patterns.

Methods

Patient recruitment, inclusion/exclusion criteria

Patients attending two ESCN referral centres in Taiwan (National Taiwan University Hospital and E-Da Hospital) were recruited with consent. Patients were included only if pathological samples (EMR, ESD, oesophagectomy specimens) were acquired at the time of their endoscopy or subsequent surgery. Patients with oesophageal ulceration were excluded. Our study complied with the Declaration of Helsinki. The institutional review board (IRB) of E-Da Hospital approved this study (IRB number: EMRP-097-022, July 2017).

Endoscopic procedures and video acquisition

Gastroscopies were performed by two expert endoscopists (W.L.W. and H.P.W.), both performing > 50 ESCN assessments per year. Endoscopies were performed using a high-definition (HD) ME-NBI GIF-H260Z endoscope with Olympus Lucera CV-290 processor (Olympus, Tokyo, Japan). The oesophageal mucosa was cleaned with a Simethicone solution prior to the interrogation of the IPCL patterns using ME-NBI. ME was performed on areas of interest at 80–100 × magnification. After assessment of the lesion, pathological samples of the imaged area were acquired through EMR or ESD. Histopathological analysis was undertaken by two expert gastrointestinal pathologists and reported according to the Vienna classification system.²⁴

Labelling ME-NBI videos and establishing an expert consensus

HD endoscopic videos were reviewed independently by three expert upper gastrointestinal endoscopists with extensive experience in the endoscopic assessment of ESCN lesions (W.L.W., R.J.H. and H.P.W.). The visualised IPCL patterns were classified by consensus based on the JES classification system. Classifications were then correlated with histology from the imaged area before a final classification was assigned to each image. Type A IPCLs were considered normal; types B1, 2 and 3 IPCLs were considered abnormal. Videos were sampled at 30 fps to generate sequential still images which were stored as the lossless.png format.

Rationale for selecting images as a data input

An AI system for use in vivo needs to be able to classify IPCL patterns at video rate. A video is simply a rapid sequence of images and can be broken down into individual frames. Individual frames provide more data to train the CNN and is acceptable as a data input for training AI networks for video classification.

Image quality control

Images were manually quality-controlled by a senior clinician study member (M.E.); those that were blurred, or if there was obscuration of the mucosa by blood or mucus, were removed from both the training and testing datasets. The black borders surrounding the endoscopic image were also removed

Formation of image datasets

The full dataset consisted of 7046 images (8193 excluded), with resolutions ranging from 458 × 308 to 696 × 308 pixels. We employed five-fold cross-validation to generate five distinct datasets with different combinations of images (Figure 2(a)). Five-fold validation serves two purposes: first, to maximise the number of images that can be used for training, validation and testing. For any given fold, images of patients used within the training dataset were not used in the validation or testing dataset (summarised in Tables 1 and 2). Secondly, each fold generates a distinct CNN; by developing multiple networks simultaneously, one can interrogate that the designed network architecture performs well across a range of patients and is not simply performing by ‘fluke’ on one particular combination of patient images.

Figure 2.

(a) Representative example of how five-fold cross-validation was applied to the datasets used in our study. (b) Schematic representation of our study workflow and CNN design. CNN: convolutional neural network; eCAMs: explicit class activation maps; IPCL: intrapapillary capillary loop; ME-NBI: magnification endoscopy-Narrow Band Imaging® (Olympus, Japan)

Table 1.

Number of frames in each fold used for cross-validation containing normal intrapapillary capillary loop patterns.

Fold	Training	Validation	Testing
1	2620	201	577
2	1792	891	715
3	1822	685	891
4	1792	715	891
5	1559	685	1154
Average	1917	635	646

Table 2.

Number of frames in each fold used for cross-validation containing abnormal intrapapillary capillary loop patterns.

Fold	Training	Validation	Testing
1	2803	258	587
2	2205	739	704
3	1549	1360	739
4	1912	961	775
5	1754	743	1151
Average	2045	812	791

CNN design

A full description of our technical methodology has previously been described by our group²⁵ (Figure 2(b)). Quality-controlled ME-NBI images of ESCNs were used as the input data. Each image was labelled as representative of a specific IPCL subtype. In training, input images at various magnifications are passed through layers of the CNN. Images pass through five CNN layers, with a fixed learning rate of 1⁻⁶, momentum of 0.9 and a batch size of 1. Stochastic gradient descent was the optimiser. All folds were trained with a maximum iteration of 4 × the number of images in the training set. Explicit class activation maps (eCAMs) are generated to depict visually areas of each image that the CNN finds most discriminative when classifying them as normal or abnormal. Images were not resized but global average pooling was employed after the decoder and prior to classification. In a final output layer the CNN classifies the IPCL pattern as normal or abnormal based on a 1 × 1 convolution. If the prediction is correct compared to the clinician-labelled input image, filters are conserved. Once training is complete, new, unseen images are entered into the CNN and its classification performance is assessed.

Statistical analysis

Accuracy, F1 scores (a weighted average of precision and sensitivity), sensitivity and specificity for abnormal IPCL classification were calculated. Power calculations were not deemed necessary for this study because it is entirely novel and a proof-of-concept work. We are attempting only to benchmark the currently achievable diagnostic performance of AI systems in the classification of IPCL patterns.

Results

Patient characteristics

Seventeen patients were included in our dataset: 10 with ESCN and seven with a normal squamous oesophagus. All patients were deemed to have endoscopically resectable lesions; a summary of the histological invasion depth is given in Table 3.

Table 3.

Summary of demographics and lesion information for patients recruited.

Patient demographic
ESCN characteristics
IPCL patterns	Type A	7
	Type B1	5
	Type B2	5
Histology	Normal	7
	HGIN (M1)	1
	Lamina propria (M2)	4
	Muscularis mucosa (M3)	4
	Submucosa (SM1)	1

ESCN: early squamous cell neoplasia; HGIN: high-grade intraepithelial neoplasia; IPCL: intrapapillary capillary loop.

CNN performance for IPCL classification

Our CNN operates at video rate, capable of classifying sequential HD images acquired from endoscopic videos. The classification interval time varied according to the size of the images analysed, but ranged from 26.17 ms to 37.48 ms. The CNN demonstrated a mean accuracy for the differentiation of abnormal IPCL patterns (B1/B2/B3) from normal type A patterns of 93.3% (range, 86.2% to 98.3%). The average F1 score for identifying ESCN based on the IPCL pattern was similarly high at 92.7% (range, 85.4% to 98.2%). Our algorithm achieved a sensitivity and specificity for abnormal IPCL classification of 89.7% (range, 78.1% to 100%) and 96.9% (range, 92% to 99.7%), respectively. There was some variability in performance statistics between folds, which likely represents that the CNN is still not adept at distinguishing all the varied IPCL features that represent abnormal tissue (types B1/2/3). While our dataset is the largest in the published literature, it is highly likely that more images are required to further train the network to recognise the full spectrum of variability in IPCL patterns. Our CNN performance statistics are summarised in Table 4.

Table 4.

Summary of convolutional neural network performance statistics for detection of abnormal intrapapillary capillary loop patterns.

Fold	Accuracy (%)	Sensitivity (%)	Specificity (%)	F1 score (%)
1	86.2	80.4	92	85.4
2	89.0	78.1	99.7	87.6
3	97.7	100	95.9	97.6
4	98.3	99.4	97.3	98.2
5	95.1	90.6	99.6	94.9
Average	93.3	89.7	96.9	92.7

Analysis of eCAMs provides a clinically interpretable neural network output

The distribution of abnormal IPCL patterns are heterogeneous between images; eCAMs were therefore generated to elucidate which visual features within the images the CNN was using to discriminate healthy and unhealthy tissue. They also provide a clinically interpretable network output. This served as a further validation step to ensure that the CNN was using the IPCL patterns to classify images as normal or abnormal, rather than other subtle discriminative features. Analysis of the eCAMs patterns (Figure 3), offers several interesting insights into the decision-making process of our CNN. Firstly, as expected, our CNN appears to use the IPCL patterns as its most discriminative feature. Secondly, it is capable of recognising discrete areas of abnormal IPCL patterns within otherwise normal mucosa and ignores other uninformative features such as specular reflections. Interestingly, the CNN did not use avascular areas between the grossly abnormal B2 IPCLs seen in Figure 3, but remained able to discriminate the abnormal vessels highly selectively. We also note that within some images of healthy tissue the CNN has a tendency to classify the deep submucosal vessels as abnormal, which may potentially lead to classifications of normal tissue as abnormal.

Figure 3.

Input images (left column) with corresponding explicit class activation maps (right column) illustrating visual features recognised by the convolutional neural network (CNN) when classifying images. (a) Recognition of abnormal intrapapillary capillary loop (IPCL) patterns. (b) Specular reflections are ignored by the CNN. (c) High selectivity between normal mucosa and abnormal IPCLs.

Discussion

We introduce the first application of CAED and AI for the accurate, video rate characterisation of ESCN through the classification of IPCLs seen during ME based on the JES system.

Accurate assessment and characterisation of ESCN lesions is vital to predict histology and invasion depth to guide appropriate intervention. Lesions confined to the mucosa have low rates of local lymph node metastasis (<2%) and so are amenable to endoscopic therapy,^5–8 with reported five-year survival rates of 75% to 100% for mucosal ESCN.⁹

Several classification systems exist to assist clinicians with the characterisation of IPCL patterns. Inoue et al. in 2001 were the first to propose a five-part classification of IPCL patterns.²⁶ Types I–III IPCL patterns were associated with normal mucosa, inflammation or low-grade intraepithelial neoplasia. Type IV IPCLs were suggestive of high-grade dysplasia. Type V IPCLs were subdivided into V₁ (M1 carcinoma in situ), V₂ (M2 carcinoma in situ), V₃ (M3 or early SM1 invasion) and V₄ (invasion to at least SM2) based on the progression of abnormal IPCL morphology.²⁶ In one study, the Inoue classification enabled identification of early mucosal lesions (M1–2) with 89.5% sensitivity.¹⁴ The Inoue classification sensitivity for prediction of ESCN invasion into SM1 and SM2 was much lower at 58.7% and 55.8%, respectively.¹⁴ In 2005 Arima and colleagues proposed an alternative four-part classification system with types 1, 2, 3 and 4 IPCLs representing normal mucosa, inflammatory changes, M1–2 carcinoma in situ and >M3 invasion, respectively.²⁰

We elected to use the JES IPCL classification of ESCN as the input and output for our CNN because it is the most contemporary and accurate system and is widely used in areas of high ESCN prevalence. We believe that a CNN capable of using the JES classification will remain clinically relevant as it becomes more widely used. We suggest that the classifications proposed by Inoue and colleagues²⁶ and Arima et al²⁰ are complex and may not be as intuitive to use.

Our study, the first reported use of CNNs for this purpose, uses sequential still images captured from HD videos of endoscopic examinations to train a CNN to characterise ESCN lesions based on the IPCL patterns visualised at endoscopic assessment. This CAED platform achieved an accuracy of 93.7% for the characterisation of abnormal IPCL patterns. The sensitivity and specificity of our CNN compares favourably to other published work at 89.7% and 96.9%, respectively. Our CNN operates at video rate, but it should be noted that more work is required for this to be put to clinical use, including training a CNN to remove uninformative frames without the need for manual selection.

We envisage that our CNN could assist endoscopists in the recognition of abnormal IPCL patterns in patients undergoing assessment for suspected ESCN and consideration of endoscopic treatment. Such a system could shorten the time required for endoscopic evaluation, reduce interobserver variability and importantly inform in vivo clinical decision making on what lesions require or are amenable to endoscopic resection. Wang et al. in 2018 demonstrated that among a non-expert panel of endoscopists accuracy of histology prediction using the JES classification ranged from 48% to 57% after a short training program.²⁷ In 2017, Kim and colleagues also demonstrated an overall accuracy of 78.6% for the correct prediction of histology.²² Our CNN's overall accuracy of 93.7% therefore compares very favourably to this. We would therefore expect that our system would most benefit non-expert endoscopists or centres where the endoscopic assessment of ESCN is less common and whose clinicians may feel less confident in recognising and classifying IPCL patterns.

Our study has several limitations, all of which we argue are to be expected in a feasibility study, but have nonetheless attempted to mitigate. Firstly, our patient sample size is small, using 7046 images generated from 17 patients. However, this is currently the largest image dataset used to train a CNN for this specific application reported in the literature. To mitigate this, we utilised five-fold cross-validation, as outlined in Table 4, to benchmark the ability of the CNN to recognise a range of IPCL patterns under different operating conditions. This method ensures that our network was trained and tested on all available images, while ensuring that no images were used both for training and testing. Our novel use of eCAMs also acts as an additional form of validation; we have demonstrated that the ICPL patterns are the visual features the network finds discriminative when classifying images. We therefore expect that with greater patient numbers we should continue to see such high accuracies for IPCL characterisation.

Secondly, our system accurately differentiates normal (type A) compared to abnormal (types B1–3) IPCL patterns, but is not yet able to differentiate individual subtypes with sufficient accuracy for clinical use. This is comparable to other studies using the Inoue and Arima classifications which reported less accurate characterisation of >SM1 lesions based on IPCL patterns. We have included only images of types A, B1 and B2 IPCLs as an input to our CNN for this study because we currently seek to develop a system that highlights abnormal areas of potentially resectable lesions to clinicians in the first instance.

Lastly, our gold standard used three expert clinicians. We feel that for a pilot study this is sufficient to provide accurate ground truth to develop a CNN, particularly since the consensus classifications were correlated with histological results. We recognise that future work is needed to validate this CNN's performance against a larger panel of clinicians of varying experience before it could be used in a clinical environment. It is important to note that published work on the JES classification reports interobserver variability of between 0.61²⁷ and 0.86.²² A CNN developed to classify IPCL patterns should therefore be expected to obtain this level of variability only during external validation. An automated method of removing uninformative frames may also reduce the selection bias introduced by a clinician removing uninformative frames.

Further work currently underway aims to produce a CNN capable of more precise classification based on the individual JES subgroups. Such a validated system would also improve the characterisation of ESCN and could allow clinicians to triage, in real time, lesions that would be amenable to EET, compared to lesions that would be too advanced to be cured endoscopically. This would prevent patients with advanced disease undergoing unnecessary endoscopic therapy, while ensuring that they are referred promptly for surgical management.

Declaration of conflicting interests

R.J.H. has received research grant support from Pentax Medical, Cook Endoscopy, Fractyl Ltd, Beamline Ltd and Covidien plc to support research infrastructure. The other authors have nothing to declare.

Ethics approval

Our study complied with the Declaration of Helsinki. IRB of E-Da Hospital approved this study (IRB number: EMRP-097-022, July 2017).

Funding

This work was supported by Wellcome Trust (WT101957, 203145Z/16/Z), EPSRC (NS/A000027/1, NS/A000050/1), National Institute for Health Research Biomedical Research Centre University College London (UCL) Hospitals/UCL High Impact Initiative and a UCL Engineering and Physical Science Research Council Centres for Doctoral Training Scholarship Award (EP/L016478/1).

Informed consent

Patients attending two ESCN referral centres in Taiwan (National Taiwan University Hospital and E-Da Hospital) were recruited with consent.