A deep learning system for quantitative assessment of microvascular abnormalities in nailfold capillary images

Abstract

Objectives

Nailfold capillaroscopy is key to timely diagnosis of SSc, but is often not used in rheumatology clinics because the images are difficult to interpret. We aimed to develop and validate a fully automated image analysis system to fill this gap.

Methods

We mimicked the image interpretation strategies of SSc experts, using deep learning networks to detect each capillary in the distal row of vessels and make morphological measurements. We combined measurements from multiple fingers to give a subject-level probability of SSc.

We trained the system using high-resolution images from 111 subjects (group A) and tested on images from subjects not in the training set: 132 imaged at high-resolution (group B); 66 imaged with a low-cost digital microscope (group C). Roughly half of each group had confirmed SSc, and half were healthy controls or had primary RP (‘normal’). We also estimated the performance of SSc experts.

Results

We compared automated SSc probabilities with the known clinical status of patients (SSc versus ‘normal’), generating receiver operating characteristic curves (ROCs). For group B, the area under the ROC (AUC) was 97% (94–99%) [median (90% CI)], with equal sensitivity/specificity 91% (86–95%). For group C, the AUC was 95% (88–99%), with equal sensitivity/specificity 89% (82–95%). SSc expert consensus achieved sensitivity 82% and specificity 73%.

Conclusion

Fully automated analysis using deep learning can achieve diagnostic performance at least as good as SSc experts, and is sufficiently robust to work with low-cost digital microscope images.

Rheumatology key messages.

• The image interpretation challenge limits uptake of nailfold capillaroscopy for diagnosis of SSc.

• Automated analysis of nailfold images could provide objective information for rheumatologists at the point-of-care.

• We describe fully automated analysis, showing diagnostic performance that equals or exceeds that of experts.

Introduction

Nailfold capillaroscopy plays a key role in the assessment of the patient presenting with RP. Abnormal capillaries are highly suggestive of an underlying SSc-spectrum disorder, whereas normal capillaries (in the absence of other clinical features pointing to CTD) are reassuring. Abnormal nailfold capillaries score 2 of the 9 points required to fulfil the 2013 ACR/EULAR classification criteria for SSc [1], so all rheumatologists should have access to nailfold capillaroscopy at the point-of-care. However, results of a recent UK survey suggest that almost 60% of rheumatologists do not currently perform or have access to capillaroscopy in their own departments [2]. There are two ‘bars’, or challenges, to rheumatologists performing nailfold capillaroscopy themselves. First is image acquisition: not all rheumatologists have the equipment or expertise to acquire images, though this could be overcome by training and the use of low-cost devices [3]. Second is image interpretation: what exactly is meant by the ‘enlarged capillaries and/or capillary loss with or without capillary haemorrhages at the nailfold’ referred to in the 2013 classification criteria [1]? Although the international community has made strides in attempting to standardize image interpretation [4, 5], there remains a large subjective component in the qualitative assessment of nailfold capillary images.

Automated analysis could play an important role in addressing the challenge of interpreting nailfold capillary images, providing rheumatologists with objective assessment at the point-of-care. Our aim was to develop and validate a complete automated analysis system taking, as input, nailfold images from several of a patient’s fingers, and producing, as output, a patient-level probability of SSc that can be used directly to inform diagnosis.

Subjects and methods

Subjects

Images were obtained from subjects participating in a current study and two previous studies [6, 7]. All subjects had signed written informed consent (current study approved by National Research Ethics Service (NRES) Committee London – Brighton & Sussex and previous studies approved by NRES Committees North West – Liverpool Central [6] and North West – Greater Manchester East [7]). Subjects were divided into three mutually exclusive groups (A, B and C), with each group containing roughly equal numbers of subjects labelled as patients with SSc, and subjects labelled as normal [patients with primary RP (PRP) or healthy controls, expected to have normal capillaries]. Details and demographic data for the three groups are given in Table 1. All patients with SSc were attending a tertiary centre for SSc (Northern Care Alliance NHS Foundation Trust) and all fulfilled the 2013 ACR/EULAR criteria for early SSc [1]. Patients with PRP had no clinical or serological features suggestive of an underlying CTD. We chose to label subjects by clinical status (SSc or normal), rather than using expert assessment of vessel abnormality, because it avoided the uncertainty associated with expert interpretation and addressed, directly, the clinical challenge. Data from subjects in group A were used to train the analysis system; data from subjects in group B and C were used to evaluate the system under different imaging conditions.

Table 1.

Demographics of the three patient groups

	Group A			Group B			Group C
	SSc	PRP	HC	SSc	PRP	HC	SSc	PRP/HC
Subjects, n (%)	61 (55)	19(17)	31 (28)	65 (49)	11 (8)	56 (42)	29 (44)	37 (56)
Mosaics/images	536	166	208	577	110	520	206	218
Age, median (IQR)	64 (57–68)	41 (32–50)	45 (38–55)	60 (51–66)	39 (36–55)	49 (40–56)	60 (48–67)	48 (34–59)
Female, n (%)	51 (84)	16 (84)	21 (68)	57 (88)	8 (73)	37 (66)	25 (86)	24 (65)
RP years, median (IQR)	13 (9–25)	6 (4–10)		16 (9–26)	10 (3–40)		10 (6–20)
SSc years, median (IQR)	9 (4–18)			11 (4–21)			6 (4–12)

A: training group; B: main test group; C: Dinolite test group. For groups A and B one mosaic was recorded per nailfold per visit, and for group C one or more images were recorded per nailfold per visit. PRP: primary RP; HC: healthy controls; IQR: interquartile range; RP years: years since onset of RP; SSc years: years since onset of other SSc clinical manifestation(s).

Image acquisition

All images were acquired at a nominal magnification of ×200 with a working distance of approximately 10 mm. For group A, images were acquired using a 5 frames per second (f.p.s.) video microscope (KK Technology, Honiton, UK; cost over £8000) and green LED illumination, with custom capture software that stitched video frames together to create a panoramic mosaic of the whole nailfold, with a resolution of 1.25 µm per pixel [8]. We also had expert annotation of these images, which provided the position and width of each distal apex [6]. For group B, 47% of images were acquired using the same system as group A, 53% using a system developed in-house [9, 10] (cost of parts £7500) which uses a 120 f.p.s. camera to capture a video sequence, and novel software to generate a high-quality mosaic of the whole nailfold, with a resolution of 1.0 µm per pixel. For group C, one or more 1280 × 1024 pixel fields of view, with a resolution of 1.35 µm per pixel, were acquired for each nailfold, using a hand-held 30 f.p.s. USB microscope (Dinolite, AnMo Electronics Corporation, Taiwan; cost approximately £500). For subjects in groups A and B, mosaics were normally recorded from 10 fingers; for group C images were recorded from 4 fingers only (middle and ring from both hands) to limit time taken for the investigation. For analysis, all images were rescaled to a resolution of 2.5 µm per pixel.

Assessing microvascular abnormality

Capillary-level measures

We structured the analysis to mimic the interpretation strategy of an SSc expert. A pipeline of deep learning networks was used to detect capillaries and measure their morphology in each mosaic/image (Fig. 1A–C). This involved the following steps: (i) generating an initial set of candidates for the locations of capillary apices across the whole image; (ii) extracting a patch centred on each candidate location and either rejecting the candidate or designating it as the apex of a distal or non-distal capillary; and (iii) for each distal capillary, estimating its apical width and assigning a shape score to quantify capillary tortuosity. The shape score was computed by first segmenting the vessel from the tissue background, then finding the vessel orientation at each point on its centre-line and measuring the spread of orientations.

Figure 1.

Image analysis and subject classification results. (A) Original nailfold mosaic. (B) Apex candidates. (C) Candidates classified as true distal apices. (D) ROC curves, showing median and 90% CIs: group B (blue), AUC 97% (94–99%), EER sensitivity/specificity 91% (86–95%); group C (red), AUC 95% (88–99%), EER sensitivity/specificity 89% (82–95%); expert consensus (black dot), sensitivity 82%, specificity 73%. ROC: receiver operating characteristic; AUC: area under the ROC; EER: equal error rate

Two types of deep learning model were used in the analysis pipeline: U-Nets [11] for apex candidate generation and vessel segmentation, and ResNet34 [12] for candidate classification (rejection/non-distal/distal) and width estimation. All models were built using patches extracted from the 910 mosaics in group A (Table 1), split into separate training (456 mosaics) and validation (454 mosaics) sets. The U-Nets for apex candidate generation and vessel segmentation were trained using 450 and 150 patches of size 256 × 256 pixels, respectively, while the ResNet34 networks for candidate classification and width prediction were trained using 23 823, and 5723 patches of size 224 × 224 pixels, respectively. Note that since U-Nets learn to classify individual pixels, they need fewer training patches than ResNet34 networks, which learn to classify whole patches.

Image-level measures

The morphological features extracted at each capillary were combined at the image level to compute mean apical width, maximum apical width and mean shape score. The number of capillaries divided by the distance in mm from the left-most to right-most capillary was computed to quantify vessel density. Other investigators have included micro-haemorrhages [13, 14]; we did not because they are difficult to define and are not necessary for diagnosis of SSc [1].

Subject-level microvascular abnormality probability

For each subject, mean apical width, maximum apical width, mean shape score and vessel density were averaged over all available images acquired at a single visit. These subject-level measures were used to build a logistic regression model to distinguish subjects with SSc (positives) from ‘normals’ (healthy controls or subjects with PRP—negatives). This model was trained on the 111 subjects in group A and tested on the 132 subjects in group B and 66 subjects in group C, providing a capillaroscopy-based probability of SSc for each subject. We used these probabilities together with subjects’ clinical status (SSc vs normal) to plot receiver operating characteristic (ROC) curves. These summarize the classification performance of the automated system, showing the trade-off between sensitivity and specificity as the probability above which subjects are classified as SSc is varied (for example, subjects with probabilities ≥0.5 might be classified as SSc, <0.5 as normal). We computed two summary performance measures for each ROC: the area under the curve (AUC), which provides an overall measure of classification performance (50% = random classification, 100% = perfect classification), and the sensitivity and specificity at the point of equal error rate (EER). We obtained median values and 90% CIs for the ROC curves and summary measures [quoted in the text as: median (5th–95th centile)] using a 10 000-sample bootstrap.

Expert performance

To provide a benchmark, we computed the performance of SSc experts, using data from the international study that provided images for groups A and B [6]. In that study, 2 or more of 10 experts graded each of 1650 nailfold mosaics from 173 subjects, blinded to disease status and finger, on a scale of ‘1: Normal’, ‘2: Early SSc’, ‘3: Active SSc’, ‘4: Late SSc’, 5: ‘Ungradable condition’ (capillaries too abnormal to assign a disease stage), ‘Non-specific abnormality’ (capillaries not entirely normal, but no definite SSc pattern) or ‘Ungradable quality’ (capillaries not sufficiently well visualized to be gradable). In total there were 3038 mosaic assessments (Normal: 681; Early: 157; Active: 283; Late: 253; Ungradable condition: 152; Non-specific: 898; Ungradable quality: 614). Our analysis was based on the 1397 mosaics where at least one expert assigned a grade other than ‘Ungradable quality’. Of these 747 had one expert assessment, while the remaining 650 had two or more. To arrive at an expert consensus, each of a subject’s mosaics was classified as abnormal if the majority of experts graded it 2, 3, 4 or 5 (normal in case of a normal/abnormal tie), and the subject was classified as abnormal if any of their mosaics was classified as abnormal. The expert consensus was compared with clinical status for each subject, to give sensitivity and specificity values. ‘Non-specific abnormality’ grades were grouped with ‘Normal’ because grouping with the abnormal categories gave very low specificity (less than 10%).

Results

Fig. 1A–C shows an example of distal apex detection in a nailfold mosaic. Fig. 1D gives the ROC curves for subject-level prediction of SSc vs normal capillaries for groups B and C, showing the trade-off between sensitivity and specificity. For group B, the AUC was 97% (94–99%), with EER sensitivity/specificity 91% (86–95%). For group C, the AUC was 95% (88–99%), with EER sensitivity/specificity 89% (82–95%). The CIs for group C are wider than for group B, because these preliminary results are for a smaller dataset. There is no statistically significant difference in AUC or EER sensitivity/specificity between groups B and C. For comparison, we have plotted the sensitivity (82%) and specificity (73%) achieved by expert consensus. This is poorer than the automated system for both groups (lies outside the 90% ROC confidence band).

Discussion

We have developed a complete automated system that analyses nailfold capillary images, and returns a subject-level probability of SSc, based on microvascular morphology. We have validated the system using high-quality images from subjects not involved in training the system, and have shown it can discriminate between patients with SSc and subjects with normal capillaries at least as well as human experts. We have also shown that the system is robust to differences in the imaging conditions, including working with images obtained using a low-cost, hand-held USB microscope.

Despite longstanding interest in automation [9, 15], we know of only three clinical systems that have been developed and systematically evaluated. AUTOCAPI measures capillary density for a single field of view (preselected by the operator) [16]. Capillary.io [13] and a system developed by Swiss researchers [14] both detect vessels and signs of abnormality (e.g. giant capillaries, microhaemorrhages) from a single field of view (as opposed to a nailfold mosaic). Given the objective of supporting non-specialist rheumatologists, we believe an overall capillaroscopy-based probability of SSc is critically important. AUTOCAPI and Capillary.io provide no such output; the system reported by Garaiman et al. [14] does, but it performs significantly worse than our current system (80% sensitivity/specificity vs 91%). A key difference between our approach and those of Gracia Tello et al. [13] and Garaiman et al. [14] is that they detect the presence/absence of discrete phenomena, whereas we extract continuous measurements of vessel morphology, allowing more subtle signs of microvascular abnormality to be detected.

The challenge of interpreting nailfold capillary images has been well documented [17], with substantial inter-observer variability not only in grading images but also in selecting which are ‘evaluable’ [9]. A recently described method to simplify and standardize interpretation, particularly for non-experts, is the ‘fast-track’ algorithm [18], which classifies as SSc-pattern any nailfold image in which there are one or more giant capillaries or areas of capillary density of fewer than 3 capillaries/mm. Although the reported inter-observer agreement was high, this approach risks being specific but not sensitive for detecting SSc (sensitivity was not reported), given that several investigators have reported densities higher than 3/mm in most patients with SSc [19, 20]. Automation, thus, remains one of the most promising approaches to achieving widespread and effective use of capillaroscopy in clinical practice.

In conclusion, using a deep learning approach holds promise for accurate separation between patients with SSc and those with normal capillaries, bypassing the need for human interpretation. Although not included in this study, it may also be applicable to secondary RP more generally, for example in DM and SLE. Taking an approach that mimics the interpretation strategies of human experts, as we have here, can provide the basis for rheumatologist-friendly explanation of the results. Future challenges (and next steps) are to encourage and facilitate more widespread use of nailfold capillaroscopy among rheumatologists, so they can become confident in the acquisition of images to which automated analysis can be applied.

Data availability

De-identified participant data will be available to qualified researchers after approval of a proposal by the Sponsor, and the signing of a data sharing access agreement with the trial sponsor. The code and anonymized data used to generate the ROCs and quantitative results quoted in the paper are available at https://gitlab.com/manchester_nailfold/papers/rheumatology_rhe_22-2222.

Funding

This work was supported by the National Institute for Health and Care Research (NIHR) Invention for Innovation [grant number II-LB-1117-20006].

Disclosure statement: The authors have declared no conflicts of interest.