Dorsal Finger Fold Recognition by Convolutional Neural Networks for the Detection and Monitoring of Joint Swelling in Patients with Rheumatoid Arthritis

Abstract

Digital biomarkers such as wearables are of increasing interest in monitoring rheumatic diseases, but they usually lack disease specificity. In this study, we apply convolutional neural networks (CNN) to real-world hand photographs in order to automatically detect, extract, and analyse dorsal finger fold lines as a correlate of proximal interphalangeal (PIP) joint swelling in patients with rheumatoid arthritis (RA). Hand photographs of RA patients were taken by a smartphone camera in a standardized manner. Overall, 190 PIP joints were categorized as either swollen or not swollen based on clinical judgement and ultrasound. Images were automatically preprocessed by cropping PIP joints and extracting dorsal finger folds. Subsequently, metrical analysis of dorsal finger folds was performed, and a CNN was trained to classify the dorsal finger lines into swollen versus non-swollen joints. Representative horizontal finger folds were also quantified in a subset of patients before and after resolution of PIP swelling and in patients with disease flares. In swollen joints, the number of automatically extracted deep skinfold imprints was significantly reduced compared to non-swollen joints (1.3, SD 0.8 vs. 3.3, SD 0.49, p < 0.01). The joint diameter/deep skinfold length ratio was significantly higher in swollen (4.1, SD 1.4) versus non-swollen joints (2.1, SD 0.6, p < 0.01). The CNN model successfully differentiated swollen from non-swollen joints based on finger fold patterns with a validation accuracy of 0.84, a sensitivity of 88%, and a specificity of 75%. A heatmap of the original images obtained by an extraction algorithm confirmed finger folds as the region of interest for correct classification. After significant response to disease-modifying antirheumatic drug ± corticosteroid therapy, longitudinal metrical analysis of eight representative deep finger folds showed a decrease in the mean diameter/finger fold length (finger fold index, FFI) from 3.03 (SD 0.68) to 2.08 (SD 0.57). Conversely, the FFI increased in patients with disease flares. In conclusion, automated preprocessing and the application of CNN algorithms in combination with longitudinal metrical analysis of dorsal finger fold patterns extracted from real-world hand photos might serve as a digital biomarker in RA.

Introduction

Biomarkers are defined as objectively measurable values to diagnose, monitor, or predict pathologic processes [1]. Ideally, biomarkers are easy to observe, cheap, and disease-specific. To monitor disease activity in rheumatoid arthritis (RA), currently available biomarkers are limited to serum C-reactive protein and blood sedimentation rate [2]. Digital biomarkers are considered non-invasive and should be able to provide a continuous picture of a disease status. In rheumatology, digital biomarkers most commonly include information obtained from wearables or smartphones, such as data from gyroscopes, accelerometers, tab speed, or geolocation [3]. Thermography (where temperature is recorded by a camera), in combination with artificial neural networks, has recently been used to differentiate between RA patients and healthy individuals, but so far has not been shown to monitor disease activity sufficiently [4]. The core problem of biomarkers for RA is the insufficient use of real-world data and the low specificity of patient-reported outcomes, respectively. Thus, there is a need for more specific and easier to collect data to measure inflammation.

Joint swelling due to underlying synovitis and effusion is a hallmark of arthritis and is only quantifiable clinically by ultrasound or magnetic resonance imaging. Clinical grading of joint swelling in RA does not improve the performance of disease activity scores and is not recommended in practice [5, 6]. We postulated that hand joint swelling leads to an early and reversible change of dorsal finger fold patterns. Automated assessment of finger folds in RA and other forms of arthritis on hand photographs over time may be used as a supplement for existing biomarkers and clinical assessment, respectively. If captured by patients themselves, such a tool might empower patients and improve self-management strategies [7, 8]. In this proof-of-concept study, we investigated dorsal finger fold patterns of proximal interphalangeal (PIP) joints in patients diagnosed with RA according to the EULAR/ACR criteria [9].

Methods

Hand photographs of patients with RA were taken during consultations by smartphone cameras in a standardized manner (single hand on a white DIN A4 paper, no direct light, at a distance of 25 cm) after obtaining informed consent. 190 PIP joints were categorized as either swollen or not swollen based on clinical judgement and ultrasound. For preprocessing, PIP joints were cropped, and dorsal finger folds from PIP joints were identified and extracted from the hand photographs by a subsequent pipeline algorithm consisting of open-source hand keypoint detection, contrast limited adaptive histogram equalization, and Canny edge detection [10] algorithms (Fig. 1a, b). A convolutional neural network (CNN) was trained to classify the resultant (cropped and preprocessed images) into swollen versus non-swollen joints on an open access platform www.giotto.ai (Fig. 1d), after dividing the images into a training (80%, 152 images) and validation set (20%, 38 images). The network architecture we used is a resnet34 to which we postpended a fully connected layer with 1,000 input nodes and 2 output nodes. We trained the model using stochastic gradient descent with a constant learning rate of 0.001 for 40 epochs. We used a cross-entropy loss to optimize the parameters of the model.

Fig. 1.

Automated finger fold recognition to monitor RA. Hand photographs are taken by a smartphone in a standardized manner (a). Hands, and subsequently PIP joints, are automatically recognized and extracted. Finger fold lines are isolated from the images, measured and related to the joint diameter (b, c). A convolutional deep neural network was used to train a model for classification of extracted finger fold patterns into swollen versus non swollen joints (d) A heatmap of the cropped PIP joints confirmed finger folds as the region of interest for correct classification (e).

Results

In swollen joints, the number of automatically extracted double-contoured, deep skinfold imprints was significantly reduced compared to non-swollen joints (1.3, SD 0.8 vs. 3.3, SD 0.49, p < 0.01). The joint diameter/deep skinfold length ratio (=finger fold index, FFI) was significantly higher in swollen (4.1, SD 1.4) versus non-swollen joints (2.1 SD 0.6, p < 0.01) (Fig. 1c). Finger folds in swollen joints appeared less curved than in non-swollen joints.

Accordingly, the CNN model successfully differentiated swollen from non-swollen joints based on extracted finger fold patterns with a validation accuracy of 0.84, a sensitivity of 88%, and a specificity of 75% (Fig. 1d). The main reasons for misinterpreted images were hairy skin or oblique finger positioning. A heatmap of the original images obtained by an extraction algorithm confirmed finger folds as the region of interest for correct classification (Fig. 1e).

Next, we metrically compared 8 representative deep finger folds from different swollen PIP joints of four RA patients with significant clinical and ultrasound improvement after disease modifying antirheumatic drug ± corticosteroid therapy in patients achieving a good DAS28 response (−1.2, DAS28 ≤ 3.2) (Fig. 2). The mean FFl decreased from 3.03 (SD 0.68) before therapy to 2.08 (SD 0.57) after therapy. Curving reappeared in some skin folds after therapy. Conversely, the finger fold index increased in patients with disease flares and increased joint swelling.

Fig. 2.

Longitudinal metrical analysis of horizontal finger folds. a Representative images of finger fold patterns before and after treatment or arthritis flares. (b) The joint diameter/finger fold length ratio reduced in a panel of patients after successful treatment with disease modifying antirheumatic drugs and/or cortisone.

Discussion

Here we provide a proof-of-concept to use automated dorsal finger fold recognition via standard smartphone cameras as a potential real-world digital biomarker for joint swelling in RA. Both the CNN model and specific metrical analyses were able to differentiate swollen from non-swollen joints. Furthermore, the metrical analysis demonstrated a clear change in the joint diameter/finger line ratio after successful therapy and resolution of inflammation. To our knowledge, this is the first application of CNNs to real-world images in the setting of RA or osteoarthritis, whilst in previous studies, CNNs were applied to radiographs and ultrasound images [11, 12]. Such a patient-controlled biomarker could complement patient reported outcomes and wearables and thus empower patients in monitoring their disease. How such a digital biomarker will finally be deployed in rheumatic patient care needs to be clarified. A shifting point of care towards interoperable, patient-led digital platforms could be one possible scenario. Potentially, flare detection by finger fold patterns in combination with patient-reported outcomes will further increase the accuracy of flare detection (and prediction) and thus lead to earlier clinical visits. On the other side, fewer C-reactive protein measurements or ultrasound exams during the patient journey might be required and costs could be saved. Future algorithms could potentially combine CNN classification tasks and longitudinal observations of finger line metrics in order to monitor arthritis and predict disease flares. In return, treat-to-target strategies or drug holidays could be monitored more closely both by the patient and the rheumatologist and necessary decisions might be taken earlier. The pilot study here has a focus on PIP joints in RA but also could be used for distal interphalangeal joints in hand osteoarthritis. Our model also captures metacarpal phalangeal joint swelling via longitudinal reliefs of extensor tendons (data not shown).

Clearly, to validate this digital biomarker, it needs to be correlated with disease activity scores such as DAS28-CRP in a larger, prospective study. In the literature, clinical exam for the detection of synovitis has a sensitivity of 43% and a specificity of 89% [13]. In this study, the sensitivity for the detection of joint swelling by finger fold patterns is substantially higher at 88%, with a slightly lower specificity of 75%. Given the limited sensitivity of clinical detection and quantification of joint swelling, automated finger fold analysis could be an adjunct to clinical examination and improve inter-clinician assessment, respectively.

We observed a high variability of finger fold patterns due to age, and there is a possible intraindividual circadian variation. Variation in background lighting, the influence of skin pigmentation, hydration status, and the use of lotions and cremes on extracted hand folds need to be clarified in further studies. In any case, education via the user interface (app) itself, the rheumatologist or health care professionals will be necessary to ensure a sufficient quality of captured images. Whether the user interface will be used as a standalone application, remote patient monitoring tool, or in combination with a digital therapeutic, needs to be clarified. In any case, an optimal user experience and interaction with the treating rheumatologist as well as interoperability, e.g., with electronical records, will be required for a successful integration into standard care.

Finger fold patterns seem more useful for monitoring intraindividual swelling, rather than for new diagnosis of arthritis. To this end, the CNN model needs more training to provide a more sophisticated classification of finger swelling. Swelling might also be monitored in other forms of arthritis or dactylitis, notably in psoriasis arthritis, where skin or nail lesions could be integrated into the algorithm [14].

Taken together, this pilot study demonstrates the application of computer vision and image recognition as a potential digital biomarker for RA. Even more importantly than the CNN models themselves, the quality of the obtained data will ultimately determine the utility of the biomarker.

Statement of Ethics

Written informed consent was obtained from all participants and the study was approved by the Commission Cantonale d'Ethique de la Recherche sur l'Etre Humain, Canton de Vaud, Lausanne, Switzerland (CER-VD ID-2020-00033). The study was conducted ethically in accordance with the World Medical Association Declaration of Helsinki.

Conflict of Interest Statement

Thomas Hügle is patentholder of DETECTRA. Thomas Hügle and Marc Blanchard are scientific board members of Atreon S.A.

Funding Sources

This study was supported by a grant from the Fondation pour l'Innovation Technologique (FIT) Lausanne.

Author Contributions

Thomas Hügle: Development of the theory and data acquisition and statistical analysis. Contribution to the CNN model via Giotto. Data interpretation. Preparation of the manuscript. Leo Caratsch and Matteo Caorsi: Algorithm development via Giotto. Data interpretation and preparation of the manuscript. Jules Maglione: Development of the heatmap. Data interpretation and preparation of the manuscript. Diana Dan and Alexandre Dumusc: Data acquisition and data interpretation. Marc Blanchard: User interface development. Data interpretation. Preparation of the manuscript. Gabriel Kalweit and Maria Kalweit: Algorithm development, data interpretation and preparation of the manuscript.

Data Availability Statement

All data generated or analysed during this study are included in this article. Further enquiries can be directed to the corresponding author.