Assessment of inflammation in patients with rheumatoid arthritis using thermography and machine learning: a fast and automated technique

Abstract

Objectives

Sensitive detection of joint inflammation in rheumatoid arthritis (RA) is crucial to the success of the treat-to-target strategy. In this study, we characterise a novel machine learning-based computational method to automatically assess joint inflammation in RA using thermography of the hands, a fast and non-invasive imaging technique.

Methods

We recruited 595 patients with arthritis and osteoarthritis, as well as healthy subjects at two hospitals over 4 years. Machine learning was used to assess joint inflammation from the thermal images of the hands using ultrasound as the reference standard, obtaining a Thermographic Joint Inflammation Score (ThermoJIS). The machine learning model was trained and tuned using data from 449 participants with different types of arthritis, osteoarthritis or without rheumatic disease (development set). The performance of the method was evaluated based on 146 patients with RA (validation set) using Spearman’s rank correlation coefficient, area under the receiver-operating curve (AUROC), average precision, sensitivity, specificity, positive and negative predictive value and F1-score.

Results

ThermoJIS correlated moderately with ultrasound scores (grey-scale synovial hypertrophy=0.49, p<0.001; and power Doppler=0.51, p<0.001). The AUROC for ThermoJIS for detecting active synovitis was 0.78 (95% CI, 0.71 to 0.86; p<0.001). In patients with RA in clinical remission, ThermoJIS values were significantly higher when active synovitis was detected by ultrasound.

Conclusions

ThermoJIS was able to detect joint inflammation in patients with RA, even in those in clinical remission. These results open an opportunity to develop new tools for routine detection of joint inflammation.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Thermography is a fast, non-invasive imaging technique that creates an image of the heat emitted by bodies.

• Warmth is one of the cardinal signs of inflammation. Previous preclinical and clinical research showed thermographically detectable changes in inflamed joints.

WHAT THIS STUDY ADDS

• The analysis of thermal images of the hands with a novel machine learning-based algorithm assesses joint inflammation in patients with rheumatoid arthritis instantaneously, accurately and automatically.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• This novel method could help to assess subclinical inflammation in the rheumatologist’s office quickly and automatically.

• This novel method constitutes an easy approach to assessing joint inflammation remotely.

Introduction

Rheumatoid arthritis (RA) is an inflammatory disease characterised by chronic synovitis, joint destruction and disability. Current therapies and treat-to-target strategies make remission an achievable goal.^{1 2} Several definitions of clinical remission have been proposed, mainly using composite indices of disease activity, with the strictest being the American College of Rheumatology (ACR)/European Alliance of Associations for Rheumatology (EULAR) Boolean definition of remission.^3–5 Many patients in clinical remission continue having subclinical synovitis irrespective of whether the 28-joint count Disease Activity Score (DAS28), Simplified Disease Activity Index (SDAI), Clinical Disease Activity Index (CDAI) or ACR/EULAR Boolean definition remission is used. Subclinical synovitis has been associated with a higher risk of flares, progression of structural damage and unsuccessful drug tapering, especially when Doppler activity is present.^6–15 Imaging modalities such as MRI and ultrasound are more sensitive than clinical assessment for detecting inflammation.^16–19 However, ultrasound and MRI have disadvantages such as operator dependency for interpretation of the images, limited availability for routine clinical use, a steep learning curve and scanning time.^{20 21} In this context, new techniques that enable detection of subclinical inflammation in a fast and automated way could improve assessment of inflammation in routine clinical practice.

Thermography is a fast, non-invasive imaging technique that works by capturing the intensity of long wave infrared radiation emitted by bodies that increases with temperature.^22–24 Given that warmth is one of the cardinal signs of inflammation, thermography could be useful for detecting arthritis. Previous research (both preclinical and clinical) has demonstrated thermographically detectable changes in inflamed joints.^25–30

The aim of this study was to validate a novel machine learning-based computational method to automatically assess joint inflammation in patients with RA using thermal images of the hands.

Methods

Patients

The study population comprised 595 consecutive subjects recruited at outpatient visits to the departments of rheumatology and radiodiagnosis of two hospitals between March 2018 and March 2022. The inclusion criteria were a diagnosis of RA, psoriatic arthritis, undifferentiated arthritis, arthritis of the hands secondary to other diseases and osteoarthritis of the hands (OA). Subjects without a previous diagnosis of rheumatic disease were also recruited as healthy subjects (HS). Exclusion criteria were: age under 18 years; subjects with wounds, infection or trauma in the dorsal side of the hands; and subjects using bandages, cosmetics or other substances that could affect the thermal pattern prior to data collection. Data from the patients with RA whose thermal image was acquired with the Thermal Expert TE-Q1 camera were used to evaluate the performance of the method (validation set). Data from the other subjects were used for training and tuning of the machine learning model (development set).

The study complied with the Declaration of Helsinki.

Thermography

A thermographic image of the hands was taken using a Flir One Pro or a Thermal Expert TE-Q1 camera with a 6.8 mm lens. Both cameras use the same type of detector and capture infrared radiation on the same wavelength band (see detailed specifications in online supplemental table S1). Thermal cameras were connected to a smartphone, and a custom mobile application was developed to acquire the raw thermal images (ie, infrared wave intensity). Thermography was performed at the outpatient visits before ultrasound and physical examination and without an acclimatisation process or controlled room temperature in order to reproduce real-world conditions. The dorsal images of both hands were recorded with the fingers spread. No fixed distance between the camera and the hand was required, although the researcher was instructed to frame and focus the image.

Ultrasonography

Ultrasonography of both hands was performed in all patients except HS and was used as a reference standard for the detection and quantification of synovitis. Ultrasound was performed by three examiners (IM-I, CM and JAN) using a GE Logiq 9 with a 9-MHz to 14-MHz linear array transducer (Milwaukee, Wisconsin, USA). Both the patient and the probe were positioned according to EULAR guidelines.³¹ All participants underwent an ultrasound assessment (blinded with respect to other study results) consisting of a systematic examination (in B-mode and power Doppler mode) of the wrist (radiocarpal, midcarpal, distal radioulnar joint, using the highest score as representative), metacarpophalangeal joint 1–5, and proximal interphalangeal joint 1–5 of both hands. Each joint was scored using the OMERACT-EULAR semi-quantitative scoring system (0–3) for grey-scale synovial hypertrophy (GS) and for power Doppler (PD).^{32 33} At the patient level, an ultrasound sum score of the joints explored was made for GS (GS sum score) and PD (PD sum score). GS and PD sum scores were set to 0 in HS. Patients with a GS sum score grade >1 and PD sum score >0 were labelled as having active synovitis.^34–36

Clinical and laboratory assessment

Clinical and laboratory assessments were performed in the validation set and included the number of swollen and tender joints in the standard 28-joint count examination (SJC28 and TJC28), Patient Global Assessment (PGA) and Evaluator Global Assessment of disease activity based on a Visual Analogue Scale score (0–10), erythrocyte sedimentation rate (ESR) and C-reactive protein value (CRP). The criteria for clinical remission were applied, with remission being defined as follows: DAS28 <2.6^{37 38}; CDAI ≤2.8; SDAI ≤3.3^{5 39 40}; and the ACR/EULAR Boolean definition of remission (all ≤1: TJC28, SJC28, CRP in mg/dL and PGA).⁴⁰ Clinical and laboratory assessments were not performed on the development set, as these variables were not required to train or tune the machine learning model.

Thermal features extraction

Thermal images were resized to 160×120 pixels and processed to set the intensity values to an eight-bit grey-scale image. Additionally, images were improved by means of noise reduction, background removal and contrast enhancement (figure 1). For each thermal image, a set of regions of interest (ROIs) was obtained. ROIs were defined as local regions of the image with a large variation in intensity in all the directions, such as corners or blobs. The corners or blobs were detected in various sizes using a scale-space representation of the image. Local thermal features were extracted using scale-invariant and rotation-invariant descriptors from the ROIs detected. These features contained highly distinctive patterns, which may elucidate the underlying inflammatory process. Given that ROIs were selected and features extracted automatically by the algorithm, no human intervention was needed to process the thermal images.

Figure 1.

Eight-bit grey-scale thermal images of the hands of a patient with rheumatoid arthritis from the validation set after noise reduction, background removal and contrast enhancement.

Machine learning

Each thermal feature obtained from the development set was labelled with the PD sum score of the hand of the patient from whom the feature was extracted. A k-nearest neighbours algorithm was used to evaluate whether an unseen feature is characteristic of synovitis (ie, the average of the k-nearest features of the development set was used to assign a synovitis score to new features). A Thermographic Joint Inflammation Score (ThermoJIS) was assigned to each patient by averaging the synovitis scores of its features. The hyperparameters (eg, number of neighbours) were tuned using a leave-one-out cross-validation within the development set. The ThermoJIS raw values of the validation set were normalised to be in a more intuitive range (mean of 5 and a SD of 2.5). The higher the ThermoJIS value, the greater the confidence of having active synovitis. The analysis was performed using in-house software (Singularity Biomed).

Statistical analysis

Subject characteristics were described using means with SD, medians with IQR and frequencies with proportions, where appropriate. The performance of ThermoJIS was evaluated in the validation set, which is completely independent of the development set, to prevent overfitting. Subjects with missing ultrasound or thermographic data were excluded. Patients with missing clinical or laboratory data from the validation set were moved to the development set. Sample size for the development set was unknowable, so we recruited as many participants as possible to maximise training. The number of participants in the validation set exceeded the sample size calculations to provide robust results. The correlations of the scores were calculated using Spearman’s rank correlation coefficient. Differences between group medians were tested for significance using the Mann-Whitney test. The area under the receiver operating characteristic curve (AUROC) and the average precision were used to evaluate the diagnostic performance. Sensitivity, specificity, positive predictive value (PPV), negative predictive values (NPV) and F1-score were also calculated. Statistical significance was set at p<0.05 (two-sided). The statistical analysis was performed using Python V.3.7, NumPy V.1.19, Scikit-learn V.0.24, and SciPy V.1.4.

Results

Characteristics

All subjects tolerated the procedure well, and no adverse effects were observed. No participants were excluded due to lack of ultrasonography or thermography data. However, seven patients from the validation set were moved to the development set due to missing clinical or laboratory data. A diagram showing the flow of the participants and the description of each group is reported in online supplemental figure S1. Thermal images of the hands were acquired with a FLIR ONE Pro in 71% of cases in the development set. Demographic and ultrasound data from the development set are reported in table 1.

Table 1.

Demographic and ultrasound data: development set

	RA (n=169)	PsA (n=39)	UA (n=30)	SA (n=35)	OA (n=22)	HS (n=154)
Age (years)	61±15	58±13	61±14	64±15	61±11	52±17
Female sex (%)	75.1	59.0	60.0	60.0	90.9	57.8
Active synovitis (%)	43.8	20.5	56.7	57.1	4.5	NA
Active synovitis (GS sum score)	5 (3, 8)	6 (2, 9)	5 (3, 14)	4 (3, 5)	2 (2, 2)	NA
Active synovitis (PD sum score)	2 (2, 4)	3 (2, 5)	3 (2, 8)	2 (1, 3)	1 (1, 1)	NA

Distributions are presented as mean±SD or median (IQR).

GS, grey-scale synovial hypertrophy; HS, healthy subjects; OA, osteoarthritis; PD, power Doppler; PsA, psoriatic arthritis; RA, rheumatoid arthritis; SA, arthritis of hands secondary to other diseases; UA, undifferentiated arthritis.

Table 2details the demographic, clinical and laboratory data and disease activity of patients included in the validation set.

Table 2.

Demographic, clinical, laboratory assessment and ultrasound data: validation set

	All (n=146)	Active synovitis (n=77)	No active synovitis (n=69)
Age (years)	57±14	59±15	54±12
Female sex (%)	80.1	80.5	79.7
TJC28	1 (0, 4)	4 (1, 7)	0 (0, 2)
SJC28	0 (0, 3)	3 (0, 6)	0 (0, 0)
PGA	5 (2, 7)	5 (3, 8)	3 (1, 5)
EGA	3 (1, 5)	5 (2, 6)	2 (0, 3)
CRP (mg/L)	2.4 (1.0, 7.6)	5.0 (1.9, 11.0)	2.0 (1.0, 4.0)
ESR (mm/h)	19 (10, 34)	19 (10, 43)	19 (9, 27)
DAS28-CRP	3.1±1.4	3.8±1.4	2.3±0.8
CDAI	12.7±10.6	18.0±11.3	6.8±5.3
SDAI	13.4±11.1	19.0±1.9	7.2±5.3
DAS28-CRP Rem	60 (41.1%)	15 (19.5 %)	45 (65.2 %)
CDAI Rem	29 (19.9 %)	9 (11.7 %)	20 (29.0 %)
SDAI Rem	30 (20.5 %)	9 (11.7 %)	21 (30.4 %)
Boolean Rem	24 (16.4 %)	9 (11.7 %)	15 (21.7 %)
GS sum score	3 (0, 8)	7 (5, 11)	0 (0, 1)
PD sum score	1 (0, 4)	3 (2, 5)	0 (0, 0)

Distributions are presented as mean±SD or median (IQR).

CDAI, Clinical Disease Activity Index; CRP, C-reactive protein; DAS28, 28-joint Disease Activity Score; EGA, Evaluator Global Assessment; ESR, erythrocyte sedimentation rate; GS, grey-scale synovial hypertrophy; PD, power Doppler; PGA, Patient Global Assessment; SDAI, Simplified Disease Activity Index; SJC, swollen joint count; TJC, tender joint count.

Association between ThermoJIS and ultrasound scores

The correlation coefficients for ThermoJIS and the GS sum score (rho, 0.49; p<0.001) and for ThermoJIS and the PD sum score (rho, 0.51; p<0.001) were both moderate (figure 2). The correlation coefficients for ThermoJIS, GS sum score, PD sum score and the results of the clinical and laboratory assessments are detailed in table 3. The distribution of ThermoJIS is shown in online supplemental figure S2.

Figure 2.

Association between the Thermographic Joint Inflammation Score (ThermoJIS) and ultrasound. (A) Correlation between the grey-scale synovial hypertrophy (GS) sum score and the ThermoJIS; (B) Correlation between the power Doppler (PD) sum score and the ThermoJIS.

Table 3.

Correlation between ThermoJIS and ultrasound scores and the clinical and laboratory assessments

	TJC28	SJC28	PGA	EGA	CRP	ESR
ThermoJIS	0.33 (p<0.001)	0.38 (p<0.001)	0.16 (p=0.047)	0.32 (p<0.001)	0.20 (p=0.001)	0.28 (p=0.015)
GS sum score	0.52 (p<0.001)	0.79 (p<0.001)	0.35 (p<0.001)	0.60 (p<0.001)	0.40 (p<0.001)	0.22 (p=0.006)
PD sum score	0.56 (p<0.001)	0.72 (p<0.001)	0.39 (p<0.001)	0.60 (p<0.001)	0.36 (p<0.001)	0.16 (p=0.057)

CRP, C-reactive protein; EGA, Evaluator Global Assessment; ESR, erythrocyte sedimentation rate; GS, grey-scale synovial hypertrophy; PD, power Doppler; PGA, Patient Global Assessment; SJC28, swollen joints in standard 28-joint count; ThermoJIS, Thermographic Joint Inflammation Score; TJC28, tender joints in standard 28-joint count.

Detection of active synovitis

ThermoJIS had an AUROC of 0.78 (95% CI, 0.71 to 0.86; p<0.001) for detecting active synovitis (GS sum score >1 and PD sum score >0) (figure 3A). AUROC results were similar in different age and gender groups (online supplemental table S2). Sensitivity and specificity values depend on the cutoff chosen of ThermoJIS. The ThermoJIS value with maximum sensitivity and specificity was 3.56 (sensitivity, 94%; specificity, 51%; PPV, 68%; NPV, 88%; F1-score, 0.79). The probability of having active synovitis at different ThermoJIS intervals was also calculated (figure 4). ThermoJIS values between 4 and 5 showed no difference with random probabilities. Therefore, if values around this interval are considered indeterminate, the performance improves at the cost of reducing applicability of the method. In a subanalysis in which ThermoJIS values between 3.46 and 5.65 were considered indeterminate, the AUROC improved to 0.86 (95% CI, 0.78 to 0.95, p<0.001), although 43% of patients had an indeterminate result (figure 3B). In this subanalysis, the ThermoJIS value with maximum sensitivity and specificity was 5.81 (sensitivity, 87%; specificity, 82%; PPV, 81%; NPV, 88%; F1-score, 0.84). The precision-recall curves and average precisions are reported in online supplemental figure S3.

Figure 3.

Analysis of the area under the receiver operating curve (AUROC) of the Thermographic Joint Inflammation Score (ThermoJIS) for the detection of active synovitis. (A) Considering the entire validation set (AUROC, 0.78; 95% CI 0.71 to 0.86, p<0.001); (B) Considering ThermoJIS values lower than 3.46 and greater than 5.65 (AUROC, 0.86; 95% CI 0.78 to 0.95, p<0.001). TPR, True Positive Rate; FPR, False Positive Rate.

Figure 4.

Probability of presenting active synovitis at different ThermoJIS intervals in the validation set. The baseline probability (dashed line) is the proportion of patients with active synovitis in the set, that is, the random probability. ThermoJIS, Thermographic Joint Inflammation Score.

Detection of subclinical active synovitis

Some patients in clinical remission presented active synovitis: 25.0% for DAS28-CRP<2.6, 31% for CDAI ≤2.8, 30% for SDAI ≤3.3 and 37.5% for the ACR/EULAR Boolean definition of remission. The values of ThermoJIS in patients with RA in clinical remission were significantly higher in patients with active synovitis than in patients with remission determined by ultrasound (figure 5). AUROC values were 0.81 (95% CI, 0.70 to 0.92; p<0.001), 0.88 (95% CI, 0.76 to 1.0; p=0.001), 0.85 (95% CI, 0.71 to 0.98; p=0.003) and 0.92 (95% CI, 0.81 to 1.0; p=0.001), respectively.

Figure 5.

ThermoJIS distributions according to clinical remission criteria in patients with and without active synovitis. DAS28-CRP Rem (DAS28-CRP <2.6), CDAI Rem (CDAI ≤2.8), SDAI Rem (SDAI ≤3.3), and Boolean Rem (all ≤1: 28 tender joint count, 28 swollen joint count, C-reactive protein (mg/dL) and Patient Global Assessment). *p<0.05; **p<0.01; ***p<0.001. CDAI, Clinical Disease Activity Index; DAS28, 28-joint count Disease Activity Score; GS, grey-scale synovial hypertrophy; PD, power Doppler; SDAI, Simplified Disease Activity index; ThermoJIS, Thermographic Joint Inflammation Score.

Discussion

The 595 subjects recruited over 4 years for this cross-sectional study make it the largest study on thermography in rheumatology to date. Machine learning techniques generally require large amounts of data for training, so we recruited patients with different types of arthritis, OA and HS with the aim of increasing the development sample size and improving model training. Validation was performed exclusively in patients with RA in order to provide generalisable results for this disease. Furthermore, the use of two different camera models allowed us to note that the thermal patterns detected in a thermal image are not specific to a particular camera model, but are mantained between cameras with different specifications.

Our findings suggest that thermography analysis of the hands using our machine learning-based algorithm can successfully detect active synovitis. Furthermore, in patients in clinical remission, regardless of the definition used, the ThermoJIS was significantly higher if active synovitis was detected using ultrasonography. The ThermoJIS correlated moderately with ultrasound, but weakly with PGA, CRP and ESR, suggesting that ThermoJIS is not redundant with respect to symptoms and laboratory assessment and could be combined with these variables to develop new indices of disease activity. Moreover, the ThermoJIS also correlated better with ultrasound than symptoms and laboratory assessment.

Ultrasonography and MRI are sensitive methods for evaluating synovitis in RA, although routine use of these techniques is not feasible for most outpatient visits.^{41 42} In recent years, a new generation of affordable, uncooled, microbolometer-based thermal detectors has been developed. These thermal cameras are compact and perform sufficiently well for medical imaging.⁴³ Assessing joint inflammation using thermography of the hands with machine learning-based analysis is a non-invasive, instantaneous, automatic and operator-independent approach. These advantages make this promising new technique suitable for routine use in clinical practice. Given that ThermoJIS is higher in patients with subclinical synovitis than in those in ultrasound remission, it could be used to detect patients with persistent subclinical joint inflammation who have a higher risk of flares and progression of structural damage.^{8 10 44} Furthermore, the ThermoJIS could be of value in situations where the rheumatologist’s physical examination cannot be performed, since thermography can easily be performed remotely (ie, without the need to attend the clinic), even at the patient’s home.

In most previous studies using thermography to assess joint inflammation, descriptive statistics (eg, mean, SD) were used to report temperature in degrees.^28–30 These showed increased temperature in the inflamed joints. However, in our approach, the features extracted represent patterns rather than a temperature measurement, thus avoiding the need for precise calibration or the use of a blackbody to obtain accurate temperature readings. Another strength of the method is that the features extracted are not limited to regions that coincide with the anatomical sites of the joints; instead, the entire hand is analysed without human intervention.

Our study is subject to limitations. Although we validated our method internally with a validation set containing data that were not used in the development of the model, external validation is needed to avoid spectrum bias. In addition, we did not measure tenosynovitis. The inclusion of tenosynovitis could improve the results, since it would improve comprehension of the inflammatory process in the hand captured by the thermal camera. New studies are planned to externally validate performance in a new cohort of patients with RA.

In conclusion, the ThermoJIS detects active synovitis and could pave the way for the development of new tools for routine detection of joint inflammation in the rheumatologist’s office and for remote assessment of patients with RA.

Data availability statement

Data are available upon reasonable request. Anonymised patient data from the validation set, with the exception of the thermal images, will be shared upon request for research purposes depending on the nature of the request, the merit of the proposed research, the availability of the data and the intended use. In order to gain access, data requestors will need to enter into a data sharing agreement.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study involves human participants and was approved by (1) Name: CEIm del Hospital Universitari de Bellvitge ID: PR307/19 and AC044/16, and (2) Name: Comissió de Recerca del CSA ID: PR8/2019. Participants gave informed consent to participate in the study before taking part.