Validation of Fully Automatic Quantitative Software for Finger Joint Space Narrowing Progression for Rheumatoid Arthritis Patients

Abstract

In rheumatoid arthritis (RA), the radiographic progression of joint space narrowing (JSN) is evaluated using visual assessments. However, those methods are complicated and time-consuming. We developed an automatic system that can detect joint locations and compute the joint space difference index (JSDI), which was defined as the chronological change in JSN between two radiographs. The purpose of this study was to establish the validity of the software that automatically evaluates the temporal change of JSN. This study consisted of 39 patients with RA. All patients were treated with tocilizumab and underwent hand radiography (left and right hand separately) at 0, 6, and 12 months. The JSN was evaluated using mTSS (modified Total Sharp Score) by one musculoskeletal radiologist as well as our automatic system. Software measurement showed that JSDI between 0 and 12 months was significantly higher than that between 0 and 6 months (p < 0.01). While, there was no significant difference in mTSS between 0, 6, and 12 months. The group with higher disease activity at 0 months had significantly higher JSDI between 0 and 6 months than that with lower disease activity (p = 0.02). The automatic software can evaluate JSN progression of RA patients in the finger joint on X-ray.

Electronic supplementary material

The online version of this article (10.1007/s10278-020-00390-6) contains supplementary material, which is available to authorized users.

Introduction

Rheumatoid arthritis (RA) is a chronic inflammatory autoimmune disease that leads to progressive joint destruction resulting in severe disability [1, 2]. As current treatment of RA, the main management aim is to reduce the impact of the disease on patients’ lives by improving quality of life and reducing disability [3]. It is widely acknowledged that early diagnosis and aggressive treatment to control disease activity offers the highest likelihood of improving patient outcome and preserving function [4–6]. Radiographic joint damage correlates strongly with long-term functional decline in patients with RA [7], and therefore detecting joint damage progression has become a key to treatment. Thus, quantifying the subtle structural changes with high sensitivity is important in the assessment of therapeutic efficacy. Structural damage in RA has traditionally been assessed by plain radiography with modified Total Sharp Score (mTSS) as one of the most accepted visual assessments of joint space narrowing (JSN) [8]. However, traditional scoring methods are subjective and are not able to assess subtle changes with sufficient sensitivity and therefore require a longer period for clinical trials. In addition, these methods need specialized training and suffer from intra- and inter-observer variation because these are based on the visual assessment of radiologists [9, 10].

Over the years, various computer-based methods have been developed to measure JSN of metacarpophalangeal (MCP) and proximal interphalangeal (PIP) on a radiograph. Some studies showed that joint margin delineation with high success rate and joint space measurement gave highly sensitive and reproducible results. However, a false margin may be projected over or under the true margin due to the concave structure of the distal margin for the MCP joint and sometimes causes failure of margin delineation [11–14].

In a study by Ichikawa et al. [15], software assessed the JSN without margin detection using the joint space difference index (JSDI), which was defined as the chronological change in two radiographs using a temporal subtraction technique. Although the software quantified the JSN progression with high accuracy beyond human eyes even by non-expert of imaging analysis, it tends to be more time-consuming than scoring due to manual operations such as image registration and ROI location [16–19].

Recently, we developed an automatic system that can detect joint location, facilitate ROI placement, perform image registration, and compute the JSDI. In this paper, we investigated the validity of the software between JSDI measured by the software and JSN in two radiographs.

Materials and Methods

Patients

Thirty-nine patients (men/women, 4/35) who fulfilled the 1987 American Rheumatism Association classification criteria for RA were recruited for this study. All patients were treated with biological DMARDs (tocilizumab (TCZ) + methotrexate [MTX], n = 21; TCZ + MTZ + buccillamine (BUC), n = 7; TCZ + MTZ + tacrolimus (TAC), n = 3; TCZ + MTZ + Salazosulfapyridine, n = 2; TCZ + BUC, n = 2; TCZ + TAC, n = 1; TCZ monotherapy, n = 4).

Clinical and laboratory characteristics of the patients at baseline are shown in Table 1. The study was conducted in compliance with the Declaration of Helsinki and approved by the local ethics committee. Informed consent was obtained in the form of opt-out on the web-site.

Table 1.

Clinical characteristics of RA patients

Characteristic	Baseline
Total no. of patients included	39
Age, median (range) (years)	61.9 (30–88)
Sex, male/female	4/35
DAS28-CRP, mean (SD)	4.30 (1.14)

DAS28 disease activity score with 28 joints, CRP C-reactive protein, SD standard deviation

Radiography

Hand radiographs were obtained from patients at baseline, 6 months and 1 year. The median duration between baseline and follow-up was 6.4 months (range 4.9–7.9) and 13.2 months (range 10.4–19.7), respectively. All radiographs were taken following standard procedures [20] with digital X-ray equipment (DR-155HS2-5; Hitachi) under the following conditions: X-ray aluminum filter thickness = 1.5 mm, tube voltage = 42 kV, tube current = 100 mA, exposure time = 0.02 s, and film focus distance = 100 cm. The center of X-ray beam was the MCP joint of the middle finger. Radiographs were displayed as digital imaging and communications in medicine (DICOM) images with 2010 by 1490 pixels, and a 0.175 × 0.175 mm pixel size at 12-bit grayscale resolution.

Chiaki Narisawa and Kenneth Sutherland equally contributed to this study

In-house software

We used an original software application for this study. The software was developed in-house to automatically locate each joint and compute the JSDI. The software uses the pattern matching module of Matrox Imaging Library (MIL) [21] version 9.0 running on 64 bits Windows 7. MIL is a commercial C-language software development kit (SDK) often used in computer vision applications. The MIL pattern matching module attempts to locate a small model image within a large target image. The model image can optionally be rotated to better align with the source image. In addition to the (x, y) offset within the target image and rotation angle, the module also returns a pattern matching score which is an indicator (expressed as a percentage) of the pattern matching success. A pattern matching score of 100% would indicate an exact match. The search algorithm within MIL is based on a normalized correlation function.

Before the software can be used, one or more joint models must be created. The models are stored so that they can be used for multiple patients. A model is created by first selecting a typical baseline image. The typical image is selected from clinical cases and should be relatively disease free so that the target joint is not severely damaged. Furthermore, the typical image should have the same resolution and other radiographic parameters as the target images.

Three ROIs are specified (Fig. 1 and Supplemental file). A hand ROI is placed to specify the approximate left or right-hand region. The size of hand ROI should be large enough to include most of the finger bones. In our tests, the hand ROI size was 800 by 600 pixels. The hand ROI is used as a pattern match model to locate the general location of the left or right hand. A second ROI specifies the bones containing the target joint. This ROI helps prevent mistakes between adjacent fingers (to the left or right) and between MCP and PIP joints above or below. The bone ROI is used as a pattern search model within the region of the hand determined in the first step. The user then specifies a third ROI specifying the joint of interest. The size of the joint ROI was 80 by 20 pixels in our tests. The joint ROI can be rotated to match angle of the joint. The joint ROI is searched within the region of the bone ROI determined in the second step. After the baseline and follow up images have been aligned on the target joint, the JSDI is computed by considering the pixels within the joint ROI.

Fig. 1.

Model created with a typical hand radiograph with hand, bone and finger joint ROI specification. ROI region of interest

The bitmap of the hand ROI is saved in a TIFF file along with the offset and size of the bone and joint ROI. A single model is saved for each joint for the left and right hand. To account for variations of wrist anatomy (for example male and female), multiple models can be defined for each joint.

To compute the JSDI, the user first opens baseline and follow-up hand radiographs. The user then specifies one or more joint models. The software first automatically locates the approximate location of the base image carpal region using the carpal (hand) model bitmap read from the TIFF file. If multiple models were specified, the bitmap with the highest pattern matching score is used. The image is rotated to match the model as closely as possible.

Next, the offset of the joint ROI is used to initially place the joint ROI within the hand ROI on the baseline image. To account for slight variations of patient anatomy, the bitmap of the joint ROI model is then used for a secondary search within the carpal ROI region. In this step, rotation is disabled because the axial alignment of the wrist was already determined in the previous step.

The bitmap within the carpal ROI from the baseline image is then used as a model to search within the follow-up image. The follow-up image is rotated to maximize the pattern recognition score. The bitmap within the joint ROI on the baseline image is then used to fine-tune the placement of the joint ROI on the follow-up image, again without rotation.

The joint region of the baseline and follow-up images should now be aligned as closely as possible maximizing the pattern recognition score. The user can now direct the software to compute the JSDI. The JSDI is computed by calculating the average difference of each pixel inside the joint ROI. For each pixel in the joint ROI on the baseline image, the corresponding pixel on the follow-up is extracted based on the joint ROI offset and rotation determined by the pattern matching process. The absolute value of the difference is summed up and then divided by the number of pixels within the joint ROI. This average pixel difference is what we refer to as the JSDI. In this study, the width of the joint ROI was shrunken by 20% from the joint center so that JSN can be detected most sensitively.

Statistical analysis

Statistical analyses were performed in IBM SPSS version 24.0 (IBM) for Windows (Microsoft). All tests were two-tailed and the level for significance set to p < 0.05. We compared the JSDI between 0 and 6 months and that between 0 and 1 year to confirm that the JSDI can detect minute changes and not scored by observers. We next focused on disease activity at baseline. We divided the JSDI into two groups: joints with high disease activity at baseline and joints with remission, low, and moderate disease activity. The JSDI between the 0–6-month and 0–1-year groups and between the higher disease activity and lower disease activity groups were evaluated using Student t test. We also investigated the correlation between the 0–6-m and 0–1-year JSDI using Spearman’s correlation coefficient. Interphalangeal (IP) joints were not included because of high variability in JSDI values due to misregistration possibly caused by different manner of positioning during image acquisition.

Results

In total, 487 finger joints on the unilateral hand radiographs have been analyzed. Joints with ROI shifts were excluded before analysis. Figure 2 shows examples of the false case in joint location detection. The success rates of automatic ROI placement were 84.6% (264/312) and 71.5% (223/312) for MCP and PIP joints, respectively. Out of 487 joints, 414 joints had DAS-CRP data.

Fig. 2.

The excluded joints: The joint with an incorrect joint ROI. ROI region of interest

JSDI between 0 and 1 year was significantly higher than that between 0 and 6 months (p < 0.01). While, there was no significant difference in mTSS between 0, 6 months, and 1 year.

Out of 414 joints, 219 joints were assigned to the higher disease activity group and the others were assigned to the lower disease activity group. The group with higher disease activity at 0 month had significantly higher JSDI between 0 and 6 months than that with lower disease activity (p = 0.02).

There was a significant correlation between the JSDI between 0 and 6 months and that between 0 and 1 year at any joints (MCP2, r = 0.302, p < 0.01; MCP3, r = 0.300, p < 0.01; MCP4, r = 0.326, p < 0.01; MCP5, r = 0.321, p < 0.01; PIP2, r = 0.301, p < 0.01; PIP3, r = 0.325, p < 0.01; PIP4, r = 0.328, p < 0.01; PIP5, r = 0.349, p < 0.01).

Discussion

The purpose of this study was to establish the validity of the software that automatically evaluates JSN progression of finger joints on radiographs between baseline and follow-up images.

Generally, the results are satisfactory and promising. We divided the joints into the 0–6-month JSDI and the 0–1-year JSDI groups, 0–1-year JSDI was significantly higher than 0–6-month JSDI, while there is no significant difference in mTSS between baseline and follow-up. While, there was no significant difference in visual assessment between 0, 6, and 12 months. Moreover, in contrast to earlier systems [15–18], our approach is fully-automatic and the analysis time is 5~10 s per joints, which is very short compared with the previous study.

The disease activity in each joint space was measured using DAS28-CRP. A previous logistic regression analysis showed that baseline DAS28 (CRP), SDAI, and CDAI showed a significant correlation with radiographic joint damage and disease progression in patients with RA [22, 23].

Edward C et al. [24] have previously shown that DAS28-CRP scores at baseline were significant predictors of structural progression at months 6 and 12. Therefore, we considered that minute JSN progression can be predicted by the DAS28-CRP. In order to show the superiority of the software over visual assessment, the joints without score change between baseline and follow-up on mTSS were targeted. We confirmed that there was a difference in JSN up to 6 months depending on disease activity at baseline. This result was corresponding to the results of a previous study [24]. In this study, the software could automatically detect fine joint space narrowing that could not be detected by visual assessment. This observation supported the concept that the software was superior to human eyes.

Although we had a significant correlation between 0–6-month JSDI and 0–1-year JSDI groups for each joint, that was relatively small. Several reasons may explain the weakness of the correlation between 0–6-month JSDI and 0–1-year JSDI groups. In this study, all patients were treated with biological agents.

The efficacy of biological agents for symptoms and disease activity in RA has been established by extensive clinical experience [25, 26]. However, the clinical course of the disease is quite variable between individual patients. Correlation weakens as the patients’ disease progress changes in various ways.

Furthermore, some studies reported that treatment with biologic drugs is associated with a protective effect on bone mineral density in the hands [27], while chronic inflammation in RA generally causes bone loss [1, 2]. We consider that variations in bone density depending on the individual efficacy of the biological agents may also have affected JSDI which is calculated based on pixel values.

Some limitations of this study should be mentioned. First, the ROI misplacement occurs when the joints were severely damaged with ankylosis, complete luxation, or subluxation. This also occurs in the PIP that has a large positioning variation due to structural smallness and movable domain. We need to develop an automatic recognition system of severely damaged joints and increase detection accuracy in the PIP. Second, the JSDI may be affected by positioning variation. To ensure accuracy and reliability in measurements, it is necessary to perform standardized hand positioning. In this study, the hand positioning was performed according to the procedure carefully standardized before images were obtained. However, subtle finger flexion and rotation may cause on radiographs. Third, all patients had no progression of joint space narrowing in visual assessment throughout the period. Therefore, the agreement between the JSDI and the semi-quantitative score was not confirmed. Finally, our software is not well suited for detecting bone erosion.

In conclusion, we developed an automatic system that can detect finger joint location, place an ROI in the joint, perform image registration, and compute the JSDI and evaluated JSN progression on radiograph of the hand of RA patients. The software may be superior to human observers in terms of minute JSN progression detection. Therefore, the software might be useful for the evaluation of therapeutic responses in patients of RA.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethics Statement

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained in the form of opt-out on the website.