Reliability and Validity of the Tender and Swollen Joint Counts and the Modified Rodnan Skin Score in Early Diffuse Cutaneous Systemic Sclerosis—Analysis from the Prospective Registry of Early Systemic Sclerosis Cohort

Abstract

Objective

Determine inter/intra-observer reliability of tender and swollen joint counts (TJC,SJC) and Modified Rodnan Skin Score (MRSS) in diffuse systemic sclerosis (dcSSc) and assess content validity of TJC/SJC.

Methods

Ten rheumatologists completed SJC, TJC, and MRSS on 7 patients. Musculoskeletal ultrasound (MSUS) was performed.

Results

Inter-observer and intra-observer reliability for TJC was 0.97 and 0.99, for SJC was 0.24 and 0.71, and for MRSS was 0.81 and 0.94, respectively. MSUS abnormalities did not correspond with SJC/TJC.

Conclusion

We demonstrate excellent inter and intra-observer reliability for MRSS and TJC in dcSSc. However, SJC and TJC did not correspond to MSUS.

Introduction

Arthritis and tendinopathy are manifestations of systemic sclerosis (SSc) which affect up to 97% of patients [1,2] and contribute to disability and impaired quality of life [3]. Synovitis, joint contractures, and tendon friction rubs are associated with disease severity and progression in early SSc [4,5]. Despite the high prevalence and morbidity of arthropathy in SSc, its treatment has not been studied as a primary endpoint in randomized controlled trials and arthritis-specific outcome measures have not been validated.

The tender joint count (TJC) and swollen joint count (SJC) are core outcome measures used to assess disease activity in rheumatoid arthritis (RA) in the context of clinical trials and observational studies [6]. Adequate inter- and intra-observer reliability has been demonstrated for the 28-joint count (JC) in RA. [7]. However, the 28-JC can miss disease activity in psoriatic arthritis [8,9] or may over-interpret disease activity when fibromyalgia coexists with RA [10].

TJC and SJC are used as outcome measures in SSc clinical studies since they appear to have high face and content validity [11,12,13]. Our objectives were to evaluate inter- and intra-observer reliability of the 28 TJC, SJC and to assess the content validity of TJC and SJC through comparison to musculoskeletal ultrasound (MSUS) in patients with early dcSSc as this has not been previously studied. We additionally assessed the inter- and intra- observer reliability for the Modified Rodnan Skin Score (MRSS) in this context.

Methods

Study Design

A cross-sectional study was conducted at the University of Michigan (UM) in December 2014 on one day. Seven patients with early dcSSc were included from the Prospective Registry of Early Systemic Sclerosis (PRESS) cohort, a multicenter registry with participation of 12 US scleroderma centers. The protocol was approved by the IRB at UM and patients provided written consent prior to enrollment. Ten rheumatologists from PRESS centers participated in this exercise.

Training and Examinations

The rheumatologists underwent a training session by DK to perform TJC and SJC. Assessment occurred at 2 separate times on the same day for 5 subjects. One patient underwent the 10 initial examinations and one repeat examination. One patient underwent 6 initial examinations due to scheduling conflicts. These patients were excluded from intra-rater analysis. To prevent bias, each patient was examined separately, underwent MSUS, and then underwent repeat examinations.

Ultrasound

Grayscale and power Doppler MSUS of the bilateral hands and wrists was performed using high frequency linear array transducers (12–17 MHz) on clinical US machines (LOGIQ 9; GE Healthcare, Wauwatosa, WI; and iU22; Philips Healthcare, Bothell, WA). Static and dynamic images of flexor and extensor tendons at the wrist, distal radioulnar, metacarpophalangeal (MCP) and interphalangeal (IP) joints were obtained. Presence of joint effusion, synovitis, tenosynovitis, tendinosis and erosions were documented. The images were interpreted by the same musculoskeletal radiologist (GG) to reduce variability.

Statistical Analysis

Inter and intra-rater reliability were computed treating the TJC, SJC, and MRSS as continuous variables and by fitting a linear mixed model to the examiners' ratings with random effects for patient, rater and patient by rater. This methodology was chosen over the kappa statistic because of the repeated nature of the data. We compared the first SJC/TJC to MSUS of the hands and wrists (22 joints) using descriptive statistics. We calculated sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for each rheumatologist's first examination versus MSUS.

Results

The mean (SD) age of the 7 subjects was 41.6 (19.8) years and the mean disease duration from the first non-Raynaud's symptom was 2.7 (0.8) years. Three patients were female. Two (28.6%) patients had the RNAPolymerase3 antibody and one (14.2%) had the Scl-70 antibody. The mean (SD) MRSS at time of entry into the PRESS Registry was 21.1 (9.6). None had clinical overlap or RA, but RF and CCP were not tested.

Tender Joint Count

The mean (SD) 28 TJC for the patients was 4.2 (2.0). Inter-observer reliability for the 28 TJC was 0.97, while intra-observer reliability was 0.99. A second analysis was performed limited to the 22 joints of the bilateral hands and wrists assessed by MSUS. The 22 joint TJC showed an inter-rater reliability of 0.97 and intra-rater reliability of 0.99.

Swollen Joint Count

The mean SJC (SD) for the patients was 1.3 (0.8). Inter-observer reliability for the SJC was 0.24, and intra-observer reliability for the SJC was 0.71. The 22 joint SJC showed an inter-rater reliability of 0.17 and intra-rater reliability of 0.65.

MRSS

The inter-observer reliability for the MRSS was 0.81, and the intra-observer reliability for the MRSS was 0.94. Additionally, the inter-observer mean for the MRSS was 14.67 with a within patient standard deviation of 4.04. The intra-observer mean for the MRSS was 15.04 while the within patient standard deviation was 2.30.

Comparison of TJC and SJC vs. MSUS

Seven patients underwent MSUS of the hands and wrists (22 joints) resulting in US evaluation of 154 joints. Five (71.4%) patients had at least one joint with synovial thickening or synovitis on MSUS. Only 9.7% (15/154) of joints on MSUS showed synovitis or synovial thickening. No erosions were observed and only wrists and MCPs showed any abnormality. There were 1452 individual assessments of each joint for swelling and tenderness from the first round of examination limited to hands and wrists (Figure 1). Only 4.1% (60/1452) of examinations were positive for swelling and 20.4% (296/1452) were positive for tenderness, 2 % (3/150) of investigators' examinations of joints with synovial thickening or synovitis on MSUS identified swelling, and 9.3% (14/150) identified tenderness (Table 1). The mean sensitivity for the identification of a joint as swollen using the MSUS as a gold standard was 0.02 (0.05) and the mean specificity was 0.96 (0.03). The mean (SD) positive predictive value (PPV) was 0.04 (0.08) and the negative predictive value (NPV) was 0.89 (0.01) for swelling. The mean sensitivity for the identification of a joint as tender was 0.09 (0.03) and the specificity for the TJC was 0.77 (0.03). The mean (SD) PPV was 0.05 (0.01) and the NPV was 0.88 (0.01) for tenderness. For individual patient joints noted to be tender, the median proportion of investigators noting tenderness was 70%. For swollen joints this was just 10%. The median proportion of assessors noting tenderness and swelling concurrently in a joint was 0 [range 0-30%]. MSUS was used to assess for tendinopathy, osteoarthritis, and skin thickness, but did not show alternative explanations for swelling or tenderness.

Figure 1.

Flow Chart of Ultrasound and Physical Examination Findings.

Table 1. Summary of Abnormalities noted on Ultrasound and on Clinical Examination with Proportions of Agreement of Identification between Investigators and Ultrasound – Joint by Joint.

Joint Group	Wrist	MCP	PIP	Total
Total number examined by MSUS	14	70	70	154
Total number of clinical examinations of the joints	132	660	660	1452
Joints with synovitis or synovial thickening on MSUS	6/14 42.8%	9/70 12.9%	0/70 0%	15/154 9.7%
Clinical Examinations Identifying Swelling	6/132 4.6%	14/660 2.2%	40/660 6.1%	60/1452 4.1%
Clinical Examinations Identifying Tenderness	29/132 22.0%	117/660 17.8%	150/660 22.8%	296/1452 20.4%
Proportion of clinical examinations of the abnormal joints on MSUS identifying the joint as swollen	3/60 5.0%	0/90 0%	N/A - 0/0	3/150 2.0%
Proportion of clinical examinations of the abnormal joints on MSUS identifying the joint as tender	14/60 23.4%	0/90 0%	N/A - 0/0	14/150 9.3%
Proportion of clinical examinations of the normal joints on MSUS identifying the joint as not swollen	69/72 95.9%	556/570 97.6%	620/660 94.0%	1245/1302 95.6%
Proportion of clinical examinations of the normal joints on MSUS identifying the joint as not tender	57/72 79.2%	453/570 79.5%	510/660 77.3%	1020/1302 78.3%

In this table we show a joint by joint comparison for the ultrasound versus the examinations performed by the ten individual investigators. Musculoskeletal Ultrasound (MSUS) was performed on 22 joints of the bilateral hands and wrists [2 wrists, 10 metacarpal phalangeal joints (MCPs), and 10 proximal interphalangeal joints (PIPs)] of 7 patients. Ten physician investigators examined each of these joints, with some exclusions for missing data or if there was an overlying ulcer. The total number of clinical examinations of each joint group are shown. Proportions of joint abnormalities on MSUS and by examination are shown. Proportion of examinations identifying joints seen as abnormal on MSUS as swollen or tender as well as the proportion of examinations identifying joints seen as normal on MSUS as normal on examination are shown as well.

Discussion

Improved treatment of musculoskeletal manifestations of SSc is an important unmet need in the care of patients with SSc. Presently there are no validated outcome measures that objectively assess arthritis in this group. Some clinical trials (Abatacept:NCT02161406; Tocilizumab:NCT01532869) are studying the effect of biologics approved for RA in SSc, and the TJC and SJC are used as exploratory outcomes. However, SSc-arthritis is distinct from RA, and potential outcome measures for SSc-related arthritis must be validated specifically for SSc. This is the first study to assess the reliability and validity of the 28 joint SJC and TJC in early dcSSc, patients represented in clinical trials primarily assessing skin disease.[14]

In this study, we found near perfect inter-rater and intra-rater reliability for the TJC demonstrating good accuracy for this measurement in early dcSSc. The SJC, on the other hand, had only fair inter-rater reliability but substantial intra-rater reliability. In order to provide content validation of the TJC and SJC, we performed a joint-by-joint assessment using MSUS and noted poor agreement between the physical examination findings of individual joints with the MSUS findings. There was a low sensitivity and PPV, but high specificity and NPV. Discordance between clinical examination and ultrasound findings has also been observed in RA [15], and ultrasound can add additional information to the physical examination [16]. Reliability of MRSS was confirmatory of historical results[17].

This study had both strengths and weaknesses. All subjects were examined on the same day minimizing variability. We specifically recruited patients with early dcSSc not enriched for those with arthritis in order to assess TJC and SJC in the very population that would be recruited for dcSSc clinical trials. However, this resulted in subjects with low mean TJC, SJC and MSUS abnormalities which contributed to the low PPV. Agreement was high with respect to negative findings. A similar study should be undertaken in SSc patients with clinical evidence of arthritis and should include assessment of inflammatory markers and anti-CCP status, which were limitations of the present study. MSUS was used as a gold standard; however, MRI may be better suited for defining joint disease in this population.[18] Future study should also include incorporation of multiple MSK radiologists to better ascertain agreement in this population.

Current evidence in this pilot study does not support the use of TJC and SJC as outcome measures in an early dcSSc population not selected for arthritis. Further attention to the development of outcome measures for the musculoskeletal manifestations of SSc and validation of SJC/TJC in patients with SSc and arthritis are needed.