Interrater reliability of RheuMetric checklist scales for physician global assessment, inflammation, damage and patient distress

Abstract

Objective

To analyse interrater reliability of four RheuMetric checklist 0–10 visual numerical scales (VNSs) of physician global assessment (DOCGL), inflammation or reversible findings (DOCINF), organ damage or irreversible findings (DOCDAM) and patient distress or findings explained by fibromyalgia, depression or anxiety (DOCDIS).

Methods

A retrospective study was performed of data from a rheumatology fellows’ continuity clinic at Rush University. Each rheumatology patient seen in routine care with any diagnosis completed a multidimensional health assessment questionnaire (MDHAQ). Both the rheumatology fellow and attending rheumatologist independently completed RheuMetric estimates at the same visit for DOCGL, DOCINF, DOCDAM, DOCDIS and the proportion of DOCGL explained by each subglobal estimate (totalling 100%). Agreement between the two assessors was compared using paired t-tests, Spearman correlation coefficients, intraclass correlation coefficients (ICCs), Lin’s concordance correlation coefficients (LCCCs) and Bland–Altman plots.

Results

In 112 patients, mean levels of DOCINF were highest in inflammatory diseases, DOCDAM in osteoarthritis (OA) and DOCDIS in primary fibromyalgia (FM). However, mean DOCDAM was as high as DOCINF in inflammatory diseases. No statistically significant differences were seen between scores from attending rheumatologists and fellows. Agreement within 2/10 ranged from 60% for DOCGL to 71% for DOICINF and DOCDAM. Spearman correlations were 0.49–0.65, ICCs were 0.46–0.63 and LCCCs were 0.46–0.62 between rheumatologist and fellow, indicating moderate agreement; reliability was slightly higher for each subglobal VNS than for DOCGL.

Conclusion

RheuMetric 0–10 DOCGL, DOCINF, DOCDAM and DOCDIS have moderate interrater reliability and are feasible in routine care to estimate patient status beyond DOCGL for improved management decisions.

Key messages.

• Physician RheuMetric 0–10 estimates of inflammation, damage and distress feasibly quantitate patient status.

• RheuMetric estimates have moderate interrater reliability, and indicate contemporary higher damage and distress than inflammation.

• RheuMetric estimates may document the expertise of rheumatologists and improve clinical decisions in routine care.

Introduction

A RheuMetric physician checklist (Fig. 1) [1–3] for routine care documents four 0–10 visual numeric scales (VNSs) for physician global assessment (DOCGL), inflammation or reversible findings (DOCINF), damage or irreversible findings (DOCDAM) [secondary osteoarthritis (OA)] and patient distress (DOCDIS) [fibromyalgia (FM), depression, etc] [2]. An initial rationale for RheuMetric was that DOCGL, designed primarily to assess inflammatory activity (effectively) in clinical trials [4], may be interpreted variably in routine care by different rheumatologists to estimate joint damage and/or patient distress, as well as inflammation [5]. Non-inflammatory comorbidities often are unrecognized or underrecognized without formal screening [6, 7] but are found in 20–50% of patients with rheumatoid arthritis (RA) [6, 8–10] and other diseases [11, 12]. Even when recognized, damage and/or distress generally are recorded in medical records only as narrative descriptions rather than quantitative measures, limiting recognition of possible changes in status and complicating interpretation for treat-to-target in many patients with RA [13–15].

Figure 1.

RheuMetric physician checklist

A further rationale for RheuMetric is to document the expertise of a rheumatologist to recognize quantitatively whether an individual patient’s symptoms result from inflammation, damage and/or patient distress. Many patients have clinically important symptoms from two or even all three of these sources. Therefore, clinical decisions in patients with rheumatic diseases often are more complex than in many common chronic diseases with a ‘gold-standard’ biomarker, such as hypertension or diabetes [1–3].

An important consideration in introducing any new measure to medical care is validity and reliability. Face validity of RheuMetric was documented as highest DOCINF in RA, highest DOCDAM in osteoarthritis (OA) and highest DOCDIS in FM [1, 2]. However, DOCDAM was higher than DOCINF in RA patients at three sites [1, 2], reflecting effective current control of inflammatory activity. Criterion and discriminant validity of RheuMetric was documented by evidence in RA that variation of DOCINF was explained significantly and specifically by swollen joint count and inversely by disease duration, of DOCDAM by damaged joint count, radiographic scores and physical function and of DOCDIS by FM and depression [3]. This report analyses interrater reliability of RheuMetric estimates between a senior attending rheumatologist and rheumatology fellow in patients with various rheumatic diagnoses seen in routine care.

Patients and methods

Study design

A cross-sectional study was conducted of data collected at a continuity clinic for fellows in the Division of Rheumatology at Rush University Medical Center. Each patient seen at Rush rheumatology is asked to complete a multi-dimensional health assessment questionnaire (MDHAQ) at each visit while waiting to see the rheumatologist. In the fellows’ clinic, each patient is evaluated initially by a fellow and subsequently seen with an attending rheumatologist within the same hour. The rheumatology fellow and attending rheumatologist each completed a RheuMetric checklist independently for each patient. The estimates of four different fellows and two different attending rheumatologists were analysed in this study.

MDHAQ and RheuMetric data are entered into a data repository established by the Rush Division of Rheumatology in 2014. The patient and physician questionnaires are regarded as quality measures by the Institutional Review Board (IRB) of Rush University Medical Center and are exempt from informed consent requirements. The data repository hosted at a local site is also regarded as exempt from formal IRB approval.

Patients

Unselected patients with any rheumatic disease seen in the fellows clinic were included in the study. All included patients were >18 years of age.

RheuMetric checklist

A RheuMetric checklist (Fig. 1) is a one-page, two-sided physician questionnaire that is completed within the encounter. It includes a 0–10 VNS physician global assessment (DOCGL) ranging from 0 (excellent) to 10 (very poor) in 0.5 increments and three 0–10 VNS subscales ranging from 0 (none) to 10 (most severe) to estimate the level of inflammation or reversible findings (DOCINF), damage or irreversible findings (DOCDAM) and distress, i.e. symptoms explained by neither inflammation nor damage (e.g. fibromyalgia, depression) (DOCDIS) [2]. RheuMetric includes a query to estimate the proportion of DOCGL attributed to inflammation, damage and distress (totalling 100%). Other RheuMetric scales include the proportion of DOCGL explained by the rheumatic disease or other comorbidities (totalling 100%); estimates of prognosis with and without therapies, classified as excellent, very good, good, fair or poor; and diagnostic certainty as high, moderate, low or none, as well as a 28- or 42-joint count to record swelling, tenderness/pain on motion, limited motion/deformity and surgical intervention, with a notation for ‘normal’ as a convenience so that no further checks are needed, particularly as >60% of all joints are ‘normal’, or negative for surgical intervention [2]. Only 0–10 estimates for DOCGL, DOCINF, DOCDAM, DOCDIS and the percentages of DOCGL attributed to inflammation, damage or patient distress were analysed in this report.

Patient MDHAQ measures

An MDHAQ is a two-page questionnaire adapted from the original HAQ to improve the quality of clinical care and patient outcomes in routine clinical care [16]. The MDHAQ includes 10 queries to evaluate physical function (FN), scored 0 (‘without any difficulty’), 1 (‘with some difficulty’), 2 (‘with much difficulty’) or 3 (‘unable to do’), as in the original patient-friendly HAQ format. The MDHAQ also includes three queries concerning sleep, anxiety and depression in the 0–3 HAQ format and three (0–10) VNSs for pain, patient global assessment (PATGL) and fatigue [16].

The Routine Assessment of Patient Index Data 3 (RAPID3) is a composite index that includes the three patient-reported RA core dataset measures (FN, pain and PATGL), each scored 0–10 for a total of 30 [17]. MDHAQ/RAPID3 has been found informative in OA, SLE and FM, in addition to RA and other rheumatic diseases [18]. Other MDHAQ indices for FAST4 (fibromyalgia assessment screening tool), MDS2 (MDHAQ depression screen) and MAS 2 (MDHAQ anxiety screen) [18] were not analysed in this report.

Statistical analysis

Descriptive statistics, including means and s.d.s for each scale, were computed. The difference between estimates of the senior rheumatologist and rheumatology fellow for each DOCGL, DOCINF, DOCDAM and DOCDIS was calculated. A difference of ≤2/10 was used to define agreement between the two physicians. This level was chosen based on prior studies indicating considerable variability between different examiners for swollen joints and tender joints in 28-joint counts [19–23] and for many clinical estimates of all types (beyond joint counts) by physicians [24]. Differences between the two physicians were classified into three categories: attending rheumatologist > fellow by 2/10, the difference between attending rheumatologist and fellow was within 2/10 or attending rheumatologist < fellow by 2/10. Spearman’s correlation coefficients were analysed to compare the monotonic relationship between the two estimates of the two physicians. Intraclass correlation coefficients (ICCs) and Lin’s concordance correlation coefficients (LCCCs) were analysed for agreement. The LCCC compares individual paired rheumatologist–fellow scores, while the ICC compares two groups (rheumatologist vs fellow). Levels of correlation coefficients were regarded as ‘strong’ if >0.7, ‘moderate’ if 0.40–0.69 and ‘weak’ if <0.40 [25].

The mean differences between scores of attending rheumatologists and rheumatology fellows were analysed using paired t-tests with corresponding 95% CIs. Bland–Altman plots provide a visual representation of agreement, plotting the difference between the pair of measures against the mean of the pair [26]. A value closer to zero indicates better agreement between these two methods of measurement. All analyses were carried out using SAS software, version 9.4 (SAS Institute, Cary, NC, USA).

Results

The study included 112 patients who were evaluated by both an attending rheumatologist and rheumatology fellow. The patients included 39 (35%) with inflammatory diseases, 20 (18%) with OA, 12 (11%) with FM and 41 (37%) with other diagnoses (Table 1). Among the 39 patients with inflammatory diseases, 9 (8%) had RA, 12 (11%) systemic lupus erythematosus (SLE), 9 (8%) axial spondyloarthropathy (SpA), 4 (4%) vasculitis and 5 (4%) gout. In patients with inflammatory diseases, the mean DOCINF was 2.0, DOCDAM 2.9 and DOCDIS 2.7, indicating that damage and distress are as prominent as inflammation in this group of patients. In all patients, 36% of DOCGL was attributed to inflammation, 38% to damage and 26% to distress, echoing the findings for the 0–10 VNS, although inflammation was relatively higher and distress relatively lower (Table 1).

Table 1.

Mean (s.d.) for RAPID3, 0–10 visual numerical scale PATGL, DOCGL, and three subscales DOCINF, DOCDAM and DOCDIS according to diagnosis.

Primary rheumatic physician ICD-10 diagnosis	n (%)	RAPID3 (0–30)	Global estimates (0–10)		Physician subscales (0–10)			% of DOCGL attributed to…, mean (s.d.)
			PATGL	DOCGL	DOCINF	DOCDAM	DOCDIS	Inflammation	Damage	Distress
RA	9 (8)	16.8 (5.4)	6.6 (2.1)	4.7 (2.0)	3.1 (1.6)	4.6 (3.2)	3.4 (3.0)	53 (28)	34 (22)	13 (9)
SLE	12 (11)	9.6 (7.6)	4.4 (3.3)	3.0 (2.2)	2.2 (2.1)	2.0 (1.6)	1.7 (2.5)	48 (36)	24 (21)	28 (33)
SpA	9 (8)	11.7 (8.3)	4.4 (3.7)	5.5 (2.5)	4.1 (2.9)	2.1 (1.1)	3.0 (3.2)	55 (37)	20 (20)	25 (33)
Vasculitis	4 (4)	8.5 (7.9)	3.0 (2.6)	3.0 (3.5)	1.8 (2.3)	1.4 (1.3)	3.5 (4.2)	60 (28)	10 (14)	30 (14)
Gout	5 (4)	16.4 (3.3)	6.1 (1.9)	4.0 (2.3)	2.4 (1.7)	4.0 (3.2)	0.9 (0.5)	52 (37)	43 (36)	5 (5)
All inflammatory diseases	39 (35)	12.5 (7.3)	5.0 (3.1)	4.1 (2.5)	2.8 (2.2)	2.8 (2.4)	2.5 (2.9)	53 (32)	28 (24)	19 (24)
OA	20 (18)	17.3 (6.6)	7.1 (2.8)	4.5 (1.9)	0.7 (0.7)	4.1 (2.3)	2.9 (2.7)	11 (14)	67 (30)	23 (28)
FM	12 (11)	15.5 (5.5)	6.5 (1.9)	4.0 (1.9)	1.5 (1.7)	1.7 (1.2)	5.1 (2.3)	18 (30)	26 (26)	56 (33)
Other diagnosis	41 (37)	12.2 (6.0)	5.0 (2.7)	3.7 (2.0)	1.9 (1.6)	2.7 (2.3)	2.0 (2.1)	40 (32)	37 (26)	23 (28)
Total	112 (100)	13.6 (6.8)	5.5 (2.9)	4.0 (2.1)	2.0 (1.9)	2.9 (2.3)	2.7 (2.6)	36 (33)	38 (30)	26 (30)

The highest mean inflammation (DOCINF) score (2.8) was seen in patients with inflammatory diagnoses, while the highest mean damage (DOCDAM) was seen in OA (4.1) and patient distress (DOCDIS) in FM (5.1) (Table 1). However, the mean DOCDAM in RA was 4.6, higher than in OA, indicating that DOCGL was explained as much by damage, i.e. secondary OA, as by inflammatory activity in these RA patients, reflecting excellent control of inflammation in recent years. Furthermore, the mean DOCDAM was higher than DOCINF in all patients and most specific diagnoses, and DOCDIS was higher than DOCINF in all patients and almost as high even in all inflammatory diagnoses, ≥1.5/10 in all diagnostic groups except gout (Table 1).

The mean estimates for each subscale between evaluators differed by <3% of the 0–10 scale, ranging from 0.10 for DOCDAM to 0.29 for DOCGL (Table 2); none of these differences were statistically significant. Spearman’s correlations ranged from 0.49 for DOCGL to 0.65 for DOCDAM, and ICCs and LCCCs ranged from 0.46 and 0.46 for DOCGL to 0.63 and 0.62 for DOCDIS, respectively (Table 2). The two types of correlation coefficients gave very similar results, although correlation coefficients for DOCINF, DOCDAM and DOCDIS were higher than for DOCGL (Table 2).

Table 2.

Mean (s.d.) for the four physician estimates.

VAS (0–10)	Attending rheumatologist	Rheumatology fellow	Pairwise mean difference (95% CI)	Spearman’s correlation	ICC (95% CI)	LCCC (95% CI)	Attending rheumatologist and rheumatology fellows agreement by 2/10 units, n (%)
							Rheum > fellow	Rheum = fellow	Rheum < fellow
Overall DOCGL	3.67 (1.80)	3.96 (2.26)	−0.29 (−0.68, 0.11)	0.49*	0.46 (0.30, 0.59)	0.46 (0.31, 0.59)	15 (14)	67 (60)	30 (27)
DOCINF	1.91 (1.64)	2.17 (2.16)	−0.26 (−0.60, 0.08)	0.58*	0.54 (0.40, 0.66)	0.54 (0.40, 0.65)	12 (11)	80 (71)	20 (18)
DOCDAM	2.70 (2.16)	2.80 (2.24)	−0.10 (−0.47, 0.27)	0.65*	0.60 (0.47, 0.71)	0.60 (0.46, 0.71)	14 (13)	79 (71)	19 (17)
DOCDIS	2.93 (2.84)	2.65 (2.62)	0.28 (−0.16, 0.72)	0.63*	0.63 (0.50, 0.73)	0.62 (0.50, 0.73)	25 (22)	72 (64)	15 (13)

Mean (s.d.) for the four physician estimates according to attending rheumatologists and rheumatology fellows, Spearman’s correlations, ICC, LCCC and levels of concordance and discordance for each estimate in the evaluation of 112 patients.

^* P < 0.005.

The mean differences of DOCGL, DOCINF, DOCDAM and DOCDIS do not differ between estimates of attending and fellow rheumatologists.

Agreement of attending rheumatologists and rheumatology fellows within 2 units of the 0–10 scale ranged from 60% for DOCGL to 71% for DOCINF and DOCDAM (Table 2). Estimates of the attending rheumatologist were in general higher than for rheumatology fellows for DOCGL, DOCINF and DOCDAM, but not for DOCDIS (Table 2). Bland–Altman plots for each of the four VNSs demonstrate acceptable agreement of measurements by the two clinicians (Fig. 2). Agreement appears more robust at lower and higher levels of the scales and less robust at intermediate levels.

Figure 2.

Bland–Altman plots for RheuMetric physician estimates by an attending rheumatologist and rheumatology fellow for (A) DOCGL (physician global assessment), (B) DOCINF (physician assessment of inflammation), (C) DOCDAM (physician assessment of damage) and (D) DOCDIS (physician assessment of patient distress)

Discussion

This study documents moderate agreement between observers in 0–10 quantitative RheuMetric estimates of inflammation, damage and distress and their contributions to DOCGL across various rheumatic diseases. Conventional quantitative data collected in routine rheumatology care and longitudinal databases generally include only laboratory tests and other measures to assess inflammatory activity, reflecting data collected in clinical trials to identify statistically significant group differences between active and control treatments. However, RA clinical trials do not include the majority of all RA patients seen in routine care, with typically only 5–30% of all patients with the target diagnosis, while excluding patients with insufficient inflammatory activity, extensive joint damage, patient distress and other comorbidities [27, 28].

In one study of routine care of RA patients over 5 years, measures of inflammatory activity, including all seven RA core dataset measures, were stable or slightly improved while measures of damage, including deformed/limited motion joint count, radiographic scores, grip strength and walking time, indicated clinically important progression [29]. Furthermore, all RA core dataset clinical measures (i.e. excluding laboratory tests) may be elevated not only by inflammatory activity, but also by joint damage [8, 30] and patient distress, seen as FM [9] and/or depression [6, 10]. Many patients with inflammatory diseases may have not only clinically important inflammatory activity, but also organ damage or distress (some with all three), which often complicates clinical decisions for treat to target [13–15] and the description of outcomes. Indices of damage in RA have been reported [31, 32] but do not appear to be widely used.

Joint damage and patient distress may be recognized by rheumatologists in many patients but generally are noted in medical records only as narrative descriptions rather than as quantitative measures, including in those with primary diagnoses of RA, OA, FM and many other non-inflammatory rheumatic diseases. Even when DOCGL is recorded quantitatively in routine care, variability exists among rheumatologists, with some focusing solely on inflammatory activity while others may incorporate joint damage and patient distress into their estimates [5].

The higher agreement between the two rheumatologists for subglobal scores compared with DOCGL in this study (Table 2) may suggest a reduction of ambiguity through the availability of three subscales for inflammation, damage and patient distress. Higher agreement in patients with better or poorer status than those in intermediate status may reflect clinical experience with a higher complexity of management decisions in patients in intermediate status than those who are doing well or poorly, in whom management usually is more straightforward.

The observed Spearman’s correlations between RheuMetric estimates of two physicians of 0.49–0.65 and ICCs and LCCCs ranging from 0.46 to 0.63 (Table 2) are in a similar range to reported correlation coefficients of 0.15–0.52 in comparisons of different observers of swollen and tender joint counts [19–21]. The similarity of correlation coefficients to those of published reports may suggest acceptable reliability, although classified as ‘moderate’, as published instructions for joint counts have been available for decades while the RheuMetric checklist is a new measure used for less than a decade. Some reports indicated improvement in joint count interrater reliability with training [33, 34], yet no evidence is reported that such training was translated into ‘real-world’ routine clinical care. Specific incremental training beyond general educational and clinical activities generally is not feasible in usual assessment. All data in this report were collected in routine care without specific training of the participating physicians.

Although many rheumatologists might feel that specific joint counts are a superior measure to global scores, DOCGL is most likely among seven RA core dataset measures to distinguish active from control groups in clinical trials [35]. Global measures represent a substantial improvement over the absence of any quantitative measures to facilitate accurate comparisons of patient status over time. The addition of three global subscales also documents the expertise of a rheumatologist to recognize quantitatively whether an individual patient’s symptoms result from inflammation, damage and/or patient distress, including the complexity of rheumatic diseases in many patients who have clinically important signs and symptoms from two or even all three of these sources. Finally, global 0–10 estimates are feasible in routine clinical care, which is not the case for more elaborate quantitative measures used in research settings.

Several limitations were seen in this study. First, the data were collected from only a single setting; however, results indicating higher scores for DOCINF in RA, DOCDAM in OA and DOCDIS in FM are reported from three sites [1–3]. Second, comparisons between attending rheumatologists and rheumatology fellows were not explored in depth, although documentation of moderate agreement between the two suggests that these measures can be used by any rheumatologist. Third, as noted above, no formal instructions or training to score RheuMetric were given; while such training may have increased reliability, reasonable agreement without formal instruction simulates real-world routine care and recognizes the feasibility of scoring.

In conclusion, documentation of physician estimates of inflammation, damage and distress on a RheuMetric checklist offers a reliable method to assess inflammatory activity as well as non-inflammatory damage and patient distress in patients with any rheumatic disease. These quantitative estimates may provide insights to health professionals in making clinical decisions and recognizing changes over time.

Data availability

Data will be made available upon request to the corresponding author.

Authors’ contributions

J.S.: Interpretation of data, drafting and revision of the manuscript, final approval of the version to be published; I.C.: study design, data acquisition and analysis, drafting and revision of the manuscript, final approval of the version to be published; T.L.: data analysis, drafting and revision of the manuscript, final approval of the version to be published; J.A.B.: data interpretation, drafting and revision of the manuscript, final approval of the version to be published; T.P.: study design, interpretation of data, drafting and revision of the manuscript, final approval of the version to be published.

Funding

Support was provided by Medical History Services.

Disclosure statement: T.P. owns a copyright and trademark for MDHAQ and RAPID3. License fees are received from for-profit pharmaceutical and electronic medical records companies for the use of MDHAQ and RAPID3, but not from clinicians and academic researchers, who may freely use MDHAQ and RAPID3 to monitor patient status in usual clinical care. All funds from license fees are used to further develop and analyse quantitative measures by patients and doctors in routine clinical rheumatology. The remaining authors have declared no conflicts of interest.