Abstract
Purpose
To describe use of the ACR/EULAR (AE) rheumatoid arthritis (RA) remission criteria in clinical practice.
Methods
We examined remission in the US Veterans Affairs RA (VARA) registry of 1,341 patients (91% men) with 9,700 visits and a community rheumatology practice (ARCK) of 1,168 patients (28% men) with 6,362 visits. We studied cross-sectional and cumulative probabilities, agreement among various remission criteria, and aspects of reliability using Boolean definitions and CDAI and SDAI methods proposed by AE.
Results
By AE definition for community practice (swollen and tender joints ≤1, patient global ≤1), cross-sectional remission was 7.5% (6.4, 8.7) for ARCK and 8.9% (7.9, 9.9) for VARA. Cumulative or remission at any observation was 18.0% (ARCK) and 24.4% (VARA) over a mean of 2.2 years. Addition of ESR or CRP to criteria reduced remission to 5.0-6.2%, and use of CDAI/SDAI increased proportions to 6.9-10.1%. 1.8%-4.6% of patients met remission criteria at ≥2 visits. Agreement between criteria definitions was good by Kappa and Jaccard measures. Among patients in remission, the probability of a remission lasting 2 years was 6.0%-14.1%. Among all patients the probability of a remission lasting 2 years was <3%. Remission and examination results varied substantially among physicians by multilevel analyses.
Conclusion
Cross-sectional remission occurs at 5.0%-10.1%, with cumulative remission 2-3 times greater. Long-term remissions are rare. Problems with reliability and agreement limit criteria usefulness in the individual patient. However, the criteria can be an effective method for measuring clinical status and treatment effect in groups of patients in the community.
Keywords: Rheumatoid arthritis, Remission, Reliability
Remission in rheumatoid arthritis (RA) was first described and quantified by Pemberton in 1927 (1), followed by Thompson in the next decade (2). In 1981, Pinals et al. published Preliminary Criteria for Remission in Rheumatoid Arthritis (3). These criteria, which became official American College of Rheumatology (ACR) criteria, were difficult to use and did not have a clear basis in scientific measurement. Over the years, a series of different, often ad hoc criteria were proposed or used in publications. These additional criteria have been described in detail in a number of important publications (4-6).
In 2011, the ACR and European League Against Rheumatism (EULAR) jointly published The American College of Rheumatology/European League against Rheumatism Preliminary Definition of Remission in Rheumatoid Arthritis for Clinical Trials (6). In this paper the authors also suggested “that a definition of remission be developed for clinic based practice that would not require an acute phase reactant, as long as it would capture remission as stringently as the measure employed for clinical trials,” and furthermore that “… core set measures should be used to define remission and that any definition of remission in clinical trials should look toward and make possible a similar definition in clinical practice.”
Remission in clinical practice is an important issue. For groups of patients assessed in observational studies, remission can be a marker of disease severity and treatment response. The ACR/EULAR recommendation for the use of remission criteria in clinic-based practice suggests, in addition, that the determination of remission be extended to the individual patient. If applied to the individual patient, remission or the lack of it could serve as a measure of treatment success that could be used by the patient, and by 3rd party payers and regulatory authorities to characterize the quality of health care, dictate access to care or govern the use of specific therapies.
A large recent study of cross-sectional remission included 5,848 patients from 67 sites in 24 countries (4). The authors evaluated 8 different criteria, 3 of which are particularly of interest to the current study: Clinical Disease Activity Index (CDAI) (7), Disease Activity Score-28 (DAS-28)(8) and RAPID-3 remission. Of these, only CDAI remission is recognized by the new ACR/EULAR criteria. In this study of Sokka et al. the proportion in remission by CDAI criteria varied strikingly among countries, ranging from 0% to 35.3%, with a value of 18.5% in the US.
In the current study we obtained data from all patients and all clinic visits in multi-physician sites, including a private practice rheumatology specialty group and 9 US Veterans Administration outpatient rheumatology clinics that included 38 physicians. We used multi-level, repeated measures methods to examine the probability of remission at a given clinic visit, the cumulative probability of remission, the probability of a second remission, and the duration of remission for each of the ACR/EULAR definitions. In addition, we evaluated the degree of physician bias, the effect of patient global changes, and the agreement between the various definitions.
Methods
Patients and variables
From 10/5/2006 to 11/16/2010 1,435 patients with 15,152 visits were seen at the Arthritis and Rheumatology Clinics of Kansas (ARCK), a 5-physician rheumatology specialty clinic (9). All patients underwent assessments that included a count of 28 swollen joints (SJC), 28 tender joint counts (TJC), physician and patient visual analog scale (VAS) globals (PhGlobal and PtGlobal), and the Health Assessment Questionnaire Disability Index-II (HAQ-II) (10). The erythrocyte sedimentation rate (ESR) was not systematically obtained at all visits for clinical and insurance reasons. In this study we evaluated the 1,168 patients with 6,362 visits who had complete data for SJC, TJC, PtGlobal and ESR. All patients were seen as part of routine medical care.
We also evaluated patients who were part of the US Veterans Affairs RA (VARA) registry, consortium of 9 sites (Dallas, Washington DC, Omaha, Salt Lake City, Denver, Jackson, Portland, Brooklyn, Iowa City) and 38 physicians, during the period 12/11/2002 through 9/24/2010 (11, 12). Data collected included SJC, TJC, PtGlobal, PhGlobal, ESR, C-reactive protein (CRP), and the multi-dimensional Health Assessment Questionnaire Disability Index MDHAQ (13). From 1,510 patients with 11,915 visits, we studied 1,341 patients and 9,700 visits by restricting our analyses to those with complete data for SJC, TJC, PtGlobal and ESR. All patients were seen as part of routine medical care.
From the above variables we calculated indices of RA disease activity including the DAS-28 (8), the Patients Activity Scale- II (PAS-II) (14), the Routine Assessment of Patient Index Data 3 (RAPID-3) (15), the Simplified Disease Activity Index (SDAI) (7) and the CDAI (7). The PAS-II and RAPID 3 are essentially the same scale except that the PAS-II uses the HAQ-II and RAPID-3 uses the MDHAQ. The results of both scales are equivalent. The scales represent (PtGlobal + patient pain + (3 × HAQ-II or MDHAQ)) /3. The SDAI is the sum of the TJC (on a 0-28 scale), SJC (0-28), PtGlobal (0-10), PhGlobal (0-10) and CRP (mg/dL). The CDAI is the sum of the TJC (0-28), SJC (0-28), PtGlobal (0-10), and PhGlobal (0-10).
From the above scales we created the following remission criteria as defined in the ACR/EULAR (AE) remission paper (6): AE 3 = ≤1 SJC + ≤1 TJC + PtGlobal ≤1. AE 3 ESR = ≤1 SJC + ≤1 TJC + PtGlobal ≤1 + ESR <20 (men) or < 30 (women). AE CRP = ≤1 SJC + ≤1 TJC + PtGlobal ≤1 + CRP ≤1. AE 4 = ≤1 SJC + ≤1 TJC + PtGlobal ≤1 + PhGlobal ≤1. AE CDAI = CDAI ≤ 2.8. AE SDAI = SDAI ≤ 3.3.
We also evaluated criteria that were not included in the ACR/EULAR recommendations: DAS-28 remission = DAS-28 <2.6 (4). PAS-II/RAPID-3 = PAS-II ≤1 or RAPID-3 ≤1 (4). For comparison with the ACR/EULAR paper we evaluated Minimal Disease Activity (MDA) criteria: MDA was present if the patient satisfied at least 5 of the following 7 conditions: VAS pain ≤2 (range 0–10), SJC ≤1 (0–28), TJC ≤1 (0–28), HAQ ≤0.5 (0 –3), PtGlobal ≤2 (0 –10), PhGlobal ≤1.5 (0–10), and ESR ≤20 mm/hour, or if the patient satisfied the following conditions: had no swollen joints, no tender joints, and an ESR ≤10 mm/hour.
Statistical methods
Patients with complete data for the AE ESR criteria were included in Table 1; the cohorts were compared by randomly selecting an observation for each subject. We tested for differences between groups for individual variables by t-tests and chi square tests, and for joints (SJC and TJC) and other activity variables (PtGlobal, PhGlobal, HAQ and PAS) simultaneously with multivariate means tests (Stata MVTEST procedure).
Table 1.
Characteristics of RA study patients with ACR/EULAR 3 + ESR data at a random observation by group.
| ARCK Cohort | VARA Cohort | |
|---|---|---|
| Variable | Mean (SD) | Mean (SD) |
| Number of Patients | 1,153 | 1,341 |
| Age (years) | 59.3 (13.8) | 65.1 (11.3) |
| Sex (% male) | 25.8 | 90.9 |
| Disease duration (years) | 10.0 (9.8) | 13.2 (11.5) |
| Rheumatoid Factor (%+)* | 79.0 | 84.4 |
| HAQ-II / MDHAQ (0-3) | 1.1 (0.7) | 1.0 (0.6) |
| Pain (0-10) | 4.8 (2.8) | 4.3 (2.9) |
| Patient Global (0-10) | 4.4 (2.7) | 4.0 (2.5) |
| Physician Global (0-10) | 3.6 (2.1) | 3.3 (2.3) |
| Swollen joint count (0-28) | 3.0 (3.0) | 3.2 (4.7) |
| Tender joint count (0-28) | 3.5 (5.2) | 4.1 (6.2) |
| PAS-II / RAPID 3 | 4.2 (2.4) | 3.9 (2.1) |
| ESR (mm/Hr) | 22.4 (20.5) | 26.4 (23.0) |
| ESR (Women) (mm/Hr) | 23.0 (19.5) | 30.2 (25.1) |
| ESR (Men) (mm/Hr) | 19.9 (22.5) | 26.1 (22.8) |
| CRP (units) | 1.24 (1.97) | |
| Methotrexate current use (%) | 68.2 | 54.7 |
| Prednisone current use (%) | 26.0 | 42.1 |
| Biologics current use (%) | 37.0 | 33.7 |
Ever Rheumatoid Factor positive.
Groups differ at p <0.001 for all variables and grouped joint and activity variables except for Biologics (P = 0.083).
HAQ-II = Health Assessment Questionnaire II; MDHAQ = Multidimensional Health Assessment Questionnaire; PAS-II = Patient Activity Scale II; RAPID 3 = Routine Assessment of Patient Index Data 3; ESR = erythrocyte sedimentation rate; CRP = C-reactive protein
To determine the probability of remission we used all observations from each patient who met AE ESR entry criteria. We determined the probability of remission at a given observation with Stata's population averaged XTREG procedure together with the Margins procedure. Separate evaluations were performed stratified by cohort (ARCK and VARA) and patients' gender (Table 2). We also determined probabilities of cumulative remission (remission at any time during follow up), probabilities of a second remission, and the probabilities of remaining in remission for 3, 12 and 24 months. To determine the marginal probability of one or more remissions we used Stata's random effects XTREG procedure followed by determination of the Intraclass correlation and marginal probability procedure (16). Additional probabilities were calculated for non-ACR/EULAR remission criteria (Table 2). We determined the durability of study remissions by the Kaplan-Meier life table procedures (Figure 1) (17).
Table 2.
Probability of remission in rheumatoid arthritis.
| Probability of Remission % (95% CI) | Prob. 2ndRemission | ||||
|---|---|---|---|---|---|
| AE Criteria | All patients | Women | Men | All Patients Ever* | Any observation All patients |
| ARCK | |||||
| AE 3 ESR | 6.2 (5.2, 7.3) | 5.7 (4.6, 6.8) | 7.9 (5.5, 10.3) | 14.8 (12.8, 16.9) | 2.4 (1.4, 3.9) |
| AE 3 | 7.5 (6.4, 8.7) | 7.1 (5.8, 8.3) | 9.1 (6.5, 11.7) | 18.0 (15.8, 20.2) | 3.0 (1.9, 4.5) |
| AE 4 | 5.0 (4.1, 5.9) | 4.5 (3.5, 5.4) | 6.8 (4.6, 9.1) | 13.0 (11.1, 15.0) | 1.8 (0.9, 3.1) |
| AE CDAI | 6.9 (5.9, 8.0) | 6.1 (5.0, 7.3) | 9.8 (7.2, 12.4) | 17.8 (15.6, 20.0) | 2.2 (1.4, 3.5) |
| VARA | |||||
| AE 3 ESR | 5.0 (4.3, 5.8) | 6.4 (3.9, 9.0) | 4.9 (4.2, 5.7) | 16.5 (14.6, 18.4) | 1.5 (0.9, 2.4) |
| AE 3 CRP | 7.0 (5.9, 8.0) | 9.1 (5.3, 13.0) | 6.8 (5.7, 7.8) | 20.9 (18.5, 23.3) | 2.8 (1.8, 4.1) |
| AE 3 | 8.9 (7.9, 9.9) | 11.6 (7.9, 15.3) | 8.7 (7.7, 9.8) | 24.4 (22.2, 26.6) | 3.3 (2.4, 4.4) |
| AE 4 | 7.2 (6.2, 8.2) | 7.3 (4.0, 10.5) | 7.2 (6.2, 8.3) | 17.8 (15.8, 19.9) | 2.8 (1.8, 4.2) |
| AE CDAI | 10.1 (8.9, 11.3) | 10.3 (6.4, 14.2) | 10.0 (8.8,11.3) | 22.5 (20.3, 24.8) | 4.6 (3.3, 6.2) |
| AE SDAI | 9.0 (7.7, 10.3) | 9.1 (4.9, 13.4) | 9.0 (7.6, 10.3) | 21.9 (19.4, 24.5) | 4.2 (2.8, 5.9) |
| Non-AE Criteria | |||||
| ARCK | |||||
| DAS-28 | 28.3 (26.3, 30.4) | 24.9 (22.7, 27.2) | 39.4 (34.9, 43.8) | 48.1 (45.2, 51.0) | |
| PhGlobal 0 | 4.7 (3.9, 5.6) | 4.0 (3.1, 4.8) | 7.8 (5.5, 10.1) | 13.8 (11.8, 15.8) | |
| PhGlobal ≤1 | 19.1 (15.1, 17.7) | 17.3 (15.5, 19.1) | 25.1 (21.3, 29.0) | 40.1 (37.4, 42.8) | |
| PAS ≤1 | 9.2 (7.7, 10.6) | 8.5 (6.8, 10.1) | 11.5 (8.3, 14.6) | 17.0 (14.9, 19.2) | |
| MDA | 22.9 (20.9, 24.8) | 20.7 (18.5, 22.8) | 29.9 (25.6, 34.1) | 41.9 (39.0, 44.7) | |
| VARA | |||||
| DAS-28 | 24.0 (22.4, 25.6) | 19.4 (14.8, 24.1) | 24.4 (22.7, 26.1) | 48.4 (45.8, 51.0) | |
| PhGlobal 0 | 2.1 (1.6, 2.6) | 1.9 (0.5, 3.3) | 2.2 (1.6, 2.7) | 6.7 (5.4, 8.1) | |
| PhGlobal ≤1 | 20.8 (19.2, 22.4) | 24.8 (18.8, 30.7) | 20.5 (18.8, 22.1) | 43.4 (40.7, 46.1) | |
| PAS ≤1 | 9.1 (8.0, 10.2) | 11.7 (7.7, 15.7) | 8.9 (7.8, 10.0) | 21.8 (19.7, 23.9) | |
| MDA | 21.3 (19.6, 23.0) | 23.2 (17.0, 29.3) | 21.1 (19.3 22.9) | 39.6 (37.0, 42.2) | |
See methods for criteria definition. Remission estimates are adjusted for age, sex and duration of RA.
Cumulative remission: the probability of ever having a remission during a mean of 2.2 years (ARCK) and 2.1 years (VARA) of follow-up.
Figure 1.
Representative examples of lack of durability of remission. Y axis represents the probability of remaining in remission. See methods for definition of AE 3 and SDAI remission.
In order to explore whether the same patients are classified as in remission by different criteria, we assessed agreement between remission measures by the Kappa statistic and Jaccard's coefficient (18). We used the interpretation of Landis and Koch for kappa values: < 0 as indicating no agreement and 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.0 as almost perfect agreement (19). Both Kappa and the Jaccard coefficients can be interpreted as % values. Kappa can be interpreted as the % agreement after correcting for chance. The Jaccard coefficient can be interpreted as the % agreement after excluding joint negative pairs. Its utility lies in its ease of demonstrating the extent of clinically understandable agreement after exclusion of agreed upon criteria negative pairs.
To examine physician heterogeneity we determined the median odds ratio (MOR) after performing multilevel analyses using the Stata XTMELOGIT procedure and modeling physicians and patients in separate random effects equations (Table 4). We used the MOR to express examiner variance in remission criteria (20, 21). The MOR quantifies differences (i.e., variance between examiners) by comparing patients with the same covariates but from 2 randomly chosen examiners. This procedure yields a distribution of ORs, with 1 OR for each comparison pair. The MOR is the median of this distribution of pairwise ORs. That is, the MOR expresses how much (in median) the individual probability of obtaining a remission would increase if a patient were evaluated by another examiner with a higher proportion of remissions, assuming the patients had the same covariates. If the MOR is 1, then there are no differences in remission prevalence between examiners. If there are considerable examiner differences, then the MOR is large. The measure is directly comparable to fixed-effects ORs, which makes quantification of examiner variance easier to appreciate in terms of the familiar ORs (22). Roughly, given a remission proportion of 8%, MORs of 1.5 and 2.0 translate to probabilities of remission of about 12% and 16%. MOR analyses have not been used previously to assess groups of physicians.
Table 4.
Agreement and disagreement in remission proportions at a random observation in 1153 (ARCK) and 1358 (VARA) RA patients.
| Comparison | AE 3 ESR | AE CRP | AE 3 | AE 4 | AE CDAI | AE 3 ESR | AE 3 CRP | AE 3 | AE 4 | AE CDAI | Median Odds Ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Kappa | Jaccard | ||||||||||
| ARCK | |||||||||||
| Criteria | |||||||||||
| AE 3 + ESR | 1.00 | 1.00 | 1.0 | ||||||||
| AE 3 | 0.88 | 1.00 | 0.80 | 1.00 | 1.1 | ||||||
| AE 4 | 0.72 | 0.76 | 1.00 | 0.58 | 0.64 | 1.00 | 2.0 | ||||
| AE CDAI | 0.55 | 0.55 | 0.66 | 0.40 | 0.45 | 0.52 | 1.1 | ||||
| Criterion* | |||||||||||
| SJC | 2.0 | ||||||||||
| TJC | 1.7 | ||||||||||
| PtGlobal | 1.0 | ||||||||||
| PhGlobal | 2.7 | ||||||||||
| ESR | 1.4 | ||||||||||
| VARA | |||||||||||
| Criteria | |||||||||||
| AE 3 ESR | 1.00 | 1.00 | 1.8 | ||||||||
| AE 3 CRP | 0.64 | 1.00 | 0.50 | 1.00 | 2.1 | ||||||
| AE 3 | 0.70 | 0.84 | 1.00 | 0.56 | 0.75 | 1.00 | 2.2 | ||||
| AE 4 | 0.67 | 0.74 | 0.86 | 1.00 | 0.53 | 0.61 | 0.78 | 1.00 | 2.2 | ||
| AE CDAI | 0.57 | 0.71 | 0.79 | 0.76 | 1.00 | 0.43 | 0.59 | 0.68 | 0.64 | 1.00 | 2.0 |
| AE SDAI | 0.58 | 0.77 | 0.75 | 0.71 | 0.87 | 0.44 | 0.66 | 0.63 | 0.59 | 0.80 | 1.4 |
| Criterion* | |||||||||||
| SJC | 2.7 | ||||||||||
| TJC | 2.0 | ||||||||||
| PtGlobal | 1.5 | ||||||||||
| PhGlobal | 2.4 | ||||||||||
| ESR | 1.9 | ||||||||||
Criterion: Individual components of criteria, SJC ≤1, TJC ≤1, Pt Global ≤1, PhGlobal ≤1, ESR <20 (male) <30 (female).
See methods for criteria definition.
The Jaccard coefficient is defined as and represents the proportion of agreement in cases excluding those instances of agreement in absence.
CDAI: Clinical Disease Activity Index; DAS-28: Disease Activity Scale 28; PAS: Patient Activity Scale
We calculated Harrell's c to determine the predictor strength and discriminatory ability of individual and combined predictors for each set of criteria (Appendix I). Harrell's c has the interpretation of a receiver operating characteristic (ROC) area under the curve (AUC).
Statistical significance was set at p <0.05. All analyses were performed with Stata 11.1 (College Station, TX).
Results
Clinical characteristics
There were important and statistically significant differences between the ARCK and VARA groups (Table 1). The VARA group consisted of a much higher proportion of men (90.9% vs. 25.8%) was older (65.3 vs. 59.3 years), had longer duration of RA (13.2 vs. 10.0 years), higher ESR values, were more often rheumatoid factor positive, and were more often treated with prednisone (42.1 vs. 26.0%). In addition, they had higher and swollen and tender joint counts. By contrast, clinical activity measures including HAQ-II/MDHAQ, pain, PtGlobal, PhGlobal, and PAS were higher in the ARCK group. Combined tender and swollen joints and combined activity measures were significantly different between groups.
Remission probabilities by ACR/EULAR and other criteria
The probability of remission at a given clinic visit using AE 3 was 7.5% (6.4, 8.7) in the ARCK group and 8.9% (7.9, 9.9) in VARA. With other AE remission criteria definitions, probabilities varied in both groups, from 5.0% to 6.9% in ARCK and 5.0% to 10.1% in VARA (Table 2). In the VARA data set we were also able to examine probabilities for AE CRP 7.0% (5.9, 8.0) and AE SDAI 9.0% (7.7, 10.3), as these measures were recommended by ACR/EULAR for use in clinical trials. In addition, the probability of remission generally increased over the course of the study. Adjusted for age, sex, and use of prednisone, methotrexate and biologics, the annual increase in the probability of remission was: AE 3 0.9% (0.2, 1.6), AE ESR 0.9% (0.02, 1.5), AE CDAI 1.3% (0.6, 1.9) for ARCK, and AE 3 0.7% (0.2, 1.1), AE ESR 0.1% (-0.1, 1.3), AE CRP 0.7% (0.2, 1.1), AE CDAI 1.1% (0.5, 1.7), AE SDAI 0.6% (-.0.06, 1.2) for VARA. Methotrexate and biologics were not significantly associated with this increase in any model examined.
Regardless of which remission criteria set was selected, the cumulative or “ever” probability for all criteria were considerably higher, ranging from 13.6% to 17.8% in ARCK and 16.5% to 24.4% in VARA. The ever probabilities were determined over a mean follow-up of 2.2 years for ARCK, with a mean (IQR) duration of between visits of 2.7 (1.1 to 3.1) months, and 2.1 years follow-up for VARA with duration between visits of 3.8 (2.1 to 4.7) months. During this period of time the mean number of clinic visits was 10.6 for ARCK and 7.2 for VARA.
While the probability of ever meeting a remission criteria was greater than the probability of meeting criteria at a given clinic visit, the probability of any patient having 2 or more visits in remission (not necessarily contiguous) was considerably smaller. The probability of such events ranged from 1.8% to 3.0% in ARCK and 1.5% to 4.6% in VARA. To the extent that meeting criteria at least twice defines a meaningful remission, these probabilities might be used to further clarify the probability and meaning of remission in RA.
For those patients achieving remission, Table 3 provides data on the probability of remaining in remission. At 12 months after the start of remission, 19.7% to 33.8% remained in remission. At 2 years the probability of remaining in remission ranged from 6.0% to 14.1%. By contrast, the probability of remaining in MDA was 42.9% at 12 months and 24.8% at 24 months in ARCK and 43.9% at 12 months and 22.0% at 24 months in VARA. To put the remission data into perspective, less than 3% of all RA patients can be expected to experience a remission lasting 2 years or more. Figure 1 demonstrates representative Kaplan-Meier survival curves for remaining in remission.
Table 3. Probability of remaining in remission at 3, 12 and 24 months.
| Probability of Remaining in Remission (95% CI) | |||
|---|---|---|---|
| ARCK | At 3 months | At 12 months | At 24 months |
| AE 3 ESR | 82.2 (74.7, 87.7) | 33.8 (25.8, 42.0) | 10.8 (5.8, 17.5) |
| AE 3 | 82.0 (75.3, 87.1) | 33.2 (26.0, 46.6) | 14.1 (9.0, 20.5) |
| AE 4 | 79.2 (72.6, 84.4) | 22.0 (16.2, 28.5) | 7.0 (3.8, 11.7) |
| AE CDAI | 71.3 (65.5, 76.4) | 21.8 (16.8, 27.1) | 6.0 (3.3, 10.0) |
| MDA | 83.8 (79.7, 87.2) | 42.9 (37.7, 48.0) | 24.8 (20.1, 29.8) |
| VARA | At 3 months | At 12 months | At 24 months |
| AE 3 ESR | 85.0 (79.3, 89.3) | 19.7 (14.4, 25.7) | 6.6 (3.5, 11.1) |
| AE 3 CRP | 86.3 (80.9, 90.3) | 24.7 (18.9, 30.9) | 8.1 (4.6,12.9) |
| AE 3 | 85.5 (81.0, 89.0) | 24.2 (19.3, 29.4) | 8.3 (5.3, 12.3) |
| AE 4 | 90.7 (85.6, 94.0) | 23.8 (17.8, 30.4) | 9.6 (5.6, 14.9) |
| AE CDAI | 89.0 (84.2, 92.4) | 27.1 (21.3, 33.3) | 13.5 (9.0, 18.8) |
| AE SDAI | 89.6 (84.2, 93.3) | 31.3 (24.3, 38.4) | 13.3 (8.4, 19.4) |
| MDA | 89.5 (86.1, 92.1) | 43.9 (38.7, 49.0) | 22.0 (17.5, 26.8) |
See methods for criteria definition.
We also calculated non-ACR/EULAR probabilities that may be of interest (Table 2). The majority of these definitions resulting in remissions substantially higher than those achieved under the ACR/EULAR criteria. In particular DAS-28 remission was observed in 28.3% in ARCK and 24.0% in VARA. PAS remission, depending only on patients self-report, was 9.2% in ARCK and 9.1% in VARA. Finally, the minimal disease activity criterion was satisfied by 22.9% in ARCK and 21.3% in VARA.
Agreement among criteria
Similar cross-sectional probabilities do not necessarily mean that the same patients are identified by the different criteria. To investigate agreement, we selected a random observation for each patient and then applied Kappa and Jaccard statistics. AE CRP and AE SDAI had a Jaccard statistic of 0.66 (Table 4). The best Jaccard agreement for AE SDAI was with AE CDAI (0.80), as might be suspected because of the similarity of these criteria. The best Jaccard agreement with AE CRP was with AE 3 (0.75). In ARCK data, which lacked CRP, AE 3 and AE ESR had a Jaccard statistic of 0.80, a statistic of 0.64 for AE 4, and a value of 0.45 for AE CDAI.
The same pattern was noted for Kappa statistics, with generally moderate or substantial agreement beyond chance. The Kappa for AE SDAI and AE CRP was 0.77. The Kappa for AE 3 and SDAI was 0.75 and for AE 3 and AE CRP was 0.84. Because of the interest in pure patient based criteria, we evaluated statistics for the PAS-II/RAPID-3 and DAS-28. The Kappa and Jaccard coefficients between RAPID-3 remission and AE SDAI were 0.46, 0.35; RAPID-3 remission and AE CRP were 0.40, 0.29, DAS-28 remission and AE SDAI were 0.40, 0.32, and DAS-28 remission and AE CRP were 0.33, 0.25 in VARA.
We also examined the relation between MDA and the various remission criteria by determining the percent remission positive, given MDA is positive and the percent remission positive, given MDA is negative. For ARCK: AE 3 (32.2%/1.1%), AE ESR (26.7%/0.6%), CDAI (33.2%/3.0%); and for VARA: AE 3 (36.6%/3.9%), AE ESR (22.9%/2.0%), AE CRP (27.4%/3.2%), AE CDAI (42.3%/1.7%), and AE SDAI (37.0%/1.4%).
Importance of individual predictors
We calculated Harrell's c to determine the predictor strength and discriminatory ability of individual and combined predictors for each set of criteria (Appendix I). PtGlobal was the variable with the best discriminatory ability, with a value as high as 0.97. In criteria that contained PhGlobal, PhGlobal ranged from 0.90 to 0.93. Together these variables dominated the predictors in discriminatory ability. By contrast, tender and swollen joints had values between 0.74 and 0.77. Even when considered simultaneously, the tender and swollen joints scores produced c scores that were less than the globals.
PtGlobal and remission criteria positive and negative states
As PtGlobal was the strongest contributor to criteria positivity, we examined PtGlobal graphically in patients who met joint and ESR criteria for AE 3 ESR remission in Figure 2a. Among patients otherwise remission criteria positive there was a wide distribution of PtGlobal scores, including many within 1 point of the PtGlobal criterion for remission. Figure 2b demonstrates that among patients AE positive at the previous clinic visit, there were many patients no longer AE positive on the basis of slight changes in PtGlobal (>1.0).
Figure 2.
Figure 2a (above). Distribution of patient global scores for those who meet tender and swollen joint criteria and ESR criteria for remission at a random clinic visit. Patients meeting remission criteria have global scores ≤1 (horizontal line). Small changes in patient global scores would result in many patients meeting remission criteria. ARCK patients are on left, VARA on right. Horizontal line at patient global = 2 is added to enhance viewing.
Figure 2b (below). Patients who initially met ACR/EULAR criteria assessed at a follow-up clinic visit. All patents meet tender and swollen joint criteria for remission, but those with global score above 1 or ESR ≥20 (men) or ≥30 (women) no longer satisfy ACR/EULAR criteria. For most patients not meeting ACR/EULAR criteria the difference between remission criteria positive and criteria negative patients is small. ARCK patients are on left, VARA on right. Horizontal line at patient global = 2 is added to enhance viewing.
The effect of physician differences on criteria positivity
We addressed the issue of whether physicians differed in their examinations and ratings by using multilevel analyses and calculating the median odds ratio (MOR) in Table 4. In this analysis patients are nested within physicians. The MOR represents the degree of variation between examiners. The highest MOR for criteria components was found for SJC (MOR 2.0 – 2.7) and PhGlobal (2.4 - 2.7), indicating considerable physician heterogeneity. Slightly less heterogeneity was seen for TJC (MOR 1.7 – 2.0). When applied to specific criteria, observed bias was noted generally in the VARA data set, but only for AE 4 and AE CDAI in the ARCK data set. These data indicate physician differences influence remission diagnosis.
Discussion
The ACR/EULAR recently established criteria for remission in RA that were “stringent but achievable and could be applied uniformly in clinical trials (6).” Among the 2 definitions put forth were 1) ≤1 for TJC, SJC, CRP (mg/dl), and PtGlobal; and 2) SDAI ≤3.3. The group also suggested possibilities for criteria that might be used in clinical practice. The basis of this suggestion required that a definition of “remission be developed for clinic based practice that would not require an acute phase reactant, as long as it would capture remission as stringently as the measure employed for clinical trials.” Thus “… a Boolean measure comprising tender joint count, swollen joint count and patient global assessment [could provide] similar statistical results as the same measures encompassing CRP and the CDAI, [but which] does not contain CRP…” The committee indicated that such definitions of remission “may be used in clinical practice until better measures for that purpose become available.” In addition, the committee suggested clinical practice cut points for ESR (<20 for men and <30 for women) in the event laboratory tests were used in the clinical practice setting.
The central difference between remission in clinical trials (and observational research) and remission in clinical practice is that trial data refer to a group of patients while clinical practice remission refers to an individual patient and an individual examiner. In a trial where different definitions provide similar proportions in remission, it does not matter substantially which valid definition is used. In clinical practice, however, if different patients are identified by different criteria, it may matter a great deal.
It is not surprising that remission probability differs by remission definition. However, differences seem generally small and in accord with probabilities from clinical trials noted in the ACR/EULAR remission paper (6). Where our results differ, perhaps conceptually, from the ACR/EULAR results is in our observation of the tenuousness and sporadic nature of remission. ACR/EULAR regarded the duration of remission as worthy of a separate study and noted that they did not address it in the primary remission paper. We observed that within 12 months, 65-80% of those who had experienced remission no longer met remission criteria; at 24 months 6%-14% still met criteria (Table 3 and Figure 1). If, as Table 2 indicates, the probability of ever being in remission is 13.0%-24.4%, then the probability of being in remission for as long as 2 years is between 1.0 and 3.0%. These results are remarkably similar to those of Wolfe and Hawley in 1985 (23) who noted an 18.1% remission proportion by application of the 1981 ACR remission criteria to 458 patients in a clinical practice (3). In addition, they found that “only 15% of remissions lasted longer than 24 months.” Thus, only 3% of patients had a remission that lasted as long as 2 years.
Another indication of the potential tenuous nature of remission comes from our observation that 1.5% to 4.6% of RA patients had 2 or more physician visits in remission (Table 2, column 6) compared to 13.0%-24.4% of patients who ever experience a remission visit (Table 2, column 5).
We addressed several issues with respect to misclassification. First, we examined the degree of agreement among the different criteria. In the current study, Jaccard's coefficient between AE SDAI and AE CRP, the two recommended clinical trial criteria, showed 66% agreement between criteria in the VARA data set. When the ACR/EULAR recommended clinical criteria (AE 3) was compared with AE SDAI and AE CRP, Jaccard coefficients of 0.63 and 0.75 were noted. These levels of agreement, as well as the Kappa values in Table 4 are sufficient for clinical trials. However, at the level of the individual patient clinically significant misclassification can occur, underscoring the difference between group criteria and individual criteria with respect to levels of reliability.
Misclassification will also occur if physician examiners differ in their ratings. “Reliability concerns the degree to which patients can be distinguished from each other, despite measurement error. High reliability is important for discriminative purposes if one wants to distinguish among patients, e.g., with more or less severe disease (as in diagnostic applications) (24).” In general, reliability coefficients ≥ 0.9 are required to make decisions about individual patients. Values from 0.80 to 0.89 represent good reliability, suitable for research and use in groups of patients. However, there is substantial evidence that inter-rater reliability is poor with respect to the joint examination (25-28). Using the MOR in multilevel analyses, we also found evidence of important physician heterogeneity in the joint examination and in the PhGlobal rating. MOR scores of 2 (Table 4), for example, indicate that the probability of remission can vary twofold according to physician examiner irrespective of the degree of disease activity. Such rater variability is not likely to be a problem in clinical trials unless there is a systematic bias. At the clinical level, however, physician differences can lead to misclassification.
The sole patient measure used in the ACR/EULAR criteria sets is the PtGlobal. As shown in Appendix I, PtGlobal has the highest c statistic and is the best discriminatory variable among the components of the various remission definitions. Lassere et al. has shown that PtGlobal has poor test-retest reliability (Intraclass coefficient = 0.75) at the level of the individual patient (26). This finding is consistent with the data of the current study. Figure 2a suggests that when remission is first identified there are many patients with a PtGlobal score close to the remission level that do not satisfy the remission definition. And when we examined our results in the next visit for patients who had been in remission (Figure 2b), it can be seen that many previously in remission patients were no longer in remission because of changes in PtGlobal. Thus, remission in this setting depends on PtGlobal, which may be reflect true sensitivity to changes in RA activity or represent reliability issues where remission status changes while RA activity actually remains the same.
One approach that avoids physician bias is the use of the RAPID-3 (or PAS) (4). However, the components of these scales – pain, global and HAQ – also have poor reliability (26). In addition, we found unsatisfactory agreements with ACR/EULAR recommended measures, including Jaccard statistics of 0.35 (SDAI remission) and 0.29 (AE CRP).
We note a number of potential limitations to our study. Among the possible methodologic concerns is that we chose to analyze individual physicians in the MOR VARA analyses. We did this to be consistent with analyses of ARCK patients. Another approach would have been to analyze VARA sites rather than physicians. Although we did not report these analyses in this paper, we found no substantial difference when we substituted sites for physicians in sensitivity analyses.
We did not attempt to discern reasons for the differences in results between the VARA and the ARCK data, as that was not the purpose of our study. Patients mix and socio-demographic characteristics might explain some of the differences. In multivariate analyses of remission criteria we noted that men were more likely to achieve remission than women for CDAI in ARCK, but not in any other ARCK or VARA criteria described in Table 2.
Although we attributed high MOR scores (MOR >1) to physician differences, it is possible that some physicians were assigned patients with greater disease activity. That does not appear to have been a matter of policy in ARCK or VARA, and we found no evidence to support that possibility.
In summary, the proportion of patients in remission at a given visit ranged from 5.0% to 10.1%, and was 7.5% to 8.9% by the AE 3 criteria recommended by ACR/EULAR. During the ∼2.2 years of follow-up 18.4% to 24.4% entered AE 3 remission. Prolonged remissions were rare, with <3% of patients experiencing a remission lasting as long as 2 years.
Acknowledgments
The authors thank Grant Cannon, M.D. for his helpful and thoughtful comments.
Support: VARA has been supported by the VA HSR&D.
Dr. Mikuls is supported by a VA Merit grant.
Dr. Caplan is supported by a VA Career Development Award CDA 07-221.
Kaleb Michaud received partial funding from the Arthritis Foundation's New Investigator Award and NIH ARRA grant #1RC1AR058601-01.
Appendix
Appendix I. The association and discriminatory ability of criteria variables for remission criteria.
| ARCK | VARA | |
|---|---|---|
| AE 3 ESR | AUC (Harrell's c) | AUC (Harrell's c) |
| SJC ≤1 | 0.77 (0.76, 0.79) | 0.75 (0.74, 0.76) |
| TJC ≤1 | 0.75 (0.74, 0.77) | 0.76 (0.74, 0.77) |
| PtGlobal ≤1 | 0.96 (0.96, 0.97) | 0.94 (0.93, 0.95) |
| ESR <20,<30 | 0.67 (0.66, 0.69) | 0.77 (0.75, 0.78) |
| SJC ≤1 &TJC ≤1 | 0.85 (0.84, 0.87) | 0.83 (0.81, 0.84) |
| SJC ≤1 &TJC ≤1 + ESR | 0.90 (0.89, 0.91) | 0.92 (0.91, 0.93) |
| AE 3 | ||
| SJC ≤1 | 0.77 (0.76, 0.79) | 0.76 (0.75, 0.78) |
| TJC ≤1 | 0.76 (0.74, 0.77) | 0.77 (0.76, 0.78) |
| PtGlobal ≤1 | 0.97 (0.97, 0.98) | 0.96 (0.95, 0.97) |
| SJC ≤1 &TJC ≤1 | 0.86 (0.84, 0.87) | 0.84 (0.83, 0.85) |
| SJC ≤1 &TJC ≤1 + ESR | 0.80 (0.75, 0.84) | 0.70 (0.66, 0.74) |
| AE 4 | ||
| SJC ≤1 | 0.77 (0.75, 0.78) | 0.76 (0.74, 0.77) |
| TJC ≤1 | 0.75 (0.74, 0.76) | 0.76 (0.75, 0.78) |
| PtGlobal ≤1 | 0.96 (0.95, 0.97) | 0.95 (0.94, 0.96) |
| PhGlobal ≤1 | 0.93 (0.92, 0.94) | 0.93 (0.92, 0.94) |
| SJC ≤1 &TJC ≤1 | 0.85 (0.83, 0.86) | 0.83 (0.82, 0.85) |
| SJC ≤1 &TJC + ESR | 0.81 (0.76, 0.86) | 0.71 (0.67, 0.76) |
| AE CDAI | ||
| SJC ≤1 | 0.77 (0.76, 0.79) | 0.76 (0.74, 0.78) |
| TJC ≤1 | 0.76 (0.74, 0.77) | 0.77 (0.75, 0.78) |
| PtGlobal ≤1 | 0.78 (0.73, 0.84) | 0.85 (0.82, 0.89) |
| PhGlobal ≤1 | 0.90 (0.87, 0.93) | 0.88 (0.85, 0.90) |
| SJC ≤1 &TJC ≤1 | 0.86 (0.84, 0.87) | 0.83 (0.82, 0.85) |
| SJC ≤1 &TJC ≤1 + ESR | 0.79 (0.75, 0.84) | 0.72 (0.68, 0.76) |
| AE SDAI | ||
| SJC ≤1 | 0.74 (0.71, 0.76) | |
| TJC ≤1 | 0.75 (0.73, 0.78) | |
| PtGlobal ≤1 | 0.88 (0.84, 0.91) | |
| PhGlobal ≤1 | 0.86 (0.82, 0.89) | |
| SJC ≤1 &TJC ≤1 | 0.81 (0.78, 0.83) | |
| CRP ≤1 | 0.61 (0.58, 0.64) | |
| AE CRP | ||
| SJC ≤1 | 0.75 (0.73, 0.77) | |
| TJC ≤1 | 0.76 (0.74, 0.77) | |
| PtGlobal ≤1 | 0.95 (0.94, 0.96) | |
| SJC ≤1 &TJC ≤1 | 0.83 (0.81, 0.84) | |
| SJC ≤1 &TJC ≤1 + ESR | 0.71 (0.65, 0.76) | |
| CRP ≤1 | 0.65 (0.64, 0.67) | |
See methods for criteria definition.
Footnotes
Potential conflicts of interest: None.
Contribution of authors: The manuscript was drafted by F Wolfe. The statistical analyses were performed by F Wolfe and K Michaud. All authors reviewed and aided in the preparation of the manuscript, and approved submission of the manuscript. Note: Dr. Shahouri and Dr. Michaud contributed equally to this study.
Contributor Information
Shadi H. Shahouri, Arthritis and Rheumatology Clinics of Kansas, Wichita, KS, University of Kansas School of Medicine, Wichita, KS.
Kaleb Michaud, University of Nebraska Medical Center, Omaha, Nebraska, National Data Bank for Rheumatic Diseases, Wichita, Kansas
Ted R. Mikuls, Omaha VA Medical Center and University of Nebraska, Omaha, Nebraska
Liron Caplan, Denver VA Medical Center and University of Colorado, Denver, Colorado
Timothy S. Shaver, Arthritis and Rheumatology Clinics of Kansas, Wichita, KS, University of Kansas School of Medicine, Wichita, KS.
James D. Anderson, Arthritis and Rheumatology Clinics of Kansas, Wichita, KS, University of Kansas School of Medicine, Wichita, KS.
David N. Weidensaul, Arthritis and Rheumatology Clinics of Kansas, Wichita, KS, University of Kansas School of Medicine, Wichita, KS.
Ruth E. Busch, Arthritis and Rheumatology Clinics of Kansas, Wichita, KS, Wichita State University, Wichita, KS.
Shirley Wang, Arthritis and Rheumatology Clinics of Kansas, Wichita, KS, University of Kansas School of Medicine, Wichita, KS.
Frederick Wolfe, National Data Bank for Rheumatic Diseases, Wichita, Kansas, University of Kansas School of Medicine, Wichita, Kansas
References
- 1.Pemberton R, Pierce EC. Clinical and statistical study of chronic arthritis based on 1100 cases. AmJMedSci. 1927;173:31–6. [Google Scholar]
- 2.Thompson HE, wyatt BL, Hicks RA. Chronic atrophic arthritis. AnnInternMed. 1938;11:1792–805. [Google Scholar]
- 3.Pinals RS, Baum J, Bland J, Fosdick WM, Kaplan SB, Masi AT, et al. Preliminary criteria for clinical remission in rheumatoid arthritis. Arthritis Rheum. 1981;24:1308–15. doi: 10.1002/art.1780241012. [DOI] [PubMed] [Google Scholar]
- 4.Sokka T, Hetland ML, Makinen H, Kautiainen H, Horslev-Petersen K, Luukkainen RK, et al. Remission and rheumatoid arthritis: Data on patients receiving usual care in twenty-four countries. Arthritis Rheum. 2008 Sep;58(9):2642–51. doi: 10.1002/art.23794. [DOI] [PubMed] [Google Scholar]
- 5.van Tuyl LH, Vlad SC, Felson DT, Wells G, Boers M. Defining remission in rheumatoid arthritis: results of an initial American College of Rheumatology/European League Against Rheumatism consensus conference. Arthritis Rheum. 2009 May 15;61(5):704–10. doi: 10.1002/art.24392. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Felson D, Smolen J, Wells G, Zhang B, van Tuyl LH, Funovits J, et al. American College of Rheumatology/European League against Rheumatism Preliminary Definition of Remission in Rheumatoid Arthritis for Clinical Trials. Arthritis & Rheumatism. 2011 doi: 10.1002/art.30129. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Aletaha D, Smolen JS. The Simplified Disease Activity Index (SDAI) and Clinical Disease Activity Index (CDAI) to monitor patients in standard clinical care. BestPractResClinRheumatol. 2007;21(4):663–75. doi: 10.1016/j.berh.2007.02.004. [DOI] [PubMed] [Google Scholar]
- 8.van der Heijde DMFM, van 't Hof M, van Riel PLCM, Theunisse LAM, Lubberts EW, van Leeuwen MA, et al. Judging disease activity in clinical practice in rheumatoid arthritis: first step in the development of a disease activity score. Ann Rheum Dis. 1990;49:916–20. doi: 10.1136/ard.49.11.916. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Shaver TS, Anderson JD, Weidensaul DN, Shahouri SS, Busch RE, Mikuls TR, et al. The problem of rheumatoid arthritis disease activity and remission in clinical practice. J Rheumatol. 2008;35(6):1015–22. [PubMed] [Google Scholar]
- 10.Wolfe F, Michaud K, Pincus T. Development and validation of the health assessment questionnaire II: a revised version of the health assessment questionnaire. Arthritis Rheum. 2004;50(10):3296–305. doi: 10.1002/art.20549. [DOI] [PubMed] [Google Scholar]
- 11.Mikuls TR, Kazi S, Cipher D, Hooker R, Kerr GS, Richards JS, et al. The association of race and ethnicity with disease expression in male US veterans with rheumatoid arthritis. J Rheumatol. 2007;34(7):1480–4. [PubMed] [Google Scholar]
- 12.Michaud K, Mikuls T, Call S, Reimold A, Hooker R, Kerr G, et al. Poor to modest agreement between rheumatoid arthritis response measures in clinical practice. Clinical and experimental rheumatology. 27(4):633. [PubMed] [Google Scholar]
- 13.Pincus T, Swearingen C, Wolfe F. Toward a multidimensional Health Assessment Questionnaire (MDHAQ): assessment of advanced activities of daily living and psychological status in the patient-friendly health assessment questionnaire format. Arthritis Rheum. 1999 Oct;42(10):2220–30. doi: 10.1002/1529-0131(199910)42:10<2220::AID-ANR26>3.0.CO;2-5. [DOI] [PubMed] [Google Scholar]
- 14.Wolfe F, Michaud K, Pincus T. A Composite Disease Activity Scale for Clinical Practice, Observational Studies and Clinical Trials: The Patient Activity Scale (PAS/PAS-II) JRheumatol. 2005;32(12):2410–5. [PubMed] [Google Scholar]
- 15.Pincus T, Bergman MJ, Yazici Y, Hines P, Raghupathi K, Maclean R. An index of only patient-reported outcome measures, routine assessment of patient index data 3 (RAPID3), in two abatacept clinical trials: similar results to disease activity score (DAS28) and other RAPID indices that include physician-reported measures. Rheumatology (Oxford) 2008;47(3):345–9. doi: 10.1093/rheumatology/kem364. [DOI] [PubMed] [Google Scholar]
- 16.Rodriguez G, Elo I. Intra-class correlation in random-effects models for binary data. Stata Journal. 2003;3(1):32–46. [Google Scholar]
- 17.Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. Journal of the American Statistical Association. 1958;53:457–81. [Google Scholar]
- 18.Jaccard P. …comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin del la SociÈtÈ Vaudoise des Sciences Naturelles. 1901;37:547–79. [Google Scholar]
- 19.Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159. [PubMed] [Google Scholar]
- 20.Larsen K, Merlo J. Appropriate assessment of neighborhood effects on individual health: integrating random and fixed effects in multilevel logistic regression. Am J Epidemiol. 2005 Jan 1;161(1):81–8. doi: 10.1093/aje/kwi017. [DOI] [PubMed] [Google Scholar]
- 21.Merlo J, Chaix B, Ohlsson H, Beckman A, Johnell K, Hjerpe P, et al. A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena. J Epidemiol Community Health. 2006 Apr;60(4):290–7. doi: 10.1136/jech.2004.029454. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Due P, Hansen E, Merlo J, Andersen A, Holstein B. Is victimization from bullying associated with medicine use among adolescents? A nationally representative cross-sectional survey in Denmark. Pediatrics. 2007;120(1):110. doi: 10.1542/peds.2006-1481. [DOI] [PubMed] [Google Scholar]
- 23.Wolfe F, Hawley DJ. Remission in rheumatoid arthritis. JRheumatol. 1985;12:245–52. [PubMed] [Google Scholar]
- 24.Terwee C, Bot S, de Boer M, van der Windt D, Knol D, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology. 2007;60(1):34–42. doi: 10.1016/j.jclinepi.2006.03.012. [DOI] [PubMed] [Google Scholar]
- 25.Pincus T. Limitations of a quantitative swollen and tender joint count to assess and monitor patients with rheumatoid arthritis. Bulletin of the NYU hospital for joint diseases. 2008;66(3):216. [PubMed] [Google Scholar]
- 26.Lassere MN, van der HD, Johnson KR, Boers M, Edmonds J. Reliability of measures of disease activity and disease damage in rheumatoid arthritis: implications for smallest detectable difference, minimal clinically important difference, and analysis of treatment effects in randomized controlled trials. JRheumatol. 2001;28(4):892–903. [PubMed] [Google Scholar]
- 27.Klinkhoff AV, Bellamy N, Bombardier C, Carette S, Chalmers A, Esdaile JM, et al. An experiment in reducing interobserver variability of the examination for joint tenderness. JRheumatol. 1988;15:492–4. [PubMed] [Google Scholar]
- 28.Scott DL, Houssien DA. Joint assessment in rheumatoid arthritis. BritJ Rheumatol. 1996;35:14–8. doi: 10.1093/rheumatology/35.suppl_2.14. -32676. [DOI] [PubMed] [Google Scholar]