Content and Construct Validity, Reliability, and Responsiveness of the Rheumatoid Arthritis Flare Questionnaire: OMERACT 2016 Workshop Report

Abstract

Objective

The Outcome Measures in Rheumatology (OMERACT) Rheumatoid Arthritis (RA) Flare Group was established to develop a reliable way to identify and measure RA flares in randomized controlled trials (RCT). Here, we summarized the development and field testing of the RA Flare Questionnaire (RA-FQ), and the voting results at OMERACT 2016.

Methods

Classic and modern psychometric methods were used to assess reliability, validity, sensitivity, factor structure, scoring, and thresholds. Interviews with patients and clinicians also assessed content validity, utility, and meaningfulness of RA-FQ scores.

Results

People with RA in observational trials in Canada (n = 896) and France (n = 138), and an RCT in the Netherlands (n = 178) completed 5 items (11-point numerical rating scale) representing RA Flare core domains. There was moderate to high evidence of reliability, content and construct validity, and responsiveness. Factor analysis supported unidimensionality. Rasch analysis showed acceptable fit to the Rasch model, with items and people covering a broad measurement continuum and evidence of appropriate targeting of items to people, ordered thresholds, minimal differential item functioning by language, sex, or age. A summative score across items is defensible, yielding an interval score (0–50) where higher scores reflect worsening flare. The RA-FQ received endorsement from 88% of attendees that it passed the OMERACT Filter 2.0 “Eyeball Test” for instrument selection.

Conclusion

The RA-FQ has been developed to identify and measure RA flares. Its review through OMERACT Filter 2.0 shows evidence of reliability, content and construct validity, and responsiveness. These properties merit its further validation as an outcome for clinical trials.

Rheumatoid arthritis (RA) is a chronic systemic inflammatory condition characterized by pain, fatigue, stiffness, and disability¹. Episodes of clinically important worsening (disease flares) are common, with up to 57% of patients reporting a flare at or between visits^2,3,4. Growing evidence indicates that flares contribute substantially to patient burden, poorer health-related quality of life, disability, radiographic damage, and healthcare use and costs^5,6,7,8,9.

While newer therapeutics have revolutionized RA management, there is growing interest in understanding optimal approaches to taper or withdraw treatment once sustained remission is achieved. Although flares are an important endpoint in these trials, they have proven challenging to reliably identify and measure.

To date, investigators have used different flare definitions^8,10,11,12, including patient or physician assessments, worsening of American College of Rheumatology core set components, or Disease Activity Score¹³, with little attempt to measure flare severity. Lack of consensus on flare definition has made it challenging to compare studies or pool results.

The Outcome Measures in Rheumatology (OMERACT) RA Flare Group is a diverse group of international researchers, clinicians, patient research partners (PRP), and others working to create a new tool to identify and measure significant RA flares^14,15 from the patient perspective. In this paper we present validation results from testing of the RA Flare Questionnaire (RA-FQ) in several thousand people with RA in 3 countries. At our OMERACT 2016 workshop, this foundational work developing the measure was summarized and results of field testing were reviewed, using the first step of the OMERACT Filter Instrument Selection Algorithm (OFISA or “Eyeball Test”^15a) as a guide.

We sought participant endorsement that the RA-FQ adhered to OMERACT’s recommended process when reviewing outcome instruments^2,16.

MATERIALS AND METHODS

Summary of Foundational Work: Developing an Instrument

Definition of RA flare

Our steering committee (COB, EC, RA, SJB, VPB, RC, DEF, SH, AL, LM, TGW, RC) and a larger working group defined the concept of interest: RA flare. The definition was endorsed at OMERACT 9 in 2008 and included worsening of essential symptoms and effects of sufficient intensity and duration to be actionable (e.g., indicate a need for treatment change)^6,17,18. The context of use is outlined in Table 1.

Table 1.

Construct of interest and context of use for RA Flare Questionnaire.

Variables	Description
Population	People with RA who have reached an appropriate target of therapy (e.g., low disease activity or remission)
Intervention	Treatment tapering or withdrawal
Comparison	Tapering/withdrawal versus continuing treatment; different strategies of tapering
Outcome	Patients experiencing significant increases in core flare symptoms (pain, fatigue, stiffness) and effects (physical function, participation) for at least 7 days so as to indicate the need for consideration of retreatment
Time	May vary by trial depending on level of symptoms at start, anticipated pharmacodynamics of drugs, and other factors
Setting	Clinical trial

RA: rheumatoid arthritis.

Creating the measurement framework

To develop a measurement framework, we first asked 14 focus groups of patients with RA in 5 countries² about relevant domains. Candidate domains were prioritized in modified Delphi sessions with 125 patients from 10 countries and 108 clinicians from 23 countries to finalize the RA Flare Core Domain Set¹⁹. Domains included the RA core set plus 3 features — fatigue, stiffness, and participation; self-management was recognized as a contextual factor¹⁹ (Figure 1). The RA Flare Core Domain Set was ratified at OMERACT 11 in 2012; as well there was overwhelming participant agreement that the patient engagement process was sufficient (91%) and appropriate (85%)¹⁶.

Figure 1.

Rheumatoid Arthritis Flare Questionnaire conceptual model. From Bykerk, et al. J Rheumatol 2014;41:799–809; with permission.

In our initial review of existing instruments in 2010, we concluded that neither the Routine Assessment of Patient Index Data 3 (RAPID-3)²⁰ nor the Rheumatoid Arthritis Impact of Disease (RAID)²¹ adequately covered all patient-reported flare (PRF) domains (fatigue, stiffness, and participation are not included in RAPID-3, and participation and stiffness are not in RAID.) Thus we identified a need to develop a new instrument that covered all relevant domains.

Creating the measure

Based on our measurement model, we created a prototype self-administered questionnaire of the patient-reported domains of the RA Flare Core Domain Set [i.e., Preliminary Flare Questionnaire (PFQ)]. Respondents were also asked to self-identify if they were in a flare (yes/no), and if yes, to indicate its duration (days) and rate severity (0—10)¹. The PFQ was translated into 17 languages, with 23 linguistic and country-specific versions using a rigorous, forward/backward translation process with bilingual content experts (rheumatologists) and cognitive debriefing with 5 native-speaking patients in each country for each translation (Supplementary Table 1, available with the online version of this article)^22,23. During the final testing phase, RA clinicians (6 rheumatologists in Baltimore and New York, and others affiliated with OMERACT), 9 patients at an academic medical center in Baltimore, and 10 OMERACT PRP confirmed that the instrument was understandable and clear, with appropriate response choices (Supplementary Table 2, available with the online version of this article).

RA-FQ

The RA-FQ contains 5 items to rate pain, physical function, stiffness, fatigue, and participation over the past week using 11-point numeric rating scales (0 = none to 10 = severe; score range 0–50; Figure 2). The RA-FQ will be freely available through OMERACT, with descriptions of psychometric properties, scoring, and interpretation.

Figure 2.

Rheumatoid Arthritis Flare Questionnaire. The RA-FQ score is calculated as the sum of responses for items 1–5 (maximum 50).

Local ethics committees at individual institutions or sites approved all studies.

RESULTS

Does the RA-FQ Pass the Eyeball Test?

We summarized results of 5 years of field testing. Initial validation used data from a Canadian early RA observational study (CATCH; n = 849) and relied on classical test theory (CTT) methods¹. Additional validation included factor and Rasch analysis on data from 2 RA observational studies [Canada, CATCH, n = 896¹; France, Strategy of Treatment in Patients with Rheumatoid Arthritis (STPR), n = 138²⁴], and a randomized controlled trial [RCT; the Netherlands, Dose Reduction Strategy of Subcutaneous TNF Inhibitors in Rheumatoid Arthritis (DRESS), n = 178²⁵; for study descriptions see Supplementary Table 3, available with the online version of this article].

1. Is there a good match with the domain(s)?

Face and content validity

The foundational work was based on a reflective measurement model and grounded in qualitative studies with patients, thus ensuring a good match with patient-reported domains in the RA Flare Core Domain Set. Debriefing of the questionnaire throughout the process provided further evidence of match by people with RA. When combined with the field testing results described below, we concluded that this further supported the face and content validity of the RA-FQ.

2. Is it feasible?

Several thousand patients in RCT and observational trials in several countries have completed the RA-FQ paper forms and using electronic data collection systems (e.g., REDCap, US National Institutes of Health Assessment Center²⁶), with additional data collection ongoing. Among 46 patients with RA at 2 academic arthritis centers (Baltimore and New York), mean (SD) completion time was 1.5 (1.1) min. Patients with RA and OMERACT PRP agreed that the format was appropriate and easy to complete (Supplementary Table 2, available with the online version of this article). Availability in multiple languages increases feasibility of use in multinational studies. We concluded there was sufficient evidence to support feasibility in clinical and observational trials.

3. Do the numeric scores make sense?

To examine construct validity, we developed multiple ways to potentially identify RA flares in the datasets. Then, CTT and Rasch approaches were used to evaluate the factor structure to guide scoring. Next, RA-FQ scores were compared with other indicators of RA disease activity.

Construct validity: identifying flares

In the absence of a gold standard for flare, construct validation offers evidence that an instrument is measuring what it purports to measure²⁷. We hypothesized that PRF (answers “yes” to question: “Are you in a flare?”) would be moderately to highly correlated with MD-identified flare (MDF), and Disease Activity Score at 28 joints flare criteria (DAS28F; DAS28 increase > 1.2 or > 0.6 if DAS28 at previous visit was ≥ 3.2¹³). We have previously shown that in patients who were previously in remission, agreement was high (κ ≥ 0.73) for flare status among PRF, MDF, and DAS28F; in low disease activity (LDA), agreement was moderate to strong between PRF and MDF, and PRF and DAS28F (κ = 0.44–0.63)¹.

To increase confidence that the PRF represented clinically important worsening that was consistent with our definition of flare^17,28 and that would be actionable in a clinical trial, we added additional criteria that would take into account intensity (4/10 on severity scale) and duration (> 7 days). This more stringent definition of PRF (hereafter referred to as PRF-SD) was based on discussions among the steering group and members of the larger RA flare working group. Receiver⁻operation characteristic (ROC) curves were used to analyze the performance of the severity and duration cutpoints among patients where both the patient and MD agreed that the patient was in flare, supporting these cutpoints as discrimination thresholds consistent with clinically important worsening (Supplementary Figure 1, available with the online version of this article). Among CATCH patients, we also identified cases in which the patient and MD both agreed that the patient was in a flare (P-MDF). The 5 definitions of flare were used for subsequent analyses as described below.

Validity of flare domain scores

We have previously shown that PFQ domain scores were moderately to highly (r > 0.7) correlated with existing scales measuring the same or related domains¹. Domain scores were also significantly higher in those who were in a flare versus those who were not in a flare.

Validity of RA-FQ

To establish an appropriate scoring system, we used factor analysis to examine structural validity. The 5 items represented a single factor (81% of variance explained) with each item loading ≥ 0.84 (eigenvalue 4.064), supporting use of a summative score of the 5 domains (range 0 = no flare to 50 = extreme flare) to adequately represent RA flares. In CATCH patients, we compared RA-FQ mean scores and other indicators of disease activity in flaring and non-flaring patients. Flaring patients had significantly higher RA-FQ scores and disease activity indicators, except for acute-phase reactants (Table 2).

Table 2.

Mean (SD) scores of RA-FQ and other RA disease activity indicators by flare^* status in the Canadian Early Arthritis Cohort.

Variables	Flare*, n = 51	No Flare*, n = 571	Mean Difference (95% CI)
RA-FQ	29.0 (10.2)	9.4 (10.0)	19.6 (16.7–22.6)
Tender joint count	5.3 (5.8)	1.6 (3.3)	3.7 (2.0–5.4)
Swollen joint count	3.0 (5.2)	1.1 (2.4)	1.9 (0.4–3.4)
MD global	2.5 (2.4)	0.9 (1.6)	1.6 (0.9–2.3)
Patient global	5.2 (1.5)	1.7 (1.8)	3.4 (2.9–4.0)
Pain, 10 mm VAS	6.2 (2.1)	2.0 (2.2)	4.2 (3.6–4.9)
HAQ, 0–3	0.8 (0.5)	0.3 (0.5)	0.5 (0.4–0.7)
ESR	19.8 (19.5)	15.9 (16.0)	4.0 (−1.5 to 9.4)
CRP	8.0 (12.9)	5.2 (8.9)	2.8 (−1.2 to 6.8)
DAS28	4.2 (1.5)	2.5 (1.2)	1.6 (1.1–2.2)

^*Both patient and MD classified the patient as being in a flare.

RA-FQ: Rheumatoid Arthritis Flare Questionnaire; VAS: visual analog scale; HAQ: Health Assessment Questionnaire; ESR: erythrocyte sedimentation rate; CRP: C-reactive protein; DAS28: Disease Activity Score in 28 joints.

Rasch analysis

Rasch analysis was used to further analyze measurement properties and scoring of the RA-FQ²⁹ in the combined dataset. We examined response thresholds, how well the items worked together, targeted the population of interest, and reflected a unidimensional continuum using RUMM2030 (rating scale model). Reliability, local dependence (items in a scale should not be related to each other or redundant), and differential item functioning (DIF; item bias) were also examined. Items and people covered a broad continuum (> ± 3 logits), covering 99% of targeted range. Results suggested excellent fit with the Rasch model, high reliability (e.g., Person Separation Index > 0.9), 10 well-ordered thresholds for each item, minimal redundancy among items, and minimal DIF by age, sex, or country/language. Rasch results affirm that responses can be added across items to yield a total score (range 0–50) on an interval scale where higher values reflect worsening flare. (Rasch data will be described in greater detail in a separate publication.)

We concluded that results of psychometric methods offered evidence supporting the construct validity of the RA-FQ.

4. Can the RA-FQ evaluate change?

Test-retest reliability

Test scores obtained at 2 timepoints in stable patients should not change. RA-FQ obtained 48–72 h apart (a time during which no change would be anticipated) in 93 patients with RA at 2 academic centers suggested high reliability [r = 0.94; ICC (2, 1) = 0.93, 95% CI 0.90–0.95].

Responsiveness

From CATCH, DRESS, and STPR studies, we selected patients who started in remission/LDA at baseline (DAS28 < 3.2) because this would represent typical patients entering tapering/withdrawal trials. Compared with those who did not flare at the second visit, flaring patients had significantly higher RA-FQ scores using 3 flare definitions (PRF, PRF-SD, DAS28F), with moderate to large effect sizes evident (Table 3).

Table 3.

Rheumatoid Arthritis Flare Questionnaire scores by flare status at 2 consecutive visits using 3 definitions of flare.

Flare Definition	Flare		No Flare		Mean V2 Difference (95% CI)	Effect Size†
	V1	V2	V1	V2
CATCH (observational)
PRF*	13.4	22.5	9.8	8.5	14.0 (11.4–16.5)	1.39
PRF + severity + duration**	16.6	29.0	9.9	9.4	19.6 (16.7–22.6)	1.95
DAS28 flare criteria***	11.8	26.8	10.0	9.3	17.5 (13.6–21.5)	1.73
DRESS (RCT)
PRF*	17.4	22.4	11.6	12.2	10.2 (5.6–14.8)	1.10
PRF + severity + duration**	16.7	31.3	12.2	12.9	18.3 (8.9–27.8)	1.10
DAS28 flare criteria***	18.7	24.6	11.6	12.2	12.4 (7.7–17.2)	1.37
STPR (observational)
PRF*	17.5	22.7	13.0	13.0	9.6 (3.6–15.6)	1.10
PRF + severity + duration**	17.1	25.3	13.3	13.7	11.5 (2.3–20.8)	1.30
DAS28 flare criteria***	16.5	20.5	13.9	13.2	7.3 (1.4–13.2)	0.82

^†Cohen d statistic.

^*Patients answered “yes” to the question “Are you having a flare at this time?”

^**PRF AND patient-rated severity > 4/10 AND reported duration > 7 days.

^***Required increase in DAS28 > 1.2 or > 0.6 if DAS at previous visit was ≥ 3.2.

V1: visit 1; V2: visit 2; CATCH: Canadian Early Arthritis Cohort; PRF: patient-reported flare; DAS28: Disease Activity Score in 28 joints; DRESS: Dose Reduction Strategy of Subcutaneous TNF Inhibitors in Rheumatoid Arthritis; RCT: randomized controlled trial; STPR: Strategy of Treatment in Patients with Rheumatoid Arthritis.

We concluded that initial reliability and responsiveness data suggested that RA-FQ is responsive to change. However, results should be considered preliminary until the publication of additional responsiveness data from clinical trials. (These data are currently being collected with results forthcoming.)

5. Can the RA-FQ define thresholds of meaning for individual patients?

Using ROC curves, we have begun examining thresholds in RA-FQ scores to identify flare; because identification of flare may trigger retreatment, specificity (i.e., correctly identifying those not in a flare) was prioritized over sensitivity. Because a cutpoint to identify flares may differ somewhat depending on the desired outcome, population, and setting, we analyzed thresholds using multiple definitions of flare (PRF, PRF-SD, DAS28F, P-MDF). We also investigated cutpoints in relation to prespecified changes in patient global, MD global, DAS, and Clinical Disease Activity Index. Work is ongoing to establish relevant cutpoints to identify flare in various settings and RA subsets.

Results of field testing data offer evidence of feasibility, construct and content validity, reliability, and responsiveness of the RA-FQ. Strengths of our validation approach include the use of both classical and modern psychometric methods, testing done with patients similar to those with whom the measure is likely to be used, and administration across different samples of international patients with RA. There is evidence from the Rasch analysis that a simple summative score is meaningful and reliable. Limitations include the absence of a gold standard to identify flares and limited evidence that identifying and addressing flares improves longterm outcomes. Table 4^{1,2,6,14,18,19} summarizes our validation activities prior to OMERACT 2016, including the stages at which different steps have been presented and endorsed.

Table 4.

History of endorsement by OMERACT participants for OMERACT Filter 2.1 Instrument Selection Algorithm steps for the RA Flare Questionnaire.

Step	Evidence	OMERACT Endorsement
		2012	2014	2016
1. Is there a good match with domain?	Identifying RA Flare Core Domains: Focus groups6/Delphi exercises19	*
	Face and content validity1, 2, 14, 18	*	*	*
2. Is it feasible to use?	Implementation in clinical trials and observational studies from perspectives of patients, clinicians, other stakeholders1,2; translation into multiple languages		*	*
3. Do the numbers make sense?	Reliability, concurrent, discriminative, convergent, consequential validity1,2; factor analysis and Rasch analysis		*	Passed Eyeball Test
4. Can it evaluate change in patients?	Test-retest reliability, responsiveness1			Passed Eyeball Test
5. Have relevant thresholds been defined?	Preliminary results from ROC curves; work is ongoing to establish thresholds in different populations and settings			Passed Eyeball Test

OMERACT: Outcome Measures in Rheumatology; RA: rheumatoid arthritis; ROC: receiver-operation characteristic.

Small group discussions

Small group discussions during the workshop were conducted to allow more in-depth review of data and to obtain feedback from attendees. Report-backs were largely supportive of the methods used and interpretation of data, and recommendations were offered regarding formatting, presentation of results, and additional analyses to consider, to enhance use in different settings and with subsets of patients with RA.

Voting results

Initial voting at the end of the workshop focused on whether the presented data were sufficient to pass each Eyeball Test question. Consensus [defined as “Green” (no concerns; strong recommendation) PLUS “Amber” (some concerns; conditional recommendation) votes being ≥ 70%] was obtained as follows: (1) match with domain (93%), (2) feasibility (98%), (3) does score make sense (94%), (4) able to measure change (91%), and (5) thresholds of meaning (87%).

Voting results stimulated discussions between the RA flare steering committee and other participants during the remaining days of OMERACT that helped enhance the understanding of the relative strengths and weaknesses of our approach and the interpretation of the results. At the final vote, 88% of participants (70% no concerns, 18% some concerns) agreed that the RA-FQ passed the OFISA Eyeball Test.

DISCUSSION

OMERACT 2016 participants agreed that the RA-FQ fulfilled initial OFISA screening, supporting its potential as a valid and acceptable measure of RA flare. In our OMERACT 2016 plenary workshop, we showed how, by working iteratively with PRP, clinicians, and others and using a mixed-methods approach, we developed a new outcome measure in rheumatology in accordance with OMERACT Filter 2.0³⁰. The OFISA (Eyeball Test) was developed to help researchers initially screen the literature for valid and acceptable outcome measures to potentially include in Core Outcome Measurement Sets. In the plenary, we demonstrated how OFISA could also be used to organize the results of field testing activities when developing a new instrument.

During the plenary, we summarized results of psychometric testing of the RA-FQ from data obtained over 6 years with > 2000 patients across 3 countries. Factor analysis supported unidimensionality of the set of items, and Rasch analysis demonstrated that response options were appropriate, items worked well together, and that the measure was well targeted to patients with RA across the full measurement continuum. The RA-FQ performed similarly in different subgroups (age, sex) and across 3 countries and languages supporting measurement invariance. The RA-FQ is easily scored and readily interpreted by patients and physicians. All these results increase confidence that the RA-FQ can reliably and precisely identify and measure RA flares, although it remains unclear whether addressing flares promptly will improve longterm RA outcomes. Voting results supported adequate initial evidence of feasibility, reliability, validity, and responsiveness.

The RA Flare Group is acquiring additional data from several large RCT and observational studies to establish appropriate thresholds to identify RA flare for different settings and uses.