Current State of Reporting Pain Outcomes in Cochrane Reviews of Chronic Musculoskeletal Pain Conditions and Considerations for an OMERACT Research Agenda

Abstract

Objective

To assess the current state of reporting of pain outcomes in Cochrane reviews on chronic musculoskeletal painful conditions and to elicit opinions of patients, healthcare practitioners, and methodologists on presenting pain outcomes to patients, clinicians, and policymakers.

Methods

We identified all reviews in the Cochrane Library of chronic musculoskeletal pain conditions from Cochrane review groups (Back, Musculoskeletal, and Pain, Palliative, and Supportive Care) that contained a summary of findings (SoF) table. We extracted data on reported pain domains and instruments and conducted a survey and interviews on considerations for SoF tables (e.g., pain domains, presentation of results).

Results

Fifty-seven SoF tables in 133 Cochrane reviews were eligible. SoF tables reported pain in 56/57, with all presenting results for pain intensity (20 different outcome instruments), pain interference in 8 SoF tables (5 different outcome instruments), and pain frequency in 1 multiple domain instrument. Other domains like pain quality or pain affect were not reported. From the survey and interviews [response rate 80% (36/45)], we derived 4 themes for a future research agenda: pain domains, considerations for assessing truth, discrimination, and feasibility; clinically important thresholds for responder analyses and presenting results; and establishing hierarchies of outcome instruments.

Conclusion

There is a lack of standardization in the domains of pain selected and the manner that pain outcomes are reported in SoF tables, hampering efforts to synthesize evidence. Future research should focus on the themes identified, building partnerships to achieve consensus and develop guidance on best practices for reporting pain outcomes.

The comparative effectiveness research movement and regulatory agencies have challenged Outcome Measures in Rheumatology (OMERACT), an international initiative to standardize outcomes in clinical trials in rheumatology, and similar organizations, to establish truthful, discriminative, and feasible patient-important outcomes for randomized controlled trials and non-randomized studies^1,2. Partnerships with organizations working on the development and assessment of patient-important outcomes are necessary and overdue³.

The Cochrane Collaboration is perhaps the largest organization that uses outcomes as the basis to synthesize results of interventional trials to provide high quality, independent evidence to patients, clinicians, and other decision makers. A key part of a Cochrane review is the summary of findings (SoF) table, which presents the results of the major outcomes in the systematic review. Up to 7 major outcomes may be included in the SoF table; they should be those that are deemed most important from a patient/policy perspective and represent both benefit and harm^4,5. While the guidance provided in the Cochrane Handbook states that outcomes important to both patients and other decision makers should be included⁶, there is no explicit process for deciding which outcomes, which outcome instruments, and which effect size metric should be reported in the SoF table. Some Cochrane review groups and other groups such as IMMPACT (Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials) have developed guidance specific to their disease area^{7,8,9,10,11,12,13,14}. Since guidance efforts were developed mostly independently, it is not surprising that there are differences; these include different outcome domains, instruments, methods of analysis and transformations, cut points and thresholds for minimally important and clinically important differences, interpretation of clinical or policy relevance, and methods of presentation.

Patient-reported outcomes (PRO) assessing the effect of a health condition on patients’ lives constitute a major proportion of core sets for clinical trial and systematic reviews of health interventions, especially in the case of chronic, painful conditions. The Cochrane PRO methods group has a chapter in the Cochrane Handbook in which PRO are defined as “reports coming directly from patients about how they feel or function in relation to a health condition and its therapy, without interpretation by healthcare professionals or anyone else”¹⁵. A checklist for authors on important considerations when describing and assessing PRO in systematic reviews has been developed. The checklist includes considerations such as rationale for the construct, evidence for reliability and validity of the measurement instruments, and interpretability of the result. The OMERACT Filter 2.0 provides a framework for development of core outcome sets, including both the selection of appropriate domains and instruments¹⁶. Similarly, the US Food and Drug Administration (FDA) has published guidance on the development of PRO¹⁷.

The PRO domain selected for this evaluation was pain, since 23 Cochrane review groups have pain as a major outcome in their Cochrane reviews of an intervention for a chronic painful condition. It is suspected that the results of pain outcomes are not reported in a consistent manner in SoF tables. Some tables focus on the concept of intensity of pain, others on the interference of pain. Within each of these concepts, different instruments are used, and methods of analysis and presentation vary widely.

We conducted 2 studies with these objectives: (1) to assess the current state of reporting of pain outcomes by sampling SoF tables in Cochrane reviews on chronic musculoskeletal painful conditions, and (2) to elicit opinions of key participants on how best to present pain outcomes to clinicians, patients, and policymakers. The results of this work will contribute to a research platform for OMERACT in partnership with other international initiatives involved in outcome measure methodology in pain.

MATERIALS AND METHODS

Study 1: Assessment of Reporting of Pain Outcomes in Cochrane Reviews

On July 17, 2013, we conducted a search of titles/abstracts/keywords of the Cochrane Library using the key word “pain” to identify all intervention reviews (excluding overviews) in defined chronic musculoskeletal painful conditions from Cochrane review groups (Back; Musculoskeletal; and Pain, Palliative, and Supportive Care) that contained a SoF table. These 3 Cochrane review groups are responsible for the conditions that fall within the World Health Organization definition of a chronic musculoskeletal condition: “inflammatory rheumatic diseases such as rheumatoid arthritis, osteoporosis and other bone diseases, osteoarthritis and related conditions, soft-tissue periarticular disorders, back pain”¹⁸. We independently extracted data in duplicate on the pain domains and instruments and used Excel 2010 for data management and analysis.

We extracted composite measures if they contained a pain component. If a SoF table reported more than 1 pain outcome and/or instrument, all were extracted. When a SoF table reported only a standardized mean difference (SMD), this was recorded as “no instrument reported.” If a SMD was reported along with the transformation of the SMD to a specific instrument, then the instrument used for the back transformation was extracted.

Study 2: Survey and Interviews of Key Stakeholders Regarding Presentation of Pain Outcomes in Cochrane SoF Tables

The aim of our survey and interviews was to obtain information from patients, clinicians, and methodologists, on the most important aspects to consider when expressing the pain response of trial participants in chronic musculoskeletal pain intervention studies with respect to Cochrane systematic reviews. Participants were asked: (1) Which of the following domains of pain are important for reporting in a SoF table: pain intensity, pain frequency, pain interference with function, or other domains; (2) What is the best way to present measures of change; and (3) What are the important thresholds/cutoffs for identifying responders in (i) change scales and (ii) achieving predefined absolute “states.”

We used a purposive, expert sampling technique to select survey and interview participants to obtain representation from Cochrane review groups, international initiatives involved in outcome measures methodology, patients with painful musculoskeletal conditions, healthcare practitioners, and methodologists. Prospective participants were sent a link to the survey via E-mail and were asked to participate in an interview to provide more detailed comments. Two members of the project team drafted the survey [a rheumatologist/journal editor/systematic reviewer and senior outcomes researcher (PT), and a managing editor and systematic reviewer (LJM)] that consisted of open-ended text responses. We piloted our survey with 3 invitees, and revised it in response to comments. We then administered our survey using SurveyMonkey™, and invitees were sent 2 reminders to complete the survey. The interviews were conducted by a researcher trained in conducting semistructured interviews. The interview guide followed the sequence of the survey (Supplementary Table 1, available online at jrheum.org). Interviews lasted between 30 and 75 minutes. The majority of interviews (21/24, 88%) were recorded, transcribed verbatim, and then coded. Three were coded from notes taken during the interviews because the audio recording equipment malfunctioned.

The text data were analyzed by 1 researcher (LJM) and checked by a second (PT) for themes, using a directed approach of qualitative content analysis¹⁹. This directed approach was used because the survey was developed around existing ideas for themes, as identified by OMERACT executive committee members. We first ascertained the incidence of responses representing the pre-defined ideas and then identified the incidence of newly identified topics.

Ethics approval for survey and interviews was obtained from the University of Split, School of Medicine Ethical Committee, Split, Croatia.

RESULTS

Study 1: Assessment of Reporting of Pain Outcomes in Cochrane Reviews

Our search of the Cochrane Library website identified 57 reviews containing a SoF table that assessed interventions for chronic painful musculoskeletal conditions (see Supplementary Figure 1, available online at jrheum.org). All but 1 SoF table (98%, 56 of 57) reported an outcome of pain (Table 1). Half the SoF tables (41, 53%) reported only the word “pain” in the table’s outcome column, and we assumed the domain of interest was pain intensity based on the scales that were reported. All 56 SoF tables that reported pain presented a measure of pain intensity, in either single or multiple domain instruments; 20 different instruments were reported, with the visual analog scale (VAS) being the most frequent (45%). Pain intensity was measured using a continuous scale for all but 5 outcomes: 4 responder analyses and the outcome “number of people with resting pain” (Table 1). Pain interference was reported in 8 SoF tables (5 different instruments), and pain frequency was reported in 1 multiple domain instrument in a single SoF table. No SoF tables reported other aspects of pain, such as pain quality or pain affect¹⁴. In 9 (16%) SoF tables, the instrument for measuring pain intensity was not reported, and of these, 6 reported SMD with no back-transformation using a familiar instrument.

Table 1.

Pain outcome domains and instruments reported in included Cochrane summary of findings (SoF) tables.

		SoF Tables, n = 57 Reviews*
Outcome Domain/Subdomain	SoF Tables, n	Outcome Instrument	n
Pain intensity	48**	Unidimensional pain intensity scales:
		• VAS (0–10 cm or 0–100 mm)	28
		• VAS (1–9)	11
		• Verbal rating score (0–10)	1
		• 10-point Likert scale	1
		Multidimensional pain intensity scales:
		• WOMAC pain subscale score	1
		• Hospital for Special Surgery pain subscale score	1
		Dichotomous outcomes — instrument not reported
		• At least 50% improvement from baseline	2
		• Patient Global Impression of Change [in pain] much or very much improved	2
		• IMMPACT definition — any substantial pain benefit	1
		• IMMPACT definition — at least moderate pain benefit	1
		• Number of participants with resting pain	1
		• Instrument not reported (only SMD reported)	10 (6)
		• Pain reported in SoF as an outcome, but not measured in included studies	6
Pain intensity/tender joints	2	Number of tender joints	1
		Number of tender points	1
Multidomain outcomes including pain intensity	14	ACR50 response criteria	9
		ASAS40 response criteria	1
		ASAS partial remission response criteria	1
		ASES Shoulder Score	1
		Disease Activity Score (DAS28)	6
		Hospital for Special Surgery knee score	1
		Lequesne Index	1
		QUALEFFO	1
Multidomain/dimension outcomes including pain intensity and pain interference	6	Neck Disability Index	2
		DASH	3
		Fibromyalgia Impact Questionnaire	1
Multidomain/dimension outcomes including pain interference	1	SF-12	1
Multidomain/dimension outcomes including pain intensity, pain frequency and pain interference	1	Osteoporosis quality of life	1
Pain not reported in SoF table	1	Not applicable

^*More than 1 pain outcome can be reported per SoF table; 5 reviews had no included studies

^**41 reported only “pain,” but we assumed pain intensity from the scale; outcome not measured in included studies.

ACR 50: American College of Rheumatology 50% response criteria; ASES: American Shoulder and Elbow Surgeons Shoulder score; DASH: Disabilities of the Arm, Shoulder, and Hand; SF-12: Medical Outcome Study Short Form 12 Survey; VAS: visual analog scale; WOMAC: Western Ontario and McMaster Universities Arthritis Index: QUALEFFO: quality of life questionnaire in patients with vertebral fractures; SMD: standardized mean difference.

Study 2: Survey and Interviews of Key Stakeholders on Expressing Pain Outcomes in Cochrane SoF Tables

Forty-five individuals were invited to participate in a more in-depth discussion via survey and/or telephone interview. Thirty-six completed an interview and/or the survey; 10 completed both an interview and a survey. Therefore, responses were obtained from 36/45 (80%) invited individuals. All 24 interviews were conducted by the same person (LJM). Reasons for nonparticipation of invitees were not obtained, but we assumed they were unavailable. Those involved in either interview or survey included: patients with painful chronic musculoskeletal conditions (n = 3); healthcare practitioners and/or researchers with expertise in outcomes measurement representing the following fields: rheumatology (n = 12), occupational therapy (n = 2), physiotherapy (n = 1), neurology (n = 1), pain management (n = 4), pain psychotherapy (n = 2), and statisticians and methodologists with expertise in outcomes measurement (n = 11). The majority of healthcare practitioners also conduct outcomes research and thus fulfilled more than 1 role.

Each respondent was active in 1 or more of the following initiatives (Supplementary Table 2, available online at jrheum.org): ACTTION/IMMPACT (Analgesic, Anesthetic, and Addiction Clinical Trial Translations/Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials)^11,12,13,14, COMET (Core Outcome Measures in Effectiveness Trials)²⁰; COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments)^21,22; Cochrane (Editorial Unit; review groups: Back; Musculoskeletal; Neuromuscular Disorders; Pain, Palliative and Supportive Care; and methods groups: Applicability and Recommendations; and Patient Reported Outcomes); OMERACT; or VAPAIN (Validation and Application of a core set of patient-relevant outcome domains to assess the effectiveness of multimodal PAIN therapy)²³.

Tables 2 to 5 describe the 4 key themes derived from respondents, along with examples of issues that were raised. We completed the COREQ (Consolidated criteria for Reporting Qualitative studies) checklist to ensure high-quality reporting of the study (details available from corresponding author).

Table 2.

Theme 1: Which concepts (or “domains”) of chronic pain should be included as “core” in Cochrane summary of findings tables?

Key Issues Raised by Respondents	No. Respondents (denominator = 36)
Pain intensity is an important outcome to present in SoF tables for chronic conditions  • A direct measure of pain, describes the pain experience; the first issue of communicating with HCP  • There is clear and consistent evidence that improving pain results in improvements in fatigue depression, health-related quality of life and function, and work  • Existing consensus on this by IMMPACT (PI measured on a 0–10 NRS)	32
A 1-dimensional measure of PI alone does not capture the complexity of pain impact  • “This is to me more important: whether it [pain] stopped me from what I wanted to or needed to do rather than something that was just there. Rating the intensity of the pain might be impacted by whether it is preventing me from doing what I want/need to do” (quote from patient)  • The best measure for a trial because it has the best sensitivity to change (i.e., intensity) doesn’t necessarily reflect a meaningful improvement in the patient experience	10
Consideration of the phrasing and standardization of questions about PI with respect to:  • Time frame (e.g., current, last 24 hours, last month, change from previous time point)  • Type of pain (e.g., average, least, worst)  • Specification of activity (e.g., on movement, on walking, at rest)  • Location (overall or global pain, pain targeted to a joint)  • Recall bias concerns	7
Difficulties in capturing and measuring the concept of PI  • It is framed by individual experience and tolerance  • It is a qualitative construct that we are trying to quantify	5
Importance of pain frequency
• Is an important outcome to include in an SoF table	5
• “It depends” on condition, e.g., important to describe for recurrent/periodic/intermittent pain	11
Importance of pain interference with function
• Is an important outcome to include in an SoF table	28
• How does it link or overlap with a measure of function alone?	2
• Oversimplification that improving pain improves function	1
Consideration of whether generic or disease-specific pain measures should be reported
• Both	7
• Prefer generic (“pain is pain”)	4
• Prefer condition-specific	3
• Depends on the question and goal of the systematic review	5
• Generic helps to make comparisons across conditions, but a field may prefer to use condition-specific	4
Other pain-related domains for consideration:  • Pain duration, pain relief, pain behavior, pain quality, and the effect of pain on fatigue, activities of daily living, worker productivity, health-related quality of life, sexual activities, effect on partners/caregivers  • Should consider both the etiology of the pain condition and the nature of the intervention	15
Important to include patient perspective in the discussions  • Link with existing OMERACT pain working group and their discussions on pain domains and key issue: Is chronic non-cancer pain a disease in and of itself?  • Consider OMERACT Filter 2.0 framework	24

PI: pain intensity; HCP: healthcare practitioner; SoF: summary of findings; IMMPACT: Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials; NRS: numeric rating scale.

Table 5.

Theme 4: Establishing a hierarchy of pain outcome instruments.

Key Issues Raised by Respondents	No. Respondents (denominator = 36)
Important for systematic reviewers	4
• To reduce bias we need a systematic method to inform which pain outcome instrument to choose when more than 1 is reported in a trial
Different methods have been used to develop hierarchies for pain outcome instruments in OA	3
• Methods include expert opinion and responsiveness of pain outcome instruments in OA trials  • What other criteria than responsiveness should be considered?
What is the patient perspective on this hierarchy?	2
• Could use concept mapping approach to get input from patients
Important to distinguish the hierarchy of constructs from hierarchy of instruments	2

OA: osteoarthritis.

There was good agreement in some areas, such as pain intensity being an important domain to measure in a chronic painful musculoskeletal condition. Twenty-four of 36 respondents raised the issue of the importance of incorporating the patient perspective in these discussions. Analysis of other topics resulted in a range of responses, which were occasionally contradictory. For example, most respondents agreed that analysis by “responder” is ideal and when available should be presented. In cases when only mean values are available, a few respondents noted that the presentation of absolute change (e.g., treatment group improvement by 2 points more than a control group on a scale of 0–10) is easily interpretable and is useful for presenting results, while some respondents felt that these mean values should not be presented. Explanations for the latter included: the argument that presenting a mean change is not useful because the distribution response is often bimodal making an “average” change meaningless; or that mean change is not easily interpretable by patients. Others noted the importance of ensuring that treatment groups are similar at baseline in order to interpret absolute change.

DISCUSSION

The results of our 2 projects described above provide compelling evidence of the disparate use of pain outcomes, and underpin the need to establish dialogue between participants in the fields of pain measurement and outcome methodology. Such partnerships can advance development of guidance on best practices for expressing the pain response to an intervention in a way that is most meaningful to decision makers.

The SoF table is the hallmark of current Cochrane reviews, and while it is reassuring that all but 1 of the included reviews of interventions in chronic musculoskeletal pain conditions provided an outcome of pain, the results were not presented in a consistent manner. A variety of scales, cut points, and transformations were reported in the domains of pain intensity, frequency, and interference, making it difficult for the readers of Cochrane reviews to make sense of the evidence across reviews. Different pain conditions were included in the analysis and, as noted by the survey/interview respondents, it was not entirely clear whether chronic painful conditions of different etiologies could be reported similarly. As well, the nature of an intervention may affect the choice of key outcome domains.

The lack of reporting of the outcome instrument used seriously limits the interpretation of results; this was a concern in almost one-third of SoF tables (18/57) that did not report the outcome instrument. IMMPACT has published consensus recommendations for using a numerical rating scale (NRS) to measure pain intensity in chronic pain trials¹¹ and to report the proportion of patients who achieve reductions in pain intensity of ≥ 30% and ≥ 50% (reflecting what are proposed as moderate and substantial clinically important differences, respectively)¹³. For back pain, 1 consensus paper suggests a 30% reduction in pain as minimally important to patients²⁴. In spite of these recommendations, it is notable that the NRS was not reported in a single SoF table. Responder analyses were reported in 2 SoF tables; the timing of the publication of the recommendations (2005 and 2008) may be a reason for the lack of their use.

Seven global pain domains (intensity, frequency, interference, location, affect, quality, and factors associated with pain) were identified when patients with chronic pain were asked to describe their pain in their own words²⁵. An IMMPACT survey of patients with a variety of chronic pain conditions found that within the concept of pain interference, patients identified 19 aspects of pain interference with daily life (e.g., sleep, social relationships, employment, emotional well-being) as being important¹⁴. We found that the majority of SoF tables reported on pain intensity, with few assessing pain interference or frequency or any other of the pain (sub) domains identified by IMMPACT. The majority of survey and interview participants in this study raised the importance of including the patient perspective. They also noted that, although the burden on respondents must be taken into account, a more complex measure of the effect of pain, in addition to intensity, should be considered when reporting evidence for the effectiveness of a treatment for chronic pain. This is a strength of the new ICOAP instrument²⁶ (Intermittent and Constant Osteo-Arthritis Pain), which was developed based on focus groups with patients from 4 countries and used modern psychometric approaches as recommended in the OMERACT Filter 2.0¹⁶. We look forward to seeing this instrument in the updates of these systematic reviews.

The survey and interviews allowed us to generate themes for future research, based on input from a broad group of people. Participants generally agreed on some topic areas but not on others, such as the preferred method for the presentation of results, highlighting differences of opinion.

With OMERACT’s core principle of actively including patients and others in the consensus process, and through building partnerships with key organizations involved in pain and outcome measurement, there is a strong foundation and opportunity now for achieving consensus and developing guidance.

Table 3.

Theme 2: Criteria for acceptable clinimetrics/psychometrics for core endpoints for inclusion in Cochrane summary of findings tables.

Key Issues Raised by Respondents	No. Respondents (denominator = 36)
Must establish congruent language about measurement properties  • Terminology used in various groups is not consistent (e.g., meaning of “discrimination” differs in OMERACT and COSMIN contexts)	5
Need clear distinction between what to measure and how to measure  • First, what is the most important construct to measure and then to discuss what is the best instrument to measure this construct  • Need outcome instruments with acceptable clinimetric criteria before we can have a discussion on how best to express treatment response	5
Consideration of assumptions that instruments like NRS or VAS have underlying operational metrics  • Concern about use of nonlinear scales to quantify a percentage improvement and the impact on MID/MCID calculations  • Suggest attention to use of Rasch methods	3
Important to consider the instrument in terms of the intervention  • Where you expect to see variation in the scale as a result of the intervention is the place on the scale that needs to be the most sensitive  • Perhaps different scales might be needed depending on severity of pain and where we expect the intervention to act	2

COSMIN: COnsensus-based Standards for the selection of health Measurement INstruments; NRS: numeric rating scale; VAS: visual analog scale; MID/MCID: minimum important difference/minimal clinically important difference.

Table 4.

Theme 3: Which “threshold of meaning” should be presented in the summary of findings table?

Key Issues Raised by Respondents	No. Respondents (denominator = 36)
There should be a presentation of the proportion of people reaching a certain threshold (e.g., proportion of patients achieving a 50% change from baseline). How to define the threshold?	26
• MCID	3
• “Collective ‘minimum important change’ can [not] be defended scientifically or logically”	1
• 50% is a very good pain reduction and recommended by IASP	5
• Pain responses tend to be bimodal — good relief or very little — an easy discriminating point is 50%	1
• What patients want is ≥ 50% pain intensity reduction	5
• 50% is a less realistic target	2
• Interested in empirical data re bimodal response	3
Show results for various thresholds	5
• E.g., 20%, 50%, 70% responders	3
• Report all percentage improvements in cumulative frequency distribution	5
• If concerned about statistical power; might find a statistically significant difference with mean change but not in a responder analysis	4
• Want to determine a reliable way to dichotomize continuous data	3
• Why limit to one way of presentation? Consider offering Web-based automatic calculation	1
Concern that a fixed proportion like 50% will bias against those with low/better scores at baseline	2
• Unless you have similar baselines, meaning is different
There should be a presentation of proportion of people achieving a state, e.g., patient acceptable state, low or minimum state; a state of “no worse than mild pain”	17
• “Status/state” is much more important to a patient than “change”	8
• The important question for patients “Is your pain at a level now where you can function and do what you want/have to do without the pain being an issue?”	2
• It might be considered the ultimate goal of treatment as in reality NWTMP is what patients want — a manageable point vs not manageable point	3
• For many people in chronic conditions associated with pain, they will not be completely pain-free	3
• Keep magnitude and value separate, and focus on clear ways to present the data	1
Suggestions of thresholds for defining a “state”
• Magnitude of change: below 4 on 0–10 NRS or less than 3 on 0–10 scale	2
• Based on patient response: Can ask patient at end of study if they are in an “acceptable state”	6

IASP:International Association for the Study of Pain; MCID: minimal clinically important difference; NWTMP: no worse than mild pain; NRS: numeric rating scale.