Evaluating the design and reporting of pragmatic trials in osteoarthritis research

Abstract

Objectives

Among the challenges in health research is translating interventions from controlled experimental settings to clinical and community settings where chronic disease is managed daily. Pragmatic trials offer a method for testing interventions in real-world settings but are seldom used in OA research. The aim of this study was to evaluate the literature on pragmatic trials in OA research up to August 2016 in order to identify strengths and weaknesses in the design and reporting of these trials.

Methods

We used established guidelines to assess the degree to which 61 OA studies complied with pragmatic trial design and reporting. We assessed design according to the pragmatic–explanatory continuum indicator summary and reporting according to the pragmatic trials extension of the CONsolidated Standards of Reporting Trials guidelines.

Results

None of the pragmatic trials met all 11 criteria evaluated and most of the trials met between 5 and 8 of the criteria. Criteria most often unmet pertained to practitioner expertise (by requiring specialists) and criteria most often met pertained to primary outcome analysis (by using intention-to-treat analysis).

Conclusion

Our results suggest a lack of highly pragmatic trials in OA research. We identify this as a point of opportunity to improve research translation, since optimizing the design and reporting of pragmatic trials can facilitate implementation of evidence-based interventions for OA care.

Rheumatology key messages

• Only 61 self-identified pragmatic trials on osteoarthritis were published prior to August 2016.

• Existing pragmatic trials in osteoarthritis research show variable compliance with established guidelines.

• Most pragmatic trials on osteoarthritis did not meet guidelines related to practitioner expertise.

Introduction

The prevalence of OA is expected to rise with population ageing [1]. There is no cure for OA, but there are strategies that can reduce progression and mitigate symptoms [2, 3]. The challenge lies in effective implementation of these interventions, particularly since there are demonstrated practice gaps in the delivery of OA care [4]. Implementation research aims to reduce the gap between what is known to be clinically effective and what is actually delivered in clinical care [5]. Allen et al. [6] provide an overview of the design and conduct of implementation trials of interventions for OA. The authors describe conceptual frameworks (e.g. knowledge-to-action), study designs (e.g. pragmatic trials) and evaluations (both process and formative) for implementation trials.

Pragmatic trials are particularly useful in implementation research, as they are designed to determine the generalizability of interventions to routine practice [6]. Whereas explanatory trials are used to test the efficacy of interventions in controlled settings, pragmatic trials are used to demonstrate the effectiveness of interventions in real-world settings [7, 8]. In theory, pragmatic trials test interventions that are evidence-based, with flexibility for application across multiple settings with large and heterogeneous populations, looking at stakeholder-related outcomes over longer periods of time [9, 10]. In practice, this may not always be the case.

The objective of this study is to evaluate the degree to which existing pragmatic trials in OA research comply with guidelines for the design and reporting of pragmatic trials [11, 12]. We identify strengths and weaknesses of pragmatic trials in OA research and suggest ways in which pragmatic trial guidelines can be applied to OA research to achieve highly pragmatic trials. By optimizing pragmatic trial methodology in OA research, we can facilitate implementation of evidence-based interventions in routine practice and reduce care gaps.

Methods

We searched PubMed and Web of Science using the terms ‘pragmatic’, ‘trial’ and ‘OA’ (all fields) to identify publications prior to August 2016. Our search identified 63 citations from PubMed and 93 citations from Web of Science, with 96 unique citations combined (Supplementary Fig. S1, available at Rheumatology Online). We included articles that explicitly stated that the study was pragmatic in the title (36%), abstract (59%) or methods/discussion (5%). We excluded articles that were not reports of primary research, were not available in full text or English and were not related to OA. We excluded reports of trial results when reports of the trial protocol for the same study were already included. For each study, we determined whether the intervention was clinician-based (oral drug, injections, acupuncture, surgery or clinical pathways) or patient-based (diet, exercise, self-management programs, devices, topical therapies) and which joints were targeted (Supplementary Table S1, available at Rheumatology Online).

We used the pragmatic–explanatory continuum indicator summary (PRECIS) [11] and the pragmatic trials extension of the CONsolidated Standards of Reporting Trials (CONSORT) [12] guidelines to determine the parameters of an ideal pragmatic trial in OA research [13, 14]. Guidelines for optimal pragmatic trial design (PRECIS) and reporting (CONSORT) were consistent, with an additional guideline for reporting blinding in the CONSORT extension. We combined these guidelines into 11 criteria (Table 1) to evaluate each of the 61 studies reporting a pragmatic trial in OA research. Determinations were made for each criterion using a simple binary system to indicate whether the study met pragmatic criteria (yes = 1) or not (no = 0), where a maximum score of 11 could be assigned per study (Supplementary Table S2, available at Rheumatology Online). After being trained to code [15], two independent raters (K.L. and K.W.) evaluated each study. Interrater agreement of coding for a random sample of studies (n = 30) was determined to be 78%. A third reviewer (S.A.A.) evaluated any discrepancies in coding (an average of three criteria per study).

Table 1.

Application of pragmatic trial guidelines to OA research

Criteria	Design (PRECIS)	Reporting (CONSORT)	A pragmatic trial in OA research
1	Participant eligibility criteria	Participants	Captures the target population (e.g. does not exclude people with co-morbidities)
	Experimental intervention	Interventions
2	Flexibility	Generalizability	Implements an intervention that can be delivered after the study concludes
3	Practitioner expertise		Relies on a general practitioner or other typical OA care provider
	Comparison intervention	Background
4	Flexibility		Describes the current standard of care, does not alter it (e.g. by providing pamphlets)
5	Practitioner expertise		Relies on a general practitioner or other typical OA care provider
6	Follow-up intensity	Outcomes	Measures outcomes infrequently and at least 6 months following the intervention
7	Primary trial outcome	Sample size	Uses minimally invasive outcomes that are meaningful to the participant (e.g. function)
8	Participant compliance		Does not track participant compliance (e.g. with self-reports in diaries/logs)
9	Practitioner adherence		Does not monitor general practitioner/OA care provider adherence to the study protocol
10	Analysis of primary outcome	Participant flow	Includes all participants in an intention-to-treat analysis of the primary outcome
11		Blinding	Provides an explanation for blinding decisions

Summary of PRECIS [11] and CONSORT [12] guidelines showing overlap and application to pragmatic trials on osteoarthritis.

Results

None of the 61 pragmatic trials we evaluated met all 11 criteria described in Table 1. Most of the trials, for both clinician- and patient-based interventions, met five to eight of the criteria (Supplementary Fig. S2, available at Rheumatology Online). Few trials were at either extreme, meeting nine or more criteria or four or less criteria (Supplementary Fig. S2, available at Rheumatology Online). Of note, 5% of studies met nine or more criteria, suggesting that it is possible, but rare, to have highly pragmatic trials in OA research.

The criteria that most studies failed to meet were practitioner expertise for both experimental and comparison interventions. This requires the intervention be applied by practitioners ordinarily involved with the care of patients [11]. For OA patients, this typically includes general practitioners, pharmacists, family and friends. Only 10% of studies met this criterion for the experimental intervention and only 34% for the comparison intervention (Table 2). The majority of studies required additional training of practitioners delivering the intervention or included experts that would require special referral in many health care systems (e.g. physiotherapists, orthopaedic surgeons).

Table 2.

Evaluation of pragmatic trials in OA research

Criteria	Clinician-based intervention (n = 25)	Patient-based intervention (n = 36)	Combined (n = 61)
Participant eligibility criteria	12 (48)	13 (36)	25 (41)
Experimental intervention
Flexibility	13 (52)	18 (50)	31 (51)
Practitioner expertise	5 (20)	1 (3)	6 (10)
Comparison intervention
Flexibility	12 (48)	17 (47)	29 (48)
Practitioner expertise	9 (36)	12 (33)	21 (34)
Follow-up intensity	17 (68)	30 (83)	47 (77)
Primary trial outcome	19 (76)	31 (86)	50 (82)
Participant compliance	14 (56)	19 (53)	33 (54)
Practitioner adherence	21 (84)	27 (75)	48 (79)
Analysis of primary outcome	19 (76)	34 (94)	53 (87)
Blinding	8 (32)	18 (50)	26 (43)

n (%) of studies that met each criteria, separated by clinician- or patient-based intervention, and combined.

Only 41% of studies met pragmatic trial guidelines for participant eligibility criteria (Table 2). As described by Thorpe et al. [11], trials with minimal inclusion and exclusion criteria are considered pragmatic. The majority of trials we evaluated imposed specific participant eligibility criteria relating to the severity or type of OA (inclusion criteria) and the presence of co-morbidities (exclusion criteria) and seldom explained why. For example, 61% of studies recruited participants with knee OA (16% knee and hip, 5% hip, 5% did not specify a joint, 8% generalized OA, 3% hand, 2% shoulder) and many studies excluded participants who had undergone joint replacement or other surgical interventions. These design decisions may be appropriate for trials examining interventions for specific populations but do not capture the OA population with multiple morbidities due to advanced age or with persistent symptoms in the same or additional joints after surgery.

We found 48% of studies met criteria for flexibility of the comparison intervention (Table 2), where pragmatic trials use the existing standard of care as the comparison intervention [11]. This number may be inflated since many studies did not report the standard of care, so we assumed no changes were made. Many studies did change the standard of care, for example, by offering the comparison group information pamphlets. Lack of reporting was also evident for blinding procedures. Traditional single- or double-blinding may not always be possible for pragmatic trials [10], but only 43% of studies provided an explanation for the blinding decisions (Table 2).

Pragmatic trials avoid monitoring participant compliance with the intervention [11]; we found 54% of the studies met this criterion (Table 2). Several studies required participants to keep track of a behaviour using diaries or logs over extended periods of time. While compliance measures may help researchers explain effect sizes, they may also introduce an observer effect. Truly pragmatic trials accept non-compliance as a reality [13]. This relates to the flexibility of the experimental intervention, for which 51% of studies met the criterion (Table 2). Pragmatic trials have interventions that are not closely monitored, that are flexible in delivery and that accommodate variation across settings [13].

The strengths of pragmatic trials in OA research include the choice of primary trial outcome, where 82% of studies used outcomes that were minimally invasive and clinically meaningful to participants (e.g. pain, quality of life, function), and analysis of primary outcome, where 87% of studies used intention-to-treat analysis. We found 79% of studies did not monitor practitioner adherence to the study protocol, although this number may reflect a common practice to refrain from monitoring practitioners rather than a research effort to comply with pragmatic trial guidelines. We found 77% of studies met the criterion for minimizing follow-up intensity, although we allowed for up to two follow-ups and considered any follow-up by phone or mail to be pragmatic (Table 2).

Discussion

In OA research, studies that self-identify as pragmatic trials fail to meet many criteria for the design and reporting of pragmatic trials. While the PRECIS tool [11] is not intended as a method for classifying trials, it is useful for evaluating the degree to which pragmatic trials meet design recommendations [13, 15]. Our results show that most trials have both pragmatic and explanatory elements, supporting the idea of a pragmatic–explanatory continuum in trial design [11, 13].

Ideally, pragmatic trials should maximize external validity, and this requires moving away from the controlled conditions of traditional explanatory trials. In the ‘real world’, populations are heterogeneous with different stages of OA, practitioners apply protocols variably and patients may not fully comply with interventions, particularly since OA is deprioritized in clinical settings [4]. Yet for scientific rigor, trials must have some inclusion/exclusion criteria, practitioners must follow protocol to some degree, an appropriate comparison group is needed and some type of follow-up is required to measure change in outcomes. As a result, there is considerable tension for some pragmatic trials criteria, between minimizing bias and maximizing generalizability [10]. How these tensions are reconciled depends on the research question and parameters of individual studies [7].

Going forward, improved reporting of design decisions can reveal whether trials are more pragmatic, more explanatory or potentially negligent in a particular domain of trial design. We did not evaluate the overall quality of the studies included, but only what was reported, making it difficult to distinguish shortcomings in design vs reporting. Although 75% of the studies included were published after the CONSORT extension for pragmatic trials was available in 2008 [12], it appears that there are still deficiencies in reporting of pragmatic trials.

To clarify what may constitute a pragmatic trial in OA research, we identified common design decisions that are consistent with guidelines (Table 1). The list in Table 1 is not exhaustive and was formulated based on the pragmatic trials we evaluated, of which 41% were clinician-based interventions and 59% were patient-based interventions. Existing guidelines for pragmatic trials had to be flexibly applied for trials with clinician-based interventions to qualify as pragmatic. We found eligibility criteria were more specific, experimental and comparison interventions were less flexible, practitioner adherence to protocol was stricter and follow-up intensity was more frequent—out of necessity for surgical and pharmacologic interventions. Therefore, if the trial design captured as closely as possible the way in which the intervention would ultimately be delivered in usual clinical care, we considered it pragmatic.

We excluded articles that were not related to OA or declared as pragmatic trials, making our search specific, but not necessarily sensitive. Other studies may have incorporated elements of pragmatic trial design without declaring the trial type as pragmatic or may have tested interventions for joint pain without declaring an OA diagnosis. This may have resulted in undercounting of pragmatic trials in OA in our literature search. Other articles may have inappropriately declared the trial type as pragmatic, causing our results to reflect poor design and reporting and an overall lack of highly pragmatic trials. The underlying issue may be a lack of clarity and consensus in the field about what constitutes a pragmatic trial [7].

It remains unclear whether trials are not sufficiently pragmatic or whether existing pragmatic trial guidelines are not appropriate. Ultimately, pragmatic trials test the implementation of interventions in the real world, and what constitutes real world will differ depending on the intervention type (in-home for many lifestyle interventions, hospital-based for surgical interventions), the end users (patients, clinicians, policymakers) and the social, political and economic contexts in which the intervention will ultimately be delivered [16]. It is difficult to prove whether having more trials that are more pragmatic will improve implementation of evidence-based interventions [17]. Certainly without pragmatic trials and implementation research, practitioners may lack trial evidence that is amenable to their clinical context, and this may hinder their ability to operationalize clinical practice guidelines.

In conclusion, there is a lack of highly pragmatic trials in OA research, as defined by current guidelines for the design [11] and reporting [12] of pragmatic trials. Understanding existing pragmatic trial guidelines and how they can be applied to OA research may improve the use of this method in implementation research. Further efforts are needed to achieve a common understanding among researchers about what constitutes a pragmatic trial.