Improving the reporting of pragmatic trials: an extension of the CONSORT statement

Abstract

Background The CONSORT statement is intended to improve reporting of randomised controlled trials and focuses on minimising the risk of bias (internal validity). The applicability of a trial’s results (generalisability or external validity) is also important, particularly for pragmatic trials. A pragmatic trial (a term first used in 1967 by Schwartz and Lellouch) can be broadly defined as a randomised controlled trial whose purpose is to inform decisions about practice. This extension of the CONSORT statement is intended to improve the reporting of such trials and focuses on applicability.

Methods At two, two-day meetings held in Toronto in 2005 and 2008, we reviewed the CONSORT statement and its extensions, the literature on pragmatic trials and applicability, and our experiences in conducting pragmatic trials.

Recommendations We recommend extending eight CONSORT checklist items for reporting of pragmatic trials: the background, participants, interventions, outcomes, sample size, blinding, participant flow, and generalisability of the findings. These extensions are presented, along with illustrative examples of reporting, and an explanation of each extension. Adherence to these reporting criteria will make it easier for decision makers to judge how applicable the results of randomised controlled trials are to their own conditions. Empirical studies are needed to ascertain the usefulness and comprehensiveness of these CONSORT checklist item extensions. In the meantime we recommend that those who support, conduct, and report pragmatic trials should use this extension of the CONSORT statement to facilitate the use of trial results in decisions about health care.

Pragmatic trials are designed to inform decisions about practice, but poor reporting can reduce their usefulness. The CONSORT and Practihc groups describe modifications to the CONSORT guidelines to help readers assess the applicability of the results

Randomised controlled trials are used to assess the benefits and harms of interventions in health care. If conducted properly, they minimise the risk of bias (threats to internal validity), particularly selection bias.^{1 2} There is, however, considerable evidence that trials are not always well reported,^{3 4} and this can be associated with bias, such as selective reporting of outcomes.⁵

The usefulness of a trial report also depends on the clarity with which it details the relevance of its interventions, participants, outcomes, and design to the clinical, health service, or policy question it examines. Furthermore, a trial may be valid and useful in the healthcare setting in which it was conducted but have limited applicability (also known as generalisability or external validity) beyond this because of differences between the trial setting and other settings to which its results are to be extrapolated.

Schwartz and Lellouch⁶ coined the terms “pragmatic” to describe trials designed to help choose between options for care, and “explanatory” to describe trials designed to test causal research hypotheses—for example, that an intervention causes a particular biological change. Table 1 shows some key differences between explanatory and pragmatic trials. Table 2 compares a trial that was highly explanatory in attitude⁷ with one that was highly pragmatic.⁸ There is a continuum rather than a dichotomy between explanatory and pragmatic trials. In fact, Schwartz and Lellouch characterised pragmatism as an attitude to trial design rather than a characteristic of the trial itself. The pragmatic attitude favours design choices that maximise applicability of the trial’s results to usual care settings, rely on unarguably important outcomes such as mortality and severe morbidity, and are tested in a wide range of participants.^{9 10 11} As Schwartz and Lellouch wrote: “Most trials done hitherto have adopted the explanatory approach without question; the pragmatic approach would often have been more justifiable.”⁶

Table 1.

Key differences between trials with explanatory and pragmatic attitudes, adapted from a table presented at the 2008 Society for Clinical Trials meeting by Marion Campbell, University of Aberdeen

Question	Efficacy—can the intervention work?	Effectiveness—does the intervention work when used in normal practice?
Setting	Well resourced, “ideal” setting	Normal practice
Participants	Highly selected. Poorly adherent participants and those with conditions which might dilute the effect are often excluded	Little or no selection beyond the clinical indication of interest
Intervention	Strictly enforced and adherence is monitored closely	Applied flexibly as it would be in normal practice
Outcomes	Often short term surrogates or process measures	Directly relevant to participants, funders, communities, and healthcare practitioners
Relevance to practice	Indirect—little effort made to match design of trial to decision making needs of those in usual setting in which intervention will be implemented	Direct—trial is designed to meet needs of those making decisions about treatment options in setting in which intervention will be implemented

Table 2.

Comparison of trial that was highly explanatory in attitude with trial that was highly pragmatic

	Highly explanatory attitude (NASCET7)	Highly pragmatic attitude (Thomas et al8)
Question	Among patients with symptomatic 70-99% stenosis of carotid artery can carotid endarterectomy plus best medical therapy reduce outcomes of major stroke or death over next two years compared with best medical therapy alone?	Does a short course of acupuncture delivered by a qualified acupuncturist reduce pain in patients with persistent non-specific low-back pain?
Setting	Volunteer academic and specialist hospitals with multidisciplinary neurological-neurosurgical teams and high procedure volumes with low mortality in US and Canada	General practice and private acupuncture clinics in UK
Participants	Symptomatic patients stratified for carotid stenosis severity, with primary interest in severe carotid stenosis (high risk) group, who were thought to be most likely to respond to endarterectomy. Exclusions included mental incompetence and another illness likely to cause death within 5 years. Patients also were temporarily ineligible if they had any of seven transient medical conditions (eg, uncontrolled hypertension or diabetes)	Anyone aged 18-65 with non-specific low back pain of 4-52 weeks’ duration who were judged to be suitable by their general practitioner. There were some exclusion criteria, eg those with spinal disease
Intervention	Endarterectomy had to be carried out (rather than stenting or some other operation), but the surgeon was given leeway in how it was performed. Surgeons had to be approved by an expert panel, and were restricted to those who had performed at least 50 carotid endarterectomies in the past 24 months with a postoperative complication rate (stroke or death within 30 days) of less than 6%. Centre compliance with the study protocol was monitored, with the chief investigator visiting in the case of deficiencies	Acupuncturists determined the content and number of treatments according to patients’ needs
Outcomes	The primary outcome was time to ipsilateral stroke, the outcome most likely to be affected by carotid endarterectomy. Secondary outcomes: all strokes, major strokes, and mortality	Primary outcome was bodily pain as measured by SF-36. Secondary outcomes included use of pain killers and patient satisfaction
Relevance to practice	Indirect—patients and clinicians are highly selected and it isn’t clear how widely applicable the results are	Direct—general practitioners and patients can immediately use the trial results in their decision making

Calls have been made for more pragmatic trials in general,^{6 12 13} and in relation to specific clinical problems.^{14 15 16} Articles have been published discussing the characteristics and value of pragmatic trials^{17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35} or proposing improvements in the design and conduct of these trials.^{36 37 38} Patients, advocacy groups, clinicians, systematic reviewers, funders, and policymakers want to use the results of randomised controlled trials. As such, a clear description of the design and execution of the trial, the intervention and comparator, and the setting in which health care is provided may simplify their decision on the likely benefits, harms, and costs to be expected when implementing the intervention in their own situation. There is, however, no accepted standard to guide reporting on the aspects of design and conduct of trials that affect their usefulness for decision making, particularly considerations that would affect the applicability of the results.

We propose here guidance for reporting pragmatic trials, as a specific extension of the CONSORT statement. Our aim is to identify information which, if included in reports of pragmatic trials, will help users determine whether the results are applicable to their own situation and whether the intervention might be feasible and acceptable. Reporting this information is crucial for any trial that is intended to inform decisions about practice.

CONSORT initiative

The original CONSORT statement (www.consort-statement.org), last revised in 2001, was developed by clinical trialists, methodologists, and medical journal editors to help improve the reporting of parallel (two) group randomised trials.³⁹ The objective of the statement is to enable readers to critically appraise and interpret trials by providing authors with guidance about how to improve the clarity, accuracy, and transparency of their trial reports. It consists of a 22-item checklist and a diagram, detailing the flow of participants through the trial. It is a living document that is updated as needed, incorporating new evidence.⁴⁰ The guidelines have been endorsed by more than 300 journals,⁴¹ and by several editorial groups, including the International Committee of Medical Journal Editors.⁴² The CONSORT statement has been translated into several languages.⁴³ Since its original publication in 1996 the quality of reports of controlled trials has improved.⁴⁴

The CONSORT recommendations are intentionally generic, and necessarily do not consider in detail all types of trials. Extensions of the CONSORT statement have been developed for non-inferiority and equivalence,⁴⁵ cluster randomised designs,⁴⁶ reporting of abstracts,⁴⁷ data on harms,⁴⁸ trials of herbal interventions,⁴⁹ and of non-pharmacological interventions,^{50 51} but not yet for the reporting of pragmatic trials, although some issues pertaining to pragmatic trials were discussed in the CONSORT explanation and elaboration paper.⁴

Methods

In January 2005 and in March 2008, we held two-day meetings in Toronto, Canada, to discuss ways to increase the contribution of randomised controlled trials to healthcare decision making, focusing on pragmatic trials. Participants included people with experience in clinical care, commissioning research, healthcare financing, developing clinical practice guidelines, and trial methodology and reporting. Twenty four people participated in 2005 and 42 in 2008, including members of the CONSORT and Pragmatic Trials in Healthcare (Practihc) groups.⁵²

After the 2005 meeting a draft revised checklist for the extension was circulated to a writing group, including some of those invited to the meeting but unable to attend. After several revisions the writing group produced a draft summary paper. At the 2008 meeting the draft was discussed and modified. It was circulated to the CONSORT group for feedback, modified, and submitted for publication.

Recommendations for reporting pragmatic trials

Meeting participants agreed that no items needed to be added to the CONSORT checklist and that the flow diagram did not need modification. However, participants felt that eight items (2-4, 6, 7, 11, 13, and 21) needed additional text specific to the reporting of pragmatic trials (see table 3). Although participants discussed additional text for item 1 of the checklist (title/abstract), principally adding the word pragmatic to the title or abstract, we decided against making this recommendation because it may reinforce the misconception that there is a dichotomy between pragmatic and explanatory trials rather than a continuum. We elected not to extend item 5 (objectives), although we would encourage trialists to report the purpose of the trial in relation to the decisions that it is intended to inform and in which settings; we have included this recommendation in connection with the extension of item 2 (background).

Table 3.

Checklist of items for reporting pragmatic trials

Section	Item	Standard CONSORT description	Extension for pragmatic trials
Title and abstract	1	How participants were allocated to interventions (eg, “random allocation,” “randomised,” or “randomly assigned”)
Introduction
Background	2	Scientific background and explanation of rationale	Describe the health or health service problem that the intervention is intended to address and other interventions that may commonly be aimed at this problem
Methods
Participants	3	Eligibility criteria for participants; settings and locations where the data were collected	Eligibility criteria should be explicitly framed to show the degree to which they include typical participants and/or, where applicable, typical providers (eg, nurses), institutions (eg, hospitals), communities (or localities eg, towns) and settings of care (eg, different healthcare financing systems)
Interventions	4	Precise details of the interventions intended for each group and how and when they were actually administered	Describe extra resources added to (or resources removed from) usual settings in order to implement intervention. Indicate if efforts were made to standardise the intervention or if the intervention and its delivery were allowed to vary between participants, practitioners, or study sites
			Describe the comparator in similar detail to the intervention
Objectives	5	Specific objectives and hypotheses
Outcomes	6	Clearly defined primary and secondary outcome measures and, when applicable, any methods used to enhance the quality of measurements (eg, multiple observations, training of assessors)	Explain why the chosen outcomes and, when relevant, the length of follow-up are considered important to those who will use the results of the trial
Sample size	7	How sample size was determined; explanation of any interim analyses and stopping rules when applicable	If calculated using the smallest difference considered important by the target decision maker audience (the minimally important difference) then report where this difference was obtained
Randomisation—sequence generation	8	Method used to generate the random allocation sequence, including details of any restriction (eg, blocking, stratification)
Randomisation—allocation concealment	9	Method used to implement the random allocation sequence (eg, numbered containers or central telephone), clarifying whether the sequence was concealed until interventions were assigned
Randomisation—implementation	10	Who generated the allocation sequence, who enrolled participants, and who assigned participants to their groups
Blinding (masking)	11	Whether participants, those administering the interventions, and those assessing the outcomes were blinded to group assignment	If blinding was not done, or was not possible, explain why
Statistical methods	12	Statistical methods used to compare groups for primary outcomes; methods for additional analyses, such as subgroup analyses and adjusted analyses
Results
Participant flow	13	Flow of participants through each stage (a diagram is strongly recommended)—specifically, for each group, report the numbers of participants randomly assigned, receiving intended treatment, completing the study protocol, and analysed for the primary outcome; describe deviations from planned study protocol, together with reasons	The number of participants or units approached to take part in the trial, the number which were eligible, and reasons for non-participation should be reported
Recruitment	14	Dates defining the periods of recruitment and follow-up
Baseline data	15	Baseline demographic and clinical characteristics of each group
Numbers analysed	16	Number of participants (denominator) in each group included in each analysis and whether analysis was by “intention-to-treat”; state the results in absolute numbers when feasible (eg, 10/20, not 50%)
Outcomes and estimation	17	For each primary and secondary outcome, a summary of results for each group and the estimated effect size and its precision (eg, 95% CI)
Ancillary analyses	18	Address multiplicity by reporting any other analyses performed, including subgroup analyses and adjusted analyses, indicating which are prespecified and which are exploratory
Adverse events	19	All important adverse events or side effects in each intervention group
Discussion
Interpretation	20	Interpretation of the results, taking into account study hypotheses, sources of potential bias or imprecision, and the dangers associated with multiplicity of analyses and outcomes
Generalisability	21	Generalisability (external validity) of the trial findings	Describe key aspects of the setting which determined the trial results. Discuss possible differences in other settings where clinical traditions, health service organisation, staffing, or resources may vary from those of the trial
Overall evidence	22	General interpretation of the results in the context of current evidence

For each of the eight items we present the standard CONSORT text and additional guidance, an example of good reporting for the item, and an explanation of the issues. The selection of examples is illustrative for a specific item and should not be interpreted as a marker of quality for other aspects of those trial reports. The suggestions in this paper should be seen as additional to the general guidance in the main CONSORT explanatory paper and where relevant, other CONSORT guidance.

Item 2: introduction; background

Scientific background and explanation of rationale

Extension for pragmatic trials: Describe the health or health service problem that the intervention is intended to address, and other interventions that may commonly be aimed at this problem.

Example (a): Describe the health or health service problem which the intervention is intended to address—“Although interventions such as telephone or postal reminders from pharmacists improve compliance their effect on clinical outcome is not known. We investigated whether periodic telephone counselling by a pharmacist . . . reduced mortality in patients” receiving polypharmacy.⁵³

Explanation—Users of pragmatic trial reports seek to solve a health or health service problem in a particular setting. The problem at which the intervention is targeted should thus be described. This enables readers to understand whether the problem confronting them is similar to the one described in the trial report, and thus whether the study is relevant to them. Ideally, the report should state that the trial is pragmatic in attitude (and why) and explain the purpose of the trial in relationship to the decisions that it is intended to inform and in which settings.

Example (b): Describe other interventions that may commonly be aimed at this problem—“Sublingual buprenorphine is increasingly being prescribed by General Practitioners for opiate detoxification, despite limited clinical and research evidence. Comparing methadone, dihydrocodeine and buprenorphine it is important to note several factors which may impact upon prescribing and use of these agents”.⁵⁴

Explanation—The background of the trial report should mention the intervention under investigation and the usual alternative(s) in relevant settings. To help place the trial in the context of other settings authors should explain key features that make the intervention feasible in their trial setting and elsewhere (such as, the widespread availability of the trial drug, the availability of trained staff to deliver the intervention, electronic databases that can identify eligible patients).

Item 3: methods; participants

Eligibility criteria for participants and the settings and the locations where the data were collected

Extension for pragmatic trials: Eligibility criteria should be explicitly framed to show the degree to which they include typical participants and, where applicable, typical providers (eg, nurses), institutions (eg, hospitals), communities (or localities eg, towns) and settings of care (eg, different healthcare financing systems).

Examples—“The study population included all National Health System physicians in the Northern Region of Portugal except for those not involved in any clinical activity (eg, administrators, laboratory analysis); those working in substance abuse and rehabilitation centers or specialty hospitals (because they cover multiple geographical areas); and those working at the regional pharmacosurveillance center or any department having a specific voluntary ADR reporting program.”⁵⁵

“Our study took place in the three public hospitals (totalling 850 beds) in southern Adelaide, Australia, with a regional population of about 350 000. In Australia, entry to long term care (nursing home) can occur only after an independent clinical assessment by the aged care assessment team (ACAT), who determine level of dependency.”⁵⁶

Explanation—Treatments may perform better when evaluated among selected, highly adherent patients with severe but not intractable disease and few comorbidities. Reports of these restricted trials may be of limited applicability. Excessively stringent inclusion and exclusion criteria reduce the applicability of the results and may result in safety concerns,⁵⁷ so the method of recruitment should be completely described. This stringency seems to be reducing over time but remains a problem.⁵⁸

In some trials the unit of randomisation and intervention might be healthcare practitioners, communities, or healthcare institutions such as clinics (that is, cluster randomised pragmatic trials). In these trials volunteer institutions may be atypically well resourced or experienced, successful innovators. Since the feasibility and success of an intervention may depend on attributes of the healthcare system and setting, reporting this information enables readers to assess the relevance and applicability of the results in their own, possibly different, settings.

Item 4: methods; interventions

Precise details of the interventions intended for each group and how and when they were actually administered.

Extension for pragmatic trials: Describe extra resources added to (or resources removed from) usual settings in order to implement the intervention. Indicate if efforts were made to standardise the intervention or if the intervention and its delivery were allowed to vary between participants, practitioners or study sites. Describe the comparator in similar detail to the intervention.

Example: (a) Describe extra resources added to (or resources removed from) usual settings in order to implement the intervention—“The hospitals and a private long term care provider developed and ran the off-site transitional care facility, which was 5-25 km from the study hospitals. The private provider supplied accommodation, catering, cleaning, nursing (5.0 full time equivalents in 24 hours), and career staff (10.0 full time equivalents in 24 hours) while the hospitals provided the allied health staff (4.4 full time equivalents), medical staff, and a transitional care nurse coordinator (1.0 full time equivalent). The whole team assessed all patients on admission to the transitional care unit and had weekly case conferences. Specialist medical staff visited the site for the case conferences and reviewed all admissions. On-call medical care was available 24 hours a day.”⁵⁶

Explanation—If the extra resources to deliver the intervention are not described, readers cannot judge the feasibility of the intervention in their own setting. When relevant, authors should report details (experience, training etc) of those who delivered the intervention⁵¹ and its frequency and intensity. If multicomponent interventions are being evaluated, details of the different components should be described.

Example: (b) Indicate if efforts were made to standardise the intervention or if the intervention and its delivery were allowed to vary between participants, practitioners or study sites—“Two trained leaders introduced a structured sequence of topics using a collaborative approach. All leaders had run at least one previous group. Throughout the 12 week programme leaders received three hours of supervision each week from a certified trainer.⁵⁹

Explanation—In explanatory trials the intervention is standardised, and thus the results may not apply under usual conditions of care where no such standardisation is enforced. Pragmatic trials are conducted in typical care settings, and so care may vary between similar participants, by chance, by practitioner preference, and according to institutional policies.⁶⁰ For pragmatic trials, efforts that may reduce this natural variation in the intervention and its delivery should be described. However, if reducing variation in a care process or shifting practice patterns is itself the main purpose of the intervention, this should be explicit in the title, abstract, and introduction.

Regardless of the extent to which the intervention was standardised, pragmatic trials should describe the intervention in sufficient detail that it would be possible for someone to replicate it, or include a reference or link to a detailed description of the intervention. Unfortunately, this information is often lacking in reports of trials.⁶¹

Examples: (c) Describe the comparator in similar detail to the intervention—“Standard advice was given as for the naproxen group. Participants were provided with co­codamol for additional pain relief and an information leaflet about “tennis elbow” based on the Arthritis Research Campaign publication but omitting specific treatment recommendations.”⁶²

“Women assigned to the control group received usual care from the healthcare team and completed all outcome measures on the same time frame as the intervention group. After randomisation, this group received a two page leaflet entitled “Exercise after cancer diagnosis,” which provided safe guidelines. After the six month follow-up, these women were helped to construct their own personalised exercise plan and invited to join a local general practice exercise referral scheme.”⁶³

Explanation—In a randomised controlled trial the effects of the intervention are always related to a comparator. To increase applicability, and feasibility, pragmatic trials often compare new interventions to usual care. The chosen comparator should be described in sufficient detail for readers to assess whether the incremental benefits or harms reported are likely to apply in their own setting, where usual care may be more, or less, effective.

Item 6: methods; outcomes

Clearly defined primary and secondary outcome measures, and, when applicable, any methods used to enhance the quality of measurements (eg, multiple observations, training of assessors)

Extension for pragmatic trials: Explain why the chosen outcomes and, when relevant, the length of follow-up are considered important to those who will use the results of the trial.

Example—“The patient-based outcomes used in the evaluation were selected on the basis of empirical evidence from consumers about the most important outcomes from SDM [shared decision making] and risk communication.”⁶⁴

The total number of days off work in the year after inclusion was calculated for each patient. Days off were defined as days 100% compensated by the NIA [National Insurance Administration]. Thus, days on ASL [Active Sick Leave] were considered as days absent. After a full year of sick leave, administrative proceedings are initiated to transfer the beneficiary to other measures of rehabilitation or disability pension within the NIA system. One year of absence was therefore a proxy measure for long-term disability.”⁶⁵

Explanation—The primary outcome(s)⁶⁶ in pragmatic trials are chosen to be relevant to the participants and key decision makers at whom the trial is aimed. The length of follow-up should be appropriate to the decision the trial is designed to inform. If the target decision makers are patients and their clinicians, the primary outcome is likely to be a health outcome, while trials aimed at policymakers and institutional leaders may focus on a process or system efficiency or equity outcome. Explicitly indicating that the chosen outcome is important to decision makers, and specifying the decision makers to whom it was important will assist other readers to decide whether the results are relevant to them.

Item 7: methods; sample size

How sample size was determined; when applicable, explanation of any interim analyses and stopping rules

Extension for pragmatic trials: If calculated using the smallest difference considered important by the target decision maker audience (the minimally important difference) then report where this difference was obtained.

Example—“There were no previous data using the main outcome measure on which to base the sample size calculation, and therefore the sample size was calculated on the number of days with URTI [upper respiratory tract infection]. It was decided, in line with other rigorous pragmatic studies that the smallest difference worth detecting was a 20% reduction in number of days with URTI.”⁶⁷

Explanation—The minimally important difference (MID) is the size of a change in the primary outcome which would be important to the key decision making audience. The MID may differ between settings, consequently readers need to know what MID was considered important in the trial setting, and by whom, to contrast with their own expectations.

Item 11: methods; blinding (masking)

Whether participants, those administering the interventions, and those assessing the outcomes were blinded to group assignment

Extension for pragmatic trials: If blinding was not done, or was not possible, explain why.

Example—“Randomisation was done by telephone to an interactive voice response system. We entered and managed all data in an anonymised format; we held data on patient contacts and other administrative data in a separate database. The study was a pragmatic, randomised, prospective, open trial. In exercise studies, blinding the participants to allocation is not possible. We took steps to blind the evaluation of outcomes by having questionnaire responses in sealed envelopes and ensuring that outcome measures were taken by researchers who were not involved in exercise classes.”⁶³

Explanation—In explanatory trials blinding⁶⁸ prevents belief in the effectiveness of the intervention (by participant, clinician and/or assessor) from confounding the causal link between the intervention and the primary outcome. In pragmatic trials, as in the real world delivery of care, blinding of participants and clinicians may be impossible. Belief (or disbelief) in the intervention, extra enthusiasm and effort (or less), and optimism (or pessimism) in the self-assessment of outcomes may thus add to (or detract from) the effects of an intervention. Pragmatic trials may incorporate these factors into the estimate of effectiveness, rendering the findings more applicable to usual care settings. Authors should speculate on the effect of any suspected modifying factors, such as belief in the intervention, in the discussion (item 20). Moreover, in pragmatic trials, it is still desirable and often possible to blind the assessor or obtain an objective source of data for evaluation of outcomes.

Item 13: results; participant flow

Flow of participants through each stage (a diagram is strongly recommended). Specifically, for each group report the numbers of participants randomly assigned, receiving intended treatment, completing the study protocol, and analyzed for the primary outcome. Describe protocol deviations from study as planned, together with reasons

Extension for pragmatic trials: The number of participants or units approached to take part in the trial, the number which were eligible and reasons for non-participation should be reported.

Example—“These practices ascertained 3392 registered patients with Parkinson’s disease; 3124 were eligible for study of whom 1859 (59.5%) agreed to participate (fig 3). Twenty-three patients died during recruitment, leaving 1836 patients when the intervention began. Seventeen of the 1836 patients were not traced at the NHS central registry and are therefore not included in mortality analyses”.⁶⁹

Explanation—The more similar the participants, practitioners, or other units of intervention or randomisation are to those in usual care, the more likely that the results of the trial will be applicable to usual care. Consequently the text and/or the trial flow diagram should mention, if known, the number of participants or units approached to take part in the trial, the number whom were eligible, and reasons for non-participation. Although this information is requested in the CONSORT statement, the need for it is greater when reporting a pragmatic trial.

Item 21: generalisability (applicability, external validity)

Generalisability of the trial findings

Extension for pragmatic trials: Describe key aspects of the setting which determined the trial results. Discuss possible differences in other settings where clinical traditions, health service organisation, staffing, or resources may vary from those of the trial.

Examples—“The intervention was tailored to the specific study population and may not be as effective in a different group. The positive results may reflect in part unique aspects of the Portuguese health care system or the regional physician culture. Willingness to report adverse drug reactions may be less in countries in which there is greater concern about malpractice liability.”⁵⁵

“The incentive for implementing the clinical pathway will be different for a single-payer third-party system, as exists in Canada, in which costs of the pathway and offsetting hospital costs are realized by the same payer, than for a multiple payer system as exists in the United States, in which hospital cost offsets will be realized by the hospital and not the nursing home payer.”⁷⁰

Explanation—The usefulness of the trial report is critically dependent on how applicable the trial and its results are and how feasible the intervention would be. The authors are well placed to suggest how feasible the intervention might be, which aspects of their setting were essential to achieve the trial result, and how that result might differ in other settings. The applicability of the study result could be encapsulated here by reference to the setting (is it a usual care setting), the participants and providers (how selected were they), intensity of intervention and follow up (how much like usual care was this), adherence to the intervention and whether efforts were made to standardise its delivery, the use of intention to treat analysis, and the amount of loss to follow up. Feasibility can be encapsulated by reference to economic, political, and logistic barriers to implementation and by the range of settings and societies in which these barriers would be low.

Discussion

As demand rises for more pragmatic trials to inform real world choices,¹³ so too does the need to ensure that the results are clearly reported. Readers need to be able to evaluate the validity of the results, the extent to which they are applicable to their settings, and the feasibility of the tested interventions. The existing CONSORT statement applies fully and directly to pragmatic trials. Here we have proposed extensions for eight items in the statement to make more explicit the important attributes of pragmatic trials and thus to ease the task of users in assessing feasibility, relevance, and likely effects of the intervention in their own setting.

We reached consensus that the trial results are likely to be more widely applicable if the participants, communities, practitioners, or institutions were not narrowly selected; if the intervention was implemented without intense efforts to standardise it; if the comparator group received care or other interventions already widely used; and if the outcomes studied were of importance to the relevant decision makers. The intervention needs to be precisely described if readers are to be able to assess its feasibility.

The multiplicity and independence of the elements constituting the design of pragmatic trials guarantee that pragmatism is not an all or none attribute; rather, it might be best conceived as a continuum along several dimensions. For example, a randomised trial could have broad inclusion criteria for participants but rely primarily on a short term, physiological outcome rather than one that is more meaningful to the participants. Alternatively, a trial might include a wide range of participants, meaningfully assess the effect, but evaluate an intervention that is enforced or tightly monitored and thus not widely feasible. Other permutations probably exist. It is not the case that more pragmatic is always better; a trial’s design should be such that the results will meet the needs of the intended users. A trial intended to inform a research decision about the biological effect of a new drug is likely to be more explanatory in design. At a later date, a trial of that same drug aimed at helping patients, practitioners, or policymakers to decide whether it should be prescribed is likely to be more pragmatic in design. To help display this multidimensionality, we have developed of a tool, primarily intended to be used in designing a trial, for characterising where it will stand along the pragmatic-explanatory continuum in relation to each design decision.⁷¹

We hope that these reporting guidelines will help editors, reviewers, trialists, and policy makers in reporting, reviewing, and using pragmatic trials. Journals that have endorsed the CONSORT statement could also support CONSORT for pragmatic trials, by including reference to this extension paper in the journal’s instructions to authors. We also invite editorial groups to consider endorsing the CONSORT extension for pragmatic trials and encourage authors to adhere to it. Up to date versions of all CONSORT guidelines can be found on the CONSORT website (www.consort-statement.org).

Cite this as: BMJ 2008;337:a2390