Pragmatic vs explanatory trials: the Pragmascope tool to help measure differences in protocols of mental health randomized controlled trials

Abstract

In the pragmatic-explanatory continuum, a randomized controlled trial (RCT) can at one extreme investigate whether a treatment could work in ideal circumstances (explanatory), or at the other extreme, whether it would work in everyday practice (pragmatic). How explanatory or pragmatic a study is can have implications for clinicians, policy makers, patients, researchers, funding bodies, and the public. There is an increasing need for studies to be open and pragmatic; however, explanatory trials are also needed. The previously developed Pragmatic-explanatory continuum indicator summary (PRECIS) was adapted into the Pragmascope tool to assist mental health researchers in designing RCTs, taking the pragmatic-explanatory continuum into account. Ten mental health trial protocols were randomly chosen and scored using the tool by three independent raters. Their results were compared for consistency and the tool was found to be reliable and practical. This preliminary work suggests that evaluating different domains of an RCT at the protocol level is useful, and suggests that using the Pragmascope tool presented here might be a practical way of doing this.

Background

The randomized controlled trial (RCT) has long been recognized as the most robust, technique for evaluating the effects of mental health care interventions.¹ Sometimes these trials are impossible, sometimes unethical, and sometimes impractical. The first, RCT was the 1948 Medical Research Council Streptomycin Trial.² In the austere times of bankrupt, post-war England, just as the National Health Service was being established, it, was suggested that, a new drug would be of value for treatment of tuberculosis. The only equitable way to distribute this scarce resource was through randomization and then, by the establishment of good evidence, to encourage those funding health care to support its use. There are many interesting and important examples preceding this date,³ but this landmark and courageous trial radically changed the pathway of evaluation of medical treatments.

Mental health has a fine tradition of using trials to evaluate treatments.⁴ The MRC Streptomycin trial coincided with the discovery of psychoactive compounds that were potentially therapeutic, as well as an increasing push towards deinstitutionalization. Mental health professionals discovered, discussed, and, largely, embraced the use of RCTs. Up to the advent of antipsychotic drugs such as chlorpromazine, psychiatric care had been likened to “little more than zoo keeping”⁵ and, perhaps because of that stinging criticism, those undertaking the trials did not necessarily follow the path of that first tuberculosis trial. Factors combined to largely direct mental health trials along another route. There was the yearning for rigorous science, collective subspecialty insecurity, and also the needs of regulatory authorities. Mental health trials drifted towards use of as rigorous diagnoses as possible, rather rigid regimens of care and use of fine-grained outcome measures that, are not usually part of routine practice. This planted the RCT firmly in the realm of researchers, and there it has stayed. The needs of regulatory authorities did have to be met, but, there was less consideration of needs of clinicians and of recipients of care and their families.

This was not at all unique to mental health, but it. took leaders in the fields of cancer care,⁶ heart disease,⁷ and perinatal medicine⁸ to recall and refine the techniques of generous inclusion, simple treatment, and routine data collection that underpinned the MRC trial of 1948. Many examples now exist in these areas of RCTs where entry criteria are broad and encompass as many relevant people as possible, the treatment packages are those that would be given in everyday care, and outcomes are essentially routinely recorded data. Examples of such open work were rare in mental health until relatively recently. The description of “pragmatic” or “practical” is increasingly employed of trials in psychiatry or psychology but there are clearly different interpretations of what, this really means.

A recent series of papers has highlighted the problems in interpretation of the explanatory/pragmatic domains in trials and presented some practical solutions.⁹ It is not a simple continuum from explanatory through to pragmatic. There are many elements of design that should be considered to allow a judgment to take place about whether a randomized trial is investigating whether, in ideal circumstances, a treatment could work (explanatory) or, at the other extreme, whether this accessible treatment would work in everyday practice (pragmatic). This is not a purely academic exercise. There are good reasons to make these judgments. To use one example, funders, on receiving a proposal, may wish to consider whether the proposed trial fits with the ethos in which that support was proffered. For example one funding body may be interested in discovering potentially new treatments. In this instance, explanatory studies, undertaken in very rigorous circumstances with fine measures of outcome to highlight, any - even modest, - effects, may be best. On the other hand, another funder, using public money, may wish to consider whether the study is likely to produce evidence of practical importance regarding accessible treatments relevant, to the majority of people suffering with the condition in that, community.¹⁰ Here a much more pragmatic study would be desired. How explanatory or pragmatic a study (or a group of studies) is has also obvious and direct implications for clinicians, policymakers, patients, and the public.

The main goal of this study is to adapt the instrument described by Thorpe et al⁹ (PRECIS) to assist, researchers in making those judgments in the protocol stage of RCTs in mental health (the Pragmascope tool).

Methods

The Pragmascope tool

This tool is based on the ten domains described in the development, of the Pragmatic-explanatory continuum indicator summary (PRECIS).⁹ It can be used to assess applicability of results from any given RCT, based on what, was planned at the protocol stage.

Each included RCT protocol^11-19 was scored in ten domains by three independent, reviewers (GT, KSW, CEA). The reviewers made a judgment and rated the protocol from 1 (most explanatory) to 5 (most, pragmatic) by reading the details of the protocol. If the protocol did not contain any information on which to base the decision, these domains were rated as zero. The average scores for each included protocol were placed on the wheel diagram and the dots joined for visual clarity (Figure 1).

Figure 1. Examples of output. Reproduced from ref 9: Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464-475. Copyright © 2009 Elsevier.

Selection of RCT protocols

We searched the Cochrane Schizophrenia Group Trials Register and Medline (November 2010) for references of RCT protocols and chose a random sample of 10 protocols dealing with schizophrenia, depression, post-traumatic stress disorders, and psychiatric rehabilitation.^11-19

Scoring the Pragmascope tool

Three independent reviewers (GT, KSW, CEA) scored each included RCT protocol. The overall score can be from 0 to 50 and a diagram illustrating how open (pragmatic) or restrictive (explanatory) the study is likely to be was created using the average score of the three independent reviewers.

Our initial interpretation of the scores was of 0 to 30 for an explanatory study investigating whether the experimental intervention will work in ideal circumstances and a total score >35 for a more pragmatic study focusing mostly on whether, in routine practice, an intervention has a meaningful effect. A total score between 31 and 39 were interpreted as an interim where trial design balances pragmatic and explanatory domains.

Data analysis

Mean and variance were calculated for each domain of the Pragmascope tool for each included RCT protocol using STATA (version 10). In addition, a weighted kappa for the domains was calculated using R.

Results

Table I presents the average score of the three raters in each one of the domains for each RCT protocols with a judgment based on the scores. Reliability among the three independent raters was high (weighted Kappa =0.72 for categories 0-30, 31-39, 40-50) suggesting that this cluster of judgments might be useful to highlight and quantify important, issues during the protocol stage of an RCT.

Table I. Average score of three raters for each one of the domains of the RCT protocol.

Domain	ACHIEVE14	CATIE11	CCEST17	DYD18	ERP16	FIAT19	PTSD- Yoga20	ROMT15	SPCCD13	TREC - SAVE
	(Low)	(Low)	(Low)	(Low)	(Low)	(High)	(Moderate)	(Moderate)	(Low)	(High) (unpublished)
Eligibility criteria	1	2	2	2	2	3	2	3	2	5
Flexibility of intervention	2	3	3	3	1	4	2	3	3	4
Practitioner expertise (intervention)	1	3	2	2	2	5	2	2	2	4
Practitioner expertise (comparison)	4	3	4	2	5	5	5	5	5	5
Flexibility of comparison intervention	3	3	3	3	5	5	5	5	4	5
Outcomes	3	2	3	3	2	4	4	3	2	4
Participant compliance	4	2	4	3	2	4	5	5	3	4
Practitioner adherence	3	2	3	3	2	4	5	4	3	5
Primary analysis of outcomes	1	2	1	3	5	4	4	4	5	4
Follow-up intensity	1	2	3	5	2	4	2	2	2	5
Total Average score	23	25	29	28	27	43	34	34	28	45

We recognize that validity is a more problematic issue, as this does depend on the rater's perspective, but, work is ongoing involving raters from very different backgrounds. In any case, we concur that, consideration of these domains is useful⁹ and suggest that the Pragmascope is one practical way of doing this.

Discussion

The world of RCTs has changed remarkably in the last 10 years. Systematic reviewing of trials, now industrially undertaken through initiatives like the Cochrane Collaboration,²⁰ has highlighted issues with poor design and inconsistent reporting. These systematic reviews are potent to guide care but, are undermined by trial evidence that is difficult or impossible to apply in the real world. For mental health, studies of increasing pragmatism are now being designed and undertaken.^21-23 Such pragmatic, real-world, practical design can be dovetailed within explanatory studies or sit independently. With maintained systematic reviews guiding practice,²¹ transparent priority setting for research funding for evaluative research,³ and the push towards defining core out come measures of agreed relevance in trials,²⁴ a great, increase in pragmatic trial activity is likely. Of course explanatory trials have an important place in the portfolio of research, but. the rigorously undertaken but highly pragmatic trial will give us the opportunity to learn much more about, the real effects of the potent, treatments we give.

Appendix. The Pragmascope (Figures 2). Explanation: This tool is based on ten domains described in the development of the Pragmatic-explanatory continuum indicator summary (PRECIS).⁹ It can be used to assess applicability of results from any given randomized controlled trial. Instructions: For each of the ten domains please make a judgement and rate the protocol from 1 (most explanatory) to 5 (most pragmatic). Your score should be based on a thorough reading of the protocol. If the protocol does not contain any information on which to base your decision we advise a default score of zero. If you feel your scores require justification - please make comments in box provided. Mark score on relevant part of the diagram and join the dots. Results: The scoring can give you an overall score (0-50) and a diagram illustrating how open (pragmatic) or restrictive (explanatory) the study is likely to be. Interpretation: Figure 1 (see main text) demonstrates an explanatory study investigating whether the experimental intervention will work in ideal circumstances (total score 0-15) and a more pragmatic study focusing mostly on whether, in routine practice, an intervention has a meaningful effect (total score >35). A total score between 16 and 35 suggest an interim where trail design balances pragmatic and explanatory domains.

Appendix. The Pragmascope (Figures 3).