Abstract
This Editorial addresses the crucial issue of which research methodology is most suited for capturing the complexity of psychosocial interventions conducted in ‘real world’ mental health settings. It first examines conventional randomized controlled trial (RCT) methodology and critically appraises its strengths and weaknesses. It then considers the specificity of mental health care treatments and defines the term ‘complex’ intervention and its implications for RCT design. The salient features of pragmatic RCTs aimed at generating evidence of psychosocial intervention effectiveness are then described. Subsequently, the conceptualization of pragmatic RCTs, and of their further developments – which we propose to call ‘new generation’ pragmatic trials – in the broader routine mental health service context, is explored. Helpful tools for planning pragmatic RCTs, such as the CONSORT extension for pragmatic trials, and the PRECIS tool are also examined. We then discuss some practical challenges that are involved in the design and implementation of pragmatic trials based on our own experience in conducting the GET UP PIANO Trial. Lastly, we speculate on the ways in which current ideas on the purpose, scope and ethics of mental health care research may determine further challenges for clinical research and evidence-based practice.
Key words: Outcome assessment, pragmatic trials, psychosocial interventions, RCTs
Introduction
Complex interventions are the rule rather than the exception in mental health care, and their effectiveness has become a pressing question in this era of limited economic and personnel resources (Becker & Pushner, 2013). The randomized controlled trial (RCT) has traditionally been considered as the main tool for addressing this issue from a research perspective. Yet, current RCT methodology in mental health research essentially responds to a relatively limited set of questions, which predominantly pertain to pharmacological treatments. Although all RCTs have much in common – in terms of design, conduct, analysis and reporting – complex intervention trials (i.e. interventions with several interacting components) (Chambless & Hollon, 1998) can pose specific challenges that require careful consideration, if they are to yield high quality and credible evidence that might modify or inform practice (Thornicroft et al. 1998).
We discuss herein some practical challenges that must be faced when designing and implementing trials aimed to provide knowledge that is really useful to disseminate evidence-based practices, and refer to our own experience in designing and conducting the GET UP PIANO Trial, a pragmatic cluster RCT that aimed to evaluate the effectiveness of a multi-component psychosocial intervention in a large epidemiologically based cohort of patients with first episode psychosis recruited from Italian public Community Mental Health Centers (CMHCs) located in a 10 million inhabitant catchment area (Ruggeri et al. 2012). The intervention lasted 9 months and consisted of an integrated package of evidence-based psychosocial interventions (i.e. cognitive behavioral therapy for patients, family intervention for family members and case management for patients and families) plus standard care (consisting of personalized outpatient psychopharmacological treatment, combined with non-specific supportive clinical management and non-specific informal support/educational sessions). The control group received standard care only.
Lastly, we speculate on the ways in which current ideas about the purpose, scope and ethics of mental health-care research may determine further clinical research challenges.
Conventional RCT application limitations in mental health care
The RCT is considered the gold-standard methodology for testing health treatment effectiveness. The importation of this approach into psychiatric research has proved to be more easily applicable to pharmacotherapy (which tends to be a more standardized and well-defined intervention) than to psychological and social interventions. Mental health care RCTs aiming to assess the many complex aspects of psychosocial interventions – which are no less important in real-world practice than psychotropic drug prescription – tend to be undermined by methodological limitations, such as: (a) poor design, which oversimplifies the variables to be examined; (b) poor conceptualization and inaccurate measurement of process variables, with change usually being measured only at the end of treatment and not also during it; (c) over-interpretation of associations, with causality assumed even for unclear directions in associations; (d) neglect in modelling the plausible influence of important pre-treatment factors (moderators) on treatment effects; (e) difficulties in identifying the variables that might actually modulate observed changes (mediators); (f) under-powerment of trial sample, due to costs and study design constraints, which interferes negatively on a trial's potential for detecting important effects.
Therefore, the rigour of the conventional RCT design, as traditionally conceived, has turned out to be unsatisfactory for studying situations of high treatment complexity (Slamoirago-Blotcher & Ockene, 2009), such as those involved in psychosocial interventions (Slade & Priebe, 2001). In fact, in ‘real world’ mental health settings, interventions are usually targeted to test their effect on a broad range of outcome domains. Furthermore, they generally include different kinds of integrated pharmacological and psychosocial interventions, whose implementation can be mediated by a series of clinical and non-clinical variables, e.g. therapist-specific characteristics (including cultural background, training, interpersonal skills, caseloads, etc.) and features of the social-relational environment within which the treatment is delivered. Difficulties in conducting psychosocial RCTs in real-world care stem not only from the challenges that are associated with a given intervention's intrinsic complexity but also from the local service context's complexity and heterogeneity (Harvey et al. 2011). Persons involved in a study – whether participants, health professionals or researchers – are influenced by their own beliefs, attitudes and experiences, and these consciously or unconsciously affect the way in which they engage with the research process. This phenomenon creates (both positive and negative) cultural expectations, which affect the ways in which people engage with a trial. A given context and service culture's impact on a trial has been acknowledged, particularly in relation to complex interventions (Campbell et al. 2007; Nastasi & Hitchcok, 2009), although trial designers have perhaps thus far failed to fully address the above-mentioned challenges when designing and implementing these types of clinical trials.
As an example, we report the procedure used in the GET UP PIANO Trial to take into consideration all these issues. Specifically, to increase the overall acceptability of the trial, we choose not to administer the experimental treatment using external psychotherapy experts, but we rather decided to train staff of routine CMHCs allocated to the experimental intervention arm. This implied performing the training of the staff to the experimental intervention in the pre-trial stage, with assessment of competencies achieved, followed by ongoing supervision during the patient enrolment phase (Ruggeri et al. 2012; Poli et al. submitted for publication). This strategy has indeed proved to be an important factor in favouring a high degree of collaboration by the CMHCs staff in all trial's procedures, with none of the participating CMHCs in the experimental arm dropping out during the study.
Beyond the conventional RCT
Clinicians have often held a negative view of conventional RCT use in mental health-care research, by considering them a reductionist tool or as being poorly representative of complex clinical practice and by claiming that RCTs disrupt clinical decision-making or clinical engagement with patients. Moreover, clinicians tend to consider the results of RCTs inapplicable or irrelevant to real clinical practice (Hotopf et al. 1999; Ruggeri et al. 2008), and, in general, they may resent having to strictly adhere to evidence-based practice, fearing that it will deprive them of leeway in employing their own competence and experience.
Conversely, clinicians have frequently assumed that qualitative methods are more suitable for use in clinical research (Bird et al. 2011). Paradoxically, however, the randomized design approach is particularly suited for testing complex treatments in a number of ways. For example, the nature of complex treatments is that of multiple potential factors, both known and unknown, which influence outcomes: only an adequately powered randomized design approach allows for the proper control of these variables. Furthermore, despite their limitations, the key principles of the RCT methodology is to date considered to represent the gold standard: in the current evidence-based era, any attempt to deprive complex psychosocial interventions of their imprimatur potentially risks undervaluing results achieved in this field (Goldbeck & Vitiello, 2011; Melfsen et al. 2011).
Hotopf et al. (1999) championed the concept of pragmatic RCTs as a major attempt to extend RCT methodology to interventions conducted in ‘real-world’ services. The approach has since become a key tool for evaluating the effectiveness of psychotropic drugs and complex psychological interventions. Pragmatic trials provide a realistic compromise between observational studies (which have good external validity, at the expense of internal validity) and conventional RCTs (which have good internal validity, at the expense of external validity). Thus, the gap between clinician's treatment concerns in everyday practice and the study designs that are able to test them may be narrowing, thereby allowing for the development of more targeted intervention and ultimately for service enhancement.
Seminal publications from the Medical Research Council (MRC, 2000; Campbell et al. 2000; Campbell et al. 2007) provide useful guidance for developing and evaluating complex interventions. The MRC underscores the need for robust and rigorous evaluation of complex interventions, particularly in the area of psychological and psychosocial treatments. They promote the use of experimental methods and also provide information on alternatives to conventional RCTs and highlight situations in which these trials are impractical or undesirable. Specific dimensions of complexity are implicated in both the development and evaluation phases of research, such as: the ways in which a given intervention leads to change, lack of impact due to implementation failure and variability in individual-level outcomes, use of multiple primary outcomes and adaptation of intervention programmes to local settings (Craig et al. 2008). In fact, conventional RCTs may be inapplicable in these instances, and assessment of treatment effectiveness may require solutions involving special experimental designs, such as cluster randomized trials (Barbui & Cipriani, 2011; Campbell et al. 2012), stepped wedge designs (Hussey & Hughes, 2007), preference trials (Torgerson & Sibbald, 1998), randomized consent designs (Zelen, 2006) and ‘N of 1 designs’ (Mahon et al. 1996). Furthermore, randomization may not always be necessary or appropriate, thus leaving a non-randomized design as the only choice (Mohr et al. 2009; Catts et al. 2010).
In the light of these considerations, the GET UP PIANO Trial deliberately adopted a cluster randomized controlled design, due to the higher feasibility of the complex interventions' implementation at the service level rather than in individual patients randomized to the experimental or the control arm. Specifically, clusters in the GET UP PIANO Trial were CMHCs operating for the Italian National Health Service in the catchment area. Notwithstanding the higher statistical complexity of this model, as described by Dunn (2013) in this issue and proved in the GET UP Trial, cluster randomization seems to be the strategy of choice when interventions that require some overall service organizational modification are to be implemented.
The MRC Guidance places new emphasis on collaboration with clinicians and service users. This approach can be considered part of a recent cultural shift that aims to view the generation of robust science-based knowledge on complexity as the core of cultural concern, rather than as a peripheral, specialized or methodologically flawed discipline. Indeed, this new MRC conceptualization is yielding many fascinating intellectual and practical challenges for clinicians and researchers alike. In the crucial pre-trial stage of treatment modelling there is the risk of oversimplified reductionism, which overlooks key aspects. Thus, during the pre-trial phase, staff training and assessment of competencies achieved must be undertaken; moreover, all treatment features must be robustly operationalized, and this procedure should result in the development of a manual. In this process, then, any form of rigidity that diminishes the possibility to respond to real patients' individual needs must be avoided by incorporating flexibility of response in the protocol itself (Hawe et al. 2004). With this regard, in the GET UP PIANO Trial, detailed intervention manuals based on international standards were developed and given to the staff providing experimental interventions as a standard to be followed for the treatment. Flexibility in the use of the manual was allowed, but a written explanation of the specific reasons for this was required. Fidelity was measured throughout the trial by using therapists' reports of their own session and audiotape recordings of therapy sessions. Therapist reports and audiotapes were planned to be rated at the end of the trial by an independent team using the Cognitive Therapy Scale-Revised (CTRS) (Blackburn et al. 2001) and the Cognitive Therapy for Psychosis Adherence Scale (CTPAS) (Startup et al. 2004), together with ad hoc checklists based on the specific trial intervention manuals. Staff were also supported in their clinical work by a team of expert psychotherapists assigned to each CMHC and received on site supervision by external experts throughout the study.
The body of knowledge and experience discussed above has allowed to further extend the pragmatic approach in order to fully capture the complexity of psychosocial interventions in real-world care (Roy-Byrne et al. 2003; Ruggeri & Tansella, 2011; Ruggeri et al. 2012). We propose to call these kinds of trials ‘new generation’ pragmatic trials. An increasing number of them are being implemented in mental health services, paving the way to the possibility of changing mental health professionals' perceptions of clinical trials, and testing the applicability of new methodological tools.
Tools to building on the pragmatic trials approach and implementing ‘new generation’ trials
Of great interest in this perspective, is the CONSORT (Consolidated Standards of Reporting Trials) Statement extension (Boutron et al. 2008a, b), which specifically addresses the complexity of non-pharmacological treatments, including psychotherapy and behavioural interventions. It is worth mentioning here its checklist item extensions concerning: (a) the description of the different intervention components (both experimental treatment and comparators), procedures for tailoring interventions to individual participants, details on the ways in which interventions are standardized and the ways in which care provider adherence to the protocol is assessed and, if necessary, improved; (b) details on the implementation of experimental treatment and comparators; (c) comparator choice, reasons for lack or partial blinding, and unequal expertise of each group's care providers and/or participating centres.
Lastly, discussion on the trial's external validity, comparators, patients, care providers and services is required. The extended CONSORT flow diagram provides a box for reporting information on the number of each group's centres and care providers, and the distribution of participants per care provider and/or centre. The authors of the present editorial tested in the GET UP PIANO Trial the feasibility and utility of the CONSORT expanded procedure and found that it is a very useful tool for identifying and monitoring the trial's context complexity and performing continuous quality checks on its implementation. Of special interest is the blinding procedure used in the GET UP PIANO Trial, where a complete blinding of patients, clinicians and raters working on site was not possible because of the cluster randomization design which implied the intervention's implementation at the CMHC level, and not at the patient level. However, every effort was made to preserve the independence of the raters: they were not involved in the treatment sessions, and any conflict of interest was accurately prevented and monitored. Primary outcomes (relapses and/or changes in psychopathology) were mostly objective clinical assessments, based on standardized instruments with clearly defined anchor points. Cross-check of the different instrument scores' congruence was performed post hoc, and, whenever raw data were to be analysed, assessments were made by paired and independent members of the research team who were blinded to the randomization arm (Ruggeri et al. submitted for publication).
Another relevant issue to be aware of is that distinguishing between explanatory and pragmatic trials in real life is not easy (Schwartz & Lellouch, 1967) The view that explanatory and pragmatic procedures are mutually exclusive is now transforming into the idea that most trials have both explanatory and pragmatic aspects, and are becoming ‘hybrid trials’ thereby (Green & Dunn, 2008). Furthermore, pragmatism is an attribute that is not merely dichotomous (absent/present), and is gradually being seen as existing on a continuum. It is noteworthy that, to provide a thorough measure of this aspect, Thorpe et al. (2009) have recently developed the PRagmatic-Explanatory Continuum Indicator Summary (PRECIS) tool.
The primary outcomes issue in ‘new generation’ pragmatic trials
When dealing with complex interventions, another key issue is that of how primary and secondary outcomes should be characterized. Conventional RCT methodology allows for only one pre-specified primary outcome measure, which tests the trial's key hypothesis (Green, 2006). Secondary outcomes and intermediate measures are permitted, but they only provide opportunities for testing wider aspects of the primary outcome. Moreover, they are used only sparingly and are considered of marginal value. This rigorous stance is aimed at obviating any post hoc temptations ‘fishing trips’, but researchers and clinicians are increasingly questioning whether this degree of rigour is actually appropriate for testing complex interventions that yield important outcomes, which are unlikely, however, to be unitary or thoroughly simple and which present similarly complex intermediate effects.
In addition to the choice of single or multiple outcomes, which influences sample size calculation, the planning of outcomes in complex intervention trials calls for the careful examination of other critical aspects. Outcomes selected for trials conducted in routine settings are frequently those inherent to patients' self-report measures, such as the GET UP PIANO Trial, where two primary outcome measures were defined in order to detect more finely tuned clinical changes. The first primary outcome was based on symptoms' severity assessed by the independent raters (as measured by using the PANSS; Kay et al. 1987), but the second was based on the subjective appraisal of psychotic symptoms as referred by patients themselves (measured by using the PSYRATS; Drake et al. 2007).
A growing body of literature has shown that patients' pre-intervention characteristics can affect patients' subjective reports of outcome, and that some heterogeneity in patient-reported outcomes may likely be driven by baseline heterogeneity in the enrolled population (Candy et al. 2011). The exploration of heterogeneity in ‘new generation’ pragmatic trials could represent an important methodological advance, because treatment effect heterogeneity might not only be simply due to outcome variability (which is controllable through statistical approaches) but also to non-random variability. This latter type of variability can be attributed to patient, treatment, provider or environmental factors, which require more complex epidemiological methods of analysis. In this context, it might happen that, paradoxically, the kind of outcomes selected contribute to the heterogeneity of treatment effects (Kent et al. 2008).
Moreover, subgroups of individuals might be more greatly impacted by treatment than other subgroups, with clinically crucial implications. In this light, subgroup analysis – which has traditionally been considered a subordinate part of the trial results (frequently relegated to the role of exploratory analysis) – can conversely be considered a major contribution to the intervention's mechanism of action. Both moderators and mediators can play an important role in this process. For example, moderators could be crucial pre-treatment variables that help identify the type of individuals and conditions in which treatment has a certain effect on outcome. Moreover, treatment could have a causal effect on mediators, which are thus the intervention's real targets, though frequently disregarded. Indeed, mediator identification can be used for speculating on modifications to treatment strategies aimed to augment effectiveness or reduce costs (Kraemer et al. 2001; Kraemer 2013). Consistently with this, in the GET UP PIANO Trial (Ruggeri et al. 2012), a series of exploratory analyses have been planned in order to compare the outcome in groups of patients with specific characteristics identified a priori (such as gender, age of onset, duration of untreated psychosis, etc.) and pathways to outcome have been analysed in these different subgroups (Ruggeri et al. in preparation).
Conclusion
In conclusion, the issue of how to best evaluate the effectiveness of complex interventions in real-world services poses some key questions: Does a specific type of intervention work? How does it work? Is it cost-effective? What components are responsible for its efficacy, and for cost- and patient-related outcomes? Can it be tailored to work more effectively or cost-effectively with particular types of patient? (Patsopoulos, 2011). The challenge for trial methodologists is to develop ways of designing trials to answer these questions without abandoning methodological rigour.
By further exploring these questions, remarkable progress can be made, and many challenges posed nowadays by the issue of trial design might represent landmark opportunities in the advancement of mental health service research and implementation of evidence-based interventions.
Acknowledgements
This Editorial has been conceived in the frame of the theoretical and experimental work conducted between 2007 and 2012 to devising and implementing the GET UP (Genetics Endophenotype and Treatment: Understanding early Psychosis) Research Programme (National Coordinator: Mirella Ruggeri), funded by the Italian Ministry of Health as part of a National Health Care Research Program (Ricerca Sanitaria Finalizzata) coordinated by the Academic Hospital of Verona (Azienda Ospedaliera Universitaria Integrata Verona). The authors wish to express their deepest gratitude to the researchers and the clinicians who have contributed to the implementation of the GET UP Program. See http://ww.psychiatry.univr.it/page_getup for the full list of those involved in the GET UP GROUP.
Conflict of Interest
None.
Financial Support
This work was supported by a grant from the Italian Ministry of Health (National Health Care Research Program (Ricerca Sanitaria Finalizzata), funds assigned to the Genetics Endophenotype and Treatment: Understanding early Psychosis) Research Programme
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
References
- Barbui C, Cipriani A (2011). Cluster randomised trials. Epidemiology and Psychiatric Sciences 20, 307–309. [DOI] [PubMed] [Google Scholar]
- Becker T, Pushner B (2013). Complex interventions in mental health services research: potential, limitations and challenges In Improving Mental Health Care: the Global Challenge (eds. Thornicroft G., Ruggeri M. and Goldberg D.), John Wiley: Chichester. [Google Scholar]
- Bird L, Arthur A, Cox K (2011). Did the trial kill the intervention? Experiences from the development, implementation and evaluation of a complex intervention. BMC Medical Research Methodology 11, 24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Blackburn IM, James IA, Milne DL, Baker C, Standart S, Garland A, Reichelt FK (2001). The Revised Cognitive therapy scale (CTS-R). Psychometric properties. Behavioural and Cognitive Psychotherapy 29, 431–446. [Google Scholar]
- Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P (2008a). Extending the CONSORT Statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Annals of Internal Medicine 148, 295–309. [DOI] [PubMed] [Google Scholar]
- Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P (2008b). Methods and processes of the CONSORT group: example of an extension for trials assessing nonpharmacologic treatments. Annals of Internal Medicine 148, W60–66. [DOI] [PubMed] [Google Scholar]
- Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P (2000). Framework for design and evaluation of complex interventions to improve health. British Medical Journal 321, 694–696. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Campbell MK, Piaggio G, Elbourne DR, Altman DG, for the CONSORT Group (2012). CONSORT 2010 statement: extension to cluster randomised trials. British Medical Journal 345, e5661. [DOI] [PubMed] [Google Scholar]
- Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, Guthrie B, Lester H, Wilson P, Kinmonth AL (2007). Designing and evaluating complex interventions to improve health care. British Medical Journal 334, 455–459. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Candy B, King M, Jones L, Oliver S (2011). Using qualitative synthesis to explore heterogeneity of complex interventions. BMC Medical Research Methodology 11, 124–132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Catts SV, O'Toole BI, Carr VJ, Lewin T, Neil A, Harris MG, Frost AD, Crissman BR, Eadie K, Evans RW (2010). Appraising evidence for intervention effectiveness in early psychosis: conceptual framework and review of evaluation approaches. Australian and New Zealand Journal of Psychiatry 44, 195–219. [DOI] [PubMed] [Google Scholar]
- Chambless DL, Hollon SD (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology 66, 7–18. [DOI] [PubMed] [Google Scholar]
- Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M (2008). Developing and evaluating complex interventions: the new Medical Research Council guidance. British Medical Journal 337, 979–983. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Drake R, Haddock G, Tarrier N, Bentall R, Lewis S (2007). The psychotic symptom rating scales (PSYRATS): their usefulness and properties in first episode psychosis. Schizophrenia Research 89, 119–122. [DOI] [PubMed] [Google Scholar]
- Dunn G (2013). Pragmatic trials of complex psychosocial interventions: methodological challenges. Epidemiology and Psychiatic Sciences, this issue. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Goldbeck L, Vitiello B (2011). Reporting clinical trials of psychosocial interventions in child and adolescent psychiatry and mental health. Child and Adolescent Psychiatry and Mental Health 5, 4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Green J (2006). The evolving randomised controlled trial in mental health: studying complexity and treatment process. Advances in Psychiatric Treatment 12, 268–279. [Google Scholar]
- Green J, Dunn G (2008). Using intervention trials in developmental psychiatry to illuminate basic science. British Journal of Psychiatry 192, 323–325. [DOI] [PubMed] [Google Scholar]
- Harvey C, Killaspy H, Martino S, White S, Priebe S, Wright C, Johnson S (2011). A comparison of the implementation of assertive community treatment in Melbourne, Australia and London, England. Epidemiology and Psychiatric Sciences 20, 151–161. [DOI] [PubMed] [Google Scholar]
- Hawe P, Shiell A, Riley T (2004). Complex interventions: how “out of control” can a randomised controlled trial be? British Medical Journal 328, 1561–1563. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hotopf M, Churchill R, Lewis G (1999). Pragmatic randomised controlled trials in psychiatry. British Journal of Psychiatry 175, 217–223. [DOI] [PubMed] [Google Scholar]
- Hussey MA, Hughes JP (2007). Design and analysis of stepped wedge cluster randomized trials. Contemporary Clinical Trials 28, 182–191. [DOI] [PubMed] [Google Scholar]
- Kay SR, Fiszbein A, Opler LA (1987). The positive and negative syndrome scale for schizophrenia. Schizophrenia Bulletin 13, 261–276. [DOI] [PubMed] [Google Scholar]
- Kent DM, Alsheikh-Ali A, Hayward RA (2008). Competing risk and heterogeneity of treatment effect in clinical trials. Trials 9, 30–35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kraemer HC (2013). Discovering, comparing, and combining moderators of treatment on outcome after randomized clinical trials: a parametric approach. Statistics in Medicine Jan 10. doi: 10.1002/sim.5734. [Epub ahead of print]. [DOI] [PubMed] [Google Scholar]
- Kraemer HC, Stice E, Kazdin A, Offord D, Kupfer D (2001). How do risk factors work together? Mediators, moderators, and independent, overlapping, and proxy risk factors. American Journal Psychiatry 158, 848–856. [DOI] [PubMed] [Google Scholar]
- Mahon J, Laupacis A, Donner A, Wood T (1996). Randomised study of n of 1 trials versus standard practice. British Medical Journal 312, 1069–1074. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Medical Research Council (2000). A Framework for the Development and Evaluation of RCTs for Complex Interventions to Improve Health. MRC: London. [Google Scholar]
- Melfsen S, Kuehnemund M, Schwieger J, Warnke A, Stadler C, Poustka F, Stangier U (2011). Cognitive behavioral therapy of socially phobic children focusing on cognition: a randomised wait-list control study. Child and Adolescent Psychiatry and Mental Health 5, 5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mohr DC, Spring B, Freedland KE, Beckner V, Arean P, Hollon SD, Ockene J, Kaplan R (2009). The selection and design of control conditions for randomized controlled trials of psychological interventions. Psychotherapy and Psychosomatics 78, 275–284. [DOI] [PubMed] [Google Scholar]
- Nastasi BK, Hitchcock J (2009). Challenges of evaluating multilevel interventions. American Journal of Community Psychology 43, 360–376. [DOI] [PubMed] [Google Scholar]
- Patsopoulos NA (2011). A pragmatic view on pragmatic trials. Dialogues Clinical Neuroscience 13, 217–224. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Roy-Byrne PP, Sherbourne CD, Craske MG, Stein MB, Katon W, Sullivan G, Means-Christensen A, Bystritsky A (2003). Moving treatment research from clinical trials to the real world. Psychiatric Services 54, 327–332. [DOI] [PubMed] [Google Scholar]
- Ruggeri M, Tansella M (2011). New perspectives in the psychotherapy of psychoses at onset: evidence, effectiveness, flexibility, and fidelity. Epidemiology and Psychiatric Sciences 20, 107–111. [DOI] [PubMed] [Google Scholar]
- Ruggeri M, Lora A, Semisa D SIEP-DIRECT'S Group (2008). The SIEP-DIRECT'S Project on the discrepancy between routine practice and evidence. An outline of main findings and practical implications for the future of community based mental health services. Epidemiologia e Psichiatria Sociale 17, 358–368. [PubMed] [Google Scholar]
- Ruggeri M, Bonetto C, Lasalvia A, De Girolamo G, Fioritti A, Rucci P, Santonastaso P, Neri G, Pileggi F, Ghigi D, Miceli M, Scarone S, Cocchi A, Torresani S, Faravelli C, Zimmermann C, Meneghelli A, Cremonese C, Scocco P, Leuci E, Mazzi F, Gennarelli M, Brambilla P, Bissoli S, Bertani ME, Tosato S, De Santi K, Poli S, Cristofalo D, Tansella M (2012). A multi-element psychosocial intervention for early psychosis (GET UP PIANO TRIAL) conducted in a catchment area of 10 million inhabitants: study protocol for a pragmatic cluster randomized controlled trial. Trials 13, 73. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Salmoirago-Blotcher E, Ockene IS (2009). Methodological limitations of psychosocial interventions in patients with an implantable cardioverter-defibrillator (ICD): a systematic review. BMC Cardiovascular Disorders 9, 56. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schwartz D, Lellouch J (1967). Explanatory and pragmatic attitudes in therapeutical trials. Journal of Chronic Disease 20, 637–648. [DOI] [PubMed] [Google Scholar]
- Slade M, Priebe S (2001). Are randomised controlled trials the only gold that glitters? British Journal of Psychiatry 179, 286–287. [DOI] [PubMed] [Google Scholar]
- Startup M, Jackson MC, Bendix S (2004). North Wales randomized controlled trial of cognitive behaviour therapy for acute schizophrenia spectrum disorders: outcomes at 6 and 12 months. Psychological Medicine 34, 413–422. [DOI] [PubMed] [Google Scholar]
- Thornicroft G, Wykes T, Holloway F, Johnson S, Szmukler G (1998). From efficacy to effectiveness in community mental health services. PRiSM Psychosis Study 10. British Journal of Psychiatry 173, 423–427. [DOI] [PubMed] [Google Scholar]
- Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K (2009). A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. Journal of Clinical Epidemiology 62, 464–475. [DOI] [PubMed] [Google Scholar]
- Torgerson D, Sibbald B (1998). Understanding controlled trials: what is a patient preference trial?. British Medical Journal 316, 360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zelen M (2006). Randomized consent designs for clinical trials: an update. Statistics in Medicine 9, 645–656. [DOI] [PubMed] [Google Scholar]