Evidence-Based Practice: Separating Science From Pseudoscience

Abstract

Evidence-based practice (EBP) requires that clinicians be guided by the best available evidence. In this article, we address the impact of science and pseudoscience on psychotherapy in psychiatric practice. We describe the key principles of evidence-based intervention. We describe pseudoscience and provide illustrative examples of popular intervention practices that have not been abandoned, despite evidence that they are not efficacious and may be harmful. We distinguish efficacy from effectiveness, and describe modular approaches to treatment. Reasons for the persistence of practices that are not evidence based are examined at both the individual and the professional system level. Finally, we offer suggestions for the promotion of EBP through clinical practice guidelines, modelling of scientific decision making, and training in core skills.

Burgeoning rates of mental disorder in most countries (including Canada) and a global shortage of psychiatrists¹ require judicious allocation of resources to address mental health problems.² The promotion of EBP ensures that health care yields maximum returns.³ The commitment to EBP in Canadian psychiatry was articulated in the submission made by the RCPSC to the Romanow Commission on Health Care in 2002. Requirements for psychiatry residency education from the RCPSC and the US Accreditation Council for Graduate Medical Education include competence in the provision of EBP, although they do not address how training should be delivered or outcomes should be measured.⁴

With the broad professional context of EBP in mind, we address the impact of science and pseudoscience on psychiatry. We begin by discussing key aspects of the EBP movement in mental health care. EBP involves initial assessment, followed by the selection of a treatment, and ongoing monitoring to determine the usefulness of the treatment for the patient. In this article, we focus on psychotherapy, presenting examples that are evidence based as well as those that are not (see online Supplemental Materials for a review of issues in evidence-based assessment). Finally, we examine reasons for the reluctance of many clinicians to embrace EBP and conclude with constructive strategies to promote EBP.

Evidence-Based Practice

There is a long-standing emphasis on the importance of research evidence in psychiatry. Almost a century ago, Eugen Bleuler criticized the “autistic and undisciplined thinking”5, p 1 in medicine that allowed unsubstantiated treatments to flourish. Bleuler suggested that the motivation to alleviate suffering, combined with limited knowledge, created conditions in which physicians and patients are fooled into believing that an ineffective treatment works. There is no ethically defensible argument against the need to offer treatments demonstrated to work for patients with similar problems and compatible with their needs and preferences. Still, a thorny question remains: What constitutes evidence that an intervention works?

One model for choosing psychotherapies is comparable to that used to evaluate medications: establish methodological criteria for evaluating studies, create lists of treatments that meet these criteria, and promote these treatments. This approach was adopted in the early 1990s, by task forces of the American Psychological Association’s Society of Clinical Psychology, to generate lists of ESTs.⁶ Classification as an EST required sufficient research of high quality (multiple RCTs or single-participant designs), demonstrating efficacy of the intervention in clients with a specific condition.

These initiatives underlined that efficacious psychosocial treatments for many disorders across the lifespan exist (see also Roth and Fonagy,⁷ Fonagy et al,^8,9 and Nathan and Gorman^10–13). This encouraging news has neither been embraced by some clinicians as positively as one might expect nor led to the widespread adoption of ESTs.^14,15 Critics of ESTs have expressed concerns about the scientific soundness of the endeavour and its potential negative impact on clinicians who do not offer ESTs.¹⁶ They noted that a treatment with a positive, but limited, evidence base could fall short of EST criteria. The EST initiative was an important step in the development and promotion of EBP, but it is not synonymous with EBP. Originating in Canada and the United Kingdom, evidence-based medicine is based on the premise that the application of empirical knowledge improves patient care.³ This movement spread to the United States¹⁷ and other countries. The term evidence based entails the synthesis of information from a wide array of sources to guide clinicians to select the best available treatment options.¹⁷ Training in EBP encompasses attention to the content of the evidence base and to the process of evidence-based decision making.⁴ Sources of information typically considered in EBP include systematic data, clinical expertise, and patient preferences.

Two North American psychological associations created task forces to operationalize evidence-based psychotherapies. The American Psychological Association Presidential Task Force on Evidence-Based Practice¹⁸ defined EBP as the integration of the best available research and clinical expertise within the context of client characteristics, culture, values, and treatment preferences. The assertion that treatment should be informed by research but determined by other clinical information, client choice, and the likely costs and benefits of treatments has sometimes been interpreted as suggesting that research evidence is the least important consideration in EBP.¹⁹ Additionally, the American Psychological Association’s statement was silent on the need to ensure that clients’ views were based on accurate rather than inaccurate information. Clients often request a treatment they have heard about from a friend or read about online, unaware that it is not backed by scientific evidence. In such situations, clinicians bear a responsibility to educate their clients to ensure that their choices are informed by sound knowledge.

Clinical Implications

• EBP is an essential safeguard against pseudoscience and potentially harmful assessment and treatment methods.

• Even interventions that appear to be plausible can be ineffective or even iatrogenic.

Limitations

• It can be difficult to distinguish between science and some forms of pseudoscience.

• Research on the effectiveness of training psychiatrists in EBP is still in its infancy.

The Canadian Psychological Association Task Force on Evidence-Based Practice of Psychological Treatments²⁰ asserted that EBP should rely, first and foremost, on peer-reviewed research. The task force recognized the potential contribution of diverse methodologies to yield evidence to guide practice, but established a hierarchy in which the strongest weight is given to research that is based on data from the highest-quality research designs and that has been scrutinized by peer review. Clinicians are exhorted to apply their knowledge of the best available research, considering client characteristics, cultural backgrounds, and treatment preferences. Further, they are expected to continuously monitor the effects of treatment and to adjust treatment when appropriate. The monitoring of treatment effects is a key element of EBP and has been shown to have a substantial positive impact on treatment outcomes.²¹ Although surveys indicate that mental health professionals underuse patient-report measures when making decisions about clinical services,^22,23 there are widely available sources that summarize psychometrically sound instruments that can be used for treatment planning and monitoring.^24,25 Additionally, recent initiatives in the United States have led to the widespread availability of psychometrically sound, patient-reported health status and social well-being measures that can also be used for service delivery decisions (see Patient Reported Outcomes Measurement Information System [PROMIS] Network²⁶).

Science, Compared With Pseudoscience

With the move toward scientifically supported practice, one commonly sees monikers, such as scientifically proven or evidence-informed, attached to descriptions of treatments. The cachet of scientific support is a potent marketing tool. Lilienfeld et al²⁷ viewed science and pseudoscience as existing along a continuum. Features of pseudoscience include the following:

1. The overuse of ad hoc hypotheses to account for negative research findings.

2. Avoidance of peer review.

3. Emphasis on confirmation rather than refutation.

4. Lack of connection with basic or applied research.

5. Overreliance on anecdotal evidence.

6. Reversed burden of proof in which proponents of a technique demand that critics refute claims of treatment efficacy.

The greater the number of such features, the more likely a practice is based on pseudoscience rather than science.

TFT, a treatment applied to mood, anxiety, and trauma-related disorders, is a prime example of practice founded on pseudoscience. TFT is based on the premise that bodily energy imbalances cause negative emotions. Treatment is purported to rectify imbalances by tapping on acupuncture meridians.²⁸ Virtually no peer-reviewed research supports this treatment rationale. With only methodologically weak reports available in the literature, the so-called science cited to support TFT is primarily anecdotal and does not rule out placebo effects.^29–31 Despite these criticisms, the TFT website³² continues to advance unsupported claims about TFT’s ability to cure almost any emotional problem.

Clinicians may find it difficult to resist the pull of anecdotal evidence, such as unsystematic clinical observations, in decision making. Dramatic changes in patient functioning following an intense session may appear to provide more compelling evidence of a treatment’s impact than do the results of RCTs or meta-analyses. Although anecdotal evidence can inform hypotheses to be evaluated systematically, such evidence should not be equated with scientific data. Without the controls afforded by scientific practices and scientific thinking, unsystematic clinical observations can lead to erroneous conclusions about the value of a clinical procedure. Lilienfeld et al^33,34 enumerated over 2-dozen cognitive errors that can contribute to spurious therapeutic effectiveness. These include naive realism (for example, “If I saw it, it must be true”), confirmation bias (looking only for evidence that supports an hypothesis and failing to look for evidence that refutes it), illusory correlation (the co-occurrence of 2 events or characteristics does not mean that they are linked), and illusory causation (the co-occurrence of 2 events or characteristics does not mean that one causes the other). Clearly, it is insufficient to think scientifically only when conducting research: one must think scientifically in all aspects of one’s professional activities, including clinical services.

Evidence-Based Treatment

Prior to the 1960s, few studies had evaluated psychotherapy outcome, but, since the 1970s, thousands of studies have filled this gap. There are now evidence-based treatments for many mental health disorders in children, adolescents, and adults.^9,12 However, less emphasis has been placed on the need to desist from delivering interventions that are inefficacious or harmful.

For example, media reports of natural and human-made tragedies almost inevitably end with the statement that counsellors will be available on site to assist. Given the woeful shortage of services for mental health problems, it is striking that such services are commonly offered.³⁵ The rationale for postdisaster debriefing is intuitively appealing. If one could protect people from developing posttraumatic stress disorder by devoting a few hours to hearing their stories, it would be time well invested. Nevertheless, literature reviews do not support the efficacy of single or multiple sessions of debriefing and suggest that it can sometimes be harmful, as it may impede natural recovery processes.^36,37

Another program based on popular belief is Scared Straight. Prison visits are designed to deter youth on a trajectory toward delinquency by arousing fear about the life they could expect if they engaged in crime.³⁸ Fuelled by anecdotal reports suggesting high success rates, Scared Straight was widely implemented in the United States. Echoing the pattern found for debriefing, reviews of RCTs provided no evidence that Scared Straight or other juvenile awareness programs are efficacious; to the contrary, they boost the odds of future offending.³⁹

An important distinction in treatment outcome research is that between efficacy and effectiveness.⁴⁰ Efficacy refers to evidence of treatment effects obtained in controlled research, whereas effectiveness refers to evidence of treatment effects as evaluated in the real world. Skeptics of EBP argue that findings from efficacy trials, often conducted in academic training centres, have limited applicability to services delivered in the exigencies of real clinical practice, in which patients have multiple diagnoses and severe problems.¹⁵ Nevertheless, Jensen-Doss and Weisz⁴¹ found no evidence of poorer treatment effects in youth with multiple problems. Kazdin and Whitley⁴² found that comorbidity was associated with greater change in young people with disruptive behaviour disorders who received evidence-based parent training or problem-solving treatments. These findings suggest that evidence-based services are useful to people with multiple problems.

Using a benchmarking strategy, in which results of efficacy trials are used as a point of comparison, Hunsley and Lee⁴³ found that effectiveness studies of evidence-based treatments for adults reported comparable completion and improvement rates to those obtained in efficacy trials. A meta-analysis of effectiveness studies of cognitive-behavioural therapy for adult anxiety disorders similarly found that the mean treatment outcome in over 50 studies was consistent with results from efficacy studies.⁴⁴ Lee et al⁴⁵ examined treatment effectiveness studies in regular clinical settings for children and adolescents. Across studies, more than 75% of patients completed the services. Improvement rates for internalizing problems were comparable to those reported in efficacy trials. There was greater variability in outcomes for parenting interventions for disruptive behaviour problems, with several studies yielding superior results compared with the benchmark, and a smaller number yielding poorer results. Overall, benchmarking studies indicate that evidence-based treatments for children and youth can be effective in routine practice settings.

Critics of EBP have raised concerns that the use of treatment manuals to guide services interferes with the therapeutic alliance. To test this hypothesis, Langer et al⁴⁶ randomly assigned youth with internalizing disorders in community clinics to receive manualized services or nonmanualized usual care. Observer ratings revealed that early on in the process, youth receiving manualized services had a stronger therapeutic alliance than did youth receiving usual care. By the end of services, there were no significant group differences. Although this study requires replication, it is noteworthy that there was no evidence that manualized services impaired the therapeutic relationship; instead, these services were associated with a stronger early alliance.

One pressing question in the delivery of EBP is to identify active treatment ingredients.⁴⁷ Chorpita and Daleiden⁴⁸ analyzed the practice elements in 322 RCTs of treatments for children and adolescents. They identified constellations of strategies consistently found to be helpful for specific problems. For example, for anxiety, treatments commonly included modules for exposure, relaxation, cognitive interventions, modelling, and psychoeducation. The authors recommended that clinicians individualize treatments by integrating treatment modules that best address a client’s problems.

The usefulness of a modular approach to the treatment for depression, anxiety, and conduct problems in people aged 7 to 13 was examined by Weisz et al.⁴⁹ Community clinicians were randomly assigned to 1 of the following: treatment as usual; an evidence-based treatment for anxiety, depression, or conduct problems; or a modularized approach in which clinicians flexibly delivered modules that integrated the 3 evidence-based approaches as needed. Modular treatment outperformed both usual care and a single problem-focused, evidence-based treatment, suggesting that this approach holds promise for treatments delivered to youth.

How Can We Understand Persistence in Offering Services That Are Not Evidence Based?

From the treatment of scurvy, to routine handwashing to prevent infection, the history of health care is replete with examples of long delays between the demonstration that a practice is efficacious and its adoption in routine care.⁵⁰ Similarly, clinicians do not always abandon a mental health service that is ineffective or harmful. To understand why clinicians remain loyal to unsupported or iatrogenic practices and why they do not embrace efficacious practices, we must consider both individual and contextual variables. Individual-level variables include challenges in keeping up with the scientific literature, perceived difficulties in accessing training, and attitudes toward life-long learning in general and in EBP in particular.⁵¹ Contextual variables include the culture of the profession,⁴ training in EBP, and institutional and professional support for EBP.⁵²

Keeping up with the literature requires clinicians to formulate a question related to patient care, seek the best research evidence that addresses the question, critically appraise the validity and applicability of research to the patient, apply the information, and systematically evaluate treatment effects.⁵³ Further, it requires recognition that the evidence base is constantly evolving, thus a practice that is currently the best available may later become obsolete or require modification. With the proliferation of efficacious treatments, psychiatrists may feel overwhelmed, believing that they require proficiency in numerous treatments to serve their patients’ needs. Clinicians who are convinced that their current practice is efficacious may not perceive a need to devote energy and time to achieve competence in new treatments. Clinicians may be most reluctant to consider new interventions that are grounded in a different theoretical framework than that in which they were trained.

The culture of a profession can influence clinicians’ openness to EBP. The antithesis of EBP is practice based on tradition and authority (facetiously dubbed eminence-based practice⁵⁴) in which recommendations are accepted because the person delivering them is regarded as an expert. Unless backed by high-quality evidence, the opinions of recognized experts are just that—opinions.⁵⁵ An apprenticeship model of training provides fertile ground for eminence-based practice, in which the views of charismatic or authoritative supervisors are adopted, regardless of their evidence base. Recent discussion in The Canadian Journal of Psychiatry has examined the susceptibility of psychiatry to fads that arouse enthusiasm but are later abandoned.⁵⁶ Some commentators (for example, see Goldberg⁵⁷) have suggested that the tradition in psychiatry to convene expert panels to offer pronouncements about practice can be problematic. Goldberg warned that psychiatrists must be aware of the risks of abandoning practices based on expert consensus rather than research evidence. The dialectic between the promotion of hope and certainty in helping professions and the need for skepticism in science is especially acute in psychiatry.⁵⁶

Responsibility also rests with researchers to ensure their findings are presented accurately. In a review of the American Association for the Advancement of Science’s online coverage of press releases in all areas of science, Yavchitz et al⁵⁸ found an unwarranted emphasis on the benefits of experimental treatments (such as focusing only on findings showing a treatment effect) in 40% of journal abstracts, 47% of press releases, and 51% of news items based on press releases. In most instances, the spin in press releases and media reports originated in journal abstracts, suggesting that researchers themselves are responsible for at least some of the overselling of findings. Gonon et al⁵⁹ examined media reports of influential studies on attention-deficit hyperactivity disorder. Most findings reported in newspapers during the 1990s were not replicated, with only 2 of the 10 most cited studies supported by later investigations. Only one newspaper article reported that the findings stemming from one of the most commonly mentioned studies were not replicated.

The practising psychiatrist faces a daunting task in offering EBP. The Draft CanMEDS 2015 Milestones Guide—May 2014⁵³ highlights that a physician must “integrate best available evidence, contextualized to specific situations, into real-time decision-making.”^{p 38, 42, 46} The foundation for critical appraisal skills is to be established in medical school, with demonstration of these skills by the time of transition to practice. The steps to acquisition of competence in formulating testable hypotheses about a patient, consulting the relevant scientific literature and appraising its relevance, and ongoing monitoring of the usefulness of a practice are elaborated in broad brushstrokes. However, the Guide offers few specific suggestions, so it is unclear how training assists in the development of this competence. Nevertheless, psychiatrists require training that promotes a scientific approach to decision making, access to research syntheses, and an environment supportive of EBP.

Constructive Strategies to Promote Use of Evidence-Based Practices

Given the rapid evolution of the knowledge base, psychiatric residents must not only master current knowledge but also learn strategies for continual questioning and updating of knowledge in a process of life-long learning. Our recommendations focus on strategies that create a context favourable for EBP.

Promoting Use of Clinical Practice Guidelines

The Draft CanMEDS 2015 Milestones Guide—May 2014⁵³ specifies that by their transition to practice, physicians must be competent in identifying, selecting, and navigating pre-appraised evidence. Psychiatrists have access to many high-quality EBP databases, including the Canadian Psychiatric Association Clinical Practice Guidelines,⁶⁰ American sites, such as the American Psychiatric Association Clinical Practice Guidelines,⁶¹ Evidence-Based Behavioral Practice,⁶² and the Society for Clinical Psychology’s site on psychological treatments,⁶³ and sites based in the United Kingdom, including the National Institute for Health and Care Excellence.⁶⁴

Despite the wealth of practice guidelines based on research syntheses, the extent to which clinicians access these resources is unknown. A survey⁶⁵ found that psychologists were moderately knowledgeable about general research databases, such as PsycINFO and MEDLINE, but were much less knowledgeable about online databases that provide integrated information on evidence-based health and mental health services. Slightly less than one-half of respondents reported feeling competent with at least one evidence-based database. Clinicians will require training in the use of databases to allow for their ongoing integration into practice. We recommend that case presentations, which are routinely used for training and consultation, should include information not only about the patient but also about how the clinician searched for and selected an intervention.⁶⁶

Modelling of Scientific Reasoning in Clinical Decision Making

In the apprenticeship model of training in psychiatry, residents consolidate their didactic learning by applying knowledge in the clinical context. Peers and supervisors model how they generate hypotheses about patients, search the literature for evidence, and monitor the usefulness of their practice.⁴ The adoption of EBP requires a commitment to a humble stance in which testable hypotheses guide a search for the best evidence, and ongoing monitoring informs the usefulness of an intervention. In this model, the capacity to generate questions, search the literature, and critically appraise evidence is more important than is knowledge of the literature.

Training in Core Skills

An increasing number of online tools have been developed to assist in EBP training. The Psychotherapy Training e-Resources based at McMaster University are designed for training and continuing education of mental health professionals.⁶⁷ The PracticeWise site⁶⁸ offers a database on evidence-based services for children and youth, as well as practice guides and tools for treatment plans and tracking progress. Expertise in using scientifically sound assessment tools, to both develop treatment plans and monitor treatment progress, is essential for true EBP. Because no clinician can expect to master all evidence-based protocols for all disorders, there is growing interest in identifying core treatment skills that can be flexibly applied to different problems. Although there was a time when EBP was largely synonymous with cognitive-behavioural therapy, interventions from other theoretical frameworks (for example, interpersonal psychotherapy and short-term psychodynamic therapies) are also efficacious for certain conditions.^69,70

Final Comment

The goal of psychiatric services is to provide the best treatment possible to people with mental disorders. Attending to the research evidence in planning these services and systematically monitoring their impact, provides the optimal means of achieving this goal. For this goal to be realized, clinicians must recognize not only the value of disciplined scientific inquiry but also that clinical judgment should be continually informed by such inquiry. By developing the attitudes, values, and skills consistent with EBP principles, people in training to provide psychiatric services will be well-placed to deliver the best services possible.

Abbreviations

EBP

evidence-based practice

EST

empirically supported treatment

RCT

randomized controlled trial

RCPSC

Royal College of Physicians and Surgeons of Canada

TFT

Thought Field Therapy