Abstract
The field of research and practice in psychotherapy has been deeply influenced by two different approaches: the empirically supported treatments (ESTs) movement, linked with the evidence-based medicine (EBM) perspective and the “Common Factors” approach, typically connected with the “Dodo Bird Verdict”. About the first perspective, since 1998 a list of ESTs has been established in mental health field. Criterions for “well-established” and “probably efficacious” treatments have arisen. The development of these kinds of paradigms was motivated by the emergence of a “managerial” approach and related systems for remuneration also for mental health providers and for insurance companies. In this article ESTs will be presented underlining also some possible criticisms. Finally complementary approaches, that could add different evidence in the psychotherapy research in comparison with traditional EBM approach, are presented.
Keywords: evidence based medicine, evidence based psychology, empirically supported treatments, psychotherapy research, common factors
Between “Common Factors” and “Empirically Supported Treatments” Perspectives
The fundamental questions for psychotherapy researchers have historically concerned the real effectiveness of psychotherapeutic treatments. The reply to this problem has been largely positive (Smith et al., 1980; Lambert and Bergin, 1994; Lambert, 2005) with some notable exceptions (Eysenck, 1952, 1961).
Then the focus of attention in research moved from a general demonstration of the effectiveness of psychotherapy to the particular examination, identification, and classification of specific treatments which have been shown to be effective in experimental settings for generally recognized psychopathologies. The question came back to “which treatment, prescribed by whom, and in which circumstances, is the most effective for this particular individual with this specific problem?’ (Paul, 1967, p. 111).
Among researchers and clinicians looking for significative results in psychotherapy research, two were the widespread approaches: on the one hand the “common factors perspective”, theories confirming the so-called “Dodo Solution” (Luborsky and Singer, 1975; Luborsky et al., 2002; Lambert, 2005), on the other hand the development of so-called “Empirically Supported Treatments”, which are based on the “Evidence Based Medicine” philosophy (Herbert, 2003; Morrison et al., 2003; Karver et al., 2005; Joyce et al., 2006).
In Lambert's opinion (2005), “common factors are those dimensions of the treatment setting (therapist, therapy, client) that are not specific to any particular technique. Research on the broader concept of common factors investigates causal mechanisms such as expectation for improvement, therapist confidence, and a therapeutic relationship that is characterized by trust, warmth, understanding, acceptance, kindness, and human wisdom. But also can be expanded to include some mechanisms that are often regarded as unique to a particular form of treatment such as exposure to anxiety-provoking stimuli, encouragement to participate in other risk-taking behavior (facing rather than avoiding situations that make the patient uncomfortable), and encouraging client efforts at mastery such as practicing and rehearsing behaviors. Such a view of common factors recognizes that while specific theories of psychotherapy may emphasize systematic in vivo or in vitro exposure to frightening situations, or social-skills training, nearly all therapies encourage people to review and discuss the things they fear and face rather than avoid such situations. Common factors, no matter how unimportant they may be from the point of view of a particular theory (theoretically inert or trivial) are central to nearly all psychological interventions in practice, if not, theory” (p. 856, 2005).
For a more in-depth analysis of “common factors perspective” please consider (Castelnuovo et al., 2004b, 2005). One comprehensive example of application of this approach is the Karver et al. (2005) model.
Evidence-Based Approach in Psychotherapy
The validation of treatments which have proved effective (empirically supported treatments, ESTs) has obtained power and credibility due to the a convergence of political economics and professional forces: Managed Health Care, developments in biological psychiatry, protocols of the American Psychological Association (APA), guidelines for accreditation within doctorate programs in clinical psychology (Lyddon and Chatkoff, 2002).
To date, the effectiveness of psychotherapy for the functional treatment of many mental disorders is empirically well demonstrated. Using an ‘‘evidence-based medicine’’ viewpoint, psychotherapy, in comparison to other medical treatments such as pharmacology, can be considered as one of the most effective therapeutic approaches (Schnyder, 2009).
Furthermore, the National Institute of Mental Health (NIMH), an important source of finance for psychotherapy research, has decided to apply the methodology used in pharmaceutical research to evaluate psychotherapy, with the development of random and controlled clinical trials (RCTs) (Duncan, 2002; Castelnuovo et al., 2004b, 2005).
About the use of randomized controlled trials (RCTs), considered as the best standard in the psychotherapy research, there is a lot of criticisms by clinical practitioners. An interesting description of the main features of this methodological procedure and of the relative critical areas has been carried out by Starcevic (2003) and his interesting analysis is reported in Table 1.
Table 1.
Features of RCTs and relative criticisms (source: Starcevic, 2003).
| Features of RCTs | Criticisms of RCTs |
|---|---|
| ‘‘Randomized controlled trials’’ (RCTs) are a methodological procedure that consists in the comparison of the group of patients in whom the usefulness of treatment is being examined (experimental group) with the group of patients who are receiving no active treatment (e.g., a placebo; control group); | In the psychotherapy studies there is no counterpart to a placebo that is used in the pharmacotherapy studies. The non-specific (and presumably placebo-like) psychological treatments, administered to patients in control groups, are not ‘neutral’ in the way that placebo is pharmacologically ‘neutral’ because they produce psychological effects, regardless of whether these are clinically significant. |
| RCTs focus on strict diagnostic homogeneity of the groups of patients and give emphasis on diagnostic precision; | Psychiatric diagnosis is usually not the main factor that determines the use of psychotherapy and diagnostic precision is not emphasized in psychotherapy. As a result, psychotherapy patients are not as diagnostically homogeneous as patients in RCTs and often have additional disorders that would exclude them from RCTs. |
| Another key feature is represented by the randomization into experimental and control groups of patients; | Randomizing patients in the psychotherapy usefulness studies is troublesome because clinical practice is not randomized; also, randomization creates an artificial situation because it ignores the fact that psychotherapy patients actively choose their own treatments. |
| It is important to carry on double-blind design of research; | A double-blind design is impossible in psychotherapy research. Patients cannot be blind as to what psychological treatment they are receiving because they actively participate in it; likewise, therapists cannot be blind because they know what treatments they administer. |
| Another ‘‘gold standard’’ of the RCTs procedure is the use of standardization of treatment procedures, so that all patients receive (or are presumed to receive) treatment in the same way. | Psychotherapy is extremely difficult to ‘standardize’ so that its procedures and techniques are used in the same way by all the therapists, regardless of their training and personality. Every encounter between the patient and the therapist has some unique features, with the potential of producing ‘something’ that cannot be predicted and entirely ‘standardized’. |
On the other hand Division 12 (Clinical Psychology) of the APA has set up a task force for the promotion and diffusion of treatment protocols which are proven experimentally effective (Task Force on Promotion and Dissemination of Psychological Procedures, 1995; Chambless and Ollendick, 2001). The work of this task force has resulted in the drawing up a minimum level of criterion required to validate as “effective” a given psychotherapy, with a list of experimentally proven treatments which have been approved or rejected (Chambless et al., 1996; Chambless and Hollon, 1998; Chambless and Ollendick, 2001). Thus criteria established by the task force have not the goal to verify the clinical effectiveness of a treatment, based on the external validity of a therapy, but only the experimental effectiveness of a treatment or the validity of experimental results (Lyddon and Chatkoff, 2002).
Criteria for Empirically Validated Psychotherapies
Before highlighting the criticisms of the evidence-based model let us consider the criteria for selection of empirically validated psychotherapies (Table 2). These comprehensive criteria are freely available on the Internet in their previous 1997 version (for example on the APA's official website1).
Table 2.
Workgroup criteria for identification of empirically supported therapies (source: Chambless and Ollendick, 2001).
| DIVISION 12 TASK FORCE CRITERIA (CHAMBLESS ET AL., 1998), GROUP A |
| Well-established treatments |
| I. At least two good between-group design experiments must demonstrate efficacy in one or more of the following ways: |
| A. Superiority to pill or psychotherapy placebo, or to other treatment |
| B. Equivalence to already established treatment with adequate sample sizes |
| OR |
| II. A large series of single-case design experiments must demonstrate efficacy with |
| A. Use of good experimental design and |
| B. Comparison of intervention to another treatment |
| III. Experiments must be conducted with treatment manuals or equivalent clear description of treatment |
| IV. Characteristics of samples must be specified |
| V. Effects must be demonstrated by at least two different investigators or teams |
| Probably efficacious treatments |
| I. Two experiments must show that the treatment is superior to waiting-list control group |
| OR |
| II. One or more experiments must meet well-established criteria IA or IB, III, and IV above but V is not met |
| OR |
| III. A small series of single-case design experiments must meet well-established-treatment criteria |
| Experimental treatments |
| Treatment not yet tested in trials meeting task force criteria for methodology |
| SPECIAL SECTION OF JOURNAL OF PEDIATRIC PSYCHOLOGY (SPIRITO, 1999) CRITERIA, GROUP B |
| Well-established treatments |
| Same as Chambless et al. (1998) |
| Probably efficacious treatments |
| Same as Chambless et al. (1998) |
| Promising interventions |
| I. There must be positive support from one well-controlled study and at least one other less-well-controlled study |
| OR |
| II. There must be positive support from a small number of single-case design experiments |
| OR |
| III. There must be positive support from two or more well controlled studies by the same investigator |
| SPECIAL SECTION OF JOURNAL OF CLINICAL CHILD PSYCHOLOGY (1998, VOL. 27, NO. 2) CRITERIA, GROUP C |
| Well-established treatments |
| Same as Chambless et al. (1998) |
| Probably efficacious treatments |
| Same as Chambless et al. (1998) except: |
| There must be at least two, rather than one, group design studies meeting criteria for well-established treatments if conducted by the same investigator |
| SPECIAL SECTION OF JOURNAL OF CONSULTING AND CLINICAL PSYCHOLOGY (KENDALL AND CHAMBLESS, 1998) CRITERIA, GROUP D |
| Efficacious and specific |
| Same as Chambless et al. (1998) for well-established treatments |
| Possibly efficacious and specific treatments |
| Same as efficacious and specific above except: Treatment only required to be found superior to rival treatment in one study |
| Efficacious and possibly specific treatments |
| Same as efficacious and specific criteria above except: Treatment was found superior to wait-list group in one study and superior to rival treatment in another study by a different team |
| Efficacious treatments |
| Same as Chambless et al. (1998) for well-established treatments except: Treatment must be demonstrated to be better than no treatment but not been shown to be better than non-specific intervention, placebo, or rival intervention |
| Possibly efficacious treatments |
| Same as Chambless et al. (1998) for probably efficacious treatments |
| WHAT WORKS FOR WHOM? (ROTH AND FONAGY, 1996) CRITERIA, GROUP E |
| Clearly effective treatments |
| I. There must be a replicated demonstration of superiority to a control condition or another treatment condition |
| OR |
| II. There must be a single high-quality randomized control trial in which: |
| A. Therapists followed a clearly described therapeutic method useable as the basis for training |
| B. There is a clearly described patient group |
| Promising limited-support treatments |
| Treatment must be innovative and a promising line of intervention |
| OR |
| Treatment is a widely practiced method with only limited support for effectiveness |
| A GUIDE TO TREATMENTS THAT WORK (NATHAN AND GORMAN, 1998) CRITERIA, GROUP F |
| Type 1 studies |
| I. Study must include a randomized prospective clinical trial |
| II. Study must include comparison groups with random assignment, blind assessments, clear inclusion and exclusion criteria, state-of-the-art diagnostic methods, and adequate sample size for power |
| III. There must be clearly described statistical methods |
| Type 2 studies |
| Clinical trials must be performed, but some traits of type-1 study were missing (e.g. trial with no double blind or group assignment not randomized) |
| Type 3 studies |
| I. These are open treatment studies that are aimed at obtaining pilot data |
| OR |
| II. These are case control studies in which treatment information was obtained retrospectively |
| TREATMENTS FOR OLDER ADULTS (GATZ ET AL., 1998) CRITERIA, GROUP G |
| Same as Chambless et al. (1998) criteria |
| TREATMENTS FOR CHRONIC PAIN (WILSON AND GIL, 1996) CRITERIA, GROUP H |
| Same as Chambless et al. (1998) criteria |
A list of ESTs (for adults) according to the criteria outlined in the previous Table 2 is reported in Table 3. For a previous version of the Well-Established Treatments and Probably Efficacious Treatments consider Chambless and Hollon (1998) and Castelnuovo et al. (2005).
Table 3.
Empirically supported treatments for adults: a summary across workgroups (source: Chambless and Ollendick, 2001).
| Category of empirical support (b, c) | |||
|---|---|---|---|
| Condition and treatment (a) | I | II | III |
| ANXIETTY AND STRESS | |||
| Agoraphobia/panic disorder with agoraphobia | |||
| CBT | A, E?, F | E? | |
| Couples communication training as adjunct to exposure | A, D | ||
| Exposure | A, D, E?, F | E? | |
| Partner-assisted CBT | D, F | ||
| Blood injury phobia | |||
| Applied tension | F | E | |
| Exposure | E | ||
| Generalized anxiety disorder | |||
| Applied relaxation | F | A, D, E | |
| CBT | A, D, E?, F | E? | |
| Geriatric anxiety | |||
| CBT | F, G | ||
| Relaxation | F | ||
| Obsessive-compulsive disorder | |||
| ERP | A, D, E?, F | E? | |
| Cognitive therapy | A, D | E | |
| RET C exposure | E | ||
| Family-assisted ERP C relaxation | D | ||
| Relapse prevention | A | ||
| Panic disorder | |||
| Applied relaxation | F | A, D, E | |
| CBT | A, D, E?, F | E? | |
| Emotion-focused therapy | F | ||
| Exposure | E? | D, E? | |
| Post-traumatic stress disorder | |||
| EMDR | A (civilian only), D | ||
| Exposure | F | A, D | |
| Stress inoculation | F | A, D | |
| Stress inoculation in combination with CTC exposure | E? | E?, F | |
| Structured psychodynamic treatment | E | ||
| Public-speaking anxiety | |||
| Systematic desensitization | A | ||
| Social anxiety/phobia | |||
| CBT | E?, F | A, D, E? | |
| Exposure | E?, | A, D, E?, F | |
| Systematic desensitization | A | ||
| Specific phobia | |||
| Exposure | A, E?, F | E? | |
| Systematic desensitization | A | ||
| Stress | |||
| Stress inoculation | A | ||
| CHEMICAL ABUSE AND DEPENDENCE | |||
| Alcohol abuse and dependence | |||
| Community reinforcement | E?, F? | A, D, E?, F? | |
| Cue exposure therapy | A, D | ||
| Cue exposure therapy C urge-coping skills | D | ||
| Cue exposure with inpatient treatment | A | ||
| Motivational interviewing | E? | E? | |
| BMT C disulfiram | E?, F? | A, D, E?, F? | |
| Social-skills training with inpatient treatment | E?, F? | A, D, E?, F? | |
| Benzodiazepine withdrawal for panic disorder | |||
| CBT | A | ||
| Cocaine abuse | |||
| Behavior therapy | A | ||
| CBT relapse prevention | A, D | ||
| Opiate dependence | |||
| Behavior therapy (reinforcement) | D | ||
| Brief dynamic therapy | A, D | ||
| CT | A, D | ||
| DEPRESSION | |||
| Bipolar Disorder | |||
| Psychoeducation | F | ||
| CBT for medication adherence | F | ||
| Family Therapy | F | ||
| Geriatric depression | |||
| Behavior therapy | E?, F | E?, G | |
| Brief psychodynamic therapy | E?, F | E?, G | |
| CBT | E?, F | A, E?, G | |
| Interpersonal therapy | F | ||
| Problem-solving therapy | F, G | ||
| Psychoeducation | F | ||
| Reminiscence therapy (mild–moderate) | F | A, G | |
| Major depression | |||
| Behavior therapy | A, F | D | |
| BMT (for those with marital discord) | F | D | |
| Brief dynamic therapy | A | E | |
| CBT | A, D, E?, F | E? | |
| Interpersonal therapy | A, E?, F | D, E? | |
| Self-control therapy | A, F | ||
| Social problem solving | A, D | ||
| HEALTH PROBLEM | |||
| Anorexia | |||
| Behavior therapy | E? | E? | |
| BFST | F | ||
| CT | E? | E? | |
| Family therapy | F | ||
| Binge-eating disorder | |||
| Behavioral weight control | F | ||
| CBT | F | A | |
| Interpersonal therapy | A, F | ||
| Bulimia | |||
| CBT | A, E?, F | D, E? | |
| Interpersonal therapy | E? | A, D, E?, F | |
| Cancer pain | |||
| CBT | H | ||
| Chemotherapy side effects (for cancer patients) | |||
| Progressive muscle relaxation with or without guided imagery | D | ||
| Chronic pain (heterogeneous) | |||
| CBT with physical therapy | A, D, H | ||
| EMG biofeedback | A | ||
| Operant behavior therapy | A, D | ||
| Chronic pain (back) | |||
| CBT | H | A, D | |
| Operant behaviour therapy | D | ||
| Headache | |||
| Behavior therapy | A | ||
| Idiopathic pain | |||
| CBT | H | ||
| Irritable-bowel syndrome | |||
| CT | A, D | ||
| Hypnotherapy | D | ||
| Multicomponent CBT | A, D | ||
| Migraine | |||
| EMG biofeedback C relaxation | D | ||
| Thermal biofeedback C relaxation training | A, D | ||
| Obesity | |||
| Hypnosis with CBT | A | ||
| Raynaud's | |||
| Thermal biofeedback | A | ||
| Rheumatic disease pain | |||
| Multicomponent CBT | A, D, H | ||
| Sickle cell disease pain | |||
| Multicomponent CT | A | ||
| Smoking cessation | |||
| Group CBT | D | ||
| Multicomponent CBT with relapse prevention | A, D | ||
| Scheduled reduced smoking with multicomponent behavior therapy | A, D | ||
| Somatoform pain disorders | |||
| CBT | F | ||
| Marital discord | |||
| BMT | A, D | ||
| CBT | D | ||
| CT | D | ||
| Emotion-focused couples therapy | A (no more than moderately distressed), D | ||
| Insight-oriented marital therapy | A, D | ||
| Systemic therapy | D | ||
| SEXUAL DYSFUNCTION | |||
| Erectile dysfunction | |||
| Behavior therapy aimed at reducing sexual anxiety and improving communication | E? | E? | |
| CBT aimed at reducing sexual anxiety and improving communication | E? | E? | |
| Female hypoactive sexual desire | |||
| Hurlbert's combined therapy | A, D | ||
| Zimmer's combined sex and marital therapy | A, D | ||
| Female orgasmic disorder/ dysfunction | |||
| BMT with Masters and Johnson's therapy | D | ||
| Masters and Johnson's sex therapy | A, D | ||
| Sexual-skills training | D | ||
| Premature ejaculation | |||
| Behavior therapy | E | ||
| Vaginismus | |||
| Exposure-based behavior therapy | E? | E? | |
| OTHER | |||
| Avoidant personality disorder | |||
| Exposure | F | ||
| Social-skills training | E? | E?, F | |
| Body dysmorphic disorder | |||
| CBT | F | ||
| Borderline personality disorder | |||
| Dialectical behavior therapy | E? | A, E?, F | |
| Psychodynamic therapy | F | ||
| Dementia | |||
| Behavioral interventions applied at environmental level for behavior problems | G | ||
| Memory and cognitive retraining for slowing cognitive decline | G | ||
| Reality orientation | G | E | |
| Geriatric care givers’ distress | |||
| Psychoeducation | G | ||
| Psychosocial interventions | E? | E? | |
| Hypochondriasis | |||
| CBT | F | ||
| Paraphilias/Sex offenders | |||
| Behavior therapy | A | ||
| CBT | F | ||
| Schizophrenia | |||
| Assertive case management | F | ||
| Behavior therapy and social learning/token economy programs | F | ||
| Clinical case management | F | ||
| CT (for delusions) | E, F | ||
| Behavioral family therapy | D, E?, F | A, E? | |
| Family systems therapy | D | ||
| Social-learning programs | F | ||
| Social-skills training | F | A, D | |
| Supportive group therapy | F | ||
| Supportive long-term family therapy | D | ||
| Training in community living program | F | ||
| Severely mentally ill | |||
| Supported employment | A, F | ||
| Sleep disorders | |||
| Behavior therapy | F | ||
| CBT (for geriatric sleep disorders) | G | ||
| Unwanted habits | |||
| Habit reversal and control techniques | A | ||
a: CBT, cognitive behavior therapy; BMT, behavioral marital therapy; ERP, exposure plus ritual prevention; BFST, behavioral family systems therapy; EMDR, eye movement desensitization and reprocessing; CT, cognitive therapy; EMG, electromyographic.
b: Category I, well-established/efficacious and specific/two type-1 studies; Category II, probably efficacious/efficacious/or possibly efficacious/one type-1 study; Category III, promising/type-2 or -3 studies. Only Groups B, E, and F listed Category III treatments.
c: Work groups: A, Task Force (Chambless et al., 1998); B, Special section of Journal of Pediatric Psychology (Spirito, 1999); C, Special section of Journal of Clinical Child Psychology (1998); D, Special section of Journal of Consulting and Clinical Psychology (Kendall and Chambless, 1998); E, What Works for Whom? (Roth and Fonagy, 1996); F, A Guide to Treatments That Work (Nathan and Gorman, 1998); G, Gatz et al (1998); H, Wilson and Gil (1996).?, unclear from author's description whether treatment belongs in Category I or II.
About the application of “manualized” protocols, the treatment manuals are traditionally considered restrictive, not able to capture the different nuances of each patient and not useful in the most complex cases (Herbert, 2003). Moreover the use of an EST approach tend to support the cognitive and behavioral therapies (15 of the 16 treatments identified as efficacious in 1998 were behavioral or cognitive-behavioral in orientation) and to limit other models that are less amenable to a manualized presentation (Beutler, 1998; Deegear and Lawson, 2003). It is also important to take into account that treatment manuals are not so flexible in front of the heterogeneity existing in any DSM-based category of disorders. In fact “there remains a wide degree of therapist and intertreatment variability within a given model of treatment, even when a manual is closely followed, and therapist effects are often quite large” (Malik et al., 2003, p. 151).
The Limited Impact of ESTs on Clinical Practice
Many authors have underlined that the real impact of lists of ESTs on clinical practice has not been so significative. For example, in the case of anxiety disorders, only a minority of patients have received an EST despite a lot of documents, reports, and papers supporting the effectiveness of such treatments (Goisman et al., 1993, 1999; Hagemoser, 2009; Jameson et al., 2009; Schnyder, 2009; Shafran et al., 2009). It is also important to take into account that, according to the RCTs approach, “supporters” of EBM consider that efficacy studies are more appropriate in the clinical field, whereas psychotherapists value effectiveness studies more suitable considering them as an accurate reflection of the reality of clinical practice (Starcevic, 2003).
Despite of this possible gap between “science” and “practice”, Stewart and Chambless (2007) noted that Division 12’s (Society of Clinical Psychology, APA) Task Force on Promotion and Dissemination of Psychological Procedures (1995) confirmed that “ESTs should be identified and disseminated to practitioners to improve patient care, and that EST research should be used to guide practice whenever possible” (Stewart and Chambless, 2007, p. 269). About the possible criticisms, the same Authors noted that “the work on ESTs has proved quite controversial for a variety of reasons. For example, Westen et al. (2004) have criticized the use of RTCs as the gold standard for EST research. Others have attacked empirically based research for not addressing the complexity of a typical clinical case or the issues and concerns of the practicing clinician (Persons and Silberschatz, 1998). Yet others believe that the style of research countenanced by EST proponents is inimical to schools of psychotherapy other than cognitive and behavioral therapies, which have a long tradition of EST-type research (e.g., Reed and Eisman, 2006). Certainly the preponderance of ESTs to date are cognitive-behavioural” (Stewart and Chambless, 2007, p. 269). Finally the same Authors gave some suggestions to clinicians with a not-cognitive-behavioral approach: “practitioners not finding their preferred treatment on the list of ESTs may look to other sources of evidence for their approach to practice such as clinical judgment, case reports, discussions with colleagues, and personal experiences” (Stewart and Chambless, 2007, p. 269).
Even if CBT has consistently demonstrated to be effective across a wide range of mental diseases due to many RCTs and meta-analyses (Ost, 2008; Ponniah and Hollon, 2008; Friedberg et al., 2009; McHugh et al., 2009; Ogrodniczuk et al., 2009; Ponniah and Hollon, 2009; Schnyder, 2009), “such ESTs are rarely available and, even when they are, they are often delivered suboptimally” (Schnyder, 2009, p. 902). Schnyder suggested some key recommendations in order to enhance the spread of CBT protocols in routine care. Firs of all “treatment developers should state how the existing trials address comorbidity and produce treatment guidelines and manuals; such manuals should be easily accessible and available at a reasonable cost” (Schnyder, 2009, p. 902). Moreover “clinicians should have easy access to training in diagnostic assessments and routine outcome measures. They should be encouraged to use outcome measures at regular intervals during treatment to monitor progress … The skill level that is required for a therapist to obtain good outcomes should be identified; this requires reliable assessment measures of competence … Methods to establish which patients would benefit from lower intensity interventions and which require more face-to-face contact are required” (Schnyder, 2009, p. 903).
Unfortunately Cohen et al. noted that “psychologists are more likely to go to workshops and read theoretical and how-to books and articles on treatment than to consult the research literature” (Stewart and Chambless, 2007, p. 268). Medline and PsycINFO are bibliographic databases fundamental in the mental health field and few psychologists are very familiar with for searching these databases efficiently (Walker and London, 2007). For example “few psychologists are aware that both databases are based upon the use of controlled vocabulary subject descriptors, which is the best way to search” (Walker and London, 2007, p. 640).
One possible conclusion of this discussion about limitations of ESTs in psychotherapy could be a sentence reported in (Deegear and Lawson, 2003): “Although there are political, societal and monetary enticements to accepting the current rendering of ESTs, sufficient evidence has cast doubt on the movement as it currently exists” (p. 276).
About the future perspectives, Eric Kandel wrote that ‘‘insofar as psychotherapy or counseling is effective and produces long-term changes in behavior, it presumably does so through learning, by producing changes in gene expression that alter the strength of synaptic connections and structural changes that alter the anatomical pattern of interconnections between nerve cells of the brain. As the resolution of brain imaging increases, it should eventually permit quantitative evaluation of the outcome of psychotherapy” (Kandel, 1998, p. 460). “Neuropsychotherapy” is a term used by Grawe (2004) and Walter et al. (2009) and could represent a possible future scenario (Schnyder, 2009). Finally modern psychotherapists are required to integrate knowledge and skills not only in traditional psychological and psychopathological theories, but also in neurosciences, anthropology (Schnyder, 2009), new technologies (Castelnuovo et al., 2003, 2004a; Castelnuovo, 2010) and methodological issues in order to evaluate their treatments following an evidence-based approach.
Complementary Approaches to the Traditional RCTs-Based EBM Approach in Psychotherapy
Comparison between treatments
The “Treatment-placebo” method of comparison when used in clinical psychology research can result in a number of paradoxes. Indeed if there was a logical reason for the placebo, if it lacked credibility, patients would soon realize its existence and it would thus lose its effect (here there is a clear difference between its properties and those of the chemical being tested). On the other hand if the control conditions regarding the placebo were to be believed, then its administration would produce results much more in line with an active treatment that with those of control conditions (Baskin et al., 2003). Thus the placebo, much used in psychotherapeutic research, become in reality a therapeutic treatment and loses its nature of placebo (Finniss and Benedetti, 2005; Benedetti, 2007; Pollo and Benedetti, 2009).
Another reason favoring comparisons between different treatments is that the current evidence-based protocols do not permit a means of treatment being removed from the list of approved protocols even if alternative procedures had proved their superiority when compared to treatments originally included on the approved list: in order to remain on the evidence-based list a treatment simply needs to work better than a placebo. For this reason clinical evidence-based research can be defined as a procedure to test out any given artificial treatment in an artificially controlled clinical context using atypical patients (Ablon and Jones, 2002).
I therefore suggest an approach which is clinically and ecologically legitimate in psychotherapy research wherein patients with similar DSM diagnoses are able to pursue different “active” treatments (without controls, placebos or waiting lists), and their progress is measured not only using the traditional tests but also a so-called “sliding scale technique” (Nardone and Watzlawick, 1993; Nardone, 1996; de Jong and Berg, 2001; Nardone and Watzlawick, 2004; Nardone and Portelli, 2005) which highlights a satisfaction shared by therapist and patient alike at the final outcomes of therapy, and which yet respects the individual nature of a given approach.
‘The sliding scale technique’ therefore allows each treatment, independently of the chosen approach, to validate the results predicted at the start of the treatment. I am talking about a measure of value, and the intrinsic coherence of each psychotherapeutic intervention respecting its uniqueness and traditions; a measure of the correlation between what every approach promises and its final outcome. Thus I can quantify the quality (of a treatment using a sliding scale) and qualify the quantity (in other words by giving a clinical meaning to the value). To better understand this approach, I quote the sliding scale technique as described by (Nardone and Portelli, 2005): “If you had to mark the improvement reached so far regarding your problem, 0 being the lowest, corresponding to when you came here asking us to help you with your problem, and 10 being the maximum – when you feel you can tell us, “Thank you, doctor(s), but I no longer need your help” – where would you place yourself now?” (p. 173).
It is important to underline that the use of such techniques is closely related to the internal workings of an advanced kind of research where key words are openness, evolution, knowledge through change, knowledge of the problem through its solution, absence of inflexible theories at the beginning. In fact action-research allows self-corrective protocols, which can be repeated and are predictable to be developed: “the protocols are simple guidelines, which are far from being rigid and preordained. Protocols and strategic interventions are designed in a way that allows self-correction at any point of the therapy, since we are aware that the only way to really get to know a particular problem is through its solution” (Nardone and Portelli, 2005, p. 170). Such a framework differs from the traditional way research is carried out, where the recurrent words are justification, interpretation arising out of strong theories and a priori explanations and therefore and therefore give rise to systems which are self-protecting.
Practice-based evidence
Another alternative to the traditional evidence-based approach is to move from an “evidence-based” practices to a “practice-based” evidence approach (Margison et al., 2000) examining the results of the psychotherapy in natural situations with treatments which vary in length and are carried out with greater flexibility allowing the therapist to exert a greater clinical influence and to have a more active role. An evaluation of the results should not only consider a reduction of the symptoms but also factors such as ability to function, disabilities, and quality of life (Margison et al. 2000).
Evidence-based practice and hermeneutic single-case efficacy design
Amongst the alternatives to the traditional evidence-based movement the Pragmatic Houses Study Method (Fiedler, 2001; Fishman and Messer, 2004; Messer, 2004) could be interesting: it is characterized by systematic and largely qualitative case studies that focus on practical results. Another useful approach is the Hermeneutic Single-Case Efficacy Design (Elliott, 2001) which comprises the use of quantitative and qualitative information, direct and indirect evidences to create a rich database with single-case records.
Same approach – different therapist
A further alternative is the evaluation of the effectiveness and the efficiency of the same approach-treatment when used by different therapists. A possible alternative would be to evaluate performance both from the point of view of the patient and of the therapist (Starcevic, 2003).
Conclusion
To conclude this contribution to the ESTs I would like to emphasize that the term evidence itself does not just have one single definition, but it depends on the various contexts where it is used (Upshur et al., 2001).
This contribution aimed to throw light on the characteristics and limits of such an Evidence-based framework and on the alternatives that can be integrated with and complement such methods of research. My position is in line with the important contribution of Westen (2005), where the author reports that “EBP > EST – that is, evidence-based practice (EBP) includes many forms of evidence other than data from RCTs (Wampold, 2001)” (Westen, 2005, p. 7).
As underlined by Gelso (2005), “Recent years have witnessed the emergence of two powerful, and seemingly contradictory, visions of what most fundamentally causes change in psychotherapy. Each vision has its share of devotees and detractors, and each has become so prominent that it may be seen as attaining the status of a worldview with respect to what psychotherapy is all about and what is most vital to treatment success. One of these visions emphasizes the primacy of therapist technique. According to this viewpoint, it is the specific methods used by the psychotherapist that account for, by far, most of the variance in treatment outcome. Other factors (e.g., therapist relational qualities, patient–therapist relationship) are secondary, at best. This viewpoint is seen most notably in what have been termed the EST and EBP movements. Advocates who are part of these movements (they really meld into one movement) conduct tightly controlled outcome studies in which specific treatments are pitted against one another or a control group and applied to the treatment of specific disorders, usually as defined in terms of DSM nosology. An effort is made to control individual differences among therapists and relational factors through the use of treatment manuals that are to be followed judiciously. Stated in the extreme, only experimental studies (called RCTs) that adhere to these requirements (specific, manualized treatments; patients who fit reliably into only one diagnostic group; random assignment of patients to treatment groups) are capable of determining which treatments work best. And only treatments that are found to work through the use of these methodological features should be taught and used. In support of this worldview, evidence has been marshaled that psychotherapy methods and techniques, broadly defined, do indeed matter and have an effect on treatment outcome. The second vision instead focuses on the patient–therapist relationship and so-called therapist-offered relationship qualities as the sine qua non of therapeutic effectiveness. Advocates of this viewpoint underscore research on the importance of relational qualities such as the therapeutic working alliance (or cohesion in groups) and therapist-offered conditions” (2005, p. 419).
I conclude by remembering that, as well as being in a position to integrate the ESTs movement with other “evidence-based” methodologies (using the word evidence in its widest sense), I can also improve the EST system itself. Drug research, from which the EBM method derives, has a built-in retroactive mechanism by which the clinical effectiveness of a given treatment in the field is controlled. The EST movement, and psychotherapeutic research in general, “does not currently predict this kind of built-in and retroactive mechanism being set up” (Lyddon and Chatkoff, 2002, p. 258). Therefore “if it is true that the EST movement's focus on the experimental effectiveness of psychotherapeutic procedures represents an important step forward…, it is also true that only a similar focus on clinical efficacy can lead to the public giving psychotherapy its complete trust”, (Lyddon and Chatkoff, 2002, p. 259).
Perhaps, as underlined by Joyce et al. (2006), the practice of psychotherapy concerns “art” and “craft”: “Should the practice of psychotherapy be regarded primarily as an art or as a craft? Alternatively, does it display characteristics of a scientific discipline? be regarded primarily as an art or as a craft? Alternatively, does it display characteristics of a scientific discipline? After a half-century of psychotherapy research, the most reasonable answer appears to be that psychotherapy is indeed a craft, but one involving critical constituent elements that can be discovered and clarified through scientific investigation. That is, science makes it possible to identify the key elements associated with change in psychotherapy. Still, the individual clinician's creativity and responsiveness with a particular patient can be a major factor in the patient's psychotherapy. This uniqueness is not often captured by the scientific method, and thus, it can be regarded as a form of artistry (p. 798).
Conflict of Interest Statement
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Footnotes
References
- Ablon J. S., Jones E. E. (2002). Validity of controlled clinical trials of psychotherapy. Findings from the NIMH treatment of depression collaborative research program. Am. J. Psychiatry 159, 775–783 10.1176/appi.ajp.159.5.775 [DOI] [PubMed] [Google Scholar]
- Baskin T. W., Tierney S. C., Minami T., Wampold B. E. (2003). Establishing specificity in psychotherapy: a meta-analysis of structural equivalence of placebo controls. J. Consult. Clin. Psychol. 71, 973–979 10.1037/0022-006X.71.6.973 [DOI] [PubMed] [Google Scholar]
- Benedetti F. (2007). What do you expect from this treatment? Changing our mind about clinical trials. Pain 128, 193–194 [DOI] [PubMed] [Google Scholar]
- Beutler L. E. (1998). Identifying empirically supported treatments: what if we didn't? J. Consult. Clin. Psychol. 66, 113–120 10.1037/0022-006X.66.1.113 [DOI] [PubMed] [Google Scholar]
- Castelnuovo G. (2010). No medicine without psychology: the key role of psychological contribution in clinical settings. Front. Psychol. Clin. Settings 1, 4. 10.3389/fpsyg.2010.00004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Castelnuovo G., Buselli C., De Ferrari R., Gaggioli A., Mantovani F., Molinari E., Villamira M., Riva G. (2004a). New tools in cybertherapy: the VEPSY web site. Stud. Health Technol. Inform. 99, 15–35 [PubMed] [Google Scholar]
- Castelnuovo G., Faccio E., Molinari E., Nardone G., Salvini A. (2004b). A critical review of empirically supported treatments (ESTs) and common factors perspective in psychotherapy, brief strategic and systemic therapy European review. Brief Strategic Syst. Ther. Eur. Rev. 1, 208–224 [Google Scholar]
- Castelnuovo G., Faccio E., Molinari E., Nardone G., Salvini A. (2005). Evidence based approach in psychotherapy the limitations of current empirically supported treatments paradigms and of similar theoretical approaches as regards establishing efficient and effective treatments in psychotherapy. Brief Strategic Syst. Ther. Eur. Rev. 1, 229–248 [Google Scholar]
- Castelnuovo G., Gaggioli A., Mantovani F., Riva G. (2003). From psychotherapy to e-therapy: the integration of traditional techniques and new communication tools in clinical settings. Cyberpsychol. Behav. 6, 375–382 10.1089/109493103322278754 [DOI] [PubMed] [Google Scholar]
- Chambless D. L., Hollon S. D. (1998). Defining empirically supported therapies. J. Consult. Clin. Psychol. 66, 7–18 10.1037/0022-006X.66.1.7 [DOI] [PubMed] [Google Scholar]
- Chambless D. L., Ollendick T. H. (2001). Empirically supported psychological interventions: controversies and evidence. Annu. Rev. Psychol. 52, 685–716 10.1146/annurev.psych.52.1.685 [DOI] [PubMed] [Google Scholar]
- Chambless D. L., Sanderson W. C., Shoham V., Bennett Johnson S., Pope K. S., Crits-Christoph P., Baker M., Johnson B., Woody S. R., Sue S., Beutler L., Williams D. A., McCurry S. (1996). An update on empirically validated therapies. Clin. Psychol. 5–18 [Google Scholar]
- de Jong P., Berg I. K. (2001). Interviewing for Solutions. Stanford: Wadsworth Publishing [Google Scholar]
- Deegear J., Lawson D. M. (2003). The utility of empirical supported treatments. Prof. Psychol. Res. Pr. 54, 271–277 10.1037/0735-7028.34.3.271 [DOI] [Google Scholar]
- Duncan B. L. (2002). The legacy of Saul Rosenzweig: the profundity of the dodo bird. J. Psychother. Integrat. 12, 32–57 10.1037/1053-0479.12.1.32 [DOI] [Google Scholar]
- Elliott R. (2001). “Hermeneutic single-case efficacy design: an overview,” in The Handbook of Humanistic Psychology, eds Schneider K. J. et al. (Thousand Oaks, CA: Sage; ), 315–324 [Google Scholar]
- Eysenck H. J. (1952). The effects of psychotherapy: an evaluation. J. Consult. Clin. Psychol. 319–324 [DOI] [PubMed] [Google Scholar]
- Eysenck H. J. (1961). “The effects of psychotherapy,” in Handbook of Abnormal Psychology, ed. Eysenck H. J. (New York: Basic Books; ), 697–725 [Google Scholar]
- Fiedler P. (2001). How myths and prejudices about psychotherapy hamper the ongoing process of psychotherapy-integration. Psychother. Psychosom. Med. Psychol. 51, 406–412 10.1055/s-2001-18159 [DOI] [PubMed] [Google Scholar]
- Finniss D. G., Benedetti F. (2005). Mechanisms of the placebo response and their impact on clinical trials and clinical practice. Pain 114, 3–6 [DOI] [PubMed] [Google Scholar]
- Fishman D. B., Messer S. B. (2004). “Case-based studies as a source of unity in applied psychology,” in The Unification of Psychology: Prospect or Pipedream?, ed. Sternberg R. J. (Washington, DC: American Psychological Association; ), 37–59 [Google Scholar]
- Friedberg R. D., Gorman A. A., Beidel D. C. (2009). Training psychologists for cognitive-behavioral therapy in the raw world: a rubric for supervisors. Behav. Modif. 33, 104–123 10.1177/0145445508322609 [DOI] [PubMed] [Google Scholar]
- Gelso C. J. (2005). Introduction to special issue. Psychother. Theor. Res. Pract. Train. 42, 419–420 10.1037/0033-3204.42.4.419 [DOI] [Google Scholar]
- Goisman R. M., Rogers M. P., Stekettee G. S., Warshaw M. G., Cuneo P., Keller M. B. (1993). Utilization of behavioural methods in a multicenter anxiety disorders study. J. Clin. Psychiatry 54, 213–218 [PubMed] [Google Scholar]
- Goisman R. M., Warshaw M. G., Keller M. B. (1999). Psychosocial treatment prescriptions for generalized anxiety disorder, panic disorder, and social phobia, 1991-1996. Am. J. Psychiatry 156, 1819–1821 [DOI] [PubMed] [Google Scholar]
- Grawe K. (2004). Neuropsychotherapie. Göttingen: Hogrefe [Google Scholar]
- Hagemoser S. D. (2009). Braking the bandwagon: scrutinizing the science and politics of empirically supported therapies. J. Psychol. 143, 601–614 [DOI] [PubMed] [Google Scholar]
- Herbert J. D. (2003). The science and practice of empirically supported treatments. Behav. Modif. 27, 412–430 10.1177/0145445503027003008 [DOI] [PubMed] [Google Scholar]
- Jameson J. P., Chambless D. L., Blank M. B. (2009). Empirically supported treatments in rural community mental health centers: a preliminary report on current utilization and attitudes toward adoption. Community Ment. Health J. 45, 463–467 10.1007/s10597-009-9230-7 [DOI] [PubMed] [Google Scholar]
- Joyce A. S., Wolfaardt U., Sribney C., Aylwin A. S. (2006). Psychotherapy research at the start of the 21st century: the persistence of the art versus science controversy. Can. J. Psychiatry 51, 797–809 [DOI] [PubMed] [Google Scholar]
- Kandel E. R. (1998). A new intellectual framework for psychiatry. Am. J. Psychiatry 155, 457–469 [DOI] [PubMed] [Google Scholar]
- Karver M. S., Handelsman J. B., Fields S., Bickman L. (2005). A theoretical model of common process factors in youth and family therapy. Ment. Health Serv. Res. 7, 35–51 10.1007/s11020-005-1964-4 [DOI] [PubMed] [Google Scholar]
- Lambert M. J. (2005). Early response in psychotherapy: further evidence for the importance of common factors rather than “placebo effects”. J. Clin. Psychol. 61, 855–869 10.1002/jclp.20130 [DOI] [PubMed] [Google Scholar]
- Lambert M. J., Bergin A. E. (1994). “The effectiveness of psychotherapy,” in “Handbook of Psychotherapy and Behaviour Change (4th Edn.), eds Bergin A. E., Garfield S. L. (New York: Wiley; ), 143–189 [Google Scholar]
- Luborsky L., Rosenthal R., Diguer L., Andrusyana T. P., Berman J. S., Levitt J. T., Seligman D. A., Krause E. D. (2002). The dodo bird verdict is alive and well-mostly. Clin. Psychol. Sci. Pract. 9, 2–12 10.1093/clipsy/9.1.2 [DOI] [Google Scholar]
- Luborsky L., Singer B. (1975). Comparative studies of psychotherapies. Is it true that “everywon has one and all must have prizes”? Arch. Gen. Psychiatry 32, 995–1008 [DOI] [PubMed] [Google Scholar]
- Lyddon W. J., Chatkoff D. K. (2002). “I trattamenti di provata efficacia sperimentale: tendenze attuali e prospettive future,” in L'approccio Evidence-Based on Psicoterapia, eds Lyddon W. J., Jones J. V. (Milano: McGraw-Hill; ), 249–261 [Google Scholar]
- Malik M. L., Beutler L. E., Alimohamed S., Gallagher-Thompson D., Thompson L. (2003). Are all cognitive therapies alike? A comparison of cognitive and noncognitive therapy process and implications for the application of empirically supported treatments. J. Consult. Clin. Psychol. 71, 150–158 10.1037/0022-006X.71.1.150 [DOI] [PubMed] [Google Scholar]
- Margison F. R., Barkham M., Evans C., McGrath G., Clark J. M., Audin K., Connell J. (2000). Measurement and psychotherapy: evidence-based practice and practice-based evidence. Br. J. Psychiatry 177, 123–130 10.1192/bjp.177.2.123 [DOI] [PubMed] [Google Scholar]
- McHugh R. K., Smits J. A., Otto M. W. (2009). Empirically supported treatments for panic disorder. Psychiatr. Clin. North Am. 32, 593–610 10.1016/j.psc.2009.05.005 [DOI] [PubMed] [Google Scholar]
- Messer S. B. (2004). Evidence-based practice: beyond empirically supported treatments. Prof. Psychol. Res. Pract. 35, 580–588 10.1037/0735-7028.35.6.580 [DOI] [Google Scholar]
- Morrison K. H., Bradley R., Westen D. (2003). The external validity of controlled clinical trials of psychotherapy for depression and anxiety: a naturalistic study. Psychol. Psychother. 76, 109–132 10.1348/147608303765951168 [DOI] [PubMed] [Google Scholar]
- Nardone G. (1996). Brief Strategic Solution-Oriented Therapy of Phobic and Obsessive Disorders. New Jersey: Aronson [Google Scholar]
- Nardone G., Portelli C. (2005). Knowing through changing: The Evolution of Brief Strategic Therapy. UK: Crown House Publishing Limited [Google Scholar]
- Nardone G., Watzlawick P. (1993). The Art of Change. Strategic Therapy and Hypnotherapy Without Trance. San Francisco: Jossey-Bass [Google Scholar]
- Nardone G., Watzlawick P. (2004). Advanced Brief Therapy. New Jersey: Aronson [Google Scholar]
- Ogrodniczuk J. S., Kealy D., Howell-Jones G. (2009). A view from the trenches: a survey of Canadian clinicians’ perspectives regarding the treatment of borderline personality disorder. J. Psychiatr. Pract. 15, 449–453 10.1097/01.pra.0000364286.63210.db [DOI] [PubMed] [Google Scholar]
- Ost L. G. (2008). Cognitive behavior therapy for anxiety disorders: 40 years of progress. Nord. J. Psychiatry 62(Suppl 47), 5–10 10.1080/08039480802315590 [DOI] [PubMed] [Google Scholar]
- Paul G. L. (1967). Strategy of outcome research in psychotherapy. J. Consult. Psychol. 109–118 10.1037/h0024436 [DOI] [PubMed] [Google Scholar]
- Persons J. B., Silberschatz G. (1998). Are results of randomized controlled trials useful to psychotherapists? J. Consult. Clin. Psychol. 66, 126–135 10.1037/0022-006X.66.1.126 [DOI] [PubMed] [Google Scholar]
- Pollo A., Benedetti F. (2009). The placebo response: neurobiological and clinical issues of neurological relevance. Prog. Brain Res. 175, 283–294 10.1016/S0079-6123(09)17520-9 [DOI] [PubMed] [Google Scholar]
- Ponniah K., Hollon S. D. (2008). Empirically supported psychological interventions for social phobia in adults: a qualitative review of randomized controlled trials. Psychol. Med. 38, 3–14 10.1017/S0033291707000918 [DOI] [PubMed] [Google Scholar]
- Ponniah K., Hollon S. D. (2009). Empirically supported psychological treatments for adult acute stress disorder and posttraumatic stress disorder: a review. Depress. Anxiety 26, 1086–1109 10.1002/da.20635 [DOI] [PubMed] [Google Scholar]
- Reed G. M., Eisman E. J. (2006). “Uses and misuses of evidence: Managed care, treatment guidelines, and outcomes measurement in professional practice,” in Evidence-Based Psychotherapy: Where Practice and Research Meet, eds Goodheart C. D. et al. (Washington, DC: American Psychological Association; ), 13–35 [Google Scholar]
- Schnyder U. (2009). Future perspectives in psychotherapy. Eur. Arch. Psychiatry Clin. Neurosci. 259(Suppl 2), S123–S128 10.1007/s00406-009-0051-z [DOI] [PubMed] [Google Scholar]
- Shafran R., Clark D. M., Fairburn C. G., Arntz A., Barlow D. H., Ehlers A., Freeston M., Garety P. A., Hollon S. D., Ost L. G., Salkovskis P. M., Williams J. M., Wilson G. T. (2009). Mind the gap: improving the dissemination of CBT. Behav. Res. Ther. 47, 902–909 10.1016/j.brat.2009.07.003 [DOI] [PubMed] [Google Scholar]
- Smith M. L., Glass G. V., Miller T. I. (1980). The Benefits of Psychotherapy. Baltimore: Johns Hopkins University Press [Google Scholar]
- Spirito A. (1999). Introduction. J. Pediat. Psychol. 24, 87–90 [Google Scholar]
- Starcevic V. (2003). Psychotherapy in the era of evidence-based medicine. Aust. Psychiatry 11, 278–281 10.1046/j.1440-1665.2003.00575.x [DOI] [Google Scholar]
- Stewart R. E., Chambless D. L. (2007). Does psychotherapy research inform treatment decisions in private practice? J. Clin. Psychol. 63, 267–281 10.1002/jclp.20347 [DOI] [PubMed] [Google Scholar]
- Task Force on Promotion Dissemination of Psychological Procedures (1995). Training and dissemination of empirically-validated psychological treatments: Reports and recommendation. Clin. Psychol. 48, 3–23 [Google Scholar]
- Upshur R. E. G., VanDenKerkhof E. G., Goel V. (2001). Meaning and measurement: an inclusive model of evidence in health care. J. Eval. Clin. Pract. 7, 91–96 10.1046/j.1365-2753.2001.00279.x [DOI] [PubMed] [Google Scholar]
- Walker B. B., London S. (2007). Novel tools and resources for evidence-based practice in psychology. J. Clin. Psychol. 63, 633–642 10.1002/jclp.20377 [DOI] [PubMed] [Google Scholar]
- Walter H., Berger M., Schnell K. (2009). Neuropsychotherapy: conceptual, empirical and neuroethical issues. Eur. Arch. Psychiatry Clin. Neurosci. 259(Suppl 2), S173–S182 10.1007/s00406-009-0058-5 [DOI] [PubMed] [Google Scholar]
- Wampold B. E. (2001). The Great Psychotherapy Debate: Models, Methods, and Findings. Mahwah, NJ: Lawrence Erlbaum Associates [Google Scholar]
- Westen D. (2005). Letter to Editor. Clin. Psychol. 58, 6–7 [Google Scholar]
- Westen D., Novotny C. M., Thompson-Brenner H. (2004). The empirical status of empirically supported psychotherapies: assumptions, findings, and reporting in controlled clinical trials. Psychol. Bull. 130, 631–663 10.1037/0033-2909.130.4.631 [DOI] [PubMed] [Google Scholar]