Abstract
It is not yet clear what mental disorders are and what are the causal pathways that lead to them. That makes it difficult to decide what the targets and outcomes of psychotherapies should be. In this paper, the main types of targets and outcomes of psychotherapies are described, and a brief overview is provided of some of the main results of research on these types. These include symptom reduction, personal targets and outcomes from the patient's perspective, improvement of quality of life, intermediate outcomes depending on the theoretical framework of the therapist, negative outcomes to be avoided, and economic outcomes. In line with the dominance of the DSM and ICD systems for diagnoses, most research has been focused on symptom reduction. This considerable body of research, with hundreds of randomized trials, has shown that for most mental disorders effective psychotherapies are available. There is also research showing that psychotherapies can result in improvement of quality of life in most mental disorders. However, relatively little research is available on patient‐defined outcomes, intermediate outcomes, negative outcomes and economic outcomes. Patients, relatives, therapists, employers, health care providers and society at large each have their own perspectives on targets and outcomes of psychotherapies. The perspective of patients should have more priority in research, and a standardization of outcome measures across trials is much needed.
Keywords: Psychotherapies, outcomes, targets, symptom reduction, quality of life, patient‐defined outcomes, intermediate outcomes, cognitive behavior therapy
Mental disorders are one of the most important public health challenges of this time1, 2. With hundreds of millions people worldwide affected by them, these disorders are associated with severe personal suffering by patients and their relatives, considerable transgenerational transmission3, 4, 5, huge economic costs6, and increased levels of physical morbidity and mortality7, 8.
It is, however, still not clear what these disorders exactly are. There are no objective tests or measures to establish the presence of a mental disorder, nor are there clear thresholds for when a patient has a disorder and when not. The dominant systems for classifying and defining mental disorders in the past decades have been the Diagnostic and Statistical Manual of Mental Disorders (DSM) and the International Classification of Diseases (ICD). Although most research on mental disorders in the past decades has been using the different versions of these systems, they have been widely criticized.
For example, there is evidence that most mental disorders should not be considered as separate entities but rather as consisting of dimensions, on which some people score high and others score low9, 10, 11. Furthermore, high levels of comorbidity are more the rule than the exception12. Some argue that the diagnostic categories in the DSM and ICD have limited validity13. Treatments are also typically not effective in just one disorder, but across several different disorders, such as pharmacotherapies in mood and anxiety disorders, and cognitive behavior therapy (CBT) in most mental disorders12.
So, if we do not yet really know what these disorders are and how they should be defined, what should be the targets of treatments and how can we measure their outcomes? The overall goal of treatments obviously is to make patients better or to help them cope with the problems they have. What this exactly means, however, and when it can be considered as accomplished, is not so clear. Not only because the nature and causes of the disorders are unclear, but also because it depends on whether one asks the patient, the clinician, patient's relatives, health insurance companies, or society at large to answer this question.
The focus of this paper is on the targets and outcomes of psychotherapies. We define the targets of a therapy as what should be tried to accomplish during the process. Outcomes are the results of a therapy. Because targets and outcomes are very much intertwined, we will consider them together in the discussion below, and often use the term “outcome” while we mean the broader concept that also includes targets.
We distinguish different types of outcomes: symptom reduction, which is the focus of most outcome research in psychotherapy; patient‐defined outcomes; quality of life improvement; intermediate outcomes based on the theoretical framework and assumptions of the therapist; negative outcomes to be avoided; and economic outcomes. A summary of the types of outcomes, the research available on these outcomes, the results obtained, and the overall status of research is presented in Table 1.
Table 1.
Summary of the main targets and outcomes of psychotherapies for mental disorders
Type of target and outcome | Research | Results | Status of research |
---|---|---|---|
Symptom reduction | Examined in hundreds of randomized trials for many types of psychotherapy for all major mental disorders |
Effective therapies exist for most mental disorders in the short term Effects are probably overestimated because of publication bias, low trial quality, lack of blinding Wide variety in measures |
Most research on the effects of psychotherapy is focused on symptom reduction |
Patient‐defined targets and outcomes | Idiographic measures of the main problems as experienced by patients, such as the Target Complaints, the Simplified Personal Questionnaire, and the Youth Top Problems | These measures are mostly used in routine care | Limited systematic research available |
Qualitative research on the personal targets and outcomes of psychotherapies | Helpful impact of therapies: awareness, insight, self‐understanding, behavioral change, solution of problems, empowerment, relief, better understanding of feelings | Limited systematic research available | |
Quality of life and related targets and outcomes | Studied as a (secondary) outcome in randomized trials | Significant effects of therapies on quality of life have been found for depression, eating disorders and anxiety disorders, but not schizophrenia | Relatively well‐studied, but more research is clearly needed |
Intermediate outcomes: mediators and working mechanisms | Each school of psychotherapy has its own theoretical framework to explain how therapy works | Mediators and working mechanisms have not been well established for any therapy, because of methodological problems | Limited systematic research available |
Negative outcomes | Have become the focus of research only recently |
Preliminary research suggests that deterioration in psychotherapies is lower than in control conditions Several types of negative effects have not been examined systematically Individual patient data meta‐analyses are a promising approach |
Limited systematic research available |
Economic outcomes | Studies include cost‐utility and cost‐effectiveness analyses |
For most mental disorders no more than one or two studies of psychotherapies are available Some more studies are available for cognitive behavior therapy in depression |
Limited systematic research available |
SYMPTOM REDUCTION AS OUTCOME OF PSYCHOTHERAPIES
Symptom reduction can be seen as the core target and outcome of psychotherapies. Not only is symptom reduction by far the most common focus of outcome research, especially randomized trials, but qualitative studies also show that it is one of the most important outcomes from the viewpoint of patients (although certainly not the only one)14. Apart from researchers and patients, symptom reduction is also a core outcome for other stakeholders, including therapists, although this depends on the model they adhere to.
The hundreds of randomized trials that have examined the effects of psychotherapies for mental disorders have mostly focused on symptom reduction as the primary outcome. In Table 2, the results are presented of some recent meta‐analyses of psychotherapies (mostly CBT) compared to control conditions for the most important mental disorders. For each disorder, three of the largest meta‐analyses published in the past five years are presented, and the type of intervention, the format of the intervention (individual, group, guided or unguided self‐help), and the type of control group are summarized. We also report the number of studies included in each meta‐analysis, the effect size (standardized mean difference), the level of heterogeneity in percentages (I2), and whether it was a conventional or a network meta‐analysis.
Table 2.
Meta‐analyses of randomized trials examining the effects of psychotherapies compared to control conditions
Intervention | Format | Comparator | N studies | SMD | 95% CI | I2 | Type | |
---|---|---|---|---|---|---|---|---|
Depression | ||||||||
Cuijpers et al15 | Any therapy |
Individual/group/ guided self‐help |
Any control | 369 | 0.70 | 0.64‐0.75 | 76 | CMA |
Mohr et al16 | Any therapy |
Individual/group/ guided self‐help |
Any control | 188 | 0.54 | 0.45‐0.64 | 82 | CMA |
Cuijpers et al17 | CBT |
Individual/group/ guided self‐help |
Any control | 94 | 0.71 | 0.62‐0.79 | 57 | CMA |
Social anxiety disorder | ||||||||
Cuijpers et al18 | CBT |
Individual/group/ guided self‐help |
Waiting list, care‐as‐usual, pill placebo |
48 | 0.88 | 0.74‐1.03 | 64 | CMA |
Mayo‐Wilson et al19 | CBT | Group | Waiting list | 28 | 0.92 | 0.51‐1.33 | NA | NMA |
Barkowski et al20 | CBT | Group | Waiting list | 25 | 0.84 | 0.72‐0.97 | 0 | CMA |
Panic disorder | ||||||||
Cuijpers et al18 | CBT |
Individual/group/ guided self‐help |
Waiting list, care‐as‐usual, pill placebo |
42 | 0.81 | 0.59‐1.04 | 77 | CMA |
Pompoli et al21 | CBT | Individual/group | Waiting list | 17 | 1.14 | 0.87‐1.41 | 61 | NMA |
Mayo‐Wilson & Montgomery22 | CBT | Guided/unguided self‐help | No treatment | 21 | 0.62 | 0.45‐0.79 | 23 | CMA |
Generalized anxiety disorder | ||||||||
Cuijpers et al23 | Any therapy |
Individual/group/ guided self‐help |
Any inactive control | 38 | 0.84 | 0.71‐0.97 | 33 | CMA |
Cuijpers et al18 | CBT |
Individual/group/ guided self‐help |
Waiting list, care‐as‐usual, pill placebo |
31 | 0.80 | 0.67‐0.93 | 33 | CMA |
Mayo‐Wilson & Montgomery22 | CBT | Guided/unguided self‐help | No treatment | 10 | 0.95 | 0.44‐1.45 | 88 | CMA |
Post‐traumatic stress disorder | ||||||||
Bisson et al24 | TF‐CBT/Exposure | Individual | Waiting list, care‐as‐usual | 28 | 1.62 | 1.21‐2.03 | 89 | CMA |
Bisson et al24 | TF‐CBT/Exposure | Group | Waiting list, care‐as‐usual | 16 | 1.20 | 0.69‐1.70 | 71 | CMA |
Gerger et al25 | CBT | Individual | Waiting list | 16 | 1.10 | 0.85‐1.36 | NA | NMA |
Obsessive‐compulsive disorder | ||||||||
Olatunji et al26 | CBT | Individual/group | Waiting list, pill or psychological placebo | 16 | 1.39 | 1.04‐1.74 | NA | CMA |
Ost et al27 | CBT | Individual/group | Waiting list | 15 | 1.31 | 1.08‐1.55 | 37 | CMA |
Ost et al27 | CBT | Individual/group | Pill or psychological placebo | 8 | 1.33 | 0.91‐1.76 | 72 | CMA |
Psychotic disorders | ||||||||
Velthorst et al28 | CBT | Individual/group | Any control | 28 | 0.09 | –0.03 to 0.21 | 63 | CMA |
Burns et al29 | CBT | Individual | Any control | 12 | 0.52 | 0.35‐0.70 | 0 | CMA |
Eichner & Berna30 | Metacognitive training | Individual/group | Any control | 11 | 0.34 | 0.15‐0.53 | 3 | CMA |
Bipolar disorder | ||||||||
Chatterton et al31 | Psychoeducation + CBT | Individual/group | Care‐as‐usual | 16 | 0.58 | –1.25 to 2.41 | NA | NMA |
Chatterton et al31 | Psychoeducation | Individual/group | Care‐as‐usual | 12 | 0.14 | –1.01 to 1.30 | NA | NMA |
Chiang et al32 | CBT | Individual/group | Any control | 13 | 0.49 | 0.03‐0.96 | 90 | CMA |
CBT – cognitive behavior therapy, TF – trauma focused, SMD – standardized mean difference, CMA – conventional meta‐analysis, NMA – network meta‐analysis, NA – not available
The effect sizes for depression, anxiety disorders (social anxiety disorder, panic disorder, generalized anxiety disorder), post‐traumatic stress disorder and obsessive‐compulsive disorder are moderate to large, with most effect sizes between 0.5 and 1.5. Effect sizes for psychotic and bipolar disorders are somewhat smaller, but that may also be related to the fact that control conditions typically consist of care‐as‐usual, which in these disorders means that most patients received intensive pharmacological treatment.
These findings clearly support the assumption that (at least some) psychotherapies have significant effects on most mental disorders when reduction of symptoms is taken as the primary outcome. However, these findings have been criticized as being too optimistic, because of publication bias33, 34, 35, low quality and validity of many trials15, 36, and problems such as “researcher allegiance”37, i.e. “the belief in superiority of an intervention and the superior validity of the theory of change that is associated with the treatment”38. Most research was also aimed at the short term, and longer‐term effects are largely unknown. Furthermore, there are indications that one type of control condition, i.e. waiting list, may overestimate the effect of a therapy39, 40.
One major problem in examining the effects of psychotherapies on symptoms is that the instruments measuring change vary widely. For example, we identified 310 randomized trials comparing psychotherapies with a control condition in people with depression15. Although the Beck Depression Inventory (BDI)41 and the Hamilton Rating Scale for Depression (HAMD)42 were the most used instruments, there were more than thirty other instruments measuring the impact of psychotherapies on depressive symptoms. As a comparison, in a recent meta‐analysis of more than 500 randomized trials of pharmacotherapy for depression, 89% used the HAMD as the primary outcome measure43.
Actually, the variety of instruments measuring an outcome was one of the main reasons why meta‐analyses were introduced44. In a meta‐analysis, the effect measured with one instrument is standardized into an “effect size”, in order to pool it with the effects using other instruments. If all studies used the same outcome measure, this standardization would not be needed, because it would be possible to simply calculate the benefit of an intervention in terms of exact points on that measure.
Another issue is whether symptoms should be measured through self‐report or clinician‐rated instruments. It could be assumed that clinician‐rated instruments provide a better estimate of the effects of an intervention, because they are applied by an independent observer (especially if the interviewer is blinded to the treatment condition). On the other hand, symptoms are experienced by patients, so one can also argue that patients themselves are the best raters of their problems. Furthermore, there are indications that outcomes rated by patients are more conservative than those rated by clinicians. We found in a meta‐analysis that effect sizes of self‐report measures were significantly smaller than clinician‐rated measures from the same studies (differential effect size of g=0.20)45.
There is no consensus about whether or not reduction of symptoms should be considered as the core outcome of psychotherapies. Therapists and researchers from the cognitive and behavioral tradition do support the notion that symptom change is the core outcome. However, therapists from the psychodynamic tradition consider personality and intrapsychic change as much more important46, even if it cannot be measured very well. For them, symptoms are only the result of these personality and intrapsychic problems. They are assumed not to be the real core problem, and to improve when the personality and intrapsychic change is obtained. Therapists from the client‐centered tradition would argue that self‐actualization is the core outcome of therapy, and that symptoms are only one of the triggers for patients to find help.
In some cases, a worsening of symptoms can even be considered a positive outcome of therapy46. For example, it has been argued that the emergence of depression during existential psychotherapy could be a sign that the patient is being more in touch with reality, which in turn motivates urgency to reevaluate priorities47.
The strong focus in research on the reduction of symptoms is in part related to the wide acceptance of the DSM and ICD, which have been dominating the field of mental health research in the past 50 years48. In recent years, however, the critique of these systems is strongly increasing. According to several authors12, 15, 49, the progress in improving outcomes of treatments of mental disorders is not being satisfactory, and, in order to change that, new systems to understand mental disorders are needed.
One of the most important new projects that challenge the dominance of the DSM and ICD is the Research Domain Criteria (RDoC) initiative, launched by the US National Institute of Mental Health50, 51. The RDoC is not based on the clinical descriptions of disorders, but considers these disorders from a translational point of view12. It starts with the fundamental, primary behavioral functions of the brain and the neural systems that are involved in the implementation of these functions. Examples are the circuits for fear and defense, for appetitive behavior such as learning to predict reward and moving toward reward, and for cognitive functions such as working memory12. The RDoC considers psychopathology as dysfunction in these systems. At this moment, it is too early to say whether this new approach will indeed result in new knowledge about if and how therapies work.
PATIENT‐DEFINED TARGETS AND OUTCOMES OF PSYCHOTHERAPIES
A completely different type of targets and outcomes of psychotherapies are those that are defined by patients themselves. Patients typically do not only come to therapy to obtain relief from symptoms, but also to address other personal problems, which may include going back to work, solving intrapersonal issues, being a better parent, or stopping the fights with their partner or their boss. Addressing these problems of the patient can be regarded as one of the main goals of therapy52, 53.
Although these individual problems have not been examined as extensively as symptom reduction, there is a long tradition of research focusing on them, going back to the 1960s54. Several standardized measures have been developed to examine the targets and outcomes that are relevant from the perspective of the patient. In this context, the difference between nomothetic and idiographic outcome measures is relevant. Most outcome measures are nomothetic, which means that items of the measure are common to all people in varying degrees, and the measure is aimed at locating where a patient scores on that dimension55. Idiographic measures, on the other hand, rely on the unique features and views of the patient. For patient‐defined targets and outcomes of therapies, idiographic measures are obviously more relevant.
The oldest of these approaches is probably the Target Complaints54. In this approach, the patient describes three target complaints in a clinical interview, and for each of these complaints both the therapist and the patient rate how significant the problem is. After the treatment, both the patient and the therapist are asked to indicate on a five‐point scale how much each of these problems has improved.
Other patient‐generated outcome measures include the Psychological Outcome Profiles (PSYCHLOPS)56, the Simplified Personal Questionnaire57, and – in the field of child and adolescent mental health – the Youth Top Problems58. These instruments differ in terms of questions, possible answers and the point in time when they are rated. But the general idea is very much comparable with the Target Complaints, in the sense that the patient indicates which problems are important, to what extent he/she is affected by them, and the improvement during treatment. These measures differ from each other in terms of reliability and validity55, but all have been found to be useful as a clinical tool.
The evaluation of these patient‐defined targets and outcomes can be helpful in clinical practice in several ways59, such as better specifying problems identified by standardized measures, focusing the attention of the therapist on these issues, and increasing patients' influence in the shaping of the agenda of therapy.
There is also some qualitative research examining the personal targets and outcomes of psychotherapies, although most of this research has been conducted in small and selective samples46. The studies included patients receiving different types of therapy, and do not point at clear, consistent types of targets and outcomes that can apply across patients. Much of this research suggests that what patients find important in therapy depends on what they need at that stage in their lives59.
One study in a small group of patients used in‐depth qualitative interviews14, and found four categories of outcomes which were most important for patients: a) establishing new ways of relating to others; b) reduction in symptoms or change in patterns of behavior that used to bring suffering; c) better self‐understanding and insight; and d) accepting and valuing oneself.
Another, more recent study was aimed at integrating the results of qualitative research on helpful impacts of psychotherapies60, 61. Several categories of helpful impacts were identified, including awareness, insight and self‐understanding, behavioral change and solution of problems, empowerment, relief, and better understanding of feelings.
QUALITY OF LIFE AND RELATED TARGETS AND OUTCOMES
There is a growing consensus that trials of psychotherapies and other treatments of mental disorders should not only focus on symptoms of disorders as targets and outcomes, but also consider the broader concept of quality of life62. What quality of life exactly means, however, is not so clear. It can be seen as a multidimensional construct encompassing physical, psychological and social dimensions of health63. It comprises a range of life domains, including social relationships, physical abilities, mental health functioning, role functioning and engagement in daily activities64.
In most outcome studies of psychotherapies, quality of life is measured by self‐report instruments. There is a considerable body of research on the effects of psychotherapies on self‐reported quality of life for most mental disorders. The results of some of the most important meta‐analyses are summarized in Table 3. Significant effects of psychotherapies on quality of life were found for depression, eating disorders and anxiety disorders, compared to control conditions. No significant effects were found for schizophrenia.
Table 3.
Meta‐analyses of randomized trials examining the effects of psychotherapies compared to control conditions on quality of life
Study | Disorder | Type of therapy | Comparator | N studies | SMD | 95% CI | I2 |
---|---|---|---|---|---|---|---|
Linardon & Brennan65 | Eating disorders | CBT | Any control | 13 | 0.39 | 0.20‐0.57 | 56 |
Laws et al66 | Schizophrenia | CBT | Any control | 10 | 0.04 | –0.12 to 0.19 | 0 |
Hofmann et al67 | Anxiety disorders | CBT | Any control | 21 | 0.56 | 0.32‐0.80 | NA |
Kolovos et al64 | Depression | Any psychotherapy | Any control | 31 | 0.33 | 0.24‐0.42 | 21 |
Kamenov et al68 | Depression | Psychotherapy | Pharmacotherapy | 8 | 0.05 | –0.19 to 0.29 | NA |
Psychotherapy | Psychotherapy + pharmacotherapy | 6 | –0.36 | –0.62 to –0.11 | NA |
CBT – cognitive behavior therapy, SMD – standardized mean difference, NA – not available
Quality of life also encompasses more concrete areas such as income level, employment and housing status. Many interventions are available for patients with mental disorders that are aimed, for example, at helping them to get employment, or supporting them with housing69, 70. These interventions are, however, outside the scope of psychotherapy.
There is some research examining the effects of psychotherapies on broader areas of quality of life. For example, some meta‐analyses found that psychotherapies for depression not only have a significant effect on depressive symptoms, but also on social support (g=0.38; 95% CI: 0.29‐0.48)71 and social functioning (g=0.46, 95% CI: 0.32‐0.60)72. There are also indications from a small meta‐analysis that psychotherapy for depressed mothers may result in improved parental functioning (g=0.67; 95% CI: 0.30‐1.04), improved mother‐child interactions (g=0.35; 95% CI: 0.17‐0.52) and improved mental health of children (g=0.40; 95% CI: 0.22–0.59)73. In these meta‐analyses, a strong association was usually found between the effects on psychopathology and on aspects of quality of life.
INTERMEDIATE OUTCOMES: MEDIATORS AND WORKING MECHANISMS
Although most research on psychotherapies has focused on symptoms of disorders as outcome, psychotherapists from different schools have very diverse views on how these improvements are realized. Each type of therapy has its own theoretical model on how change is brought about in a patient. From a research perspective, CBT is dominating the field, with by far the majority of randomized trials focusing on this type of therapy.
CBT is focused on changing biases in thinking that are postulated to cause psychopathology, and CBT therapists assume that, when they succeed in changing these biases, the therapy is successful and the symptoms are taken away.
The evidence supporting the change in these biases as a mediator of CBT is, however, not very strong. Most research in this area has been conducted in depression. A meta‐analysis of 26 randomized trials of CBT for depression found that dys‐functional thinking did indeed change as a result of that therapy74. However, it also changed with other therapies, that are not specifically aimed at dysfunctional thinking, and there was no clear difference between CBT and these other therapies. It is therefore possible that dysfunctional thinking can better be seen as a manifestation of depression, that improves when depression improves, and not as a mediator or core part of the working mechanism of CBT. As such, there is no evidence that changing biases in thinking should indeed be regarded as a target or outcome of an individual psychotherapy.
One important category of psychotherapies, the psychodynamic ones, assume that psychopathology is related to the quality of the person's early attachment relationships75, and to significant childhood experiences that may have been accompanied by frustration, shame, loss, helplessness, loneliness, or guilt76. These experiences during developmental stages shape the personality and generate the vulnerability to psychopathology later in life. Symptoms of mental disorders are not seen as the core of the problem, but as a consequence of the broader personality problems. Therapies are therefore not aimed at symptoms but at solving the deeper intrapersonal problems. They are assumed to work via the reduction of unconscious conflicts77.
There is some discussion about whether or not unconscious problems can be measured empirically77, 78. Although there is no reason why they could not be examined as a mechanism of change of psychodynamic therapies, hardly any research on these mediators or mechanisms of change is available.
A third theoretical model for how psychotherapies work is the “common factors” one53, 79, 80, 81. In this model, psychotherapies are assumed not to work through the specific techniques that are employed, but through factors that are common across all types of therapies. The relationship between patient and therapist is an important common factor, but also the hope and expectations that the problems will be solved (through the rationale given by the therapist on what the causes of the problems are and how they can be solved). So, according to this model, the development of an effective relationship with the patient is a necessary target of the therapy.
The main problem with intermediate targets and goals of psychotherapies is that randomized trials can show that a therapy works, but it is much more complicated to show how a therapy works81, 82, 83. Research on working mechanisms and mediators to date is always correlational: in order to establish that a mediator is indeed a causal factor in the recovery process, studies not only have to show that the outcome as well as the mediator improves, but also that these improvements are associated with each other. In addition to that, a temporal relationship has to be shown (change in the mediator comes before change in the outcome), a dose‐response association has to be documented (stronger change in the mediator is associated with stronger change in the outcome), and evidence has to be provided that no third variable causes change in both the mediator and the outcome. And even if this is all demonstrated, supportive experimental research and a strong theoretical framework are needed to make a convincing case that a variable may indeed be a true mediator.
Currently, no (common or specific) factor meets these criteria and can thus be considered an empirically validated working mechanism. As Kazdin83 argues, “after decades of psychotherapy research, we cannot provide an evidence‐based explanation for how or why even our most well studied interventions produce change”. This means that psychotherapies can have intermediate targets and outcomes, but there is no evidence that these targets and outcomes do indeed have an impact on mental health problems.
NEGATIVE OUTCOMES
“First do no harm” is an important injunction in all biomedical interventions84. Negative effects are a specific type of targets and outcomes, in the sense that they should be avoided instead of realized. Although the importance of negative effects of psychotherapies has been described for several decades85, 86, only recently this is emerging as one of the core issues to be prioritized in research87, 88, 89, 90. At the moment, it can be said that there is a consensus in the field of psychotherapy research that negative effects should be better examined and that they have mostly been neglected in much of this research up to now89, 91.
It is not clear how negative outcomes of psychotherapies should be defined91, 92. Important types of negative outcomes include an increased risk of deterioration during therapy90 and serious adverse events93. However, there are many other types of negative outcomes that could be considered94. For example, non‐response and drop‐out can also be considered as negative outcomes.
There are several examples of so‐called “fringe” or potentially harmful therapies, such as rebirthing, scared straight interventions, critical incidence stress debriefing, and recovered‐memory techniques87, 95. Such therapies are assumed to have overall negative effects, and should be avoided altogether. However, negative effects can also occur in evidence‐based psychotherapies. Although the mean level of symptoms may improve with these therapies more than with control interventions, this does not mean that in some individuals the therapy cannot have negative effects.
Systematic research into negative effects of psychotherapies is mostly fairly recent. A conventional meta‐analysis of controlled trials of psychotherapies for depression found that only 6% of all trials reported deterioration rates90. The pooled risk ratio (RR) of deterioration in the 18 studies (23 comparisons) that did report these rates was 0.39 (95% CI: 0.27‐0.57), meaning that patients in the psychotherapy groups had a 61% lower chance to deteriorate than patients in the control groups. Most studies defined deterioration according to the criteria proposed by Jacobson and Truax96, which indicate that the patient's levels of psychopathology have become considerably worse and meet criteria for a severe disorder.
Individual patient data (IPD) meta‐analyses are better suited to examine deterioration rates in psychotherapy trials. Randomized trials typically do not have sufficient statistical power to detect differences in deterioration rates between different conditions, because these rates are usually low. In IPD meta‐analyses, the primary data from individual trials are collected and merged into one dataset. Because the resulting datasets are usually large, they have sufficient statistical power to examine relatively rare events, such as deterioration.
In one IPD meta‐analysis, 16 trials with 1,700 depressed patients comparing CBT with antidepressant medication were included97. Five to 7% of patients showed any deterioration (an increased score on the HAMD or BDI of one point), 1% showed reliable deterioration (increase of more than 8 points on the HAMD, or more than 9 points on the BDI), and 4 to 5% showed extreme non‐response (a post‐treatment HAMD score of 21 or higher, or a BDI score of more than 31). No significant difference between CBT and antidepressant medication was found on any of these rates.
In two other IPD meta‐analyses, deterioration rates in Internet‐based guided self‐help CBT for depression were examined. In one of them, data from 18 trials with 2,079 participants were included98. The rate of reliable deterioration was 3% in CBT and 8% in the control conditions (RR=0.47; 95% CI: 0.29‐0.75). In the other meta‐analysis, focusing on Internet‐based CBT without any human support, 13 trials with 3,805 participants were included, and it was found that 6% in the CBT conditions deteriorated, compared to 9% in the control conditions (odds ratio, OR=0.62; 95% CI: 0.46‐0.83)99.
ECONOMIC OUTCOMES
In economic studies, the outcomes of therapies are often measured through cost‐utility analyses (CUAs) or cost‐effectiveness analyses (CEAs)100.
For most mental disorders, no more than one or two CEAs or CUAs of psychotherapies are available. This is the case for bipolar disorder101, obsessive‐compulsive disorder102, social anxiety disorder, panic disorder, post‐traumatic stress disorder100, 103, and generalized anxiety disorder100, 104. For depression, more studies are available105. However, most of these studies focus on CBT, while for other therapies there is hardly any research. Available evidence does suggest that CBT for depression is cost‐effective compared to pharmacotherapy in the long term105.
A growing number of CEAs and CUAs have focused on Internet‐delivered interventions, with some evidence that they are more cost‐effective as compared to waiting list, care‐as‐usual, group cognitive behavior therapy, attention control, or telephone counseling106, although this is not confirmed in all studies107.
DIFFERENT PERSPECTIVES FROM DIFFERENT STAKEHOLDERS
In this paper we described the main types of targets and outcomes of psychotherapies. But, what is the most important target or outcome? That depends very much on whom you ask this question. Most outcome research is focused on symptoms of a mental disorder. However, as we noticed, patients may not consider symptom reduction as the only or the most important outcome. Therapists also have their own perspectives on the targets and outcomes of therapies. They typically work in health systems where they are assumed to treat the mental disorder of the patient. So, one of their main targets is to reduce the symptoms of the disorder. But they also want to help the patient to solve his/her personal problems. Furthermore, they usually work within a theoretical framework, such as the cognitive‐behavioral, the psychodynamic or the “common factor” model, each of which has important intermediate targets.
But there are further stakeholders. Health insurance companies also have their own views on what the targets and outcomes of therapies should be. They want the therapy to be effective, but to the lowest economic costs. Societies at large want therapies to help individual patients, but they also expect them to reduce the societal burden of mental disorders, in terms of economic costs, but also of problems caused in the public domain, for example by patients with an antisocial personality disorder. Relatives want the best outcomes for patients, but often also have their own targets and outcomes. Employers are particularly interested in getting patients with mental disorders back to work and as productive as they were before they developed the disorder.
So, the question of what is the most important target and outcome of a psychotherapy is very much dependent on the stakeholder considered. Currently, most research is focused on symptomatology of mental disorders, but it could easily be argued that patients should have a stronger voice in deciding what the most important outcomes are. Patients are the ones who suffer from mental disorders and, as long as we do not exactly know what these disorders are or what their causes may be, we should rely on the ones who suffer from them to decide what outcomes should have the priority.
CONCLUSIONS
It is not yet clear what mental disorders are and what are the causal pathways that lead to them. That makes it difficult to decide what the targets and outcomes of psychotherapies should be. In this paper, the different perspectives on this issue and the different types of outcomes were described.
The DSM and ICD systems have dominated the research field in the past decades and have led to a strong focus on core symptoms of mental disorders as the main outcome of therapies. However, there is growing criticism of the DSM/ICD systems and, in line with this, the question is increasingly raised whether symptoms should be the core outcome of therapies. This paper highlighted that patients often have different perspectives concerning targets and outcomes of psychotherapies. Quality of life is one of the broader types of outcomes being examined in randomized trials. Therapists have other intermediate targets, and that depends heavily on the type of therapy they are implementing, while there is very little evidence that these intermediate goals are associated with outcomes. Economic outcomes are also important for patients, health care providers, and societies. Patients should ultimately have the strongest voice in deciding what targets and outcomes of psychotherapies should have priority.
It is also important that a consensus in the research field is achieved on what the core outcomes of randomized trials of psychotherapies should be. Because of this lack of consensus, many different outcomes and instruments are used across trials. Even if the instruments measure the same constructs, their heterogeneity may cause inconsistencies in reporting and difficulties in comparing and combining the findings in systematic reviews and meta‐analyses108, 109, 110. Furthermore, the quality of outcome measures varies widely, and in many cases the most reliable and valid outcome measures are not selected108. Standardization of the selection of outcomes and their measures is therefore very much needed.
Several important types of outcomes have not been examined sufficiently in psychotherapy research, including outcomes from the patients' perspective, negative outcomes, mediators and intermediate targets and outcomes, as well as economic outcomes. It is important that more research is conducted on these outcomes.
The question of what the targets and outcomes of psychotherapies should be is not easy to answer and depends on which perspective one takes. Because of the huge burden of mental disorders, this is, however, an essential question, and answering it should be one of the priorities in the next decade.
REFERENCES
- 1. GBD 2016 Disease and Injury Incidence and Prevalence Collaborators . Global, regional, and national incidence, prevalence, and years lived with disability for 328 diseases and injuries for 195 countries, 1990‐2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet 2017;390:1211‐59. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Vigo D, Thornicroft G, Atun R. Estimating the true global burden of mental illness. Lancet Psychiatry 2016;3:171‐8. [DOI] [PubMed] [Google Scholar]
- 3. National Research Council (US) and Institute of Medicine (US) Committee on Depression, Parenting Practices, and the Healthy Development of Children . Depression in parents, parenting, and children: opportunities to improve identification, treatment and prevention. Washington: National Academies Press, 2009. [PubMed] [Google Scholar]
- 4. Reupert AE, Maybery DJ, Kowalenko NM. Children whose parents have a mental illness: prevalence, need and treatment. MJA Open 2012;1(Suppl. 1):7‐9. [DOI] [PubMed] [Google Scholar]
- 5. Beardslee WR, Gladstone TR, O‘Connor EE. Transmission and prevention of mood disorders among children of affectively ill parents: a review. J Am Acad Child Adolesc Psychiatry 2011;50:1098‐109. [DOI] [PubMed] [Google Scholar]
- 6. Bloom DE, Cafiero E, Jané‐Llopis E et al. The global economic burden of noncommunicable diseases. Geneva: World Economic Forum, 2011. [Google Scholar]
- 7. Liu NH, Daumit GL, Dua T et al. Excess mortality in persons with severe mental disorders: a multilevel intervention framework and priorities for clinical practice, policy and research agendas. World Psychiatry 2017;16:30‐40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Cuijpers P, Vogelzangs N, Twisk J et al. Comprehensive meta‐analysis of excess mortality in depression in the general community versus patients with specific illnesses. Am J Psychiatry 2014;171:453‐62. [DOI] [PubMed] [Google Scholar]
- 9. Krueger RF, Hopwood CJ, Wright AG et al. Challenges and strategies in helping the DSM become more dimensional and empirically based. Curr Psychiatry Rep 2014;16:515. [DOI] [PubMed] [Google Scholar]
- 10. Krueger RF, Kotov R, Watson D et al. Progress in achieving quantitative classification of psychopathology. World Psychiatry 2018;17:282‐93. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Widiger TA, Samuel DB. Diagnostic categories or dimensions? A question for the Diagnostic and Statistical Manual of Mental Disorders – fifth edition. J Abnorm Psychol 2005;114:494‐504. [DOI] [PubMed] [Google Scholar]
- 12. Cuthbert BN, Insel TR. Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC Med 2013;11:126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Greenberg G. The book of woe: The DSM and the unmaking of psychiatry. New York: Penguin, 2013. [Google Scholar]
- 14. Binder PE, Holgersen H, Nielsen GH. What is a “good outcome” in psychotherapy? A qualitative exploration of former patients' point of view. Psychother Res 2010;20:285‐94. [DOI] [PubMed] [Google Scholar]
- 15. Cuijpers P, Karyotaki E, Reijnders M et al. Was Eysenck right after all? A reassessment of the effects of psychotherapy for adult depression. Epidemiol Psychiatr Sci 2019;28:21‐30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Mohr DC, Ho J, Hart TL et al. Control condition design and implementation features in controlled trials: a meta‐analysis of trials evaluating psychotherapy for depression. Translat Behav Med 2014;4:407‐23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Cuijpers P, Berking M, Andersson G et al. A meta‐analysis of cognitive behavior therapy for adult depression, alone and in comparison to other treatments. Can J Psychiatry 2013;58:376‐85. [DOI] [PubMed] [Google Scholar]
- 18. Cuijpers P, Cristea IA, Karyotaki E et al. How effective are cognitive behavior therapies for ma‐jor depression and anxiety disorders? A meta‐analytic update of the evidence. World Psychiatry 2016;15:245‐58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Mayo‐Wilson E, Dias S, Mavranezouli I et al. Psychological and pharmacological interventions for social anxiety disorder in adults: a systematic review and network meta‐analysis. Lancet Psychiatry 2014;1:368‐76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Barkowski S, Schwartze D, Strauss B et al. Efficacy of group psychotherapy for social anxiety disorder: a meta‐analysis of randomized‐controlled trials. J Anxiety Disord 2016;39:44‐64. [DOI] [PubMed] [Google Scholar]
- 21. Pompoli A, Furukawa TA, Imai H et al. Psychological therapies for panic disorder with or without agoraphobia in adults: a network meta‐analysis. Cochrane Database Syst Rev 2016;4:CD011004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Mayo‐Wilson E, Montgomery P. Media‐delivered cognitive behavioural therapy and behavioural therapy (self‐help) for anxiety disorders in adults. Cochrane Database Syst Rev 2013;9:CD005330. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Cuijpers P, Sijbrandij M, Koole S et al. Psychological treatment of generalized anxiety disorder: a meta‐analysis. Clin Psychol Rev 2014;34:130‐40. [DOI] [PubMed] [Google Scholar]
- 24. Bisson JI, Roberts NP, Andrew M et al. Psychological therapies for chronic post‐traumatic stress disorder (PTSD) in adults. Cochrane Database Syst Rev 2013;12:CD003388. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Gerger H, Munder T, Gemperli A et al. Integrating fragmented evidence by network meta‐analysis: relative effectiveness of psychological interventions for adults with post‐traumatic stress disorder. Psychol Med 2014;44:3151‐64. [DOI] [PubMed] [Google Scholar]
- 26. Olatunji BO, Davis ML, Powers MB et al. Cognitive‐behavioral therapy for obsessive‐compulsive disorder: a meta‐analysis of treatment outcome and moderators. J Psychiatr Res 2013;47:33‐41. [DOI] [PubMed] [Google Scholar]
- 27. Ost LG, Havnen A, Hansen B et al. Cognitive behavioral treatments of obsessive‐compulsive disorder. A systematic review and meta‐analysis of studies published 1993‐2014. Clin Psychol Rev 2015;40:156‐69. [DOI] [PubMed] [Google Scholar]
- 28. Velthorst E, Koeter M, van der Gaag M et al. Adapted cognitive‐behavioural therapy required for targeting negative symptoms in schizophrenia: meta‐analysis and meta‐regression. Psychol Med 2015;45:453‐65. [DOI] [PubMed] [Google Scholar]
- 29. Burns AM, Erickson DH, Brenner CA et al. Cognitive‐behavioral therapy for medication‐resistant psychosis: a meta‐analytic review. Psychiatr Serv 2014;65:874‐80. [DOI] [PubMed] [Google Scholar]
- 30. Eichner C, Berna F. Acceptance and efficacy of metacognitive training (MCT) on positive symptoms and delusions in patients with schizophrenia: a meta‐analysis taking into account important moderators. Schizophr Bull 2016;42:952‐62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Chatterton ML, Stockings E, Berk M et al. Psychosocial therapies for the adjunctive treatment of bipolar disorder in adults: network meta‐analysis. Br J Psychiatry 2017;210:333‐41. [DOI] [PubMed] [Google Scholar]
- 32. Chiang KJ, Tsai JC, Liu D et al. Efficacy of cognitive‐behavioral therapy in patients with bipolar disorder: a meta‐analysis of randomized controlled trials. PLoS One 2017;12:e0176849. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Driessen E, Hollon SD, Bockting CLH et al. Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta‐analysis of US National Institutes of Health‐funded trials. PLoS One 2015;10:e0137864. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Cuijpers P, Smit F, Bohlmeijer ET et al. Is the efficacy of cognitive behaviour therapy and other psychological treatments for adult depression overestimated? A meta‐analytic study of publication bias. Br J Psychiatry 2010;196:173‐8. [DOI] [PubMed] [Google Scholar]
- 35. Dal‐Ré R, Bobes J, Cuijpers P. Why prudence is needed when interpreting articles reporting clinical trials results in mental health. Trials 2017;18:143. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Cuijpers P, van Straten A, Bohlmeijer E et al. The effects of psychotherapy for adult depression are overestimated: a meta‐analysis of study quality and effect size. Psychol Med 2010;40:211‐23. [DOI] [PubMed] [Google Scholar]
- 37. Munder T, Brütsch O, Leonhart R et al. Researcher allegiance in psychotherapy outcome research: an overview of reviews. Clin Psychol Rev 2013;33:501‐11 [DOI] [PubMed] [Google Scholar]
- 38. Leykin Y, DeRubeis RJ. Allegiance in psychotherapy outcome research: separating association from bias. Clin Psychol Sci Pract 2009;16:4‐65. [Google Scholar]
- 39. Furukawa TA, Noma H, Caldwell DM et al. Waiting list may be a nocebo condition in psychotherapy trials: a contribution from network meta‐analysis. Acta Psychiatr Scand 2014;130:181‐92. [DOI] [PubMed] [Google Scholar]
- 40. Gold SM, Enck P, Hasselmann H et al. Control conditions for randomized trials of behavioural interventions in psychiatry: a decision framework. Lancet Psychiatry 2017;4:725‐32. [DOI] [PubMed] [Google Scholar]
- 41. Beck AT, Ward CH, Mendelson M et al. An inventory for measuring depression. Arch Gen Psychiatry 1961;4:561‐71. [DOI] [PubMed] [Google Scholar]
- 42. Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry 1960;23:56‐62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Cipriani A, Furukawa TA, Salanti G et al. Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta‐analysis. Lancet 2018;391:1357‐66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Hunt M. How science takes stock: the story of meta‐analysis. New York: Russell Sage Foundation, 1997. [Google Scholar]
- 45. Cuijpers P, Li J, Hofmann SG et al. Self‐reported versus clinician‐rated symptoms of depression as outcome measures in psychotherapy research on depression: a meta‐analysis. Clin Psychol Rev 2010;30:768‐78. [DOI] [PubMed] [Google Scholar]
- 46. Hill CE, Chui H, Baumann E. Revisiting and reenvisioning the outcome problem in psychotherapy: an argument to include individualized and qualitative measurement. Psychotherapy 2013;50:68‐76. [DOI] [PubMed] [Google Scholar]
- 47. Yalom ID. Existential psychotherapy. New York: Basic Books, 1980. [Google Scholar]
- 48. Lilienfeld SO, Treadway MT. Clashing diagnostic approaches: DSM‐ICD versus RDoC. Annu Rev Clin Psychol 2016;12:435‐63. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49. Borsboom D. A network theory of mental disorders. World Psychiatry 2017;16:5‐13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50. Insel T, Cuthbert B, Garvey M et al. Research Domain Criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry 2010;167:748‐51. [DOI] [PubMed] [Google Scholar]
- 51.Insel TR. The NIMH Research Domain Criteria (RDoC) Project: precision medicine for psychiatry. Am J Psychiatry 2014;171:395‐7. [DOI] [PubMed] [Google Scholar]
- 52. Kealy D, Joyce AS, Weber R et al. What the patient wants: addressing patients' treatment targets in an integrative group psychotherapy programme. Psychol Psychother 2019;92:20‐38. [DOI] [PubMed] [Google Scholar]
- 53. Wampold BE. How important are the common factors in psychotherapy? An update. World Psychiatry 2015;14:270‐7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54. Battle CC, Imber SD, Hoehn‐Saric R et al. Target complaints as criteria of improvement. Am J Psychother 1966;20:184‐92. [DOI] [PubMed] [Google Scholar]
- 55. Sales CMD, Alves PCG. Patient‐centered assessment in psychotherapy: a review of individualized tools. Clin Psychol Sci Pract 2016;23:265‐83. [Google Scholar]
- 56. Ashworth M, Shepherd M, Christey J et al. A client‐generated psychometric instrument: the development of ‘PSYCHLOPS’. Couns Psychother Res 2004;4:27‐31. [Google Scholar]
- 57. Elliott R, Wagner J, Sales CMD et al. Psychometrics of the Personal Questionnaire: a client‐generated outcome measure. Psychol Assess 2016;28:263‐78. [DOI] [PubMed] [Google Scholar]
- 58. Weisz JR, Chorpita BF, Frye A et al. Youth Top Problems: using idiographic, consumer‐guided assessment to identify treatment needs and to track change during psychotherapy. J Consult Clin Psychol 2011;79:369‐80. [DOI] [PubMed] [Google Scholar]
- 59. McLeod J. Qualitative research in counselling and psychotherapy. London: Sage, 2011. [Google Scholar]
- 60. Timulak L, Keogh D. The client's perspective on (experiences of) psychotherapy: a practice friendly review. J Clin Psychol 2017;73:1556‐67. [DOI] [PubMed] [Google Scholar]
- 61. Timulak L, McElvaney R. Qualitative meta‐analysis of insight events in psychotherapy. Couns Psychol Quart 2013;26:131‐50. [Google Scholar]
- 62. Saarni SI, Suvisaari J, Sintonen H et al. Impact of psychiatric disorders on health‐related quality of life: general population survey. Br J Psychiatry 2007;190:326‐32. [DOI] [PubMed] [Google Scholar]
- 63. WHOQOL group . The World Health Organization Quality of Life assessment (WHOQOL): position paper from the World Health Organization. Soc Sci Med 1995;41:1403‐9. [DOI] [PubMed] [Google Scholar]
- 64. Kolovos S, Kleiboer A, Cuijpers P. The effect of psychotherapy for depression on quality of life: a meta‐analysis. Br J Psychiatry 2016;209:460‐8. [DOI] [PubMed] [Google Scholar]
- 65. Linardon J, Brennan L. The effects of cognitive‐behavioral therapy for eating disorders on quality of life: a meta‐analysis. Int J Eat Disord 2017;50:715‐30. [DOI] [PubMed] [Google Scholar]
- 66. Laws KR, Darlington N, Kondel TK. Cognitive behavioural therapy for schizophrenia – outcomes for functioning, distress and quality of life: a meta‐analysis. BMC Psychol 2018;6:32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67. Hofmann SG, Wu JQ, Boettcher H. Effect of cognitive behavioral therapy for anxiety disorders on quality of life: a meta‐analysis. J Consult Clin Psychol 2014;82:375–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68. Kamenov K, Twomey C, Cabello M et al. The efficacy of psychotherapy, pharmacotherapy and their combination on functioning and quality of life in depression: a meta‐analysis. Psychol Med 2017;47:414‐25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69. Bitter NA, Roeg DP, van Nieuwenhuizen C et al. Identifying profiles of service users in housing services and exploring their quality of life and care needs. BMC Psychiatry 2016;16:419. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70. Nieuwenhuijsen K, Faber B, Verbeek JH et al. Interventions to improve return to work in depressed people. Cochrane Database Syst Rev 2014;12:CD006237. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71. Park M, Cuijpers P, van Straten A et al. The effects of psychological treatments of adult depression on social support: a meta‐analysis. Cogn Ther Res 2014;38:600‐11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72. Renner F, Cuijpers P, Huibers MJH. The effect of psychotherapy for depression on improvements in social functioning: a meta‐analysis. Psychol Med 2013;44:2913‐26. [DOI] [PubMed] [Google Scholar]
- 73. Cuijpers P, Weitz E, Karyotaki E et al. The effects of psychological treatment of maternal depression on children and parental functioning: a meta‐analysis. Eur Child Adolesc Psychiatry 2014;24:237‐45. [DOI] [PubMed] [Google Scholar]
- 74. Cristea IA, Huibers MJ, David D et al. The effects of cognitive behavior therapy for adult depression on dysfunctional thinking: a meta‐analysis. Clin Psychol Rev 2015;42:62‐71. [DOI] [PubMed] [Google Scholar]
- 75. Bowlby J. A secure base. London: Routledge, 2005. [Google Scholar]
- 76. Kohut H. The analysis of the self: a systematic approach to the psychoanalytic treatment of narcissistic personality disorders. New York: International Universities Press, 1971. [Google Scholar]
- 77. Leichsenring F, Steinert C, Crits‐Christoph P. On mechanisms of change in psychodynamic therapy. Z Psychosom Med Psychother 2018;64:16‐22. [DOI] [PubMed] [Google Scholar]
- 78. Hoffart A, Johnson SU. Psychodynamic and cognitive‐behavioral therapies are more different than you think: conceptualizations of mental problems and consequences for studying mechanisms of change. Clin Psychol Sci 2017;5:1070‐86. [Google Scholar]
- 79. Frank JD. Persuasion and healing: a comparative study of psychotherapy. Baltimore: Johns Hopkins University Press, 1961. [Google Scholar]
- 80. Wampold BE, Imel ZE. The great psychotherapy debate: the evidence for what makes psychotherapy work, 2nd ed. New York: Routledge, 2015. [Google Scholar]
- 81. Cuijpers P, Reijnders M, Huibers MJH. The role of common factors in psychotherapy outcome. Annu Rev Clin Psychol 2019;15:207‐31. [DOI] [PubMed] [Google Scholar]
- 82. Kazdin AE. Understanding how and why psychotherapy leads to change. Psychother Res 2009;19:418‐28. [DOI] [PubMed] [Google Scholar]
- 83. Kazdin AE. Mediators and mechanisms of change in psychotherapy research. Annu Rev Clin Psychol 2007;3:1‐27. [DOI] [PubMed] [Google Scholar]
- 84. Sokol DK. “First do no harm” revisited. BMJ 2013;347:f6426. [DOI] [PubMed] [Google Scholar]
- 85. Hadley SW, Strupp HH. Contemporary views of negative effects in psychotherapy. Arch Gen Psychiatry 1976;33:1291‐302. [DOI] [PubMed] [Google Scholar]
- 86. Mohr DC. Negative outcome in psychotherapy: a critical review. Clin Psychol Sci Pract 1995;2:1‐27. [Google Scholar]
- 87. Berk M, Parker G. The elephant on the couch: side‐effects of psychotherapy. Aust N Z J Psychiatry 2009;43:787‐94. [DOI] [PubMed] [Google Scholar]
- 88. Lilienfeld SO. Psychological treatments that cause harm. Perspect Psychol Sci 2007;2:53‐70. [DOI] [PubMed] [Google Scholar]
- 89. Barlow DH. Negative effects from psychological treatments: a perspective. Am Psychol 2010;65:13‐20. [DOI] [PubMed] [Google Scholar]
- 90. Cuijpers P, Reijnders M, Karyotaki E et al. Negative effects of psychotherapies for adult depression: a meta‐analysis of deterioration rates. J Affect Disord 2018;239:138‐45. [DOI] [PubMed] [Google Scholar]
- 91. Dimidjian S, Hollon SD. How would we know if psychotherapy were harmful? Am Psychol 2010;65:21‐33. [DOI] [PubMed] [Google Scholar]
- 92. Boisvert CM. Negative treatment effects: is it time for a black box warning? Am Psychol 2010;65:680‐1. [DOI] [PubMed] [Google Scholar]
- 93. Rozental A, Andersson G, Boettcher J et al. Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interv 2014;1:12‐19. [Google Scholar]
- 94. Linden M. How to define, find and classify side effects in psychotherapy: from unwanted events to adverse treatment reactions. Clin Psychol Psychother 2013;20:286‐96. [DOI] [PubMed] [Google Scholar]
- 95. Beyerstein BL. Fringe psychotherapies: the public at risk. Sci Rev Altern Med 2001;5:70‐9. [Google Scholar]
- 96. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12‐9. [DOI] [PubMed] [Google Scholar]
- 97. Vittengl JR, Jarrett RB, Weitz E et al. Divergent outcomes in cognitive behavioral therapy and pharmacotherapy for adult depression. Am J Psychiatry 2015;173:481‐90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 98. Ebert DD, Donkin L, Andersson G et al. Does Internet‐based guided‐self‐help for depression cause harm? An individual participant data meta‐analysis on deterioration rates and its moderators in randomized controlled trials. Psychol Med 2016;46:2679‐93. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 99. Karyotaki E, Kemmeren L, Riper H et al. Is self‐guided internet‐based cognitive behavioural therapy (iCBT) harmful? An individual participant data meta‐analysis. Psychol Med 2018;48:2456‐66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 100. Ophuis RH, Lokkerbol J, Heemskerk SC et al. Cost‐effectiveness of interventions for treating anxiety disorders: a systematic review. J Affect Disord 2017;210:1‐13. [DOI] [PubMed] [Google Scholar]
- 101. Pari AA, Simon J, Wolstenholme J et al. Economic evaluations in bipolar disorder: a systematic review and critical appraisal. Bipolar Disord 2014;16:557‐82. [DOI] [PubMed] [Google Scholar]
- 102. Skapinakis P, Caldwell D, Hollingworth W. A systematic review of the clinical effectiveness and cost‐effectiveness of pharmacological and psychological interventions for the management of obsessive‐compulsive disorder in children/adolescents and adults. Health Technol Assess 2016;20:43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 103. Konnopka A, Leichsenring F, Leibing E et al. Cost‐of‐illness studies and cost‐effectiveness analyses in anxiety disorders: a systematic review. J Affect Disord 2009;114:14‐31. [DOI] [PubMed] [Google Scholar]
- 104. Bereza BG, Machado M, Einarson TR. Systematic review and quality assessment of economic evaluations and quality‐of‐life studies related to generalized anxiety disorder. Clin Ther 2009;31:1279‐308. [DOI] [PubMed] [Google Scholar]
- 105. Karyotaki E, Tordrup D, Buntrock C et al. Economic evidence for the clinical management of major depressive disorder: a systematic review and quality appraisal of economic evaluations alongside randomized controlled trials. Epidemiol Psychiatr Sci 2017;26:501‐16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 106. Donker T, Blankers M, Hedman E et al. Economic evaluations of Internet interventions for mental health: a systematic review. Psychol Med 2015;45:3357‐76. [DOI] [PubMed] [Google Scholar]
- 107. Kolovos S, van Dongen JM, Riper H et al. Cost effectiveness of guided Internet‐based interventions for depression in comparison with control conditions: an individual‐participant data meta‐analysis. Depress Anxiety 2018;35:209‐19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 108. Prinsen CAC, Vohra S, Rose MR et al. How to select outcome measurement instruments for outcomes included in a “Core Outcome Set” – a practical guideline. Trials 2016;17:449. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 109. Williamson PR, Altman DG, Blazeby JM et al. Developing core outcome sets for clinical trials: issues to consider. Trials 2012;13:132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 110. Gorst SL, Gargon E, Clarke M et al. Choosing important health outcomes for comparative effectiveness research: an updated review and user survey. PLoS One 2016;11:e0146444. [DOI] [PMC free article] [PubMed] [Google Scholar]