Frequency of reporting of adverse events in randomized controlled trials of psychotherapy vs. psychopharmacotherapy

Abstract

Background

Psychopharmacology and psychotherapy are the two main therapies in mental health. It is common practice to consider adverse events (AEs) of medications, but it’s not clear this occurs with psychotherapy.

Aim

This study investigates the frequency with which reports of AEs occur in clinical trials using either psychopharmacology alone, psychotherapy alone, or combined approaches.

Methods

Forty-five articles of randomized trials published in high-impact journals were chosen from a Medline search, and separated into three groups of 15 articles: pharmacotherapy alone (M), psychotherapy alone (T) and combined studies that looked at the effect of both a psychotherapeutic (CT) and psychopharmacologic (CM) intervention. Criteria for what defines an AE were established and the papers were rated for mentions of AEs in papers as a whole and by each section.

Results

The χ²-analysis of AE mentions showed significant differences between the four study conditions in terms of each paper as a whole (χ²: 10.1, p < 0.018), and by section. Medication (M + CM) and psychotherapy papers (T + CT) were then combined into two groups to compare the odds that one was more likely to mention AEs than the other. Bivariate logistic regression yielded statistically significant odds ratios ranging from 9.33 to 20.99, with medications being far more likely to mention AEs.

Conclusion

We believe the difference in reports of AEs mirrors the attitudes researchers and providers. It’s critical to consider, and standardize the definition of, AEs in psychotherapy, and imperative to identify and address potential AEs in psychotherapy research.

1. Introduction

Psychopharmacology and psychotherapy are the two main therapeutic modalities for the treatment of behavioral and emotional problems. Each approach may be used individually or the two may be practiced concurrently, as is often the case. As adverse events can occur in any form of treatment, it is important to be aware of the nature and frequency of adverse consequences of each modality. To this end, and as mandated by the Food and Drug Administration (FDA), medications are tested and screened carefully for side effects during their development and post-marketing period [1]. It is common practice to inform patients about possible side effects and risk–benefit ratios whenever psychotropic drugs are initially prescribed. On the other hand, it is unclear whether the frequency and nature of adverse events are so rigorously explored in regard to the application of psychotherapy. Barlow has recently noted the lack of attention to this issue within the psychotherapy community, concluding that “it is time to focus attention in a more systematic manner on those unfortunate cases where harm might occur or benefit is conspicuously absent [2].” One article emerging from the STAR*D report noted an increase in suicidality after the initiation of cognitive therapy [3]. The authors state in their discussion they “thought it noteworthy that although the U.S. Food and Drug Administration warns of the emergence of suicidal ideation as a hazard following initiation of antidepressant medication, several cases of suicidal ideation occurred as serious adverse events following the initiation of cognitive therapy in our study [3].” Thus as an initial step towards improving our understanding of the potential for adverse events in psychotherapy, we investigated the frequency with which adverse events were mentioned in randomized controlled trials of both psychotherapy and pharmacotherapy. More specifically, the study outlined below investigates the frequency with which reports of clinical trials using either psychopharmacology alone, psychotherapy alone, or combined approaches, consider the incidence of adverse events in their outcome data. It is not the purpose of this paper to document the frequency with which adverse events actually occur; rather it is our intention to document the relative frequency of their consideration by the study authors. Our hypothesis is that researchers are more apt to consider and report the possibility of adverse events when addressing the use of medications. We believe this study provides a necessary first step towards better assessment of AEs in psychotherapy research.

2. Methods

2.1. Article selection

A Medline search in journals of psychiatry and psychology of randomized controlled trials limited to Axis I disorders was performed which yielded over 10,000 hits (see Fig. 1). To narrow down the selection of articles, the following inclusion/exclusion criteria were used: 1) publication in a journal with high impact factor (i.e., > 5); 2) Phase II, III &IV clinical psychopharmacology trials; 3) psychotherapy trials that studied commonly used therapeutic modalities (e.g., cognitive therapy, supportive therapy, group therapy, etc.); 4) reviews, editorials, meta-analyses, practice guidelines and brief reports were excluded. Out of the remaining pool of articles, 15 articles (see Table 1) were chosen at random for each of three groups: pharmacology trials alone (M), psychotherapy trials alone (T), and combined trials (C), in which the effects of both psychotherapy and psychopharmacology were under investigation. Of note, in some of the psychotherapy alone (T) articles, study participants may have been taking medication as well. However, the effects of medication were not the focus of those studies and were not measured for outcomes.

Fig. 1.

Outline of Journal Publication Search.

Table 1.

Articles selected for the formal analysis of adverse event mentions.

Journal	Title		Lead Author
Pharmacotherapy Trials (M)
Am J Psychiatry	1.	A Double-Blind, Randomized, Placebo-Controlled Trial of Oxcarbazepine in the Treatment of Bipolar Disorder in Children and Adolescents (2006)	Wagner, K.
	2.	Effects of Olanzapine, Quetiapine, and Risperidone on Neurocognitive Function in Early Psychosis: A Randomized, Double-Blind 52-Week Comparison (2007)	Keefe, R.
	3.	Fluoxetine Treatment for Depression in Patients With HIV and AIDS: A Randomized, Placebo-Controlled Trial (1999)	Rabkin, J.
	4.	Initial Phase 2 Trial of a Nicotinic Agonist in Schizophrenia (2008)	Freedman, R.
	5.	Multicenter Investigation of the Opioid Antagonist Nalmefene in the Treatment of Pathological Gambling (2006)	Grant, J.
	6.	Randomized, Placebo-Controlled Trial of Paroxetine Versus Imipramine in Depressed HIV-Positive Outpatients (1998)	Elliott, A
	7.	Selective Muscarinic Receptor Agonist Xanomeline as a Novel Treatment Approach for Schizophrenia (2008)	Shekhar, A.
Arch Gen Psych	8.	A Randomized, Placebo-Controlled 12-Month Trial of Divalproex and Lithium in Treatment of Outpatients With Bipolar I Disorder (2000)	Bowden, C.
	9.	Imipramine Treatment of Opiate-Dependent Patients With Depressive Disorders (1998)	Nunes, E.
	10.	Multicenter, Double-blind Comparison of Sertraline and Placebo in the Treatment of Posttraumatic Stress Disorder (2001)	Davidson, J.
	11.	Pregabalin for Treatment of Generalized Anxiety Disorder (2005)	Rickels, K.
	12.	Selecting Among Second-Step Antidepressant Medication Monotherapies (2008)	Rush, A.
J Clin Psych	13.	A Randomized, Double-Blind Study of Increasing or Maintaining Duloxetine Dose in Patients Without Remission of Major Depressive Disorder After Initial Duloxetine Therapy (2008)	Kornstein, S.
	14.	Acute Treatment of Pediatric Bipolar I Disorder, Manic or Mixed Episode, With Aripiprazole: A Randomized, Double- Blind, Placebo-Controlled Study (2009)	Findling, R.
	15.	Double-Blind Study of Dextroamphetamine Versus Caffeine Augmentation for Treatment-Resistant Obsessive– Compulsive Disorder (2009)	Koran, L.
Psychotherapy Trials (T)
Am J Psychiatry	1.	A Randomized Controlled Clinical Trial of Psychoanalytic Psychotherapy for Panic Disorder (2007)	Milrod, B.
	2.	Short-Term Psychodynamic Psychotherapy and Cognitive–Behavioral Therapy in Generalized Anxiety Disorder: A Randomized, Controlled Trial (2009)	Leichsenring, F.
	3.	Transdiagnostic Cognitive–Behavioral Therapy for Patients With Eating Disorders: A Two-Site Trial With 60-Week Follow-Up (2009)	Fairburn, C.
Arch Gen Psych	4.	A Randomized Controlled Comparison of Family-Based Treatment and Supportive Psychotherapy for Adolescent Bulimia Nervosa (2007)	le Grange, D.
	5.	A Randomized Effectiveness Trial of Interpersonal Psychotherapy for Depressed Adolescents (2004)	Mufson, L.
British J Psych	6.	Effects of cognitive therapy on psychological symptoms and social functioning in residual depression (2000)	Scott, J.
BMJ	7.	Problem solving treatment and group psychoeducation for depression: multicentre randomized controlled trial (2000)	Dowrick, C.
JAMA	8.	Cognitive Behavioral Therapy for Posttraumatic Stress Disorder in Women (2007)	Schnurr, P.
	9.	Cognitive–Behavioral Therapy for Anxiety Disordered Youth: A Randomized Clinical Trial Evaluating Child and Family Modalities (2008)	Kendall, P.
J Consult Clin Psychol	10.	Cognitive–Behavioral Treatment for Depression in Smoking Cessation (2001)	Brown, R.
	11.	Efficacy of Applied Relaxation and Cognitive–Behavioral Therapy in the Treatment of Generalized Anxiety Disorder (1993)	Borkovec, T.
	12.	Extreme Nonresponse in Cognitive Therapy: Can Behavioral Activation Succeed Where Cognitive Therapy Fails? (2007)	Coffman, S.
	13.	Short- and Long-Term Effectiveness of an Empirically Supported Treatment for Agoraphobia (2001)	Hahlweg, K.
	14.	Testing the Efficacy of Theoretically Derived Improvements in the Treatment of Social Phobia (2009)	Rapee, R.
Psychotherapy Research	15.	In vivo processes in cognitive therapy for depression: Frequency and benefits (2005)	Kanter, J.
Combined Psychopharmacology and Psychotherapy Trials (C)
Am J Psychiatry	1.	Augmentation of Behavior Therapy With d-Cycloserine for Obsessive–Compulsive Disorder (2008)	Wilhelm, S.
	2.	Cognitive Therapy Versus Medication in Augmentation and Switch Strategies as Second-Step Treatments: A STAR*D Report (2007)	Thase, M.
	3.	Combined Pharmacotherapy and Psychotherapy as Maintenance Treatment for Late-Life Depression: Effects on Social Adjustment (2002)	Lenze, E.
	4.	Suicide Attempts in Patients With Bipolar I Disorder During Acute and Maintenance Phases of Intensive Treatment With Pharmacotherapy and Adjunctive Psychotherapy (2002)	Rucci, P.
	5.	Treatment of Bereavement-Related Major Depressive Episodes in Later Life: A Controlled Study of Acute and Continuation Treatment With Nortriptyline and Interpersonal Psychotherapy (1999)	Reynolds, C.
Arch Gen Psych	6.	A Randomized Effectiveness Trial of Cognitive–Behavioral Therapy and Medication for Primary Care Panic Disorder (2005)	Roy-Byrne, P.
	7.	Chronic Depression Medication (Nefazodone) or Psychotherapy (CBASP) Is Effective When the Other Is Not (2005)	Schatzberg, A.
	8.	Cognitive Behavioral Group Therapy vs Phenelzine Therapy for Social Phobia (1998)	Heimberg, R.
	9.	Cognitive Therapy vs Medications in the Treatment of Moderate to Severe Depression (2005)	DeRubeis, R.
	10.	Efficacy of Disulfiram and Cognitive Behavior Therapy in Cocaine-Dependent Outpatients (2004)	Carroll, K.
	11.	Fluoxetine, Comprehensive Cognitive Behavioral Therapy, and Placebo in Generalized Social Phobia (2004)	Davidson, J.
	12.	Nortriptyline and Cognitive–Behavioral Therapy in the Treatment of Cigarette Smoking (1998)	Hall, S.
	13.	Psychological Intervention and Antidepressant Treatment in Smoking Cessation (2002)	Hall, S.
J Consult Clin Psychol	14.	Randomized Trial of Behavioral Activation, Cognitive Therapy, and Antidepressant Medication in the Prevention of Relapse and Recurrence in Major Depression (2008)	Dobson, K.
	15.	Social Phobia: A Comparison of Behavior Therapy and Atenolol (1994)	Turner, S.

2.2. Rating process

Two study investigators (BV and MA) reviewed the articles and rated them for mentions of possible (denoting that the authors were considering the possibility) or actual adverse events (AEs). The articles were rated for the presence or absence of AE mentions in the paper as a whole and also in each of the four article sections: Introduction, Methods, Results and Conclusion/Discussion. The 15 combined trials (C) were rated for AE mentions attributed to medications (CM) and also for AE mentions attributed to psychotherapy (CT). In sum, there were four study conditions (M, T, CM and CT) that were assessed over a total of 180 paper sections.

2.3. Identification of adverse events

An AE was defined as a deleterious result attributed directly to a treatment intervention. Sometimes, it was not clear whether a negative outcome was considered an AE of the study intervention. A system for identifying mentions of AEs was developed and is detailed below.

The coding system for rating AE mentions delineated specific linguistic terms and clinical constructs deemed to be related to AEs.

• Linguistic terms considered to explicitly denote AEs included: “adverse event(s),” “side effect(s),” “risks and benefits” (only when the risks of the intervention were stated explicitly), “safety,” “negative effect(s),” “acceptability/suitability” (only when this referred to the intervention being potentially harmful to patients, thus making it unacceptable/unsuitable), and “tolerability.”

• Mentions of worsening of mood symptoms that were explicitly attributed to an intervention were considered AEs.

• References to “poor,” “unproductive,” or “hostile therapy sessions,” along with mentions of “therapist competence,” “adherence,” or “poor therapeutic alliance” were included as AE mentions only if these phenomena were presented as possibly or actually leading to clinical deterioration or negatively impacting the tolerability of an intervention.

• Mentions of contraindications to treatment (e.g., patients with renal failure being excluded from trials with a renotoxic medication, or patients with a heart condition being excluded from trials of stressful exposure therapy for phobias) were rated as AE mentions. In other words, they were felt to be an acknowledgment of potential side effect issues.

• Terms such as “inefficacy,” “unsatisfactory effect,” “dissatisfaction,” “less well” and “inadequate improvement” were not considered AE mentions, nor were references to dropouts/attrition that were not explicitly reported as an outcome of the intervention being studied.

• Also, the phrase “informed consent” was not included as an AE mention unless the risks for which consent was given were denoted.

For combined studies with pharmacological and psychotherapeutic interventions (C), it was not always clear whether an AE reported was due to one modality or the other, as some authors stated the overall rates of AEs without specific attributions. In these cases the consideration of possible AEs was attributed to both modalities.

Prior to scoring the articles used in the analysis, inter-rater reliability was first established, with the two raters (BV and MA) using 15 articles not included in the later analysis. Five articles involved pharmacotherapy trials, five involved psychotherapy trials and the remaining five involved combination trials for pharmaco- and psycho- therapy. The presence of mentions of AEs was rated for each of the four study types listed above. As with the later formal analysis, ratings were made for the paper as a whole and for each of the four article sections: Introduction, Methods, Results and Conclusion/Discussion.

2.4. Statistical analysis

In order to compare the frequency with which AEs were mentioned across the four different study conditions (M, T, CM and CT), χ² analyses were performed for any mention of an AE in the entire paper, and also in each section of the paper. Next, the medication studies (M and CM) were grouped together, as were the psychotherapy studies (T and CT) in order to compare mentions of AEs attributable to the two overarching modalities — pharmacotherapy and psychotherapy. Bivariate logistic regressions were performed to determine the odds ratios of AE mentions between the two primary research modalities, again comparing mentions in the paper as a whole and then by each section.

3. Results

We obtained very strong inter-rater reliability on the 15 pre-analysis articles. There was 100% (15/15) agreement on the mentions of AEs in articles as a whole and 96.3% (77/80) when comparing by section of the paper. For both the M and T conditions, the agreement was 100% (20/20) across sections. In the CM condition, the agreement was 95% (19/20) and in the CT it was 90% (18/20). After assessing inter-rater reliability, the discordant ratings within the subset were discussed and differences resolved prior to the initiation of the later formal ratings.

The χ² analysis of AE mentions showed significant differences between the four study conditions in terms of each paper as a whole (χ² = 10.1, p < 0.018), and also by each section of the study (see Table 2). Medication papers (M + CM) and psychotherapy papers (T + CT) were then combined into two groups to compare the odds that one modality was more likely to contain AE mentions than the other. Bivariate logistic regression analyses of mentions of AEs yielded odds ratios ranging from 9.33 to 20.99, all with p-values < 0.05, with medications (M + CM) in all cases being far more likely to have AEs attributed to them than psychotherapies (T + CT) — in terms of both the paper as a whole, and also by section (see Table 3).

Table 2.

Number and percentage of articles and article sections that mention adverse events across different types of studies.

Section	Medication Only (M)	Psychotherapy Only (T)	Combined Med (CM)	Combined Psychotx (CT)	χ2 statistic	p-value
	N (%)	N (%)	N (%)	N (%)
Whole Paper	15 (100%)	9 (60.0%)	13 (86.7%)	9 (60.0%)	10.06	0.018
Intro	12 (80%)	2 (13.3%)	4 (26.7%)	0 (0.0%)	26.35	<0.001
Methods	15 (100%)	7 (46.7%)	13 (86.7%)	5 (33.3%)	20.40	<0.001
Results	15 (100%)	6 (40.0%)	12 (80.0%)	8 (60.0%)	15.02	<0.020
Conclusion	15 (100%)	5 (33.3%)	10 (66.7%)	5 (33.3%)	18.86	<0.001

Table 3.

Bivariate logistic regression analyses comparing proportion of articles and article sections which mention adverse events.

Section	All Medication (M + CM) Studies	All Psychotherapy (T + CT) Studies	Odds Ratio	95% CI	p-value
	N (%)	N (%)
Whole Paper	28 (93.0%)	18 (60.0%)	9.33	1.87–46.66	0.007
Intro	16 (53.3%)	2 (6.7%)	16.00	3.22–79.52	0.001
Methods	28 (93.3%)	12 (40.0%)	20.99	4.20–105.0	<0.001
Results	27 (90.0%)	14 (46.7%)	10.29	2.56–41.37	0.001
Conclusion	25 (83.0%)	10 (33.0%)	10.00	2.90–34.00	<0.001

All medication (M + CM) studies vs. all psychotherapy (T + CT) studies.

4. Discussion

The findings of this study suggest that pharmacotherapy research pays significantly greater attention to the possibility of adverse outcomes than does psychotherapy research. The data show clear and significant differences in mentions of AEs when comparing the four study conditions (M, T, CM and CT). When grouped into the two overarching categories, pharmacotherapies and psychotherapies, possible or actual AEs were nine to twenty times more likely to be mentioned in the pharmacotherapy papers. To our knowledge, this is the first paper to systematically compare the consideration and reporting of adverse outcomes in psychotherapy and psychopharmacology research.

This difference in emphasis most likely mirrors the outlook and approach of many clinicians practicing these two treatment modalities. The discussion of potential side effects and risk–benefit ratios is inherent to prescribing medications. Providers are cognizant of the likelihood for medications to worsen certain mental conditions. It should be noted that adverse outcomes in psychopharmacotherapy often become apparent well after the medications come into common clinical use. This results from the accepted practice of reporting adverse effects when using these agents, collating the data, and conducting extensive epidemiological analyses. By familiarizing themselves with these reports and studies, clinicians are able to enhance safety in the provision of clinical care. This is done by an informed discussion of both benefits and risks of taking medications, and of no treatment at all.

The presentation of information about AEs to patients with regards to psychotropic medications is standard practice. For instance, clinicians are expected to discuss black box warnings for selective serotonin reuptake inhibitors (SSRIs), the potential harm of antipsychotics (an increased susceptibility to metabolic syndrome, morbidity and mortality in the elderly, tardive dyskinesia and other movement disorders), and the addictive potential of benzodiazepines and psychostimulants. To the extent that these guidelines are followed, patients and clinicians are better able to carefully weigh the risks and benefits of a pharmacotherapeutic agent, resulting in a collaborative informed consent to treatment and increased awareness of the potential for no response or for a negative one.

By comparison, in psychotherapy it is likely that “a significant number of practitioners may either underestimate the occurrence of negative treatment effects or not realize that they can occur [4].” More pointedly, in a review by Kraus et al., an extensive analysis was done on the adverse treatment effects of “harmful therapists [5].” This under emphasis on possible adverse effects remains, despite a 40-year effort to focus on negative effects within the academic psychotherapy community [2]. The tendency to disregard adverse outcomes may be explained by a number of reasons, including the emphasis to think of the process of therapy in nomothetic or generalized terms rather than idiographic or individualized terms, enabling clinicians to de-emphasize specific cases and outcomes [2]. Our own analysis underscored this tendency since most mentions of AEs simply stated that there were no “adverse events” or “side effects,” without a discussion as to what they might be. Some of the adverse events that were explicitly noted included high density exposure therapy being dangerous for those with coronary heart disease, and too difficult for depressed patients [6]; increased suicidality with cognitive behavioral therapy [3]; increased anxiety from applied relaxation therapy [7] and exposure treatment [8,9]; and dropouts due to issues with the format of group therapy [10] and problems in the therapeutic relationship [11].

It is also notable that various steps, which might stem from the acknowledgment of negative outcomes, have not been widely fostered. This includes the question of engaging the patient in exactly the kind of risk–benefit dialogue that pharmacotherapists use. The obvious barrier to such practices is the absence of clear definitions of specific adverse outcomes, data on the relationship of identified AEs to different technical and patient variables, and the dissemination of these data to psychotherapists. To achieve greater clarity, a concerted emphasis on operationalizing AEs in psychotherapy research is critical.

Ultimately, discussion of the risk of worsening of symptoms, or even of lack of improvement, should be initiated before the onset of treatment. One might posit that this discussion is even more crucial in the area of psychotherapy where negative outcomes cannot be gauged through clear signs and symptoms such as insomnia, racing thoughts, neurovegetative signs, laboratory findings and movement disorders. Some of these same phenomena may also appear early in psychotherapy but may be attributed to factors other than the treatment. Alternatively, these changes may be viewed as an expected and necessary part of therapy. Whatever the case, psychotherapy research should examine negative outcomes, and attempt to clarify which reactions result from treatment and why they do so [12,13]. Psychotherapy will be a safer and more effective treatment modality when adverse outcomes are acknowledged, studied, and ultimately explained to patients.

An important limitation of our study is the aforementioned lack of a standardized definition for what constitutes an AE of psychotherapy, necessitating that we establish our own criteria for AE mentions. Although we provide a definition of AE mentions, this is done to provide a framework for conducting the study and is a reflection of what the authors of the various papers appeared to consider side effects, not a proposal for future efforts to define AEs in psychotherapy. The need for a precise definition of AEs in psychotherapy still remains. Another limitation was the attribution of AEs with regard to combined trials (C). When study authors did not specify whether an AE was attributable to medication or psychotherapy, our raters attributed the AE to both. A further limitation involves the small size of our sample of articles. However, the large effect size of our findings suggests we had adequate statistical power for this study. Finally, the small sample size may limit the generalizability of this study to the entire breadth of mental health journals. Nonetheless, our use of high impact journals suggests the journals selected are widely read and thereby quite influential across the mental health field.

Defining and cataloguing AEs of psychotherapy is complex, especially for the purpose of routine reporting in prospective trials. Attempts at defining AEs have been made but to date there is no accepted consensus. In a prospective review, Mohr assessed variables within the patient and therapist populations, as well as between psychotherapeutic approaches, for potential risk to the patient [14]. Other research has focused on identifying and defining negative consequences of “potentially harmful therapies” (PHTs), such as “Scared Straight,” recovered traumatic memories, and boot-camp interventions [15]. Additional adverse events observed in context of psychotherapy include onset of panic symptoms as a paradoxical effect of relaxation, deterioration during grief counseling in normal bereavement, and increased drug consumption associated with drug prevention programs [15]. More recently, Linden suggested a taxonomy for unwanted events in psychotherapy, which distinguishes the different types of AEs that can occur, and also considers the context of the event, its relation to therapy, and the degree of severity [16]. In addition, Linden assessed for contraindications to the use of different types of therapy and their potential misapplication in the treatment of certain conditions [16]. Despite efforts similar to Mohr’s and Linden’s, our data suggest that monitoring of AEs is not broadly applied in psychotherapy research.

In conclusion, our findings suggest that at present the monitoring of AEs in psychotherapy research falls behind that of psychopharmacology research. We suggest that this stems from the lack of a consensual definition of AEs in psychotherapy. Continued research is therefore needed to identify, define, and standardize potential AEs of psychotherapy. Ultimately, a reliable and valid definition should be incorporated into routine practice and clinical training to enhance safety and efficacy in the provision of psychotherapy.