Fatal flaws in a recent meta-analysis on abortion and mental health

Abstract

Similar to other reviews within the last 4 years, a thorough review by the Royal College of Psychiatrists, published in December 2011, found that compared to delivery of an unintended pregnancy, abortion does not increase women’s risk of mental health problems. In contrast, a meta-analysis published in September 2011 concluded that abortion increases women’s risk of mental health problems by 81% and that 10% of mental health problems are attributable to abortions. Like others, we strongly question the quality of this meta-analysis and its conclusions. Here we detail seven errors of this meta-analysis and three significant shortcomings of the included studies because policy, practice, and the public have been misinformed. These errors and shortcomings render the meta-analysis’ conclusions invalid.

1. Background

Several recent reviews of studies on abortion and mental health have been published [1–4]. All of these have come to similar conclusions regarding the causal association between abortion and subsequent mental health; namely, that in the aggregate, women who terminate an unintended pregnancy are not at an increased risk of mental health problems compared to women who carry an unintended pregnancy to term. These conclusions are based on a thorough review of the literature, and inclusion of studies that have very high design quality and are not biased towards finding abortion leads to mental health problems [5].

Despite these independent reviews all having similar conclusions, one recent review by Coleman came to a different conclusion. A meta-analysis concluded that abortion increases women’s risk of mental health problems by 81% and that 10% of mental health problems are attributable to abortions [6]. The methods, results, and conclusions of this meta-analysis have been seriously questioned by several researchers and scholars [1, 7–17]. Ten letters to the editor were highly critical of the methodology used [8–17], with 2 calling for a retraction of this meta-analysis [10, 16]. In addition, the final version of a review by the Royal College of Psychiatrists (RCPsych) [1] stated that (p. 18) “A number of methodological problems with the meta-analysis conducted in the Coleman review have been identified, which brings into question both the results and conclusions.” And, a commentary published in January 2012 [7], written by authors of the Royal College of Psychiatrists’ review concluded that the Coleman meta-analysis (p. 12) “cannot be regarded as a formal systematic review.” Like others [1, 7–17], we strongly question the quality of this meta-analysis of 22 papers [18–39] just as the reliability, validity, and replicability of some of the studies [e.g., 26, 37] in the meta-analysis have been questioned [40, 41].

While many [1, 7–17] have discussed the errors and shortcomings in separate critiques of the Coleman meta-analysis [6], none has presented a comprehensive summary of them in one paper. Below, we present a summary of the most serious and significant errors of this meta-analysis because policy, practice, and the public have been badly misinformed. We then briefly discuss significant shortcomings of various studies included in the meta-analysis. In addition, because the RCPsych review was published around the same time and came to different conclusions than Coleman’s meta-analysis, where appropriate, we discuss how aspects of the Coleman meta-analysis and RCPsych review differed. We also present Tables 1 and 2 to illustrate the errors of the meta-analysis and shortcomings of several of the included studies.

2. Errors

We found seven significant errors in the methods, analyses, and reasoning of Coleman’s meta-analysis [6]. They include: 1) violating guidelines for conducting meta-analyses, 2) not accounting for dependence of effect sizes, 3) calculating population attributable risk factor when not appropriate, 4) not adhering to the exclusion and inclusion criteria outlined in the methods section, 5) misclassifying the comparison group, 6) adjusting effect sizes for different factors, and 7) making invalid inferences regarding the proportion of all births that are unintended. Below we expand on each of these.

2.1. Violating guidelines for conducting a meta-analysis

Meta-analyses summarize findings in a literature across several independent studies. The Cochrane Collaboration group provides guidelines for conducting meta-analyses [42] and has a chapter dedicated to non-randomized studies [43], relevant for any meta-analysis of the abortion and mental health literature. The guidelines apply to all steps of the meta-analysis from deciding who should conduct it, how to choose studies to be included, how to extract data, and what to infer from findings.

The meta-analysis by Coleman [6] did not adhere to these guidelines in several ways. There is no information on the search terms used and no reason why some expected studies were excluded; in addition, the meta-analysis was conducted only by Coleman. In addition, a conflict of interest exists because 11 of the 22 papers included were Coleman’s own [43]. When conducting a meta-analysis and deciding whether a study conducted by oneself should be included, the Cochrane Collaboration (Section 2.6 [43]) says “there should be an independent assessment of eligibility and risk of bias by a second author [of the meta-analysis] with no conflict of interest.” Given that there was no other author of Coleman’s meta-analysis, this did not occur. In contrast to the Coleman meta-analysis [6], the RCPsych [1] review was much more transparent, detailing terms and databases searched, reasons why expected studies were excluded, and investigators contacted. In addition, two authors conducted the review of the literature, selected studies, and extracted data from these studies. Disagreements were resolved through discussion.

2. Not accounting for dependence of effect sizes

Although 22 studies are reviewed, only 14 different data sets are used. Moreover, 36 effect sizes are drawn from these 14 data sets. As noted by Higgins and Green (Section 1.2.2, [42]) “Meta-analysis is the use of statistical methods to summarize the results of independent studies.” However, Coleman treated effect sizes that were from the same data as independent, even though only 4 of 36 were independent. Table 1 provides information on whether the effect sizes were independent or not, and the data set used. Each data set should contribute only one effect size. Because there were 14 data sets, only fourteen effect sizes should have been used in this meta-analysis.

Table 1.

Studies in meta-analysis and rating by Royal College of Psychiatrists (RCPsych) report

Studies used in Coleman’s meta-analysis grouped by comparison group	Outcome	OR	P	RCPsych rating	Independe nt effect	Data used
Delivery		2.386	0.000
Coleman et al. 2009 [18]	Alco	3.390	0.001	Excluded ICPMH	Yes	Fragile Families and Child Well-being Study
Coleman et al. 2002 [19]	Alco	2.396	0.000	Excluded NUD	No	National Pregnancy and Health Survey
Coleman et al. 2002 [19]	Marij	8.554	0.000	Excluded NUD	No	National Pregnancy and Health Survey
Coleman et al. 2002 [20]	Anx	1.140	0.050	Poor	No	California Medicaid records
Coleman et al. 2002 [20]	Dep	1.160	0.087	Poor	No	California Medicaid records
Cougle et al. 2003 [21]	Dep	1.639	0.013	Excluded ICPMH	No	National Longitudinal Survey of Youth
Gissler et al. 1996 [22]	Suic	5.900	0.000	Excluded ICPMH	Yes	Finnish Records
Pederson 2008 [23]	Dep	1.750	0.337	Good	No	Young in Norway Longitudinal Study
Reardon et al. 2003 [24]	Dep	1.924	0.000	Poor	No	California Medicaid records
Reardon et al. 2002 [25]	Suic	2.540	0.023	Poor	No	California Medicaid records
No Abortion		1.592	0.000
Coleman et al. 2009 [26]	Alco	1.898	0.000	Excluded ICG	No	National Comorbidity Survey
Coleman et al. 2009 [26]	Anx	1.787	0.000	Excluded ICG	No	National Comorbidity Survey
Coleman et al. 2009 [26]	Dep	1.405	0.004	Excluded ICG	No	National Comorbidity Survey
Coleman et al. 2005 [27]	Alco	1.620	0.076	Excluded ICG	No	Washington DC Metropolitan Area Drug Study (DC-MADS)
Dingle et al. 2008 [28]	Dep	1.500	0.105	Excluded ICG	No	Mater-University Queensland Study
Dingle et al. 2008 [28]	Alco	2.100	0.002	Excluded ICG	No	Mater-University Queensland Study
Dingle et al. 2008 [28]	Anx	1.500	0.105	Excluded ICG	No	Mater-University Queensland Study
Dingle et al. 2008 [28]	Marij	1.500	0.120	Excluded ICG	No	Mater-University Queensland Study
Pedersen 2007 [29]	Alc	2.000	0.028	Good	No	Young in Norway Longitudinal Study
Pedersen 2007 [29]	Marij	3.400	0.000	Good	No	Young in Norway Longitudinal Study
Rees & Sabia 2007 [30]	Dep	2.150	0.047	Excluded ICG	No	Fragile Families and Child Well-being Study
Steinberg & Russo 2008 [31]	Anx/NCS	0.914	0.689	Good	No	National Comorbidity Survey
Taft & Watson 2008 [32]	Dep	1.220	0.065	Excluded NMEG	Yes	Australian Longitudinal Survey on Women’s Health
Unintended Birth		1.551	0.000
Coleman 2006 [33]	Alco	5.720	0.029	Excluded ICPMH	No	National Longitudinal Survey of Adolescent Health
Coleman 2006 [33]	Marij	9.000	0.004	Excluded ICPMH	No	National Longitudinal Survey of Adolescent Health
Cougle et al. 2005 [34]	Anx	1.340	0.017	Fair	No	National Survey of Family Growth
Fergusson et al. 2008 [35]	Suic ideation	1.610	0.168	Very good	No	Christchurch Health and Development Study
Fergusson et al. 2008 [35]	Alco	2.880	0.047	Very good	No	Christchurch Health and Development Study
Fergusson et al. 2008 [35]	Anx	2.130	0.006	Very good	No	Christchurch Health and Development Study
Fergusson et al. 2008 [35]	Dep	1.310	0.317	Very good	No	Christchurch Health and Development Study
Gilchrist et al.1995 [36]	Self-harm	1.700	0.016	Good	Yes	Royal Colleges Study
Reardon & Cougle 2002 [37]	Dep	1.540	0.108	Excluded ICMHM	No	National Longitudinal Survey of Youth
Reardon et al. 2004 [38]	Alco	1.720	0.073	Excluded ICPMH	No	National Longitudinal Survey of Youth
Reardon et al. 2004 [38]	Marij	2.000	0.010	Excluded ICPMH	No	National Longitudinal Survey of Youth
Schmiege & Russo 2005 [39]	Dep	1.190	0.308	Excluded ICMH	No	National Longitudinal Survey of Youth
Steinberg & Russo 2008 [31]	Anx/NSFG	1.210	0.190	Very Good	No	National Survey of Family Growth

Notes: First four columns taken directly from Coleman’s meta-analysis [6]. The first column presents the studies used, grouped by the comparison group reported in the meta-analysis. The unintended birth group in the first column was labeled “Unintended pregnancies” in Coleman’s Figure 3, which we have corrected here. The second column shows the mental health outcome used in Coleman’s meta-analysis: Alco= Alcohol measure, Marij = Marijuana measure, Anx = Anxiey measure, Dep = Depression measure, Suic ideation = suicidal ideation, Suic = Suicide. The fifth column shows the RCPsych rating if it was included in the RCH Psych review. If it was not included in the RCPsych review, this was noted and so was the reason for exclusion. Reasons for exclusion in column 5: ICPMH = inappropriate control of prior mental health, ICMH = inappropriate control for mental health, ICMHM = inappropriate control of mental health measure, NMEG = not mutually exclusive groups, ICG = inappropriate comparison group, NUD = no useable data. The sixth column addresses whether the study contributed an effect independent of all other effects in the meta-analysis (Yes signifies it did, No signifies it did not). The seventh column presents the data used in the study.

2.3. Calculating population attributable risk when not appropriate

A population attributable risk factor provides an estimate of the proportion of the outcome that would not have occurred should the predictor variable not have occurred. It shows the degree of causality between the predictor variable and outcome. In this meta-analysis, Coleman concludes that 10% of mental health problems would not have occurred should women not have had abortions. However, even if correlation exists, causation cannot be assumed, particularly given that there are other more plausible explanations for why a correlation between abortion and mental health might exist [40]. Therefore, it is not appropriate to calculate a population attributable risk factor unless causation is demonstrated.

In addition, the formula Coleman uses to estimate population attributable risk is an approximation that works well only when the outcomes are rare, so that the odds ratio approximates the relative risk. However, most of the mental health outcomes in the meta-analysis are definitely not rare. In fact, 12-month estimates of the proportion of U.S. women with depression, anxiety disorders, and substance use disorders are 8.6%, 23.4%, and 11.6%, respectively, and over their lifetime 20%, 36%, and 30% of women will have depression, anxiety disorders, and substance use disorders, respectively [44, 45].

2.4. Not adhering to the stated inclusion and exclusion criteria

We identified one paper [46] that met Coleman’s criteria and rules for extraction, but it was not included. In addition, Coleman violates the spirit of the criteria in another instance. Under “Rules for extraction and synthesis of effects,” Coleman states (p.181), “(b) When studies had more than one comparison group, selection rules were employed to provide more weight to comparisons wherein the control group was most closely matched to the abortion group. Specifically, if ‘unintended pregnancy delivered’ was used the results relative to this group were selected,” and “(d) When particular authors use the same sample and variables in more than one publication, only the most recent publication was selected.” However, in two papers [5, 21] included in the review, the same group of authors used the National Longitudinal Survey of Youth to examine the relationship between abortion and subsequent depression using the same variables; the abortion group was the same sample in both studies; and in one study the comparison group was all first births and in the other it was unintended first births (a subset of all first births).

2.5. Misclassifying the comparison group

Coleman conducted separate meta-analyses by comparison groups (shown in Table 1) and concludes (p. 184) “When the abortion group was compared with the no pregnancy group and with the unintended pregnancy delivered group, the magnitude of the effects was very close. This finding challenges the generally accepted belief that unintended pregnancy delivered represents the only or most appropriate control group for studies designed to explore the impact of abortion on mental health. Use of a no pregnancy delivered group may be a cleaner control group, since many women experience postpartum depression and/or anxiety following childbirth. From a practical standpoint, a no pregnancy comparison group should be considerably easier to secure than a group of women who deliver an unintended pregnancy.” There are two important errors of this conclusion. First, even though there was one study that used a “no pregnancy” comparison group ([28] see below), the meta-analysis never reported that there was a “no pregnancy” or a “no pregnancy delivered” comparison group; instead, it reported a no abortion group, an unintended birth group, and a birth group (not considering pregnancy intention). Second, once a woman has an unintended pregnancy, she does not have the option to go back and not have become pregnant. She must decide whether to carry the pregnancy to term or not. Therefore, for research to inform policy it is inappropriate to compare women who have abortions to women who have never been pregnant. Moreover, for purposes of informing policy, it is more appropriate to use the comparison group of women who have unintended pregnancies ending in birth rather than women who have births, regardless of pregnancy intention.

In addition, 10 effect sizes from 4 studies were classified incorrectly (see Table 2) [23, 28, 31, 35]. First, Coleman classified the comparison group of Fergusson et al. [35] (four effect sizes) as women who had unintended births, when in fact it was women who had no abortions. Second, Coleman reports that Steinberg and Russo [31] (1 effect size), in their analysis of the National Comorbidity Survey, used the comparison group of women having no abortion, but they used women who delivered. In addition, Coleman classified the comparison group of Dingle et al. [28] (four effect sizes) as no abortion, when this study compared women who aborted to women who were never pregnant, a category not included by Coleman. Finally, Coleman classified the comparison group of Pedersen [23] (1 effect size) as women who delivered, but it was women who had no abortions. These errors are significant because in the meta-analysis summary statistics are computed and conclusions are drawn about how the effect size changes as a function of the comparison group. Because some studies are classified incorrectly, conclusions based on the analyses of effect sizes by comparison group cannot be drawn.

Table 2.

Studies in meta-analysis by correct comparison group and prior mental health

Study (by correct comparison group)	Mental health outcome(s) measure used	Did abortion precede mental health outcome?	How was prior mental health controlled?
Never pregnant
Dingle et al. 2008 [28]	Lifetime measures of depression, anxiety disorders, alcohol misuse and dependence, and marijuana misuse and dependence with the Composite International Diagnostic Interview (CIDI) for DSM-IV disorders	No, not necessarily	Covariates of self-report anxious and depressive symptoms at age 14
No Abortion
Coleman et al. 2009 [26]	Lifetime measures of alcohol abuse, depression, and anxiety with the Modified version of Composite International Diagnostic Interview Schedule (CIDI) for DSM-III-R disorders	No, not necessarily	It was not controlled
Coleman et al. 2005 [27]	Did use alcohol at any point during current pregnancy (that ended in childbirth)	Yes	It was not controlled
Fergusson et al. 2008 [35]	Alcohol dependence, anxiety disorders, depression, and suicidal ideation with the Composite International Diagnostic Interview (CIDI) for DSM-IV disorders	Yes	Covariates of pre-pregnancy anxiety, depression, and suicidal ideation, at age 15 and number of mental health problems in prior data collection period
Pedersen 2007 [29]	Alcohol abuse with a cut-off between 9–10 on Alcohol Use Disorders Identification; Single item question on the use of marijuana at any point in previous 12 months	Yes	Covariate of pre-pregnancy depression at approximately 12 and 7 years before the pregnancy event
Pederson 2008 [23]	Depression with a cut-off between 8 and 9 on Kandel and Davies’ Depressive Mood Inventory	Yes	Covariate of pre-pregnancy depression at one time point
Rees & Sabia 2007 [30]	Depression as measured by the Composite International Diagnostic Interview Schedule- Short Form (CIDI-SF)	Yes	Covariate of pre-pregnancy depression two years before outcome
Taft & Watson 2008 [32]	Depression as measured by a score of 10 or greater on the Center for Epidemiological Studies Depression Scale (CESD-10)	Yes	Covariate of pre-pregnancy depression as reported by doctor four years before outcome
Delivery
Coleman et al. 2009 [18]	Was there a day in the last month in which consumed 5 or more alcoholic beverages?	Yes	It was not controlled
Coleman et al. 2002 [19]	Use of alcohol or marijuana at any point during current pregnancy that just ended in childbirth	Yes	It was not controlled
Coleman et al. 2002 [20]	Outpatient psychiatric treatment for ICD-9 anxiety states and composite of various ICD-9 depressive states from California Medicaid records in four years after pregnancy	Yes	Excluded women with psychiatric admission 12–18 months before pregnancy event
Cougle et al. 2003 [21]	Depression with a score of 16 or greater on the Center for Epidemiological Studies Depression Scale (CESD)	Yes	Covariate of pre-pregnancy locus of control at one time point
Gissler et al. 1996 [22]	Suicide according to Finnish death records in year after pregnancy	Yes	It was not controlled
Reardon et al. 2003 [24]	Composite of various psychiatric admissions for depressive disorders from California Medicaid records in four years after pregnancy	Yes	Excluded women with psychiatric admission 12–18 months before pregnancy event
Reardon et al. 2002 [25]	Suicides from California Department of Health Services death certificates in 8 years after pregnancy	Yes	It was not controlled
Steinberg & Russo 2008/NCS [31]	Anxiety disorders of PTSD and social anxiety with Composite International Diagnostic Interview (CIDI) for DSM-III-R disorders	Yes	Covariate of pre-pregnancy anxiety at any time before pregnancy
Unintended birth
Coleman 2006 [33]	Alcohol use in past 12 months; Marijuana use in past 30 days	Yes	It was not controlled
Cougle et al.2005 [34]	Generalized anxiety with a non-validated measure of anxiety symptoms	Yes	Excluded women with pre-pregnancy anxiety at any point before pregnancy
Gilchrist et al. 1995 [36]	Deliberate self-harm classified by ICD-8 codes	Yes	Excluded women with any prior psychiatric diagnoses
Reardon & Cougle 2002 [37]	Depression with a score of 16 or greater on the Center for Epidemiological Studies Depression Scale (CESD)	Yes	Covariate of pre-pregnancy locus of control at one time
Reardon et al. 2004 [38]	Score of 2 or more on an 11-item non-validated scale of alcohol use; Any use of marijuana in past 30 days	Yes	Covariates of pre-pregnancy locus of control and self-esteem at one time
Schmiege & Russo 2005 [39]	Depression with a score of 16 or greater on the Center for Epidemiological Studies Depression Scale (CESD)	Yes	It was not controlled
Steinberg & Russo 2008/NSFG [31]	Anxiety symptoms with a non-validated measure of anxiety symptoms	Yes	Covariate of pre-pregnancy anxiety at any time before pregnancy

Notes: This table shows the correct classification of studies by comparison group. The first column presents the studies by the comparison group actually used in the meta-analysis. The second column presents how the mental health outcome(s) were measured in each study. The third column depicts whether the study measured mental health after the abortion, and the final column shows whether and how prior mental health was controlled for in analyses. Steinberg and Russo appears twice because this paper included two studies in which different comparison groups were used.

2.6. Adjusting effect sizes for different factors

The effect sizes in the meta-analysis were adjusted for different factors. When effects are adjusted for such a wide variety of different factors, summarizing across effects does not provide any useful information because they have very different meanings and interpretations. For example, some were adjusted for no factors (e.g., [22]), or for only age and race (e.g., [34]), while others were adjusted for 22 different factors (e.g., [26]). Other reviews, including that by the RCPsych, have not conducted meta-analyses in part because studies in the abortion and mental health literature adjust for different factors [1–5]. Instead, these reviews discuss the importance of adjusting for certain core factors such as prior mental health or adverse experiences.

2.7. Making invalid inference regarding percent of births that are unintended

In the introduction, Coleman states, “at least half of all pregnancies in the USA are classified as unintended and among adolescents and women over 40 years old the percentage is over 75%, meaning the majority of women in the control groups in studies comparing abortion with term pregnancy actually delivered unintended pregnancies even if the variable was not directly assessed.” In addition to this statement being logically incorrect because an inference about unintended births is being obtained from facts about unintended pregnancies, U.S. data show that 35% of all births result from unintended pregnancies (78% at ages <20, 44% at ages 20–24, 27% at ages 25–29, and 22% at ages 30–44) [47], whereas 95% of abortions result from unintended pregnancies [48]. These data illustrate the importance of comparing women who abort to women who deliver unintended pregnancies.

3. Significant shortcomings of individual studies included in the meta-analysis

Several of the studies included in Coleman’s meta-analyses have serious shortcomings; therefore, the included studies vary widely in design quality and bias [1, 4, 5]. The Cochrane Collaboration, Handbook of Research Synthesis and Meta-analysis [49, 50] (which was used by Coleman [6]), and MOOSE Group [51] discuss what to do with studies of varying quality and bias, which are common in the abortion and mental health literature [1, 4, 5]. Analyses of studies of varying quality and bias should be conducted separately. For instance, results from studies with the most bias towards finding abortion leads to mental health problems could be compared to those with less bias towards finding this result. The meta-analysis conducted by Coleman [6], however, did not conduct separate analyses of studies by level of bias or quality.

In the RCPsych review [1], all included studies were given a quality-rating on a 5-point scale ranging from very good to very poor. Rather than conducting a meta-analysis on all 44 reviewed studies, only 4 were included in these meta-analyses. Moreover, 13 studies included in Coleman’s meta-analysis (Coleman was an author of 7 of these) were excluded from the RCPsych analysis that compared women who aborted to other women for reasons shown in Table 1 [18, 19, 21, 22, 26–28, 30, 32, 33, 37–39]. Of the other 10 studies¹ that were included in Coleman’s meta-analysis, 3 were rated as poor [20, 24, 25], 1 as fair [34], 4 as good [23, 29, 31, 36], and 2 as very good [31, 35] by the RCPsych review. Table 1 presents the studies classified by the comparison group Coleman reports the study used and includes the RCPsych’s ratings of each study.

At the very least, effect sizes should be summarized by different levels of the following characteristics: 1) timing of mental health outcomes relative to the abortion, 2) measurement of mental health outcomes, and 3) whether or how prior mental health was considered in analyses. These aspects cannot be ignored in a meta-analysis because they influence the study’s quality for drawing inferences regarding the hypothesis that abortion causes mental health problems. Below we discuss how individual studies vary in these characteristics, and in Table 2 we present each study’s characteristics for the effect sizes used in Coleman’s meta-analysis.

3.1. Timing of mental health outcome relative to the abortion

In 2 of the studies [26, 28], comprising 7 of the 36 effect sizes in the meta-analysis, temporal precedence of the abortion relative to the mental health outcome was not ascertained. That is, the mental health outcome may have occurred before the abortion, in which case the abortion cannot be causing mental health problems [52]. In fact, Steinberg and Finer [40] show that a large proportion of the mental health outcomes used in one paper by Coleman et al. [26] first occurred before the abortion. Certainly a study that does not determine temporal precedence of the abortion relative to the mental health problem cannot be used to infer causality.

3.2. Measurement of mental health outcomes

Coleman fails to adequately distinguish conceptually among mental health outcomes, giving equal weight to risk behavior outcomes like alcohol or marijuana use during pregnancy [19, 27] or any marijuana use [33] or any alcohol use [18, 33] as to more severe psychiatric outcomes like suicide [22]. In addition, methods of assessment such as clinical diagnoses made by structured psychiatric interviews [e.g., 26, 28, 31, 35] and those of single item measures [e.g., 18, 19, 27, 33, 38] are not distinguished. These conceptual and measurement issues influence a study’s validity and reliability regarding the effect of abortion on mental health.

3. Method of controlling for prior mental health

Coleman does not distinguish between methods of controlling for prior mental health. As shown in Table 2, two effect sizes used a covariate of mental health at any time before the pregnancy [31]. Seventeen effect sizes used a covariate of mental health or a related construct at one point in time (e.g., at age 15) or for a period (but not all) of time before the pregnancy [21, 23, 28, 29, 30, 32, 35, 37, 38]; 5 effect sizes excluded women with a mental health problem for a period of time (e.g., 12–18 months) before the pregnancy or for all the time before the pregnancy [20, 24, 34, 36]; and 12 effect sizes did not control for prior mental health at all [18, 19, 22, 25, 26, 27, 33, 39]. A study that adequately controls for mental health before the pregnancy cannot be equated with one that does not.

4. Conclusions

In conducting a meta-analysis, authors must extract information from a study. However, if the study from which data are extracted is of questionable quality or the information is drawn incorrectly, then the meta-analysis is flawed. The RCPsych review excluded 13 of the studies included in Coleman’s meta-analysis [18, 19, 21, 22, 26–28, 30, 32, 33, 37–39] because their quality was lower than very poor for reasons shown in Table 1. Unfortunately, much of the published research (including that meta-analysis) claiming to find an association between abortion and mental health is wrought with methodological and analytical errors and shortcomings, and it is used to misinform policy and clinical practice, both of which have consequences for women’s reproductive decision-making, health, and well-being.

When conducting research on the relationship between abortion and subsequent mental health, researchers must make careful decisions regarding aspects of study design and statistical analyses. It is important that researchers are aware of how these decisions influence their findings. For example, in the National Comorbidity Survey [52] data on mental health diagnoses are coded for various time periods such as lifetime, one-year, or one-month (current). Instead of using mental health outcomes that ensure that the abortion occurred before the mental health outcomes (i.e., one-month/current mental health), Coleman and colleagues [26] chose to use lifetime diagnoses [51], a choice that does not ensure the abortion occurred before the mental health outcomes. However, nowhere do Coleman et al. [26] present this design fact in their paper. Instead, in their method section (p. 772) they state that they used psychiatric diagnoses that “were assessed as “present” or “absent” at the time of data collection providing assurance that in most cases, the abortion preceded the diagnosis.” Studies such as this are misleading and provide little useful information for understanding the relationship between abortion and mental health. While some of the studies included in the meta-analyses were conducted for the purpose of examining this relationship, many were biased towards finding an association between abortion and mental health [5].

Coleman makes the point that a meta-analysis that quantifies the effect of abortion on mental health is unbiased. However, 13 of the 23 studies included by Coleman did not even merit inclusion in the RCPsych review because they were of lower than very poor quality. A meta-analysis cannot be used to make good science out of (mostly) bad science. As pointed out repeatedly in the RCPsych review, it is inappropriate to conduct a meta-analysis of the abortion and mental health literature because studies varied widely in quality and bias, and the best available evidence shows that abortion does not increase women’s risk of mental health problems relative to an unintended birth. Research on the relationship between abortion and subsequent mental health (like all research on topics with important public health implications) should be subjected to great scrutiny before publication during the peer-review process, and even if published, independent reanalysis can reveal (and in this case has revealed) errors that can be fatal. We found here, after publication, that the analyses and methods contained many errors which render the conclusions invalid.