Diagnostic accuracy of the Edinburgh Postnatal Depression Scale (EPDS) for detecting major depression in pregnant and postnatal women: protocol for a systematic review and individual patient data meta-analyses

Abstract

Introduction

Studies of the diagnostic accuracy of depression screening tools often used data-driven methods to select optimal cut-offs. Typically, these studies report results from a small range of cut-off points around whatever cut-off score is identified as most accurate. When published data are combined in meta-analyses, estimates of accuracy for different cut-off points may be based on data from different studies, rather than data from all studies for each cut-off point. Thus, traditional meta-analyses may exaggerate accuracy estimates. Individual patient data (IPD) meta-analyses synthesise data from all studies for each cut-off score to obtain accuracy estimates. The 10-item Edinburgh Postnatal Depression Scale (EPDS) is commonly recommended for depression screening in the perinatal period. The primary objective of this IPD meta-analysis is to determine the diagnostic accuracy of the EPDS to detect major depression among women during pregnancy and in the postpartum period across all potentially relevant cut-off scores, accounting for patient factors that may influence accuracy (age, pregnancy vs postpartum).

Methods and analysis

Data sources will include Medline, Medline In-Process & Other Non-Indexed Citations, PsycINFO, and Web of Science. Studies that include a diagnosis of major depression based on a validated structured or semistructured clinical interview administered within 2 weeks of (before or after) the administration of the EPDS will be included. Risk of bias will be assessed with the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Bivariate random-effects meta-analysis will be conducted for the full range of plausible cut-off values. Analyses will evaluate data from pregnancy and the postpartum period separately, as well as combining data from all women in a single model.

Ethics and dissemination

This study does not require ethics approval. Dissemination will include journal articles and presentations to policymakers, healthcare providers and researchers.

Systematic review registration

PROSPERO 2015:CRD42015024785.

Strengths and limitations of this study.

• The study will use individual patient data that will allow estimate of diagnostic accuracy for all relevant cut-off scores of the Edinburgh Postnatal Depression Scale (EPDS) using data from all patients at each cut-off score. This will overcome limitations related to selective cut-off reporting in many primary study publications.

• The study will conduct analyses that exclude patients with current diagnoses of depression or who are undergoing mental health treatment at the time of study enrolment, since these patients would not be screened in clinical practice. This will overcome potential bias in primary study publications where these patients are often included.

• The study will generate predictive models that estimate the probability that a patient has depression based on EPDS scores and relevant covariates. This will facilitate more informed clinical decision-making than can be done with standard diagnostic accuracy metrics.

• A potential limitation is that the success of the study depends on the ability to obtain large amounts of individual patient data. Thus, we do not know the proportion of eligible data sets that will be included in the present study.

Introduction

Depression is a leading cause of disability among women.¹ Although the 7–13% prevalence of major depression during pregnancy and postpartum^2–5 is similar to rates among women during non-childbearing periods,^{4 6–10} perinatal depression is associated with adverse outcomes for the developing child, the mother–infant relationship and marital quality.^11–13 Most women with depression in the perinatal period, however, do not receive adequate care.^14–16 Thus, improving care of women with depression is a high clinical priority.¹⁷

Routine screening for depression, which involves the use of self-report questionnaires to identify patients with unrecognised depression who have not been otherwise identified as at risk for depression, has been proposed as a way to improve perinatal depression identification and management.^18–20 However, screening for depression is controversial because it has not been shown to benefit patients in a well-designed and conducted randomised controlled trial, would lead to misidentification and overtreatment in some patients, and would consume large quantities of scare resources without evidence of benefit.^21–23 Indeed, recommendations, policy and implementation are inconsistent.

In 2010, the American College of Obstetricians and Gynecologists recommended that screening for depression be ‘strongly considered’ in pregnancy and the postpartum period; the report noted that there was not sufficient evidence to support a ‘firm recommendation’.¹⁸ Also in 2010, the American Academy of Pediatrics recommended that pediatricians screen new mothers for depression during well-child visits in the 6 months following birth.¹⁹ Neither of these recommendations, however, was based on a systematic review of the evidence. In the UK, the National Screening Committee determined in 2001, and again in 2010, that there is no evidence that postnatal screening would improve health outcomes.^{24 25} A 2014 UK National Institute for Health and Care Excellence guideline recommended that healthcare providers consider asking two questions about depression at several points during pregnancy and postpartum,²⁰ but this recommendation was not based on evidence from randomised controlled trials that this would improve health outcomes.

In Canada, postpartum depression screening is routine in Alberta,²⁶ but it is only considered for women at risk in Quebec²⁷ and not recommended in Ontario.²⁸ In 2013, the Canadian Task Force on Preventive Health Care recommended against screening for depression in primary care settings, including for women in pregnancy and postpartum. The Task Force noted that existing evidence likely overstates the accuracy of depression screening tools in practice and emphasised the need for better-quality estimates of screening tool accuracy.²⁹

Successfully screening for perinatal depression would require a screening tool with demonstrated accuracy. The 10-item Edinburgh Postnatal Depression Scale (EPDS) is the most commonly used screening tool in pregnancy and postpartum.^{14 26 27} Most existing studies of the accuracy of depression screening tools, including the EPDS,³⁰ however, have been conducted in samples too small to precisely estimate accuracy. In a population with prevalence of 10%, for example, a screening study sample of 200 patients would only have approximately 20 patients to generate an estimate of sensitivity. Many studies have used data-driven methods to select ‘optimal’ cut-offs with these small samples and have selectively published accuracy results from high-performing cut-offs, but not other cut-offs, even when the other cut-offs are standard.^{31 32} Few studies have excluded women who are already being treated for depression and would not be screened in actual practice, thus exaggerating the estimated number of previously unidentified patients who would be detected by depression screening.³³ Ideally, primary studies that assess the diagnostic accuracy of the EPDS would include only women who have not already been identified as depressed, and results from all potentially relevant cut-off scores would be reported.

Meta-analyses can overcome small sample sizes in primary studies, but are unbiased only if all data from all relevant cut-off scores are included. Some meta-analyses^{34 35} have synthesised data to obtain a single accuracy estimate based on different cut-offs from different primary studies. In those meta-analyses, accuracy data from non-standard cut-offs have been included when primary studies only report optimally accurate cut-offs, but do not report less optimal accuracy results from standard cut-offs. Other meta-analyses^{30 36 37} have examined results from multiple cut-offs, but have similarly been limited to using published accuracy outcomes for each cut-off. The bias that can occur from synthesising results from selectively reported cut-offs is highlighted by some traditional meta-analyses,^{30 36} in which estimates of sensitivity actually increased as the cut-off score increased, suggesting that more patients would be detected when more severe symptoms were required for a positive screen. This is mathematically impossible if complete data are available. Beyond selective reporting, existing meta-analyses have not addressed limitations in primary studies from including already diagnosed and treated patients in study samples.³³

Individual patient data (IPD) meta-analysis can potentially address some of these problems. IPD meta-analysis involves using actual patient data obtained from researchers who conducted primary studies, rather than summary results from published or unpublished study reports.³⁸ The steps involved in conducting a systematic review with an IPD meta-analysis, in terms of defining a research question, establishing study inclusion and exclusion criteria, identifying and screening studies, and analysing data, is similar to a traditional systematic review and meta-analysis and diverges only in analysing individual level data rather than summary data.³⁹ When implemented effectively, IPD meta-analyses have particular benefits in that they can conduct analyses, including some subgroup analyses, that cannot be addressed using study-level data available in original reports. In the context of evaluating the diagnostic accuracy of depression screening tools, IPD meta-analysis has a number of major advantages compared to traditional meta-analyses. First, IPD meta-analysis can address bias from the selective reporting in publications of only well-performing cut-off thresholds since accuracy can be evaluated across all relevant cut-off scores. Second, IPD meta-analysis allows the exclusion of already-treated patients, for whom the tool would not be used to screen for unidentified depression, in at least some primary data sets. Third, IPD meta-analysis with large numbers of patients and large numbers of depression cases allows the incorporation of factors that may influence screening accuracy (eg, age, pregnancy vs postpartum) and study variables (eg, study setting, risk of bias factors). Additionally, a large IPD meta-analysis would be able to generate estimates of the probability of having depression based on patient characteristics and actual EPDS scores, rather than classifying patients as simply negative or positive. This is an important consideration since a patient with a score of 0 on the EPDS, for example, would almost certainly have a lower likelihood of having depression than a patient with a substantially higher, but subthreshold, score of 10, although typically both would be classified as negative screens.

Potential downsides of IPD meta-analyses are that they are resource intensive and require substantial time to identify and obtain original data, clarify data-related issues with data providers, and generate a consistent data format across studies.^{38 39} Furthermore, if primary data sets cannot be obtained, results from the IPD meta-analysis can be biased. In some circumstances, the inability to obtain enough primary data may make it impossible to reasonably conduct an IPD meta-analysis.^{40 41} Currently, our team is conducting an IPD meta-analysis of the Patient Health Questionnaire depression screening tool, which is the first IPD meta-analysis of the diagnostic accuracy of a depression screening tool.⁴² In that study, which is still in process, as of 26 July 2015, we had identified 77 eligible primary data sets. Of these, 68 investigators had agreed to contribute their data sets, and 45 had already transferred data sets. This suggests that investigators are generally willing to provide primary data from studies of the diagnostic accuracy of depression screening tools for use in IPD meta-analyses. Preliminarily, we are aware that at least 70 eligible studies with the EPDS exist, and more may be identified with the complete search strategy.

Thus, the primary objective of this IPD meta-analysis is to determine the diagnostic accuracy of the EPDS to detect major depression among pregnant and postpartum women.

Methods and analysis

This systematic review has been funded by the Canadian Institutes for Health Research (Funding Reference Number KRS-140994). The protocol has been registered in the PROSPERO prospective register of systematic reviews (2015:CRD42015024785), and any changes to the study protocol will be registered as amendments with PROSPERO.

The IPD meta-analysis has been designed and will be conducted in accordance with best-practice standards as elaborated in the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy⁴³ and other key sources.^{38 39 44} Results will be reported in concordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.^{45 46} To conduct the meta-analysis, primary data sets will be sought that allow comparisons of EPDS scores to diagnoses of major depression.

Sources of evidence

The search strategy was developed by a medical librarian and peer-reviewed by another medical librarian. Medline, Medline In-Process & Other Non-Indexed Citations, PsycINFO on the OvidSP platform, and Web of Science (Web of Knowledge platform) will be searched. The Medline search strategy was validated by testing against already-identified publications from preliminary searches. The strategy was then adapted for PsycINFO and Web of Science. The search strategy will be limited to these databases based on research showing that adding other databases (eg, EMBASE) when the Medline search is highly sensitive does not identify additional eligible studies.⁴⁷ The Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy⁴³ suggests combining concepts of the index test and the target conditions, but this was redundant for depression screening tools as these tests are limited to testing for depression. Thus, the search strategy for electronic databases was comprised of two concepts: the index test of interest and studies of screening accuracy. There are no published search hedges, or filters, designed specifically for mental health screening, but several key articles were consulted in developing search terms.^48–50 Search strategies use a combination of subject headings, when available in the database, as well as keywords in the title, abstract, or anywhere else in the record. The search will be limited to the year 2000 forward since it is unlikely that many older data sets will be obtainable. See online supplementary file 1 for detailed information on searches. To supplement electronic searches, reference lists of all included publications and relevant reviews will be searched. In addition, a related articles search will be conducted for included papers indexed in Medline using the PubMed ‘related articles’ search feature. Researchers who have published on the topic will be contacted to obtain information about additional, unpublished studies. Search results will be initially uploaded into the citation management database RefWorks (RefWorks, RefWorks-COS, Bethesda, Maryland, USA), and the RefWorks duplicate check function will be used to identify citations retrieved from multiple sources. Unique citations will then be uploaded into the systematic review program DistillerSR (Evidence Partners, Ottawa, Canada), and DistillerSR will be used to store and track search results and to track results of the review process.

To identify relevant data sets, articles published in any language will be reviewed. Data sets will be sought for inclusion if they compare results from the EPDS to Diagnostic and Statistical Manual (DSM) or International Classification of Diseases (ICD) criteria for major depression. ICD criteria are similar to DSM criteria and are generally used outside of North America. In DSM-5, there is not a separate diagnosis for depression during pregnancy or postpartum. Rather, cases of major depression are specified as having ‘peripartum onset’, which includes pregnancy and the postpartum period. Diagnoses must be based on a validated structured or semistructured interview (eg, Structured Clinical Interview for DSM,⁵¹ Composite International Diagnostic Interview⁵²) administered within 2 weeks of the administration of the depression screening tool, since major depression criteria are for symptoms in the last 2 weeks. Data sets where some patients were administered the screening tools within 2 weeks of the diagnostic interview and some patients were not will be included if the original data allow us to select patients administered the diagnostic interview and screening tools within the 2-week window. Data from studies where all patients are known to have psychiatric diagnoses will be excluded, with the exception of patients treated for substance and alcohol abuse, for whom depression screening may be considered. The coding manual for inclusion and exclusion decisions is shown in online supplementary file 2.

Two investigators will independently review titles and abstracts for eligibility. If either reviewer determines that a study may be eligible based on title or abstract review, then a full-text article review will be completed. Disagreement between reviewers after full-text review will be resolved by consensus, including a third investigator as necessary. Chance-corrected agreement between reviewers will be assessed with the Cohen's κ statistic. Translators will be consulted to evaluate titles/abstracts and articles for languages other than those for which team members are fluent (English, French, Spanish, Dutch, Greek). See online supplementary file 3 for preliminary PRISMA flow of studies figure.

Transfer of data and data set management

Authors of studies containing data sets that meet inclusion criteria will be contacted to invite them to contribute primary data for inclusion. Data will only be used from studies that received ethics approval and all data that are transferred will be properly de-identified prior to transfer. All individual patient data that are obtained will be cleaned and coded to make patient data as uniform as possible across data sets, then entered into a single STATA database (StataCorp, College Station, Texas, USA). A preliminary codebook has been developed for coding data from original studies of the EPDS. For each study to be included in the data set, two investigators will independently determine the coding protocol, based on the codebook, with any discrepancies resolved by consensus. Actual data coding and transfer from original studies into the IPD database will be done by a supervised staff or trainee member of the team, and the resulting data sets will be compared to the original reports in order to identify any potential discrepancies. In addition to obtaining original patient-level data, data will also be extracted from the published articles of included studies. Any inconsistencies will be discussed with the original authors. Corrections will be made as necessary.

Quality assessment

The Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool⁵³ will be used to assess risk of bias factors in primary studies, and these factors will be included as study-level variables in analyses. QUADAS-2 incorporates assessments of risk of bias across four core domains: patient selection, the index test, the reference standard, and the flow and timing of assessments. Two reviewers will independently assess risk of bias with any discrepancies resolved by consensus.

Data analysis

Analyses will estimate sensitivity and specificity, which will be used to generate estimates of positive and negative predictive value, which are more useful clinically. A bivariate random-effects meta-analysis will be fit, estimated via Gauss-Hermite adaptive quadrature, as described in Riley et al,⁵⁴ for the full range of plausible cut-off values, which for the EPDS will be considered cut-off scores from 9–15.³⁰ This approach appropriately models sensitivity and specificity simultaneously and also accounts for variation in within-study precision.⁵⁴ Data will be analysed using a random-effects model so that sensitivity and specificity are assumed to vary across studies. For each cut-off, separately, this model will provide an overall pooled sensitivity and specificity and an overall pooled diagnostic OR.⁵⁴

Analyses will be conducted that evaluate data from pregnancy and the postpartum period separately, as well as that combine data from all women in a single model. We anticipate that most primary studies include only one data point per woman. However, we expect that a small number of studies will provide data for more than one time point. To include all assessments, we will fit random effects models that account both for between study heterogeneity and between patient heterogeneity. Additionally, a subgroup analysis will be conducted that includes only data from countries listed as ‘very high development’ on the United Nation's Human Development Index⁵⁵ in order to capture studies from countries with healthcare resources and outcomes reasonably similar to those of Canada.

Heterogeneity will be quantified for each cut-off analysis by reporting the estimated variances of the random effects for sensitivity and specificity, as well as by estimating R. R is the ratio of the estimated SD of the pooled sensitivity from the random effects model to the estimated SD of the pooled sensitivity from the fixed effects model.⁵⁶ We will explore underlying reasons for heterogeneity using patient-level (eg, age) and study-level factors (eg, QUADAS-2 risk of bias ratings). In diagnostic accuracy this can easily be accomplished by including the factors or interaction terms in the random effects model described above.⁵⁴ These analyses take advantage of the richness of individual patient data. When analysed at the patient-level, accounting for correlation between patients from the same study, and for the correlation between sensitivity and specificity via the random effects model, they are more powerful to detect interactions and not vulnerable to ecological bias compared to traditional meta-analyses.^57–61 It is not known if there are factors that influence the accuracy of depression screening tools in pregnancy or postpartum, but many risk factors have been identified for the presence of depressive symptoms. These may include lower education, lower income, lack of a partner, domestic violence, unintended pregnancy or complications in pregnancy, for example.^{62 63} Data relevant to established risk markers from primary study authors will be requested, although these are not likely to be consistently available. Many of these risk markers are associated with age below a threshold, and age below 23 or 25 has been used to stratify risk of depression.^64–66 Thus, age (<25 vs ≥25 years) will be evaluated as a patient-level variable potentially associated with heterogeneity. It may be considered that, ideally, accuracy estimates would account for cultural or racial/ethnic differences between patients. However, existing studies have not been based on large enough samples to know if this would be the case. If there is enough group-specific data to adjust estimates in the IPD meta-analysis, we will do so. Study-level covariates, including risk of bias factors described in QUADAS-2,⁵³ may also be evaluated. QUADAS-2 factors include patient selection factors, blinding of reference standard to index test results, type of reference standard (eg, semistructured diagnostic interview; structured diagnostic interview), and timing of administration of index test and reference standard (eg, same day, delay of 1–7 days, delay of >7 days).

In addition to estimating sensitivity and specificity for each relevant cut-off, we will build predictive models that use the score on the screening questionnaire (combined model or separately for pregnancy and postpartum) and any other key factors that account for substantial heterogeneity to estimate the probability that a woman has major depression. The models will be evaluated in terms of their calibration (eg, slope of linear predictor; are average, low and high predictions correct?) and discrimination (eg, c-statistic; are low risk participants distinguished from high risk participants?).⁶⁷ Validation with the same participants used to develop a model results in overly optimistic performance. Internal validation will be assessed via the bootstrap method, which is preferable to split sample validation approaches (eg, developing the model in half the sample and evaluating it in the other half).⁶⁸ Although there are advantages to external validation, given the wide range of study populations that will be used it would be unlikely that there would be another comparable data set large enough for validation. Thus, assessment of internal validity via bootstrapping will help develop an understanding of how the model will likely perform in a clinical setting, and by using the regression coefficients adjusted for optimism (ie, the shrinkage estimates), will maximise actual accuracy. Based on our pilot work, it is anticipated that missing data will be minimal for the variables of primary interest. Regardless, multiple imputation will be done using chained equations^{67 69} to impute data for both binary and continuous variables, considering study as a fixed effect in the imputation model.⁷⁰ This will allow imputation both for variables missing for entire studies as well those missing more sporadically.

In a sensitivity analysis, studies included in the IPD meta-analysis will be compared to eligible studies that do not provide data in terms of sensitivity and specificity using published summary data from the studies that do not provide data. A sensitivity analysis will also be conducted that includes aggregate summary estimates of sensitivity and specificity from the studies that do not provide individual patient data in the main meta-analysis.⁵⁴ If there are a large number of studies that do not contribute primary data, this analysis may become the primary analysis.

Ethics and dissemination

This IPD meta-analysis does not require ethics approval, although only individual studies that obtained ethical clearance and informed consent will be included. The reasons that the IPD meta-analysis does not require ethics review is that the objectives of the IPD meta-analysis are consistent with the objectives of the primary studies, which already received ethics approval, and only anonymised data will be provided by the investigators of the original studies.

The main outcomes of the IPD meta-analysis reflect knowledge that will influence policy and clinical practice. The proposed project is a result of a collaborative process, which involved research team members and knowledge users at the national, provincial, regional and local levels of Canada. Currently, as described in the 2013 Canadian Task Force on Preventive Health Care guideline on screening for depression, policymakers must make decisions about screening for depression in the absence of well-designed and conducted clinical trials and with primary evidence on depression screening tool accuracy that does not appear to reflect what would likely occur in practice. Regional and local planners and individual clinicians are faced with the same problem, as well as inconsistent guidance on whether screening for depression should be included in clinical care. The present IPD meta-analysis will provide high-quality evidence on screening tool accuracy to these stakeholders. It will also increase the ability of researchers to incorporate better evidence-informed and potentially individualised screening protocols into trials of depression screening programmes.

At the national level, the key knowledge user is Dr Marcello Tonelli, who is the Chair of the Canadian Task Force on Preventive Health Care. At the provincial, regional and local levels, the research team has partnered with a key provincial level policymaker with an important role in perinatal mental health policy (Dr Nicholas Mitchell, Alberta, Canada), an advisor with a provincial institute that provides policy-relevant expertise to health authorities on a range of topics, including perinatal health (Dr Liane Comeau, Quebec, Canada), a key regional planning officer with experience in perinatal policy implementation (Ms Joy Schinazi), and a psychiatrist whose practice focuses on mental illness in pregnancy and postpartum (Dr Simone Vigod). These partnerships will facilitate successful dissemination of the results of this IPD meta-analysis to provincial health agencies, non-governmental organisations, clinical programmes and university training programmes. Internationally, team members will work to disseminate findings to national guideline-making and policy-making bodies outside of Canada, such as the National Institute for Health and Care Excellence and the National Screening Committee in the UK, for instance.

Strategies for effective dissemination and specific outputs will be based on research showing how to best tailor research outputs to different user groups,^71–76 including research on improving the usefulness of reports of systematic review and meta-analyses for healthcare managers and policymakers.^{74 76} Dissemination will include publication of results in high-impact medical journals with open access, as well as presentations in seminars and symposia to policymakers, healthcare providers and researchers at national and international conferences.

Traditional methods, however, such as journal articles and conference presentations are typically not highly effective in disseminating clinical research to healthcare providers, including physicians.⁷⁷ To increase uptake of the findings from this study, a brief video and podcast will be created that is similar to the Cochrane Library podcasts (http://www.cochrane.org/podcasts/). These will summarise the key findings of these meta-analyses and their implications for practice. They will also provide an introduction to an online calculation tool that will be created and made freely available to estimate the probability that a given patient has major depression based on depression screening results and patient characteristics. An example of a tool that is based on robust research evidence and effectively disseminated is the FRAX Fracture Risk Assessment Tool (http://www.shef.ac.uk/FRAX/index.aspx). The tool that will be made from the results of this study will be modelled on this tool and presented similarly in an easy-to-use fashion with tablet and app versions. In addition, simpler nomogram-based presentations, which are user-friendly graphical depictions of positive and negative predictive value by prevalence, will be generated and made available.