Global Burden of Disease estimates of depression – how reliable is the epidemiological evidence?

Summary

Objectives

To re-assess the quality of the epidemiological studies used to estimate the global burden of depression 2000, as published in the GBDep study.

Design

Primary and secondary data sources used in the global burden of depression estimate were identified and assigned to country of origin. Each source was assessed with respect to completeness and representativeness for national/regional estimates and against the inclusion criteria used by the scientific team estimating GBDep.

Setting

Not applicable.

Participants

Not applicable.

Main outcome measures

Not applicable.

Results

First, National estimates: The 28 scientific sources cited in the GBDep study related to 40 of the 191 WHO member countries. The EURO region had studies relating to 15 of 52 countries whereas AFRO region had studies for only three of 46 countries. Only six of the 40 countries had data drawn from a nationally representative population: the three AFRO country studies were based on a single village or town and, likewise, SEARO region had no nationally representative data; second, GBDep criteria: GBDep inclusion criteria required study sample size of more than 1000 people; 19 (45%) of the 42 studies did not meet this criterion. Sixteen (44%) of 36 studies did not meet the requirement that studies show a clear sample frame and method. GBD estimates rely on estimates of incidence; only two of the 42 country studies provided incidence data (Canada and Norway), the remaining 34 studies were prevalence studies. Duration of depression is based on three studies conducted in the USA and Holland.

Conclusions

Most studies exhibit significant shortcomings and limitations with respect to study design and analysis and compliance with GBDep inclusion criteria. Poor quality data limit the interpretation and validity of global burden of depression estimates. The uncritical application of these estimates to international healthcare policy-making could divert scarce resources from other public healthcare priorities.

Background

The Global Burden of Disease (GBD) Study began as a collaboration between the World Bank, Harvard School of Public Health and World Health Organization (WHO) with the aim of providing summary measures of population health to inform priority-setting in health policy interventions at national and international level. Since its inception in 1993, the GBD project has become an important and widely used source of information on the burden of specific diseases: the WHO regularly publishes GBD estimates; countries use national burden of disease estimates to adjust their national health policies and priorities, to allocate resources, and to target specific disease interventions and treatments (e.g. national mental health programme in India, Tanzania Essential Health Interventions Project).¹

GBD calculations of disease burden are derived from epidemiological estimates of life-years lost and life-years lived with disability due to a specific disease. The disability-adjusted life year (DALY) measure incorporates epidemiological data of incidence, life expectancy at the age of incidence, duration of disease, and a disability weighting. The disability weights are defined by a group of experts and are adjusted for the severity of the episode. This composite measure is used to rank diseases according to their contribution to disease burden at national, regional and worldwide level.

The 1993 GBD estimates ranked mental disorders as a leading cause of disability even in low- and middle-income countries, with depression contributing 3.7% of all DALYs.² The 2000 GBD estimates stated that neuropsychiatric disorders contribute around 14% to the total disease burden and ranked depression as the fourth major cause of disease burden worldwide.³ The 2001 World Health Report was devoted to mental health and in 2007 The Lancet published a series on Global Mental Health, calling for increased attention to mental health globally, particularly in low- and middle-income countries citing Global Burden of Disease as an important reason for the focus on mental health.^4,5 In 2008, the WHO published its strategy for ‘Scaling up care for mental, neurological, and substance use disorders’ – the Mental Health Gap Action Programme⁶ – and a website of an international scientific and social movement on global mental health was launched.⁷

However, the GBD is not without its critics. The value judgements used in weighing DALYs^8,9 and the epidemiology have been highlighted as a cause for concern. A recent review of 16 leading causes of global burden of disease in 2000–2002 revealed of the order of 10–90%, median 41%.¹⁰ Uncertainty was due to unreliability of epidemiological estimates of incidence, prevalence and disease severity and from uncertainty in the estimates. The authors concluded that uncertainty was highest for low-income countries and for diseases without appropriate health surveillance systems and there was a need for better epidemiological data and caution when interpreting global comparative epidemiological assessments.

Questions have been raised about the usefulness of the estimates for sub-Saharan Africa in health policy and the uncritical referencing of GBD in academic studies:

Can useful policy decisions for any region of the world be made without knowing any validity of the burden of disease measures? … The GBD study is rapidly being held up as the reference standard, despite the caveats in the accompanying multi-volume text. If these data are wrong the consequences are likely to be most damaging for the very populations under-represented in the fact-gathering process.¹¹

Depression was not included in the review of 16 GBD estimates, but given its increasing significance in policy-making¹ it merits examination not least because mental health or illness raise a number of cross-cultural and methodological issues. For this reason we set out in this paper to evaluate the quality of epidemiological sources used in the GBDep study² estimating the global burden of depression.^a

Data and methods

The GBD estimate for depression is based on 24 published studies and four unpublished studies where information was collected via personal communication. We contacted the authors of the unpublished studies: the authors of the studies conducted in Morocco, the Czech Republic and Russia did not reply and it is not known whether their data are unpublished studies or expert estimates. The author of the Nepal study replied to the effect that the study (unpublished) covered onevillage in rural Nepal and was not designed to allow generalization to national level.

Of the 24 published studies, 19 are epidemiological country studies and five multi-country studies – WHO Multi-country Survey 2000–2001,¹³ WHO International Consortium in Psychiatric Epidemiology 2000, Weissman et al.,¹⁴ Copeland et al.¹⁵ and Ayuso-Mateos et al.¹⁶ The multi-country studies provided data on depression for 19 countries. With the exception of WHO Multi-country Survey 2000-2001, the other four multi-country studies drew on primary country-level studies published separately. We went back to the primary data sources and original papers to extract information on study design and sampling.

Using the WHO regional framework we assigned each published and unpublished study to their country and region of origin. For the purpose of our analysis, the country-level data from primary sources used in the multi-country studies are treated as separate studies, i.e. the umbrella studies are ignored. For each study the population sample was extracted, and compared with the regional and country-level estimates of population derived from World Health Report 2000.

Objectives

Our first two objectives are to analyse data quality with respect to:

• (1)completeness and representativeness of data for regional/national estimates;
• (2)GBDep inclusion criteria: (a) population-based studies with sample size greater than 1000; (b) clear specification of sampling method to yield nationally representative estimates; (c) prevalence studies with a specified period covered; and (d) incidence studies with specified age and gender distribution.²

Third, to review individual studies with respect to:

• (3a)GBD estimate of duration of illness for the calculation of incidence from prevalence studies;
• (3b)study design and comparability, response rates, case definition, and interview process.

Results

Objective 1: representativeness of studies for regional/ national estimates

The 28 sources cited in the GBDep study give country data on 40 of the 191 countries in six regions (Table 1). No WHO region had complete country coverage. EURO had the highest coverage with data on 15 out of 52 countries, AFRO region had the lowest with data on only three out of 46 countries (see Table A in Appendix 1 for the list of country-level sources for each region – http://jrsm.rsmjournals.com/cgi/content/full/jrsm.2010.100080/DC1 ). The number of respondents in each region is very small ranging from 49 per million in EURO to 2 per million in AFRO.

Table 1.

Population coverage in the depression estimate, by region

WHO region	Countries in regiona	Countries included in the GBDep study (n)	Total population of regionb	Population in GBD depression estimatec: number of respondents	Regional coverage per 100,000 population
AFRO	46	3	616,438,000	1029	0.2
AMRO	35	8	813,061,000	29,500	3.6
EMRO	21	3	484,488,000	7783	1.6
EURO	52	15	872,622,000	42,987	4.9
SEARO	10	4	1,508,241,000	6016	0.4
WPRO	27	7	1,673,575,000	40,084	2.4
Total	191	40	5,968,425,000	127,399

^aSource: Mathers (2004; Annex Table 1)³⁵

^bSource: World Health Report 2000³⁶

^cSource: GBDep study;² does not include data from Peru, Japan, Cambodia, Russia, the Czech Republic, Nepal or Morocco

Note: In the GBDep study, Pakistan and Singapore are included in the SEARO region instead of EMRO and WPRO, respectively, as per WHO regions. Therefore, the recalculated population numbers are: EMRO 332,157; SEARO 1,664,094; and WPRO 1,671,053. Recalculated numbers for the regional coverage per 100,000 population are: EMRO 2.3; SEARO 0.4 (unchanged); and WPRO 2.4 (unchanged)

Of the 40 countries with data, 22 countries did not have nationally representative data and the national representativeness of the included studies could not be determined for a further six countries. In EURO, the region with the highest coverage, only six of 15 studies had a nationally representative sample. The three AFRO studies were based on a small village or town. Likewise South-East Asia region had data on only four countries but none of the samples were nationally representative. Of the 6016 respondents: 5145 were in one Indian state (Andhra Pradesh); 612 from three constituencies in Singapore; 259 from a semi-urban village in Pakistan and Nepal could not be ascertained (Table 2). Overall, only 12 of the 36 (33%) studies for which we have data used a nationally representative sample.

Table 2.

Data from SEARO region

	Singapore	India	Pakistan	Nepal
Source	Kua37	WHO multi-country survey, 2000–2001	Husain38	Personal communication
Study design (duration)	Prevalence (unclear)	Prevalence (unclear)	Prevalence (unclear)	Prevalence (unclear)
Sample population	Three constituencies	Andhra Pradesh	One semi-urban village	One village
Sample size (respondents)	1000(612)	5000(5145)	259	n/a
Age range (years)	65+	18+	18+	n/a
Sampling framework	Electoral roster	Electoral roster	Electoral role	n/a
Random sampling	Yes	Yes	Yes	n/a
Stratification/weighting	No	Yes	Yes	n/a
Response rate (%)	61.2	98	98	n/a

Objective 2: GBDep inclusion criteria

a. Sample size

The GBDep study required a sample size of more than 1000 for inclusion. However, in 19 of the 42 (45%) studies either this GBDep criterion was not met or the sample size was not known (Table 3). Of those 19 studies, 12 had a sample size of 1000 or less (three AFRO studies, studies from Canada, Lebanon, Finland, Greece, Pakistan and twoGerman studies). For the remaining seven we could not determine the sample size. These included Morocco, the Czech Republic, Russia and Nepal – where data had previously been provided by personal communication to authors of the GBDep study. We could not access the Peruvian study; and the data and sample for Japan and Cambodia are not described in sources referenced in the GBDep study.

Table 3.

Matching characteristics of included data sources against the four GBDep inclusion criteria

GBDep criterion	AFRO	AMRO	EMRO	EURO	SEARO	WPRO	Total (n=42)
1. Sample size
> 1000		7	1	9	1	5	*23
< = 1000	3	1	1	5	2		*12
Unknown		1	1	2	1	2	*7
2. Prevalence studies
Period specified	3	6	1	9		4	23
Period unknown		2	2	7	4	3	18
3. Incidence studies with age and gender distribution
Known		1		1			2
Unknown
4. Sampling method
Specified	3	7	2	12	3	5	32
Unknown or unclear		2	1	4	1	2	10
Nationally representative sample		3	1	6		2	12

b. Sampling method

The GBDep study required ‘a clearly specified method for sampling (a design that would yield a probabilistic national/regional representative sample) and implementation’.² Of the 30 studies, which presented their sampling method, 29 used random sampling and one addressed the whole targeted population. Thirty studies sampled adults aged 18 years and over, one study was of children and three studies surveyed over-65s only. For eight studies the age group was not given, was unclear, or we were unable to contact authors of the study. Although all 36 studies stated their target population, in 16 the sampling frame was not clear.

c. Prevalence studies with a specified period covered and incidence studies with specified age and gender distribution

Of the 42 country studies included in the GBDep study, two provided incidence data (Canada and Norway); no information was available for six studies. The remaining 34 studies were prevalence studies but for 18 country studies the duration period was unclear or unknown.

The Canadian incidence study conducted in Stirling County reported incidence for men and women in two age groups: 40–64 years and over 65 years. The sampling method and gender distribution as compared to the population from which the sample was drawn was not described in the referenced study (but reported elsewhere). The incidence study from Norway reported incidence data for men and women over 18 years in urban (borough of Oslo) and rural (the islands of Lofoten) population. The sample was representative of the population from which it was drawn.¹⁷

Objective 3a: GBD estimates of duration of disease from prevalence studies

The GBD measure of average duration of illness for an episode was assumed to be six months. However, duration was derived from three studies only, two from the USA^18,19 and one from the Netherlands.²⁰

Objective 3b: other issues concerning study design

Study design and comparability

All studies except the WHO multi-country survey, which provided data for six countries, aimed to determine prevalence or incidence of depression and other mental disorders in specified populations. The WHO multi-country survey (2000–2001) focused on health systems performance, measuring population health, health inequality, fairness in financial contribution, responsiveness and efficiency and data on prevalence of depression are not presented in the final publication.

Five multi-country studies, which provided data on 19 countries, were included. Although all multi-country studies attempted to collect comparable data, methods varied across countries. Sampling methods and results for individual countries were reported and published separately and the multi-country studies then referred to primary data sources (see Table C in Appendix 1 online – http://jrsm.rsmjournals.com/cgi/content/full/jrsm.2010.100080/DC1 ) but reported ‘processed’ data, e.g. the Weissman et al. prevalence estimates from all study countries are adjusted to the US age and sex distribution rather than the country of origin.¹⁴

Response rates

In some studies the reported response rate did not agree with the initial and final sample that was reported. For example, for the household survey in Egypt, 5000 individuals were selected but the final sample was reported as 4486.¹³ This suggests a much lower response rate than 99% as reported. Reported sample sizes, numbers of respondents and response rates are summarized in Table A in Appendix 1 online. Based on the reported numbers, response rate was greater than or equal to 90% for nine studies, between 70–90% for 18 studies, less than 70% for four studies and unknown for 11 studies.

Case definition: epidemiological measures of depression

One source of major potential bias is that the studies used a range of different depression scales and measures: among these were highly standardized structured interviews conducted by lay people, which do not allow room for clinical judgement and additional questioning,²¹ such as the ‘Diagnostic Interview Schedule’ (DIS) and the ‘Composite Diagnostic Interview Schedule’ (CIDI). In addition semi-structured interviews were undertaken by psychiatrists with room for additional questioning and were not as standardized, such as the Present State Examination.²²

The most common measures used were CIDI (in 13), DIS (in nine) and the Present State Examination (in three). Many studies use different versions of the same measures and in the WHO International Consortium in Psychiatric Epidemiology study²³ which was the source of data for Mexico and Turkey, the use of different versions of CIDI was acknowledged as the source of wide variation in lifetime prevalence estimates across countries.

Interview process

The epidemiological studies typically used multistage sampling method with face-to-face psychiatrist-lead interviews in the second stage and nurse or other trained interviewers involved in the first screening phase. The comparative cross-country study by Weissman et al. that provided data-sets for eight countries was a two-stage study (but without the screening phase) and the interviews were conducted by trained lay people.¹⁴ The WHO multi-country survey that provided data-sets for six countries used different sampling strategies (national census data, electoral rosters: Random Walk Procedure exceptionally accepted in Georgia) as well as different modes of data collection (household individual interviews used in the six countries; in addition, in Egypt and of province of China, a postal survey was conducted).¹³

Discussion

The quality of epidemiological data in the GBD estimate

The GBDep study is characterized by lack of proper referencing and inaccessible data including four unpublished studies. The GBD estimates are highly problematic. In the absence of incidence data for most countries (there were only two incidence studies, one in Canada²⁴ and one in Norway¹⁷) GBDep relied on converting prevalence data into incidence data on the basis of the estimate of disease duration. The methodology was not fully explained in the GBDep study. Crucially, the average disease duration of six months was derived from three studies undertaken in the West, two in the USA and one in the Netherlands. In the absence of other data, these estimates of disease duration were then applied to derive estimates of burden of depression worldwide.

The authors of the GBDep study state that ‘a systematic review of all available published and unpublished papers of meaningful population studies on depressive disorders’² was conducted to obtain up-to-date prevalence and incidence data, no details of the systematic review and included studies are provided. We had to find the primary and secondary studies to extract details on samples and study design. For data collected via personal communication we could contact only one author and we do not know whether the data from the other three unpublished studies were based on population-based studies or ‘informed estimates’. We could not locate data on Japan because the source was not given in the GBDep study and the sources of data on Cambodia listed in the GBDep study did not contain any information on sample, prevalence or incidence of depression. GBDep did not provide a list of studies which were excluded and it was outside the remit of this study to examine them.

Coverage and national representativeness

Only 40 out of 191 countries contributed data to the global burden of depression measures. The AFRO region had least coverage, with data relating to only three out of 46 countries and a sample of around 1200 people. The South-East Asia region had data on only four countries with about 6300 people sampled: of these 5000 were residents of one Indian state (Andhra Pradesh) and the remaining 1300 from a single village or region in Singapore, Pakistan and Nepal. The EMRO region was also poorly covered with about 8000 people sampled in two countries (over 7000 in Egypt and less than 1000 in Lebanon); details for the Morocco study were not available.

Overall, only 12 countries had nationally representative samples; all of the other country data are based either on unrepresentative samples or extrapolated from data from other countries. For two regions (AFRO and SEARO), none of the studies met the requirement of ‘a probabilistic national/regional representative sample’² and for 20 studies the sample was sometimes only a village, small area, or based on a GP register limiting the applicability of the data to the rest of the country population.

Moreover, 12 of the 42 studies fail to meet the GBDep inclusion criteria with respect to population sample of minimum 1000 people; the three AFRO studies were based on samples of 181, 456 and 600, respectively. The authors do not provide an explanation for their decision to include smaller studies nor were details given of studies which were excluded.

Quality of studies and reliability of data

While 27 studies had a response rate of 70%; for 11 studies, the response rate was not given or unclear, and one of the German studies had a response rate of 27%.¹⁵ These studies should not have been included.

While the Global Burden of Disease project as a whole acknowledges that there is a serious need for more reliable data,²⁵ as Cooper et al.¹¹ point out, the results of the global burden of disease are largely treated as straightforward facts, without considering their limitations. A typical response to criticisms of methodology is:

Disciplines such as demography and economics often aim to make the best possible estimates using the available data … GBD 1990 has been criticised for using “estimates” rather than “actual data”. This is not a relevant discussion for comparative burden estimates because all epidemiological data relating to population are “estimates of varying degrees of precision or uncertainty”.²

These data suggest that scientists and epidemiologists have been prepared to depart from their normal standards and rely upon non-existent or unsatisfactory data. One explanation is that lack of evidence forces a make-do attitude. Another is that researchers themselves are co-opted into the project by funding and academic requirements. Studies publishing new GBD estimates have an impressive citation record. For example, according to Scholar Google the GBDep study published in the British Journal of Psychiatry in 2004 had been cited in 198 studies by August 2008. Other articles in the same issue of the journal had from six to 71 citations. The high number of citations is, however, not uncommon for other GBD estimates, e.g. Kearney et al. study published in The Lancet in 2004²⁶ had 478 citations by August 2008.

Case definition and ascertainment

The prevalence and incidence studies used in the GBDep paper are also problematic because they lack a standard case definition, use different measures or different thresholds for reporting, and different methods for ascertainment and collecting the data. Other cross-national studies highlight this as a limitation of the applicability of the results^14,23 but the GBDep study does not mention this serious limitation affecting prevalence and incidence.

The heterogeneity in the expression of mental health across cultures and countries raises questions about the validity of using standardized tools and instruments. The importance of the sociocultural context in which measures are used and the problems of using standardized tools as a substitute for clinical judgement has been highlighted.²⁷ Sadness or depression may be associated with a range of factors including poverty and distress.²⁸ It is problematic to apply a standard weight to a disease like depression, which is experienced subjectively. In different cultures and economies the weight, significance and importance of different diseases may vary and so may the ability to diagnose.

Use of GBD to set treatment priorities and public healthcare strategies

It is important to note that the lack of representative and reliable data are not limited to depression. Policymakers should be aware of the uncertainties and biases in using GBD estimates to set priorities. For example, the Disease Control Priorities Project initiated by the WB and backed by the WHO and the Bill and Melinda Gates Foundation among others uses GBD estimates of prevalence and incidence as ‘the baseline epidemiological situation that would prevail without treatment’ for the analysis and selection of cost-effective treatment strategies to reduce disease burden.²⁹ Although the composite measures are often preferred by policymakers and are a useful advocacy tool³⁰ they conceal the compromises, assumptions and extrapolations that are necessary due to the lack of data especially in low-income countries. There is a serious risk that in the absence of sound data there will be under- or over-treatment, or misallocation of resources within or across disease categories.

The GBD does not and indeed cannot acknowledge the role of social determinants such as poverty, economic distress, sanitation, hunger and malnutrition and the association between poverty and diseases. Nor do single measures recognize the complex relationships between diseases. The result is a vertical approach to health and illness and a narrow focus on disease-specific interventions ignoring the need to incorporate the broader determinants of health and horizontal approaches to healthcare.³¹

Current initiatives to collect more reliable data

Attempts are ongoing to collect better data – to include countries without previous descriptive epidemiology on mental health disorders, to collect comparable data on severity of disorders and disability caused by mental disorders, and to gather detailed information on access to and quality of treatment.³² Cross-country comparable data-sets for 27 countries are now available through the World Mental Health Initiative. For the 2010 estimates of GBD these data will need to be complemented by other sources and the concerns about the quality, overall representativeness and comparability remain valid. As Patten has highlighted, analytically-oriented epidemiologists and decision-makers need to be aware of the numerous interpretive difficulties in trying to understand and use prevalence rates for depression, and that international comparisons and measures are vulnerable to different types of measurement bias.³³

Conclusion

GBDep estimates have limited value in international policy-making. GBDep estimates are epidemiologically flawed in terms of representativeness and quality. Recent research initiatives focus on improving data collection and methods for the 2010 update of GBD estimates but there is a need to put in place disease surveillance and vital registration systems in advance of such studies.³⁴ This review provides a template for evaluating the quality of the studies used in 2010 estimates. Moreover, for neuropsychiatric disorders such as depression where the pathophysiologic process cannot be directly observed but must be inferred, there is a need to be sensitive to the numerous interpretative difficulties in determining rates of disease and better understanding of depression in the context of cultural, sociological and economic conditions. Single composite measures of depression are highly problematic: they conceal and hide uncertainty, compromise biases and distortions in epidemiological data. Crucially single measures of disease burden distance mental wellbeing from the social determinants of health such as poverty and economic status and fail to recognize their cultural and social significance let alone their interaction with other disease states.

Footnotes

DECLARATIONS —

Competing interests None declared

Funding This paper emerged from the collaborative research project Tracing Pharmaceuticals in South Asia (2006–2009) that was jointly funded by the Economic and Social Research Council and the Department for International Development (RES-167-25-0110). The project team comprised: Soumita Basu, Gitanjali Priti Bhatia, Samita Bhattaraj, Petra Brhlikova, Erin Court, Abhijit Das, Stefan Ecks, Ian Harper, Patricia Jeffery, Roger Jeffery, Sakshi Khurana, Rachel Manners, Allyson Pollock, Santosh MR, Nabin Rawal, LizRichardson and Madhusudhan Subedi. Neither ESRC nor DFID is responsible for views advanced here

Ethical approval Not applicable

Guarantor AMP

Contributorship All authors made substantial contributions to: conception and design, had access to all data and contributed to analysis and interpretation of data; drafting the article or revising it critically for important intellectual content; and final approval of the version to be published

Reviewer Dominic Murphy

Supplementary Material

Additional material

Appendix 1, Tables A, B, and C