Reliability and validity of depression assessment among persons with HIV in sub-Saharan Africa: systematic review and meta-analysis

Abstract

OBJECTIVES

To systematically review the reliability and validity of instruments used to screen for major depressive disorder or assess depression symptom severity among persons with HIV in sub-Saharan Africa.

DESIGN

Systematic review and meta-analysis.

METHODS

A systematic evidence search protocol was applied to seven bibliographic databases. Studies examining the reliability and/or validity of depression assessment tools were selected for inclusion if they were based on data collected from HIV-positive adults in any African member state of the United Nations. Random-effects meta-analysis was employed to calculate pooled estimates of depression prevalence. In a subgroup of studies of criterion-related validity, the bivariate random-effects model was used to calculate pooled estimates of sensitivity and specificity.

RESULTS

Of 1,117 records initially identified, I included 13 studies of 5,373 persons with HIV in 7 sub-Saharan African countries. Reported estimates of Cronbach’s alpha ranged from 0.63–0.95, and analyses of internal structure generally confirmed the existence of a depression-like construct accounting for a substantial portion of variance. The pooled prevalence of probable depression was 29.5% (95% CI, 20.5–39.4), while the pooled prevalence of major depressive disorder was 13.9% (95% CI, 9.7–18.6). The Center for Epidemiologic Studies-Depression scale was the most frequently studied instrument, with a pooled sensitivity of 0.82 (95% CI, 0.73–0.87) for detecting major depressive disorder.

CONCLUSIONS

Depression screening instruments yielded relatively high false positive rates. Overall, few studies described the reliability and/or validity of depression instruments in sub-Saharan Africa.

INTRODUCTION

HIV and major depressive disorder are frequently comorbid (1) and are both highly prevalent in sub-Saharan Africa. While treatment scale-up has led to dramatic gains in life expectancy (2), HIV remains one of the leading causes of disease burden throughout sub-Saharan Africa (3). Depressive disorders are also a major contributor to the burden of disease throughout the region (4, 5). The two conditions are mutually reinforcing: symptoms of depression may increase the probability of HIV acquisition (6), while biological factors (7) and psychosocial aspects of living with HIV (e.g., stigma (8, 9), poverty (10, 11), and food insecurity (8, 12)) may exacerbate psychological distress and increase the probability of developing depressive disorders. The public health significance of identifying and adequately treating depressive disorders among persons with HIV is further magnified by their important adverse HIV-related impacts, including delays in treatment initiation (13), treatment non-adherence (14, 15) and disease progression (16, 17).

Specific diagnostic criteria exist to guide the identification of persons with the syndrome of depression, but these are described explicitly as “guidelines to be informed by clinical judgment” that are “not meant to be used in a cookbook fashion” (18) (p.xxiii). Making the diagnosis of major depressive disorder additionally requires the exercise of clinical judgment, assessment of symptom severity and functional impairment, as well as the exclusion of alternative medical or psychiatric explanations for the symptoms. Where relying on such clinical expertise is often infeasible, those managing large-scale studies or programs may instead use structured instruments. These are frequently used to generate clinician ratings of patient symptoms but can often be self-administered (19). The rating scores are then used either to define cases of a depressive syndrome or measure the severity of a depressive syndrome (20). Some instruments, such as the Patient Health Questionnaire (21), are used for both. Importantly, however, there is little evidence to suggest that routine depression screening improves health outcomes (22–26).

While prior reviews focused on sub-Saharan Africa have examined depression assessment in general population samples (27, 28) and during pregnancy and postpartum (29), the potentially overlapping symptoms of HIV and depression pose important challenges. Namely, somatic symptoms of HIV can masquerade as somatic symptoms of depression, particularly among persons with advanced HIV disease (30–32). In one national study of persons with HIV in the U.S., depression was underdiagnosed by a factor of two (33). Several studies have also found that depression is undertreated among persons screening positive for mood or anxiety disorders (34–37). Comparable studies have yet to be conducted in sub-Saharan Africa, but it is likely that the same patterns of underdiagnosis and undertreatment prevail given the pervasive disparities in resources allocated to mental health systems throughout the region (38, 39). At the same time, somatic symptoms represent a common class of presenting symptoms of common mental disorders in many sub-Saharan African countries (27, 40–47). The purpose of my study, therefore, was to systematically review the reliability and validity of instruments used to screen for major depressive disorder or assess depression symptom severity among persons with HIV in African settings.

METHODS

Systematic search protocol

All study procedures were reviewed by the Partners Human Research Committee and deemed exempt from full review because the study was based on anonymous, public-use data with no identifiable information on participants. The systematic evidence search was conducted January–May 2012. Seven bibliographic databases were used: African Journals Online, the African Journal Archive, the Cumulative Index to Nursing and Allied Health Literature, Embase, the Medical Literature Analysis and Retrieval System Online (MEDLINE), PsycINFO, and the World Health Organization African Index Medicus (Table S1 in Supplemental Digital Content, http://links.lww.com/###). The MEDLINE search was updated in December 2013 to identify articles published in the interim period. After all citations were imported into EndNote reference management software (version X5, Thomson Reuters, New York, NY), the “Find Duplicates” algorithm was used to exclude duplicate references. The titles and abstracts, and then the full texts of the articles, were sequentially screened to select articles for inclusion. To identify other potentially relevant studies, I searched the reference lists of selected articles and queried colleagues in departments of psychiatry and psychology at two African academic institutions.

Selected articles had to have met each of three criteria: (a) the study was based on data collected from HIV-positive adults in any African member state of the United Nations; (b) an instrument was administered to assess for depressed mood, such as a diagnostic interview schedule, screening measure, or symptom rating scale; and (c) the article presented evidence of the reliability and/or validity of the instrument. Pregnant women were excluded given that they were the focus of a recently published review (29). There were no language restrictions. A wide range of reliability and validity evidence was considered acceptable for this review, including: evaluations of linguistic, conceptual, or metric equivalence (48); analyses of measurement reliability, such as test-retest reliability or internal consistency; or studies that confirmed hypothesized relationships between the instrument and other variables of interest (49), such a reference criterion standard (e.g., diagnosis of major depressive disorder consistent with the Diagnostic and Statistical Manual of Mental Disorders [DSM]) or variables conceptually thought to be related to depression (e.g., HIV stigma (50) and social support (9)). Because virtually any study estimating the association between depression and another variable of interest could potentially be considered as presenting evidence of construct validity, and because Cronbach’s alpha coefficients are near-universally reported, studies in which these were the sole form of evidence presented were excluded from consideration.

Data analysis

For each selected article, data were extracted regarding the study population, sampling strategy, sample size, inclusion criteria, depression instrument, and type of reliability and/or validity evidence provided. The numbers of cases of probable depression (study participants whose scores on a screening instrument or symptom rating scale exceeded a specified threshold) and of major depressive disorder (study participants who met diagnostic criteria according to the DSM) were also extracted. To calculate pooled estimates of probable depression and major depressive disorder, the variances of the raw proportions were first stabilized using a Freeman-Tukey-type double arcsine transformation (51, 52), and then the proportions were pooled using random-effects meta-analysis (53). Between-study heterogeneity was assessed with the I² statistic (54). Because HIV treatment is known to have important effects on depression (7) and other psychosocial outcomes (10, 55), random-effects meta-regression models (56) were fit to the data to explore the extent to which differences in HIV treatment status could explain heterogeneity in prevalence across studies. Treatment status was categorized as currently on HIV antiretroviral therapy [ART] vs. other (i.e., not on ART, treatment-naïve patients newly initiating ART, or mixed samples in general HIV care). Small sample size-related bias was investigated by visually inspecting graphical plots of the transformed prevalence estimates against the standard error of the transformed prevalence estimates, and also by using the Begg and Mazumdar (57) rank correlation test and the Egger et al. (58) linear regression test.

For studies that provided evidence of criterion-related validity, I assessed quality according to the Quality Assessment of Diagnostic Accuracy Studies tool (59). For the critical mass of studies that provided evidence of criterion-related validity for the Center for Epidemiologic Studies-Depression (CES-D) scale, data were extracted on the numbers of participants classified as true positives, true negatives, false positives, and false negatives according to the threshold values specified by the authors. These numbers were then employed to construct 2 × 2 tables and compute the estimated sensitivity and specificity values. Pooled estimates of sensitivity and specificity, and their associated 95% confidence intervals, were calculated using the bivariate random-effects model (60, 61). The summary receiver-operating characteristic (ROC) curves were then constructed to produce a 95% confidence ellipse within the ROC curve space (62). Between-study heterogeneity was assessed with the I² statistic for the pooled diagnostic odds ratio (54). Small sample size-related bias was investigated by plotting the logarithm of the diagnostic odds ratios against the inverse square root of the effective sample size, and by fitting the accompanying regression model of the logarithm of the diagnostic odds ratios against the inverse square root of the effective sample size, weighting by the effective sample size (63). All statistical analyses were implemented with the use of the Stata software package (version 12.1, StataCorp LP, College Station, Tex.).

RESULTS

Studies identified for this review

Of the 1,117 records returned from the electronic database search, 110 duplicates were excluded, along with 880 records that did not appear to meet inclusion criteria based on the titles and/or abstracts alone (Figure 1). Full text appraisal was completed for 127 records. Of these, 112 did not meet inclusion criteria and were excluded. Colleagues at African academic institutions suggested one additional journal article. Three studies reported findings across multiple publications; to avoid double counting, these findings were aggregated and then assigned to the first publication by calendar year. The final sample included 15 journal articles and 1 Ph.D. dissertation (that was matched to a subsequently published journal article), representing 13 unique studies.

Figure 1.

Quality of Reporting of Meta-Analyses (QUORUM) flow chart depicting the number of reports screened and included in the systematic review

Summary statistics for the selected studies are provided in Table 1. The 13 studies enrolled 5,373 persons with HIV in 7 different sub-Saharan African countries. Study sites were primarily located in southern or eastern Africa, with South Africa and Uganda accounting for more than one-half of the studies. Although I did not exclude studies from northern or western Africa none were identified, and only one study was based on data from central Africa (Cameroon). The median sample size was 368 (interquartile range, 200–610). Most studies were based on data collected from outpatients currently on ART or newly initiating ART, or from mixed samples of outpatients in general HIV care (e.g., either on ART or pre-ART).

Table 1.

Summary statistics (N=13)

Study characteristic	Number (%)
Country of origin
South Africa	4 (31)
Uganda	4 (31)
Other †	5 (38)
Number of study participants, median (interquartile range)	368 (200–610)
Study setting ‡
Outpatient	11 (85)
Community	2 (15)
Inpatient	1 (8)
Date of publication
2008–11	4 (31)
2012–14	9 (69)
HIV treatment status
Yes	5 (38)
Newly initiating	4 (31)
Mixed	4 (31)
No	1 (8)
Instrument assessed‡
Center for Epidemiologic Studies Depression Scale	5 (38)
Patient Health Questionnaire	3 (23)
Other ¶	10 (77)
Type of evidence provided‡
Reliability	10 (71)
Criterion-related validity	9 (64)
Factor structure	9 (64)
Construct validity	5 (36)
Equivalence	2 (14)

^†Includes Cameroon, Ethiopia, Kenya, Rwanda, and Zambia (1 study each)

^‡Percentages may not add up to 100, as categories are not mutually exclusive

^¶Includes the Beck Depression Inventory II, Brief Symptom Inventory, Hopkins Symptom Checklist for Depression, Hospital Anxiety and Depression Scale (2 studies), K6/K10 non-specific psychological distress scales (2 studies), Self Reporting Questionnaire, and visual analogue scale

Prevalence of depression

Altogether, the reliability and validity of nine different depression instruments were assessed (Table 2; also see Table S2 in Supplemental Digital Content, http://links.lww.com/###). While the CES-D and the Patient Health Questionnaire were the most frequently studied, none of the instruments had been the subject of study in more than three different countries. The prevalence of probable depression (i.e., as determined on the basis of screening instruments) among 4,461 participants enrolled in 11 studies varied from 6.5 to 75 percent, with a pooled prevalence of 30.2 percent (95% confidence interval [CI], 19.7–41.8; I²=98.4) (Figure 2). Publication bias did not appear to be present (P-values ranged from 0.88–0.97). Meta-regression suggested that HIV treatment status was an effect modifier (P=0.048): in studies of participants on HIV treatment, the pooled prevalence was 17.5 percent (95% CI, 6.5–32.3), whereas in studies of treatment-naive persons or mixed (treated/untreated) samples, the pooled prevalence was 37.8 percent (95% CI, 27.7–48.5).

Table 2.

Reliability and/or validity of depression assessment among persons with HIV in sub-Saharan Africa

Scale	Reference	Country	Treatment status	Factor structure	Reliability	Sensitivity / Specificity	Criterion standard	Depressed by scale †	Depressed by criterion	Related constructs
BDI-II	Kagee et al. (96)	South Africa	Yes	3 factors	0.90			83/185
BSI	Shacham (97)	Kenya	Mixed	1 factor	0.95					PHQ-9, stigma
CES-D	Akena et al. (98) ‡	Uganda	Mixed			0.88 / 0.81	MINI
	Chishinga et al. (99)	Zambia	Initiating	4 factors	0.84	0.73 / 0.70	MINI	221/649	62/649
	Kaharuza et al. (100)	Uganda	Initiating	4 factors	0.90			478/1017		Health
	Myer et al. (101)	South Africa	Mixed			0.79 / 0.61	MINI	206/465	62/465
	Singh et al. (102)	South Africa	Initiating			0.91 / 0.44	Clinical	15/20	12/20
HADS	Reda (103)	Ethiopia	Yes	1 factor	0.87
	Wouters et al. (104)	South Africa	Yes	2 factors	0.63			72/716		Social support, stigma ¶
HSCL	Epino et al. (105)	Rwanda	Initiating	1 factor	0.87					Health
K10	Akena et al. (98) ‡§	Uganda	Mixed			0.83 / 0.72	MINI
	Spies et al. (106)	South Africa	Mixed		0.87	0.67 / 0.77	MINI	112/429	58/429
PHQ-2	Akena et al. (98) ‡	Uganda	Mixed			0.83 / 0.70	MINI
	Monahan et al. (107)	Kenya	Mixed			0.91 / 077	PHQ-9	110/347
PHQ-9	Akena et al. (98)	Uganda	Mixed			0.91 / 0.81	MINI	116/368	64/368
	Pence et al. (108)	Cameroon	Yes			0.27 / 0.94	CIDI	26/400	11/400
	Shacham (97) ||	Kenya	Mixed	1 factor	0.78			45/347		Health, BSI, stigma
SRQ	Nakimuli-Mpungu et al. (109)	Uganda	Yes	2 factors	0.84	0.84 / 0.93	MINI	32/200	24/200
Visual scale	Akena et al. (110)	Uganda	Mixed			0.75 / 0.71	MINI	35/94	17/94

BSI = Brief Symptom Inventory; CES-D = Center for Epidemiologic Studies Depression Scale; CIDI = Composite International Diagnostic Interview; HADS = Hospital Anxiety and Depression Scale; HSCL = Hopkins Symptom Checklist; K6/K10 = Kessler non-specific psychological distress scale; MINI = Mini International Neuropsychiatric Interview; PHQ = Patient Health Questionnaire; SCID = Structured Clinical Interview for the DSM-IV; SRQ = Self-Reporting Questionnaire

^†For the pooled estimate of prevalence, only the best-performing scale was used (i.e., for studies that provided prevalence estimates according to more than one scale).

^‡Study provides estimates of depression prevalence according to multiple scales but only one estimate of depression prevalence according to the criterion standard, and this is reported on the row corresponding to the PHQ-9.

^¶These findings were eported separately in Pappin et al. (111).

^§Study also provides evidence for validity of K6.

^||Some of the data from this dissertation were published in final form in Shacham et al. (112) and in Monahan et al. (107).

Figure 2.

Forest plot depicting the distribution of estimates of probable depression, with pooled estimates stratified by HIV treatment status

Evidence for reliability and/or validity

The most frequently described types of reliability and/or validity evidence supplied in these studies were of scale reliability (9 [69%]), criterion-related validity (8 [62%]), and factor structure (8 [62%]). The reported Cronbach’s alpha coefficients ranged from 0.63–0.95, with only one study reporting an estimate below the conventional threshold for “good” internal consistency. Across contexts, depression instruments had statistically significant associations with related constructs, including HIV stigma, social support, and health status. Analyses of internal structure generally confirmed the existence of a depression-like construct accounting for a substantial portion of variance. Of the five studies in which a multi-factor structure was thought to best fit the data, four identified a factor related to somatic symptoms. Only two studies, both using qualitative methods, assessed aspects of linguistic or technical equivalence.

Among the 7 studies that employed a criterion standard to determine a DSM-consistent diagnosis of major depressive disorder, the prevalence of major depressive disorder varied from 2.7 to 18.1 percent, with a pooled prevalence of 14.5 percent (95% CI, 8.9–21.2; I²=93.4). Publication bias did not appear to be present (P-values ranged from 0.13–0.19). HIV treatment status was not a statistically significant effect modifier (P=0.16); in treated samples, the pooled prevalence was 6.7 (95% CI, 0.6–18.5) whereas in naïve/mixed samples, the pooled prevalence was 17.9 (95% CI, 11.8–24.9).

Too few studies assessed the performance of the same screening instrument (irrespective of setting) to permit pooled estimates of sensitivity and specificity, with the exception of four studies that examined the criterion-related validity of the CES-D. When summarized within ROC curve space, the data suggested a pooled sensitivity of 0.82 (95% CI, 0.73–0.87) and a pooled specificity of 0.73 (95% CI, 0.63–0.80) at threshold values ranging from 16–22 (Figure 3). There was substantial between-study heterogeneity underlying these estimates, as suggested by I² values of 50.2 and 92.4, respectively, but there was limited ability to adequately investigate this heterogeneity given the small number of studies. Examination of the log-diagnostic odds ratios plotted against the inverse square root of effective sample size, and the accompanying linear regression test (P=0.63), did not suggest small sample size-related bias.

Figure 3.

Summary receiver-operating characteristic (ROC) curve plot of diagnosis of major depressive disorder based on CES-D threshold values selected by authors. The solid line depicts the summary ROC curve from the bivariate random-effects model. The solid square depicts the summary operating point, i.e., summary values for sensitivity and specificity. The dotted line depicts the 95% confidence region for the summary operating point.

The quality assessment demonstrated several areas in which the studies of criterion-related validity tended to have methodological shortcomings (Table S3 in Supplemental Digital Content, http://links.lww.com/###). Specifically, most of these studies were assessed to be at risk of bias due to use of a screening threshold that was not pre-specified (i.e., the reported threshold was selected to optimize sensitivity and/or specificity). Otherwise, most studies were at low risk of bias on the other domains. In general, few concerns were noted about applicability.

DISCUSSION

In this systematic review and meta-analysis, there were several important findings. First, I identified only 13 unique studies of nine different instruments used to assess depression among persons with HIV in sub-Saharan Africa. Second, screening instruments were generally found to reliably measure depression-like constructs and to correlate with related constructs in the expected fashion. Third, depression was highly prevalent, particularly in studies of treatment-naïve persons or of untreated/mixed samples. However, the prevalence of probable depression (as determined by screening) exceeded the prevalence of DSM-consistent diagnoses of major depressive disorder by a factor of two. These findings have important research and programmatic implications for persons with HIV in sub-Saharan Africa.

Before each of these findings are discussed in detail, it is important to note that the data identified in this systematic search do not permit conclusions about the reliability or validity of depression assessment in general population samples in African countries. For example, investigators have employed qualitative methods to elicit local concepts of distress and then have validated newly developed scales in Rwanda (64–67), Uganda (67–70), and Zimbabwe (43, 71–73). While these studies make important contributions to understanding cultural concepts of distress in African settings, particularly aspects of linguistic and conceptual equivalence, they would not have been included in the present review, as they were not based on samples of persons with HIV. Because the primary incremental contribution of validation studies conducted with samples of persons with HIV relates primarily to our understanding of construct and criterion-related validity, it is likely that excluding studies conducted in the general population would result in a sample of studies less focused on aspects of equivalence.

Keeping this limitation in mind, I identified relatively few studies describing the reliability or validity of instruments used to screen for major depressive disorder or assess depression symptom severity among persons with HIV in African settings. Though the evidence search was not limited to sub-Saharan Africa, no studies from northern Africa were identified, and Cameroon was the only study site in western or central Africa. In settings where data were identified, depression instruments were found to correlate with conceptually related constructs in the expected fashion, providing some support for the idea that these Western-derived instruments are measuring the same construct in different cultures (74). Among multi-factor instruments, a majority identified a somatic factor, confirming the importance of careful symptom assessment among persons with HIV so that false positives can be minimized (30, 31).

Among the participants in the studies reviewed, depression screening was associated with relatively high false positive rates: the pooled prevalence of probable depression by screening was 30.2 percent, while the pooled prevalence of major depressive disorder was 14.5 percent. This twofold difference is notable given that screening studies are frequently but inappropriately cited in support of impact statements describing the high prevalence of depressive disorders in the context of the HIV epidemic in sub-Saharan Africa. Yet, while screening instruments may generate overestimates, the 14.5 percent pooled prevalence rate of major depressive disorder estimated in my study still exceeds the 4–6 percent age-standardized prevalence in the region (5). Thus, despite the relatively high rate of false positives identified through screening, the burden of depression among persons with HIV in sub-Saharan Africa should still be considered relatively high.

HIV treatment status appeared to explain some of the variation in outcomes, as the prevalence of depression was greater in treatment-naïve samples. This finding is consistent with prior work describing declines in depression symptom severity among persons with HIV undergoing antiretroviral treatment (7, 8, 75–80). The mechanisms underlying these observed effects are unclear but may be related to biological (7), economic (10, 11), or psychosocial (55) changes associated with treatment. If confirmed, the “antidepressant” effects of HIV treatment could lend further support to treatment initiation as an HIV prevention strategy (81, 82).

Interpretation of my findings is subject to three primary limitations, in addition to those mentioned previously. First, as with all systematic reviews, my evidence search may have missed some relevant studies. A comparison of my sample with that of a related review by Sweetland et al. (83), however, suggests this possibility is unlikely. They searched two bibliographic databases for validity studies conducted among adults in sub-Saharan Africa and published prior to 2012. They identified 4 studies of persons with HIV (compared to 6 studies included in my review that were published prior to 2012) and 11 studies of pregnant or postpartum women (compared to 25 studies included in a recently published review of perinatal depression (29)). A second limitation, which applies to interpretation of the pooled prevalence estimates, is that this review was focused specifically on studies of reliability and/or validity. Conventional studies of depression prevalence among persons with HIV that did not also provide evidence of reliability and/or validity, such as those by Nakasujja et al. (78) (53.9 percent with probable depression) and Kinyanda et al. (84) (8.1 percent with major depressive disorder) in Uganda, would have been excluded. Therefore it is likely that the studies in my review do not represent the universe of prevalence studies. At the same time, however, this is unlikely to have biased my pooled prevalence estimates in either direction, given that reliability and validity studies are unlikely to systematically overestimate (or underestimate) the prevalence of either probable depression or major depressive disorder. A third limitation is that too few studies of the same instrument were identified to permit comparisons between instruments. While the CES-D was the most frequently studied instrument, few studies of other instruments were found.

The most important policy and programmatic implications of this systematic review and meta-analysis relate to the possibility of integrating depression assessment and treatment into HIV programs in sub-Saharan Africa. The estimates presented here further underscore the potential public health impacts of effective depression treatment as recognized in the International Association of Physicians in AIDS Care guidelines on improving care for persons with HIV (85). While effective depression treatment may carry important spillover benefits for persons with HIV (86, 87), global disparities in HIV (3) are paralleled by disparities in mental health systems and human resources for mental health (39, 88). Shifting responsibility for mental health screening and/or treatment monitoring to non-specialist lay health workers is a priority area of research (89) that has been proposed as one potential mechanism for extending limited human resources (90). Such tasks will require brief, locally validated screening instruments (91, 92) whose use can be integrated cleanly into the lay health workers’ overall workflow without causing undue burden (26, 93). In many sub-Saharan African countries, mental health system resource constraints introduce additional concerns. First, in the absence of appropriate depression care support systems (94), the value of depression screening strategies remains unclear. Second, while screening instruments are generally expected to yield relatively high false positive rates, an excess of false positives could easily overwhelm the capacity for outpatient mental health care delivery (95). The findings presented in this review, while based on a paucity of evidence overall, suggest that more work needs to be done to identify valid and discriminating instruments that can be used for screening and treatment monitoring in these contexts.