Improving Detection of Depression in People Living with HIV: Psychometric Properties of the South African Depression Scale (SADS)

Abstract

Most measures developed in high income countries to screen for major depressive disorder (MDD) among people living with HIV (PWH) demonstrate suboptimal psychometric properties when utilized in non-western, resource limited settings due to their high false positive rates. For standardized MDD screening to be implementable in local settings, a measure is needed that reduces diagnostic burden by being highly sensitive while limiting false positives. This study sought to evaluate the ability of the locally developed South African Depression Scale (SADS) to screen for MDD in PWH in Cape Town. The SADS was administered along with the SCID-5-RV as gold standard to 236 PWH. It demonstrated good discriminating ability in detecting MDD with an area under the curve of 0.85. A cut-off of 27 yielded 78.2% sensitivity and 54.4% PPV. Given its robust psychometric properties, routine use of the SADS in community clinics to screen at-risk PWH, combined with evidence-based depression treatment, could improve the health outcomes and well-being of PWH in South Africa.

INTRODUCTION

Major depressive disorder (MDD) remains largely undiagnosed and untreated globally, despite accounting for 40.5% of disability-adjusted life years (DALYs) (1). Rates of MDD are particularly high in at-risk populations such as in people living with HIV (PWH) who face additional challenges, both biological and social, as a result of their status (2,3). In South Africa, the country with the highest number of PWH in the world, a prevalence rate between 14 – 30% has been documented for MDD in this population (4,5).

Health policymakers in countries such as South Africa are recognizing the critical need for mental health treatment, particularly among PWH. MDD has been found to negatively impact health outcomes in PWH including poor adherence to antiretroviral therapy (ART), faster disease progression, and increased mortality (6–8). Early detection is a critical first step to reduce the burden of untreated MDD and improve outcomes. Yet, resources and funding for mental health care are limited in low- and middle-income countries (LMICs) (9). The expansion of mental health services, including case detection, in primary care facilities will require the use of already existing resources—making task-sharing a useful approach. Task-sharing typically consists of upskilling less highly trained personnel to perform tasks usually assigned to personnel with more specialized training to allow for more efficient use of existing resources (10).

Health care personnel working in community clinics, including nurses and counselors, could be trained to detect common mental disorders such as MDD using appropriate screening measures. Petersen and colleagues have demonstrated that a task-shared, stepped care model could be successfully implemented in primary care to improve the detection and treatment of common mental disorders in patients with comorbid chronic illnesses (11). This approach could also allow for the integration of standardized screening for MDD in at-risk populations, such as PWH with unsuppressed viral loads, in community clinics throughout South Africa. Only patients who screen positive for MDD would be referred to more specialized personnel for a thorough diagnostic assessment and possible treatment. However, the success of this referral pathway would require the use of a screening measure that is able to detect a high number of true cases of depression while not mistakenly identifying a high number of false positives. Identifying a high number of false positives will result in a significant increase in diagnostic burden on an already overburdened health care system (12).

Currently, most measures available in South Africa for screening of MDD in PWH were developed in high income countries and translated into the relevant local language(s). This approach has resulted in the use of measures that have either not been formally and/or comprehensively validated in this context, or have demonstrated very high false positive rates (4,13). Conflicting psychometric properties have been reported on the PHQ-9 in screening for depression among PWH in South Africa. Cholera and colleagues found strong psychometric properties including, at a cut-off of 10, sensitivity of 82.4, specificity of 70.8, and a low false positive rate (5). However, Bhana et al. found, at a cut-off of 10, sensitivity of 48.1, specificity of 95.6, and they did not report the positive predictive value (PPV) (14). The Centre for Epidemiological Studies Depression Scale (CES-D), the Beck Depression Inventory, and the Kessler Psychological Distress Scales (K10/K6), when administered in South Africa, have all demonstrated PPVs (an indicator of the amount of true positive cases), between 21 and 24% (4,13,15). This means that in approximately 3 out of 4 cases who screen positive for MDD are false positives. In other words, in 3 out of 4 cases, the patient does not in fact have the disorder.

For screening of depressive disorders to be feasibly and sustainably integrated into community clinic services in South Africa, measures are needed that are better able to distinguish between those who are and those who are not depressed, which will prevent flooding the under resourced healthcare system with misclassified cases. As Patel and his colleagues stipulated, ideally, a measure used in resource limited clinical settings should identify a large number of true cases (high sensitivity) with few false positives (high positive predictive value) (12). They set minimum criteria of identifying a cut-off score on a measure that allows for at least 50% sensitivity and 50% PPV to be deemed acceptable (12).

The South African Depression Scale (SADS) was developed using an emic approach (i.e., grounded in cultural specificity), with the aim to improve the discriminating ability of a measure for MDD in PWH specific to this context (16). The SADS was borne primarily from qualitative interviews conducted with isi-Xhosa speaking PWH who met diagnostic criteria for MDD as determined by a clinical psychologist (16). The internal consistency, convergent validity, and factor structure of the SADS was established among PWH with clinical depression in a previous study in Cape Town, South Africa (17). However, the ability of the SADS to discriminate between those who meet and those who do not meet diagnostic criteria for MDD has yet to be established. The purpose of the current study was therefore to determine the criterion validity of the SADS in identifying cases of MDD in PWH in South Africa, and to identify a recommended cut-off value that balances sensitivity and PPV.

METHOD

Participants

The sample was drawn from a larger study examining rates of comorbid mental disorders in PWH in South Africa. Participants were recruited from a district hospital’s Infectious Disease Clinic and a primary care clinic in two peri-urban communities in Cape Town, South Africa. Participants were eligible for participation if they were adults (18 years of age or older), able to communicate in English, and were receiving antiretroviral therapy (ART). Participants were all seeking HIV care at the time of study entry. Being on ART was determined by the information obtained in their medical file including which ART regimen they were prescribed. Participants were excluded if they had an untreated severe mental illness or were unable to provide informed consent. The interviewers, who were master’s psychology students, used their judgment to identify patients who showed symptoms of a severe mental health condition such as detachment from reality, delusions, etc. The interviewers were also alert to those patients who displayed an inability to understand what was required of them to participate in the study. No participants were excluded for this reason. Patients who could not communicate in English did not volunteer for the study and were not able to be screened, so no data were obtained on these patients.

Procedure

Flyers with information about the study were distributed at the study sites by a clinic nurse. Patients who were interested in participating were referred to the research assistant who was located in a private room in the clinic. The research assistants first determined study eligibility and then obtained informed consent from the patient, including permission to audio record the assessment and to access the patients’ medical file. The research assistant then administered the diagnostic interview and the battery of measures which included demographic information, the SADS, and the MDD module of the Structured Clinical Interview Schedule for the DSM 5 Research Version (SCID-5-RV). Data entry occurred on tablets using a web-based platform. Participants were provided with a grocery voucher as compensation for their time. Over 50% of the audio recorded diagnostic assessments (SCID-5-RV) were reviewed by the principal investigators, who are both psychologists, for quality assurance and to provide ongoing supervision on administration of the SCID.

Measures

Demographic information was obtained through self-report. This information included gender, language, education, work situation, and monthly income.

SCID-RV.

MDD was diagnosed with the use of the depression module from the SCID for DSM-5, the Research Version (SCID-5-RV) (18). The SCID-5-RV is a comprehensive semi-structured psychiatric interview guide that includes diagnostic assessment of all major DSM-5 disorders including subtypes, severity, and course specifiers. The Research Version allows for the customization of the interview to the specific needs of a study. It was designed for administration by a clinician or trained mental health professional familiar with the DSM-5 diagnostic system. However, non-clinicians can administer the SCID-5-RV with appropriate training and supervision (18). In the current study, the SCID-5-RV was administered by trained master’s psychology student interviewers. Training included receiving in-person instruction from two registered psychologists with specialized SCID training, viewing instructional videos, and participating in role plays and mock interviews. The master’s psychology student interviewers also received ongoing supervision from two registered psychologists who listened to the audio-recordings of the SCID interviews.

SADS.

The SADS is a brief, 16-item screener for MDD available in English and in isiXhosa that can either be self-administered or administered via interview (17). In the current study it was self-administered on a tablet with assistance from the trained interviewer when needed. The SADS consists of nine local idioms of distress (e.g., thinking too much, carrying a lot of weight on my shoulders, pain in my heart) and six traditional symptoms of depression (e.g., sadness, difficulties sleeping, anhedonia). These idioms of distress were combined with a final 16th item (isingqala – deep sorrow) which was added from consultation with isiXhosa speaking mental health providers. The response options on the SADS are scored on a 4-point Likert scale indicating the number of days in the past week the person has experienced the symptom (0 days, 1–2 days, 3–4 days, 5 or more days). The scores range from 0 to 48 with higher scores indicating greater severity. There is one final open-ended question at the end of the measure that asks when the symptoms started.

Ethical Considerations

Participants who indicated psychological distress, including those who met diagnostic criteria for a mental disorder on the SCID, or who wished to receive mental health services were provided with a referral to their local clinic or mental health center for treatment. The study was approved by the Stellenbosch University Health Ethics Committee (N17/08/063). Interviews were conducted in a private space and all de-identified data were securely stored on a password protected web-based server. Written informed consent was obtained from all participants prior to data collection.

Data Analysis

Data analysis was conducted using the Statistical Package for the Social Sciences (SPSS) version 26.0 (19). The criterion validity of the SADS was established against the SCID-5-RV. Receiver Operating Characteristic (ROC) curve analysis was conducted to evaluate the ability of the SADS to correctly identify cases of MDD. We calculated sensitivity, specificity, positive and negative predictive values, and the area under the curve (AUC). The AUC provides an estimated probability of a scale’s ability to discriminate between true cases of depression versus true non-cases. AUC values range from 0.5 to 1.0. A value of 0.5 means the result is due to chance, while a value of 1.0 indicates perfect discrimination. An AUC value above .9 indicates an excellent measure, above .8 indicates a good measure, and above .7 indicates a fair measure (20). An AUC below 0.6 is considered poor (21). Consistent with the approach by Patel et al. (12), the recommended cut-off for the SADS was based on the optimal balance between sensitivity and PPV, with both sensitivity and PPV needing to be above 50% to be deemed acceptable.

RESULTS

Sociodemographic Characteristics

The total sample size was 236. Twenty three percent of the sample met criteria for MDD as determined by the SCID-5-RV. Participants ranged in age from 18 to 74, with a mean age of 41 years old (see Table I). The sample consisted mainly of females, 81%. The majority of the sample self-identified as Black (83%) and the most commonly spoken first language was isiXhosa (61%), followed by Afrikaans (18%), and Shona (12%). Only 34% of the sample completed matric (i.e., graduated high school), and 3.4% attended university but did not graduate. With regards to employment, 39% of the sample was unemployed, 44% were employed full-time, and 12% were employed part-time. Only one participant had a combined monthly income in their household above R10,000. The other 99.6% of the sample had a combined monthly income less than R10,000.

Table I.

Sociodemographic Characteristics

	N with MDD	% with MDD	N no MDD	% no MDD	N total	% total
Gender
Female	44	23.0	147	76.9	191	80.9
Male	11	24.4	34	75.6	45	19.1
First Language
isiXhosa	29	20.1	115	79.9	144	61.0
Afrikaans	16	38.1	26	61.9	42	17.8
Shona	3	10.7	25	0.89	28	11.9
Sotho	3	37.5	5	62.5	8	3.4
English	2	50.0	2	50.0	4	1.7
Other	2	20.0	8	80.0	10	4.1
Relationship Status
Single	28	28.3	71	71.7	99	41.9
Married/ Living together	15	15.6	81	84.4	96	40.7
Widowed	4	23.5	13	76.5	17	7.2
Separated	5	41.7	7	58.3	12	5.1
Divorced	3	25.0	9	75.0	12	5.1
Level of Education
No formal education	0	0	1	100	1	0.4
Primary school	0	0	9	100	9	3.8
High school but did not complete matric	39	28.3	99	71.7	138	58.5
Completed matric	15	18.6	65	81.3	80	33.9
Attended university but did not graduate	1	12.5	7	87.5	8	3.4
Employment Status
Full-time	19	18.3	85	81.7	104	44.1
Part-time	7	0.25	21	0.75	28	11.9
Unemployed	28	30.1	65	69.9	93	39.4
Student	1	16.7	5	83.3	6	2.5
Retired	0	0	5	100	5	2.1
Monthly income
Less than R10,000	55	23.4	180	76.6	235	99.6%
R10,000 – 40,000	0	0	1	100	1	0.4%
	Mean	S.D.	Mean	S.D.	Mean	S.D.
Age	40.86	11.07	39.65	8.62	40.59	10.56

Characteristics of the SADS

Total scores on the SADS ranged from 0 to 48, with a mean score of 19.72 (SD = 14.93). The most endorsed item and the item with the highest mean score was ‘I was thinking too much’. The next two most highly endorsed items were: ‘Things were not going well in my life’ and ‘I had stress.’ However, the item with the second highest mean was ‘I was carrying a lot of weight on my shoulders’. The least endorsed item and the item with the lowest mean was ‘I cried’. The SADS demonstrated excellent internal reliability with a Cronbach’s alpha of .96.

ROC analysis

The AUC for the SADS was 0.85 for MDD and was significant at p < .001 (see Figure I). This finding suggests that the SADS is effective in discriminating between persons who do and do not meet the criteria for MDD. The sensitivity, specificity, PPV, and NPV for a range of cut-off scores can be seen in Table II. To minimize diagnostic burden, a cut-off score of 27 was chosen as it resulted in an optimal balance between sensitivity (78.2%), and PPV (54.4%). Both sensitivity and PPV were deemed acceptable as they were above the 50% mark (see Table III). At the cut-off score of 27, the specificity was 80.1% and the NPV was 92.4%. When only balancing sensitivity and specificity, which is a common approach, the optimal cut-off was 22. This resulted in a slightly improved sensitivity of 81.8% and specificity of 69.1%. However, at a cut-off score of 22, the PPV was sacrificed slightly (44.6%) which is not ideal for a resource-constrained setting.

Figure I.

¹ROC curve for the SADS predicting cases of DSM-V major depressive disorder

¹Sub-analyses conducted on the isiXhosa subsample yielded the same AUC and a similar ROC curve.

Table II.

ROC analysis for the SADS

Cut-off score	Sensitivity	Specificity	PPV	NPV
22	0.82	0.69	0.45	0.93
23	0.78	0.70	0.44	0.91
24	0.78	0.71	0.45	0.92
25	0.78	0.76	0.50	0.92
26	0.78	0.77	0.51	0.92
27	0.78	0.80	0.54	0.92
28	0.76	0.81	0.55	0.92
29	0.75	0.83	0.57	0.92
30	0.69	0.83	0.56	0.90
31	0.69	0.87	0.62	0.90
32	0.64	0.88	0.63	0.89
33	0.62	0.90	0.65	0.89

Table III.

Recommended cut-off for the SADS and Cronbach’s alpha

MDD	Cronbach’s Alpha	AUC	Cut-off	Sensitivity	Specificity	PPV	NPV
SADS	0.96	0.854	27	0.782	0.801	0.544	0.924

Discussion

This study aimed to determine the ability of a novel, locally developed depression screening measure, the SADS, to discriminate true cases of MDD from true non-cases. Based on the ROC analysis, the SADS appears to be a good measure for detecting MDD in PWH in South Africa, with an AUC of 0.85. The optimal cut-off score that balances sensitivity and PPV was 27 yielding 78.2% sensitivity, 54.4% PPV, and 80.1% specificity. We accepted a lower sensitivity to reduce the number of false positives and thus lower the diagnostic burden.

The psychometric properties of the SADS appear promising compared to measures that were developed in high income countries and translated into isiXhosa. The AUC of the SADS (0.85) is higher than those reported for the CES-D (0.76), the Beck Depression Inventory (BDI; 0.77), and the K10/K6 (0.71/0.70), though the AUC is in line with those reported for the PHQ-9 (0.88/0.85). In addition, with a cutoff of 27, the SADS has a PPV more than double that of the CES-D, BDI, K10, and K6, while maintaining high sensitivity (78.2%). However, future studies are needed to compare the psychometric properties of these measures within the same South African sample. Furthermore, the SADS still incorrectly classified just under half the cases. Approximately 1 out of 2 cases who screen positive are false positives, so just under 50% of those who would be referred to a clinician for a diagnostic assessment would not meet criteria for MDD. It is debatable whether community clinics could handle such a high diagnostic load if all PWH were being screened. Focusing screening on the higher risk groups for MDD, for example ART-users who are unsuppressed and attending the risk of treatment failure (ROTF) groups, would be one way to limit the diagnostic burden when using the SADS, making standardized screening more feasible.

The brevity of the SADS and the fact that it can be self-administered or administered by a paraprofessional makes it ideal for task-sharing. Literate patients could complete the screener themselves and a nurse or counselor could easily be trained to administer the SADS when needed. The nurse or counselor could be trained to score the SADS and then only refer patients with a score of 27 or above for assessment by a clinician. This task-shifting approach could improve the feasibility of standardized screening if done on a focused at-risk population.

The SADS demonstrated excellent internal reliability with an alpha of 0.96. Symptoms of depression that are common globally (e.g., sadness, loss of energy, and sleep and appetite disturbances) (22), were consistently endorsed on the SADS. However, the most endorsed item in this sample and the item with the highest mean (I was thinking too much) and the item with the second highest mean was (I was carrying a lot of weight on my shoulders) are both local idioms of distress that were also highly endorsed in a previous study on the SADS. The local items could account for the lower false positive rate of the SADS. By using South African idioms of distress, the SADS is perhaps better able to distinguish between true cases and true non-cases than measures which were developed in high income countries. The least endorsed item (I cried) was also consistent with the previous study (17) and has been identified as a universal symptom of depression despite it not being included as a DSM-V criterion (22).

Twenty three percent of the sample met the criteria for MDD as determined by the SCID-5-RV. Although this rate of MDD in PWH is equal to or lower than rates reported in systematic reviews of studies of MDD in PWH across sub-Saharan Africa (23,24), it is high compared to previous studies conducted in the Cape Town area. Studies in the Cape Town area that have used structured psychiatric interviews typically report a prevalence rate of between 14 and 17 percent (4,25). The higher prevalence rate documented in the current study could be due to the high representation of women in the current sample. Women tend to have higher rates of MDD than men (26). It could also be due to the assessments being conducted by non-clinicians, although this is less likely given the extensive training and supervision provided by the study psychologists.

There are a few limitations in the current study. The sample consisted only of PWH who could communicate in English as the measures were administered in English. Although this limitation has potential implications with regards to generalizability, the demographic characteristics of the sample were consistent with studies conducted with PWH in Cape Town where the sample included non-English speakers. The majority of the sample were women, were first language isiXhosa and Afrikaans speakers, unemployed or part-time employed, with a reported combined income of less than R10,000 per month (25,27). The other potential limitation is that all participants were receiving antiretroviral therapy (ART). It is possible that rates of depression differ among PWH who are not receiving ART, which, with the new test and treat guidelines, would mainly consist of PWH who are either resistant to initiating ART or who have discontinued ART. It is highly probable that PWH experiencing severe depression would be among this group (5,7,28).

CONCLUSION

It has been well documented that PWH experience higher rates of depression than the general population, including in South Africa. To date the measures used to screen for depression in most LMICs, including South Africa, have been developed in high income countries and translated for use in the relevant languages. These measures consistently demonstrate high false positive rates. To our knowledge, the SADS is the first measure of depression for PWH in South Africa that was developed from an emic approach and comprises local idioms of distress. The current study supports the strength of the SADS in identifying true cases of MDD. This, along with its brevity and ease of administration, makes the SADS a more promising measure for inclusion in a standardized screening protocol for MDD among PWH with unsuppressed viral loads in community clinics in South Africa.