Validation of Screening Instruments for Common Mental Disorders and Suicide Risk in South African Primary Care Settings

Abstract

Introduction

In South Africa, there is limited mental health infrastructure and resources. Valid screening tools are needed to facilitate identification and linkage to care. We evaluated the performance of Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 (GAD-7), Primary Care Post Traumatic Stress Disorder Screen for DSM-5 (PC-PTSD-5), and the Columbia Suicide Severity Rating Scale (C-SSRS) among adults in South Africa against a diagnostic gold standard.

Methods

Adults present at healthcare facilities were screened with the PHQ-9, GAD-7, PC-PTSD-5, and the C-SSRS. Nurses used a structured diagnostic interview to identify depression, anxiety, panic disorder, PTSD and elevated suicide risk. We assessed the internal consistency, criterion validity, and the sensitivity and specificity of these tools.

Results

Of the 1,885 participants, the prevalence of common mental disorders and suicide risk was 24.4% and 14.9%, respectively. The PHQ-9, GAD-7, and PC-PTSD-5 showed good internal consistency (0.80–0.89). All screeners demonstrated good criterion validity. For depression, a cut-off of ≥5 on the PHQ-9 yielded sensitivity of 84.24%, while ≥10 yielded sensitivity of 48.77%. For anxiety, the GAD-7 performed similarly. A cut-off of ≥4 on the PC-PTSD yielded sensitivity of 61.96%. The C-SSRS yielded lower sensitivity than expected.

Limitations

The prevalence data is not generalizable to the larger South African adult population given the use of a targeted, healthcare facility-based sampling and recruitment strategy.

Conclusions

The performance of the PHQ-9, GAD-7, and PC-PTSD-5 demonstrated good internal consistency and criterion validity, though sensitivity and specificity trade-offs were enhanced with lower cut-offs. Further research into suicide risk screening is warranted.

Introduction

Mental health and substance use disorders are leading contributors to disability adjusted life years globally (Arias et al., 2022; GBD Mental Disorders Collaborators, 2022; Rehm and Shield, 2019; WHO, 2017). They are highly prevalent in low- and middle-income countries (LMICs), such as South Africa, where mental disorders and suicide risk are less often identified and treated than in high-income countries despite their contributions to a large proportion of the disease burden (WHO, 2017). Moreover, mental health disorders persist as a prevalent comorbidity in people living with HIV (PLHIV) and patients with tuberculosis (TB), which has particular relevance for countries located in sub-Saharan Africa (Bernard et al., 2017; Doherty et al., 2013). Over two-thirds (67%) of the 37 million PLHIV and almost one-third (29%) of the 9 million TB cases worldwide reside in countries located in sub-Saharan Africa (Rezaei et al., 2019; UNAIDS, 2021).

However, sub-Saharan Africa has limited health care infrastructure and human resource capacity to address the burden of mental health. Most countries in sub-Saharan Africa have less than 1 psychiatrist per 1 million people; only an estimated one in four people requiring mental health disorder receive any treatment (Seedat, 2009; WHO, 2021; Williams et al., 2008). In response to this gap, task shifting approaches that work to integrate mental health management into primary health care systems are progressively being adopted throughout the region (Petersen et al., 2014; Seedat, 2009; WHO, 2021). Developing the capacity of non-psychiatric specialists such as primary health clinicians and lay health workers to effectively screen and manage mental health disorders is an effective and cost-effective way to scale up mental health services, reduce the treatment disparity and address the high level of stigma traditionally associated with psychiatric disorders (Williams et al., 2008).

Brief screening tools have been developed for non-psychiatric specialists to administer and integrate into routine care. Most mental health screening tools were designed and tested in HICs, and many have not been locally validated in LMICs, such as South Africa. In a nationally representative survey, almost a third (30.3%) of individuals in South Africa were estimated to have experienced a common mental disorder (CMD) in their lifetime, with a 12-month prevalence estimate of any disorder of 16.5% – 26.2% of whom were classified as severe cases (Herman, 2009; Williams et al., 2008). Even so, true estimates on the prevalence of specific mental health disorders are unknown due to the challenges around ascertaining population level statistics in South Africa, where a lack of routine screening exists. It is estimated that one-in-five people living with HIV (PLWH) have a comorbid mental disorder (Myer et al., 2008; Zuma et al., 2022) and around one-in-three individuals with TB – with or without HIV coinfection - experience severe psychological distress (Janse Van Rensburg et al., 2020; Peltzer et al., 2012, 2013; Walt and Moyo, 2018).

Local validation of mental health tools in South Africa, with consideration for language and cultural setting, is necessary to ensure the accuracy of existing screening tools and to promote efficacious identification and diagnosis of mental health disorders (Juhasz et al., 2012). While mental health disorders are ubiquitous diseases, they have remarkable regional and cultural differences in both the prevalence and manifestation profile (Heim et al., 2017; Juhasz et al., 2012). These tools are often written in English or European languages; many African languages, including Zulu and Xhosa in South Africa do not have words equivalent to the bio-medical terms used in the original tools (Sweetland et al., 2014). It is crucial for tools to establish their diagnostic utility when translated, ensuring their accuracy remains consistent during the translation process.

While several mental health screening tools for common and severe mental disorders and suicide risk have been validated in South Africa, most of these brief screening tools have been validated in inadequately designed studies (Mughal et al., 2020). For example, validation studies were restricted to specific sub-populations, such as PLHIV, adult women in the antenatal period, and schizophrenic patients, suggesting a need for validation in general populations (Bhana et al., 2015a; Cholera et al., 2014b; van Heyningen et al., 2018). As such, there is a need to validate measures for a broad-spectrum of mental disorders in South Africa.

In order to address this research gap, we aimed to evaluate the performance of four mental health screening instruments among adults in Eastern Cape, South Africa against a diagnostic gold standard. We specifically investigate the sensitivity and specificity of these tools at standard cut-off points or definitions of a positive screen. Finally, we investigate their performance stratified by language of administration, sex, HIV status, lifetime TB history, and health seeking status.

Methods

This study was part of a larger study validating a battery of mental health screeners. (Stockton et al., 2024) This manuscript describes the performance of three screeners for common mental disorders and one screener for suicide risk.

Study Setting

Data were collected from four primary care clinics and one tertiary facility within the Buffalo City Metro (BCM) Health District Department of Health in the Eastern Cape, South Africa from February to May 2022.

Study Population

Adults 18 years or older able to communicate in isiXhosa or English who were either seeking care themselves or accompanying a patient at the study health facilities were eligible to participate.

Data Collection

A team of eight nurses and five research assistants (RAs) were trained on the mental health screening battery and Mini International Neuropsychiatric Interview (MINI). The team was trained to identify mental disorders, administer the screeners, and confirm psychiatric diagnoses using the MINI over the course of two weeks. All of the nurses had received psychiatric training during their nursing formation and had over 10 years of clinical experience. A local psychiatrist led the MINI training and provided ongoing supervision and support over the course of the study.

The RAs approached individuals entering the facility grounds, conducted an eligibility screening, informed them about the study, and invited them to participate. Interested participants met with RAs in gazebos on the facility grounds. RAs guided interested participants through the informed consent process. The RAs then screened those who provided written informed consent with the battery of mental health screeners, reading the questions aloud to participants and recording their responses. Following the screening, nurses – blinded to the results of the screening – administered the MINI in a separate gazebo. We used a targeting sampling strategy to ensure a gender-balanced sample of at least 50 individuals per disorder and at least 100 individuals without any disorder. Given the higher rate in which women attended our study clinics, RAs were instructed to target recruiting all available men to ensure gender-balanced sample. Further, as the parent study aimed to include cases of severe mental disorders as well, targeted recruitment from tertiary care facility that offered specialized psychiatric services was also employed.

MINI-diagnosed cases of mental health disorders were managed according to the facility policies. Namely, the study nurses referred cases to on-site psychiatric staff or, in the absence of on-site specialized psychiatric services, to clinic nurses to use the integrated chronic services manual (ICSM) in line with the facility Operational Managers’ purview.

Measures

All of the instruments were translated using a robust process of forward-and-backward translation and bilingual study team review. The study coordinator conducted the first translation in collaboration with two colleagues fluent in both isiXhosa and English. During staff training, the translation and meaning of the questions were further assessed by the study staff, and small grammatical and clarification corrections were made. Then, the mental health screeners were back translated from isiXhosa into English by individuals unfamiliar with the questionnaires. The study team and study psychiatrist reviewed the back-translations and no further changes were made.

Mental Disorder Diagnosis

Mental disorder diagnoses were made with modules from the MINI, a structured diagnostic interview that capture the Diagnostic and Statistical Manual of Mental Disorders-5 (DSM-5) major psychiatric disorders and has been widely-used as a reference standard across many contexts (Ali et al., 2016; Sheehan et al., 1998). We administered the following modules: major depressive episode; dysthymia; suicidality (low, moderate or high suicide risk); (hypo) manic episode; panic disorder; obsessive-compulsive disorder; PTSD; Alcohol abuse/dependence; non-alcohol psychoactive substance use disorders (with a revised the list of drugs to appropriately categorize and include local drugs); psychotic disorders and mood disorder with psychotic features; and generalized anxiety disorder. All disorders were diagnosed with the relevant MINI-V modules, with the exception of PTSD and suicide risk which were diagnosed with the MINI-Plus modules for simplicity and brevity. All diagnoses were for current disorder. For suicide risk, we dichotomized the MINI-diagnosed suicide risk into those with no risk compared to those with low-to-high risk and those with no-to-low risk vs those with moderate-to-high risk.

Mental Health Screeners

Participants were screened with the Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 (GAD-7), Primary Care Post Traumatic Stress Disorder Screen for DSM-5 (PC-PTSD-5) and Columbia Suicide Severity Rating Scale (C-SSRS). Administration of the PHQ-9 and GAD-7 allowed for assessment of the brief, pre-screeners (PHQ-2, GAD-2, and PHQ-4)(Kroenke et al., 2009) and the ninth question of the PHQ-9 (Na et al., 2018; Simon et al., 2013).

Common Mental Disorder Screeners

• Depression: The PHQ-9 is a nine-item questionnaire that assesses the presence and frequency of the nine DSM symptoms of major depression (American Psychiatric Association, 2000). The first two questions of the PHQ-9 are known as the PHQ-2, which ask about the two key symptoms of depression – depressed mood or anhedonia (Löwe et al., 2005). The PHQ-2 yields scores ranging from 0–6 and the PHQ-9 yields scores ranging from 0–27. For the PHQ-2, a score >0 is indicative of possible depression. For the PHQ-9 a total score of 5–9 and ≥10 are considered indicative of mild depression and moderate to severe depression, respectively (Kroenke et al.). The PHQ-9 may also be delivered using a two-step approach where only those who endorse the PHQ-2 (score >0) respond to the remaining 7 questions (Levis et al., 2020). Using the two-step approach, a PHQ-2 score of >0 combined with a total score of 5–9 and ≥10 are considered indicative of mild depression and moderate to severe depression, respectively.

• Anxiety: The GAD-7 is a seven-item questionnaire that assesses the presence and frequency of the DSM symptoms of generalized anxiety disorder (Spitzer et al., 2006). The first two questions of the GAD-7 are known as the GAD-2, which ask about the two key symptoms of anxiety – nervousness and uncontrollable worrying (Kroenke et al., 2007). The GAD-7 may also be delivered using a two-step approach where only those who endorse the GAD-2 (score >0) respond to the remaining 5 questions (Plummer et al., 2016). Using the two-step approach, a GAD-2 score of >0 combined with a total score of 5–9 and ≥10 are considered indicative of mild and moderate to severe anxiety, respectively.

• PTSD: The PC-PTSD-5 includes six questions; a screening question that asks about previous traumatic experiences followed by a five-item screener that captures the core DSM-5 symptoms of PTSD.(Prins et al., 2016) Scores of ≥ 4 are considered indicative of probable PTSD.

Suicide Risk Screeners:

• Columbia Suicide Severity Rating Scale (C-SSRS): The C-SSRS is a six-item screener, with one additional sub-item, that captures suicidal ideation and suicidal behavior (Posner et al., 2011). Participants are categorized as having no, low, moderate or high risk based on their affirmative answers to the questions (The Columbia Lighthouse Project, 2016). We dichotomized individuals into those with no risk vs low-to-high risk and those with no-to-low risk vs moderate-to-high risk based on their C-SSRS categorization.

• Question 9 of the PHQ-9: The 9^th question of the PHQ-9 captures suicidal ideation by asking “During the past two weeks, how often have you been bothered by thoughts that you would be better off dead or of hurting yourself in some way?” Endorsing this question (score >0) is considered indicative of reporting suicidal ideation.

Demographic and general health measures

We collected self-reported sociodemographic information (age, gender, marital status, living situation, education, religion, monthly household income, occupation, and ethnicity) and health history (noncommunicable and infectious diseases, pregnancy, and parity). Responses to all measures were recorded via tablet using the REDCap platform (Harris et al., 2009).

Statistical analysis

We excluded participants with incomplete responses to the MINI or mental health screening battery from the analysis.

To assess the internal consistency of scales, Cronbach’s alpha and McDonald’s omega were calculated for each scale with more than two items in the whole sample. Although Cronbach’s alpha and McDonald’s omega often lead to similar results in applied settings, McDonald’s omega is subject to fewer and more realistic assumptions and has less risk of over- or under-estimation (Dunn et al., 2014). Accordingly, both were calculated along with bootstrapped 95% confidence intervals to follow best practices for assessing internal consistency (Hayes and Coutts, 2020).

To assess criterion validity, we compared the total scores, representing the number and frequency of endorsed symptoms, with the reference standard MINI diagnoses using a receiver operating characteristic curve (DeLong et al., 1988) and calculated the area under the receiver operating characteristic (AUROC) (Pence et al., 2005). We specifically calculated and present the sensitivity and specificity for the standard cut-off point or definitions of a positive screen for each screener (as described in the measures section). All confidence intervals (CIs) were calculated with exact methods.

Finally, test characteristics (sensitivity, specificity and positive and negative likelihood ratios [LR+, LR−]) at each cut-off point for each of the screening tools against their respective diagnosis from the MINI were calculated (RH Fletcher, 1996). The cut-off score with the optimal sensitivity and specificity tradeoff was identified using the maximum value returned by Youden’s Index, a measure for optimizing sensitivity and specificity (Ruopp et al., 2008). These analyses are presented Annex 8 of the supplementary material.

Ethical Considerations

Participants provided written informed consent as approved by the New York State Psychiatric Institute Institutional Review Board (Protocol #8272), the Foundation for Professional Development Research Ethics Committee (8/2021) and the Eastern Cape Department of Health Research Committee (EC_202110_015).

Results

Participant Characteristics

A total of 1,885 participants were enrolled in our study and screened with the mental health screening battery and MINI (Table 1). Participants were aged 18–88 years (mean = 39; SD = 13.1), while 34.5% (n = 651) identified as male, 65.4% (n = 1232) as female, and 0.1% (n = 2) as trans or non-binary. Nearly all participants’ dominant language was isiXhosa (n = 1836; 97.4%), which was congruent with isiXhosa also being the most common language used for the screening (n = 1792; 95.1%). Only 10% (n = 188) of participants reported a tertiary education attainment and 58.6% (n = 1104) reported unemployment. Most participants presented to the clinic seeking care for themselves (n = 1364; 72.4%).

Table 1:

Participant Sociodemographic Characteristics

Characteristic	Mean (SD) or N (%)
Age (Range:18–88)	39 (13.1)
Sex
Male	651 (34.5)
Female	1232 (65.4)
In perinatal period	217 (7.3)
Trans/Non-binary	2 (0.1)
Marital Status
Single	1114 (59.1)
In a relationship/married	624 (33.1)
Separated, Divorce, Widowed	147 (7.8)
Dominant language
isiXhosa	1836 (97.4)
English	15 (0.8)
Other	34 (1.8)
Language of Screening Administration
English	92 (4.9)
isiXhosa	1792 (95.1)
Interchangeable	1 (0.1)
Race
White	17 (0.9)
Black	1834 (97.3)
Colored	31 (1.6)
Other	2 (0.2)
Education Attainment
Grade 7 (Primary) or less	219 (11.6)
Grade 8–11 (Before matric)	733 (38.9)
Grade 12 (Matric)	745 (39.5)
Tertiary	188 (10)
Residence
Informal Dwelling	551 (29.2)
Formal House	1329 (70.5)
Other	6 (0.3)
Monthly Household Income
< R1000	1061 (56.3)
> R1000	824 (43.7)
Employment
Unemployed	1104 (58.6)
Employed	532 (28.2)
Other, Unpaid Volunteer, Self-Employed, Student	249 (13.2)
Health Seeking
Seeking Care	1364 (72.4)
Accompanying	521 (27.6)

Participants also self-reported prior mental health, TB, HIV and other chronic condition diagnoses. Fourteen percent (n=268) reported a prior mental disorder diagnosis including depression, anxiety, PTSD, bipolar disorder, panic, suicidality, alcohol or substance abuse, and/or schizophrenia. Of those 268, 85.1% (n=228) reported being on treatment for the mental disorder. The lifetime TB prevalence was 15.1% (n=284) and prevalence of HIV was 25.8% (n=487). With respect to other chronic conditions, 30.6% (n=577) reported heart disease, diabetes, epilepsy, asthma, arthritis, ulcers, cancer, headaches, and/or being “disabled.”

Prevalence

The prevalence of any MINI-diagnosed current mental or substance use disorder was 38.7% (n = 730) (Table 2). The prevalence of these disorders stratified by self-reported HIV status, TB history, and healthcare seeking status are presented in Annex 1–2. Common mental disorders were prevalent in 24.4% (n = 459) of our population, with diagnoses of Major Depressive Episode (n = 406; 21.5%) and Generalized Anxiety Disorder (n = 121; 6.4%) being the most common. Women (n = 351; 28.5%) and participants living with HIV (n = 154; 31.6%) were more likely to be diagnosed with common mental disorders when compared to men (n = 107; 16.4%) and those without known HIV status (n = 249; 20.8%). There was a low prevalence of alcohol or substance use disorder (n = 206; 10.9%) in the study population. However, alcohol and substance use disorders was more prevalent in males (n = 98; 15.1%) compared to women (n=108; 8.8%), participants living with HIV (n = 77; 15.8%) compared to those without HIV (n,=107; 9.0%) and participants with a history of TB (n = 54, 19%) compared to those who had never had TB (n=152; 9.5%). Nearly 10% (n = 187) of participants had a severe mental disorder. Of note, the lifetime prevalence of psychotic disorder was 6.9% (n=131), while current prevalence of psychotic disorder was 5% (n=95). Finally, 14.9% (n=280) presented with elevated suicide risk.

Table 2:

Prevalence of Current MINI-diagnosed Mental and Substance Use Disorders, by Sex

n(%)	Total	Sex
	(N=1885)	Male (n=651)	Female (n=1232)
Common Mental Disorders *	459 (24.4)	107 (16.4)	351 (28.5)
Major Depressive Episode	406 (21.5)	93 (14.3)	312 (25.3)
Dysthymia	14 (0.7)	2 (0.3)	12 (1)
Generalized Anxiety Disorder	121 (6.4)	23 (3.5)	97 (7.9)
Panic Disorder	14 (0.7)	2 (0.3)	12 (1)
PTSD	163 (8.6)	30 (4.6)	132 (10.7)
Alcohol or Substance Use Disorder	206 (10.9)	98 (15.1)	108 (8.8)
Alcohol Use Disorder	180 (9.5)	75 (11.5)	105 (8.5)
Substance Use Disorder	31 (1.6)	26 (4)	5 (0.4)
Severe Mental Disorders	152 (8.1)	78 (12)	74 (6)
Psychotic Disorder	95 (5)	66 (10.1)	29 (2.4)
Hypomanic Episode	47 (2.5)	13 (2)	34 (2.8)
Manic Episode	20 (1.1)	7 (1.1)	13 (1.1)
Suicide Risk	279 (14.8)	60 (9.2)	219 (17.8)
Low Risk	165 (8.8)	40 (6.1)	125 (10.1)
Moderate Risk	33 (1.8)	7 (1.1)	26 (2.1)
High Risk	81 (4.3)	13 (2)	68 (5.5)
Obsessive Compulsive Disorder	13 (0.7)	0 (0)	13 (1.1)

^*Excludes Dysthymia

Psychometric Performance

Internal Consistency

The PHQ-9, PHQ-4, GAD-7, and PC-PTSD-5 all showed good internal consistency (0.80–0.89). (Table 3).

Table 3.

Internal consistency of each measure in overall sample.

	Cronbach’s α (95%CIs)	McDonald’s ω t (95%CIs)
PHQ-9
PHQ-9 Score (Range 0–27)	0.892 (0.882, 0.901)	0.894 (0.885, 0.903)
GAD-7
GAD-7 Score (Range 0–21)	0.865 (0.851, 0.879)	0.868 (0.852, 0.879)
PHQ-4
PHQ-4, Mean (Range 0–12)	0.811 (0.792, 0.830)	0.816 (0.796, 0.832)
PC-PTSD-5	0.827 (0.802, 0.847)	0.830 (0.807, 0.849)

Criterion Validity, Sensitivity, Specificity,

We next evaluated performance of each measure compared to corresponding MINI diagnoses (Table 5-7). Results are presented by common mental disorders, severe mental disorders, and suicide risk. Tables presenting results restricted to those who received the tools in isiXhosa and then stratified by sex, HIV status, TB history, and healthcare seeking status are available in Annex 3–7. The performance of the tools at every cut-off point is presented in the Annex 8.

Table 5:

Sensitivity and specificity at standard cut-off points for the suicide risk screening tools

Tool	Disorder (Prevalence)	Cut-Off or Positive Screen	AUC (95% CIs)	Sensitivity (95% CIs)	Specificity (95% CIs)
C-SSRS
	Low-High Suicide Risk (15%)	Low-High	n/a	55.56 (49.51–61.48)	90.85 (89.33–92.21)
	Moderate-High Suicide Risk (6%)	Moderate-High	n/a	54.39 (44.79–63.74)	96.16 (95.16–97.01)
PHQ-9
Q9	Low-High Suicide Risk (15%)	>0	0.76(0.73–0.79)	63.44 (57.49–69.10)	86.99 (85.24–88.59)
	Moderate-High Suicide Risk (6%)		0.83(0.79–0.87)	80.70 (72.25–87.49)	83.40 (81.58–85.10)

Common Mental Disorders:

All of the screeners for CMDs demonstrated at least good criterion validity (AUCs >0.7). (Table 4) The sensitivities and specificities of each tool for the varying disorders are presented in Table 4.

Table 4:

Sensitivity and specificity at standard cut-off points for each common mental disorder screening tool

Tool	Disorder (Prevalence)	Cut-Off Point	AUC (95% CIs)	Sensitivity (95% CIs)	Specificity (95% CIs)
PHQ-9
PHQ-2	Depression (21%)	> 0	0.80(0.78–0.82)	94.33 (91.62–96.38)	43.34 (40.8–45.91)
PHQ-9		≥ 5	0.83(0.81–0.86)	84.24 (80.32–87.64)	67.21 (64.75–69.6)
		≥ 10		48.77 (43.81–53.75)	90.33 (88.71–91.79)
PHQ-2/9 Two-Step		≥ 5	0.84(0.82–0.86)	83.50 (79.52–86.98)	69.78 (67.37–72.11)
		≥ 10		48.77 (43.81–53.75)	90.87 (89.29–92.29)
GAD-7
GAD-2	Anxiety (6%)	> 0	0.77(0.74–0.79)	86.78 (79.42–92.25)	45.18 (42.84–47.54)
GAD-7		≥ 5	0.78(0.74–0.81)	71.9 (63.01–79.69)	70.69 (68.51–72.81)
		≥ 10		24.79 (17.4–33.46)	93.14 (91.86–94.28)
GAD-2/7 Two-Step		≥ 5	0.74(0.70–0.79)	67.77 (58.67–75.98)	71.83 (69.66–73.92)
		≥ 10		23.97 (16.68–32.57)	93.42 (92.17–94.54)
GAD-2	Anxiety or Panic (7%)	> 0	0.72(0.68–0.76)	87.6 (80.64–92.74)	45.39 (43.04–47.75)
GAD-7		≥ 5	0.79(0.75–0.82)	73.64 (65.16–81.01)	71.01 (68.83–73.13)
		≥ 10		25.58 (18.31–34.01)	93.28 (92.01–94.41)
GAD-2/7 Two-Step		≥ 5	0.76(0.71–0.80)	69.77 (61.06–77.54)	72.15 (69.99–74.24)
		≥ 10		24.81 (17.63–33.18)	93.56 (92.31–94.67)
PHQ-4
PHQ-4	Depression (21%)	> 0	0.82(0.80–0.84)	96.31 (93.98–97.92)	37.32 (34.85–39.84)
	Anxiety (6%)		0.76(0.73–0.80)	95.04 (89.52–98.16)	31.8 (29.63–34.03)
	Anxiety or Panic (7%)		0.77(0.73–0.81)	95.35 (90.15–98.27)	31.95 (29.77–34.19)
	Anxiety or Depression (23%)		0.81(0.79–0.83)	95.80 (93.45–97.49)	37.71 (35.21–40.25)
	PTSD (9%)		0.77(0.74–0.80)	95.42 (93.09–97.15)	38.29 (35.76–40.87)
	Any CMD (24%)		0.80(0.78–0.83)	96.31 (93.98–97.92)	37.32 (34.85–39.84)
	PC-PTSD-5
Screener	PTSD (8%)	Trauma +	n/a	77.3 (70.10–83.49)	62.2 (59.86–64.49)
PC-PTSD-5		≥ 4	0.76(0.72–0.8)	61.96 (54.04–69.44)	81.88 (79.98–83.67)

Any CMD=Depression, Anxiety, Panic or PTSD

With respect to depression: a cut-off of >0 on the PHQ-2 yielded a sensitivity and specificity of 94.33 (95% CIs: 91.62–96.38) and 43.34 (95% CIs: 40.8–45.91), respectively. The standard cut-off of ≥ 5 for mild-to-severe depression on the PHQ-9 yielded a sensitivity and specificity of 84.24 (95% CIs: 80.32–87.64) and 67.21 (95% CIs: 64.75–69.6), respectively; while the standard cut-off of ≥10 for moderate-to-severe depression yielded lower sensitivity 48.77 (95% CIs: 43.81–53.75), and higher specificity 90.33 (95% CIs: 88.71–91.79). Using the two-step method, where only those who endorse the PHQ-2 would respond to the remaining seven questions of the PHQ-9, led to similar performance at both cut-offs. A cut-off of ≥7 yielded optimal sensitivity-specificity tradeoff as assessed by the Youden’s Index.

For anxiety and anxiety or panic disorder, a cut-off of >0 on the GAD-2 yielded sensitivities >80 and specificities <50. The standard cut-off for mild-to-severe anxiety of ≥5 on the GAD-7 demonstrated sensitivities and specificities >70. However, the standard cut-off for moderate-to-severe anxiety of ≥10 yielded lower sensitivities (<30), and higher specificities (>90). Using the two-step method, where only those who endorse the GAD-2 would respond to the remaining seven questions of the GAD-7, led to similar performance at both cut-off points. A cut-off score of ≥ 4 yielded optimal sensitivity-specificity tradeoff as assessed by the Youden’s Index for both anxiety and anxiety or panic.

For PTSD, the first screening question for the PC-PTSD-5, which asks participants whether they have ever experienced any trauma and ultimately triggers the administration of the PC-PTSD-5, yielded a sensitivity and specificity of 77.3 (86.99 (85.24–88.59)70.10–83.49) and 62.2 (95% CIs: 59.86–64.49), respectively. Using the standard cut-off of ≥ 4, the PC-PTSD-5 yielded a sensitivity and specificity of 61.96 (95% CIs: 54.04–69.44) and 81.88 (95% CIs: 79.98–83.67), respectively. A cut-off score of ≥3 yielded optimal sensitivity-specificity tradeoff assessed by the Youden’s Index.

The PHQ-4 yielded sensitivities 95.0–96.3 and specificities 31.8–38.3 for all the CMDs.

Suicide Risk:

The C-SSRS yielded sensitivities of 54.4–55.6 and specificities of 90.8–96.2 for identifying low-to-high suicide risk or moderate-to-high suicide risk using the standard categorization. (Table 5)

The ninth question of the PHQ-9 (which captures suicidal ideation) demonstrated at least good criterion validity (AUC >.7) for both low-to-high suicide risk and moderate-to-high suicide risk. Endorsing question 9 at any level (score >0) yielded a sensitivity of 63.44 (95% CIs: 57.49–69.1) and specificity of 86.99 (95% CIs: 85.24–88.59) for low-to-high suicide risk. For moderate-to-high risk, however, endorsing question 9 at any level yielded higher sensitivity 80.70 (95% CIs: 72.25–87.49) and similar specificity 83.4 (95% CIs: 81.58–85.10)

Discussion

A recent systematic review and meta-analysis of 100 studies assessing the PHQ-9 against a reference standard found a pooled sensitivity and specificity of .85 and .85, respectively, with an optimal cut off of 10 (Negeri et al., 2021). Most studies came from high income countries (87%) and only 13 came from LMICs. Another systematic review of 49 studies from 26 LMICs (15 from 10 African countries) found a somewhat lower, but acceptable sensitivity and specificity of 76% and 76%, respectively (Carroll et al., 2020); however, cut-offs that were used varied across studies. Of the eight African studies that assessed sensitivity and specificity against a reference standard, six found cut-offs of 10–11 to be optimal (Cameroon, South Africa, Uganda, Ethiopia, Zimbabwe, Nigeria)(Adewuya et al., 2006; Akena et al., 2013; Chibanda et al., 2016; Cholera et al., 2014a; Gelaye et al., 2013a; Pence et al., 2012). Three studies from South Africa, Uganda and Ethiopia found conflicting results with suboptimal performance at cut-offs between 4–5 (Hanlon et al., 2015; Nakku et al., 2016) or low sensitivity at cut-off of 9 (Bhana et al., 2015b). Our results are consistent with the last study which found the higher cut off of 10 yielded a comparably low sensitivity (52%) but high specificity (92%) (Bhana et al., 2015b). Both prior studies from South Africa were also conducted in primary care settings, but in geographically, linguistically, and ethnically different parts of the country. Our sample of 1885 was also considerably larger than either of the previous studies (n=397 and n=676). Taken together, our findings and the larger literature suggest the PHQ-9 does vary in performance across, and sometimes even within, countries.(Sweetland et al., 2014) Further, in non-western settings such as the sub-Saharan region, it is possible that the lower cut-off scores may optimize the trade-off between sensitivity and specificity; further research should investigate variability in recommended cut-offs with attention to the presence of somatic symptoms as well as symptom under-reporting.

Similar disparities emerged between our findings and others assessing the validity of the GAD-7 in other countries. A systematic review and meta-analysis of 12 studies, only 2 of which were from LMICs and none from sub-Saharan Africa, found an optimal pooled sensitivity and specificity using a cut off of 8 (83% and 84%, respectively) (Plummer et al., 2016). In our sample, using such a cut off yielded an extremely low sensitivity (35%) and high specificity (90%), whereas with a much lower cut-off of 4, we were able to obtain a sensitivity and specificity of 82% and 63%, respectively. To our knowledge, only three validation studies of the GAD-7 have been conducted in Africa. A study from Zimbabwe found a sensitivity and specificity of 89% and 73%, respectively, using a cut off of 10 (Chibanda et al., 2016). Another study from Mozambique among adolescents found a sensitivity and specificity of 81% and 72%, using a cut-off 5.(Lovero et al., 2022)

Using a higher or lower cut-off scores or the two-step administration process with the PHQ-9 and the GAD-7 has important trade-offs that need to be considered in real-world settings, particularly ones with limited mental health resources. For both instruments, using a lower cut off would ensure the detection of more people with moderate depression and anxiety, but a higher proportion of false positives could lead to a misallocation of scarce resources. Using a higher cut off would risk missing more people with moderate depression requiring treatment, but a majority individuals with more severe depression would likely be identified and prioritized for treatment. From an implementation stand-point, our results suggest the two-step method of administering the PHQ-9 could provide practical utility, shortening the number of questions any given patient screened would need to answer. This is consistent with other findings suggesting that using an endorsement of the PHQ-2 is an effective first step in a two-step process for identifying depression (Levis et al., 2020). In this strategy, endorsing the PHQ-2 provides high sensitivity, but also can be used to filter out non-depressed individuals who have not endorsed either of the two key symptoms of depression, before administering the complete PHQ-9 (Basaraba et al., 2023). While this has also been a suggested approach for administering the GAD-7 and identifying those with anxiety disorder (Plummer et al., 2016), our results again suggest careful consideration must be given to the reduction in sensitivity. Ultimately, the one-step approach captures the full breadth of symptoms for these two disorders, but necessitates answering a larger number of questions and may be more useful in research activities than for routine clinical practice. With respect to real-world integration of screening into routine primary or community care, the two-step approach may be a timesaving, resource-efficient means to facilitate linkage to mental health care.

While the PHQ-9 has been validated in South Africa for diagnosing depression in primary care settings (Bhana et al., 2015a; Cholera et al., 2014b), the ninth question regarding suicidal ideation has not been independently assessed nor validated as a suicide risk screener in South Africa or the sub-Saharan region. However, one study in Zimbabwe did identify difficulties in endorsing the suicidal ideation question (Chibanda et al., 2016; Gelaye et al., 2013b). To the best of our knowledge, only a few studies in LMICs have validated or adapted the C-SSRS(Chahine et al., 2020; Kilincaslan et al., 2019; Zakhour et al., 2021), and within sub-Saharan Africa, the C-SSRS has only been evaluated in Mozambique.(Lovero et al., Under Review) Sub-Saharan Africa experiences the highest rate of suicides in the world and South Africa in particular has the 4^th highest age-standardized suicide rate in the region of 23.2 per 100,000 population (WHO, 2022). As such, there exists the need for enhanced detection of individuals at risk through culturally-validated screening tools in the region (Ali et al., 2016). As suicidal ideation and attempts are strong predictors of suicide deaths, early identification of individuals at risk through validated screening tools and resulting linkage to care is a key strategy to improve the wellbeing of those experiencing suicidal ideation or behavior and ultimately preventing suicides (Bachmann, 2018; Runeson et al., 2010; WHO, 2014).

Our results suggest the ninth question of the PHQ-9 would function well as a first-step, single-item screener for identifying individuals at any risk of suicide, but further assessment would be necessary to effectively triage and manage suicide risk. For example, in Malawi a suicide risk assessment protocol is used to triage suicide risk and facilitate appropriate clinical management following an endorsement of PHQ-9’s suicidality question.(Landrum et al., 2023) While the C-SSRS yielded sub-optimal sensitivity, it is worth noting that the both the C-SSRS and MINI-plus suicidality module categorize risk based on participant responses to six questions that probe on past-month suicidal ideation and behavior and lifetime suicide attempt. While the C-SSRS and MINI-plus questions are similar, the questions vary slightly in their wording and their weight in determining suicide risk. As such, we recommend cognitive interviewing to assess how the questions in the C-SSRS and the MINI-plus suicidality module are understood and further investigation into how questions from both the C-SSRS and MINI-plus be used to identify those at elevated risk of suicide.

Limitations

We used a targeted sampling strategy and the prevalence data reported in this manuscript should not be interpreted as generalizable or representative of the larger South African adult population. As our prevalence data are not representative, we did not calculate the positive predictive values or negative predictive values. While the study sample could be considered a “sick” population (e.g. healthcare seeking), the analysis stratified by healthcare seeking status suggests the tools will perform similarly in the general population. The sample of individuals who were screened in English was too small to separately validate the screeners in English. Inter-rater reliability between the nurses was not assessed. The participants received the screeners prior to the MINI, which may have biased the MINI diagnostic interviews. The use of MINI as the gold standard may not be appropriate because it was developed and validated in Western settings and not evaluated specifically in South Africa. However, the MINI was designed to directly align with DSM-5 diagnoses and is one of the most widely used referent gold-standards across cultures and context. (Ali et al., 2016; Sheehan et al., 1998) Further, the nurses who administered the MINI were empowered to exercise their clinical judgment in interpreting the participants’ responses and making final diagnoses.

Conclusion

The PHQ9 and GAD7 and PC-PTSD all demonstrated good internal consistency and criterion validity. However, optimal trade-offs of sensitivity and specificity was achieved with lower than standardized cut-offs for all tools. Recognizing the inherent tradeoff in between missed cases and over identification, the choice of cut-off should be guided by the purpose of the screening and severity of the condition. Further research into the performance of screening tools for suicide risk is warranted.

Highlights.

• The performance of PHQ-9, GAD-7, PC-PTSD-5 was acceptable among adults in South Africa

• Sensitivity and specificity trade-offs were enhanced using lower cut-offs

• The C-SSRS yielded lower sensitivity than expected