Skip to main content
PMC home page
Cambridge Prisms: Global Mental Health logoLink to Cambridge Prisms: Global Mental Health
. 2023 Dec 21;11:e4. doi: 10.1017/gmh.2023.89

Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa

Melissa Ann Stockton 1,✉, Ernesha Webb Mazinyo 2,14, Lungelwa Mlanjeni 2, Kwanda Nogemane 3, Nondumiso Ngcelwane 3, Annika C Sweetland 4,5, Cale Neil Basaraba 6,7, Charl Bezuidenhout 8, Griffin Sansbury 9, Kathryn L Lovero 10, David Olivier 11, Christoffel Grobler 12, Melanie M Wall 4,5, Andrew Medina-Marino 11,13, Phumza Nobatyi 3, Milton L Wainberg 4,5
PMCID: PMC10808975  PMID: 38283876

Abstract

In low-resource settings, valid mental health screening tools for non-specialists can be used to identify patients with psychiatric disorders in need of critical mental health care. The Mental Wellness Tool-13 (mwTool-13) is a 13-item screener for identifying adults at risk for common mental disorders (CMDs) alcohol-use disorders (AUDs), substance-use disorders (SUD), severe mental disorders (SMDs), and suicide risk (SR). The mwTool-13 is administered in two steps, specifically, only those who endorse any of the initial three questions receive the remaining ten questions. We evaluated the performance of mwTool-13 in South Africa against a diagnostic gold standard. We recruited a targeted, gender-balanced sample of adults, aged ≥18 years at primary and tertiary healthcare facilities in Eastern Cape Province. Of the 1885 participants, the prevalence of CMD, AUD, SMD, SR, and SUD was 24.4%, 9.5%, 8.1%, 6.0%, and 1.6%, respectively. The mwTool-13 yielded high sensitivities for CMD, SMD, and SR, but sub-optimal sensitivities for AUD and SUD (56.7% and 64.5%, respectively). Including a single AUD question in the initial question set improved the tool’s performance in identifying AUD and SUD (sensitivity > 70%), while maintaining brevity, face-validity, and simplicity in the South African setting.

Keywords: developing countries, assessment tools, psychometric evaluations, primary care, cross-cultural

Impact statement

Valid, translated mental health screening tools for non-specialists are necessary for identifying patients with psychiatric disorders in need of critical mental health care. The Mental Wellness Tool-13 (mwTool-13) is a 13-item screener for identifying adults at risk for common and severe mental disorders, alcohol-use and substance-use disorders, and suicide risk. This study validated and improved the mwTool-13 against diagnostic gold standard. The modified SA-mwTool-12 yielded high sensitivities, maintaining brevity, face-validity, and simplicity in the South African setting. Findings from this study support the continued expansion of mental health screening in South Africa at the primary- and community-care level and may inform other validation efforts.

Introduction

Mental disorders cause substantial disease burden worldwide (Whiteford et al., 2016; GBD 2019 and Mental Disorders Collaborators, 2022; WHO, 2020). This disease burden is disproportionately borne on low- and middle-income countries (LMICs) which lack the psychiatric infrastructure, workforce, and policy to support the high demand for mental health treatment (Alloh et al., 2018; WHO, 2020). Whereby mental disorders increase the risk for other health conditions, such as HIV and tuberculosis (TB), and vice versa, there exists the need for integration of services into non-psychiatric settings to combat exacerbated poor health outcomes with comorbid conditions (Prince et al., 2007; Collins et al., 2013; Oh et al., 2017; Sweetland et al., 2018; Hayward et al., 2022). In LMICs, task-sharing mental health care to primary care providers can improve the accessibility of psychiatric services (WONCA, 2008; Javadi et al., 2017; Lovero et al., 2019). Such task-sharing efforts necessitate short mental health screening tools for non-specialists that can identify a broad-spectrum of mental and substance-use disorders (SUDs) and facilitate effective linkage to critical mental health care (Vythilingum et al., 2013; Ali et al., 2016).

In South Africa, lifetime prevalence of any mental disorder in South Africa in 2002 was estimated to be 30.3%; categorized by type of disorder, the lifetime prevalence of anxiety disorders is 15.8%, mood disorders is 9.8%, and alcohol-use disorders (AUDs) or SUDs is 13.4% (Stein et al., 2008). South Africa bears a heavy and unsustainable burden of both TB and HIV individually, and nearly 60% of individuals with TB are also living with HIV (South African National Department of Health, 2021). It is further estimated that one-in-five people living with HIV (PLWH) have a comorbid mental disorder (Myer et al., 2008; Zuma et al., 2022) and approximately one-in-three individuals with TB – with or without HIV coinfection - experience severe psychological distress (Peltzer et al., 2012, 2013; Walt and Moyo, 2018; Janse Van Rensburg et al., 2020). The high rates of mental disorders and infectious diseases, particularly HIV and TB, highlight the need for task-sharing mental health services and validated screening tools.

Unmet need for mental health treatment across the spectrum of mental disorders is high; only one-quarter of South Africans with a mental disorder receive treatment within a given year (Seedat et al., 2008). Lack of specialized providers, inequity in the allocation of both tangible and human resources between provinces, underdeveloped community-based services, and low mental health literacy contribute to the sub-optimal delivery of psychiatric care (Lund et al., 2010; Petersen and Lund, 2011). South Africa has identified mental health task-sharing as a promising strategy to increase access to mental health services while reducing stigma and mental health disparities (Mendenhall et al., 2014).

Brief comprehensive screening for mental disorders is critical to any task-sharing strategy because it enables less-trained providers to identify the presence and severity of mental disorders and make referrals for further clinical evaluation and/or mental health services (Murray et al., 2014). Unfortunately, most screening tools are specific to single disorders, thus requiring multiple screening tools to assess for more than one condition. Such an approach is not optimal, nor is it feasible in under-resourced settings. Moreover, clinical presentations of mental disorders vary in sub-Saharan Africa in comparison to Western settings due to differences in idiomatic descriptions of distress and emotions, and the somatization of psychiatric symptoms (Sweetland et al., 2014; Ali et al., 2016). In South Africa, as in other non-Western countries, there is need for existing screening tools to be linguistically and culturally validated to ensure appropriateness in their patient populations. Additionally, validation in the South Africa epidemiologic setting will allow for reliable identification of psychiatric conditions within a high-burden infectious disease context.

To facilitate broad-spectrum mental and substance-use screening at the primary- and community-care levels, the Mental Wellness Tool-12 (mwTool-12) was recently developed in Mozambique to identify symptoms of common mental disorders (CMDs – depression, anxiety, and post-traumatic stress disorder), severe mental disorders (SMDs – psychosis and mania), AUD, and suicide risk (SR) (Lovero et al., 2021). The mwTool-12 originally included screening questions for SUD, however, the low prevalence of SUD in the Mozambican sample prevented their validation (Lovero et al., 2021). While the mwTool-12 offers substantial utility as a mental and SUD screener in other low-resource settings, the mwTool-12 has yet to be assessed outside of Mozambique. Embedded within a larger study validating a battery of mental health screeners, we sought to evaluate the performance of the mwTool-12 augmented with an additional SUD item – henceforth the mwTool-13 – in Eastern Cape Province, South Africa, against the Mini International Neuropsychiatric Interview (MINI) diagnostic gold standard. For those questions that yielded sub-optimal sensitivity, we sought to improve the mwTool-13’s performance while prioritizing high sensitivity, brevity, and face-validity.

Methods

Study setting

Data were collected from four primary care clinics within the Buffalo City Metro (BCM) Health District Department of Health in Eastern Cape Province, South Africa from February to April 2022. At these facilities, nurses provide primary care, emergency, and outpatient mental health services, for disorders such as depression, anxiety and posttraumatic stress disorder (PTSD). Specialist Care for more severe mental health conditions is typically rendered in a hospital setting following a referral from a primary or community health facility. In order to capture sufficient numbers of individuals with SMD, additional data were collected from one tertiary care facility in BCM District in May 2022. Eastern Cape Province has a particularly high HIV prevalence (25.2%; 95% CI: 19.8%-31.5%), high TB incidence (1236 per 100,000 persons; 95% CI: 945-1526), and poor HIV, TB, mental health, maternal-child health, and health service delivery indicators (Massyn et al., 2017; National Department of Health, 2018; Massyn et al., 2019; Microbiologically Confirmed Pulmonary TB – Centre for Tuberculosis, 2019; Simbayi et al., 2019; Massyn et al., 2020)

Study population

Adults (patients and their accompaniers age ≥18 years) at study health facilities were eligible to participate. Individuals were excluded if they were unable to sufficiently communicate in isiXhosa or English.

Measures

All instruments (the mwTool-13 and MINI) were translated into isiXhosa through a robust process of forward and backward translation, and thorough review by the study investigators, research staff, and local psychiatrist to ensure the face-validity of the instruments.

Mental disorder diagnosis and classification

Current mental disorders were diagnosed with the MINI, a structured diagnostic interview that has been widely used as a reference standard across many contexts (Sheehan et al., 1998). With the exception of PTSD and SR, which were diagnosed with the MINI-Plus modules for simplicity and brevity, all other disorders were diagnosed with the relevant MINI-V modules. Details on the small modifications made to the MINI modules can be found in Supplementary Annex 1. We classified participants into the following five categories based on responses to relevant MINI modules:

  • SMD: manic episode (mania), hypomanic episode (hypomania), psychotic disorder (psychosis)

  • CMD: major depressive episode (depression), PTSD, general anxiety disorder (anxiety)

  • AUD: alcohol abuse or dependence

  • SUD: substance abuse or dependence

  • SR: moderate-to-high SR

Mental wellness tool-13

The mwTool-13 includes the original mwTool-12 items augmented with one additional SUD item (for a total of 13 items; Lovero et al., 2021; Smith et al., 2010). The mwTool-13 is meant to be administered in two steps. Step 1: patients respond to an initial three questions (Q1-3) to identify those who have any disorder. Step 2: those who endorse any of the initial three items in step one then respond to the remaining 10 questions and are classified into CMD (positive response to Q1 and/or Q3), AUD (positive response to Q4 and/or Q5), SUD (positive response to Q6), SMD (positive response to any of Q7-10), and SR (positive response to any of Q11-13) groups. The mwTool-12 was developed using a data-driven item-selection method; these 12 items were identified from a battery of 99 items across nine commonly used mental disorder and functioning assessments (Bebbington and Nayani, 1995; American Psychiatric Association, 2000; Spitzer et al., 2006; Posner et al., 2011; Prins et al., 2016) using a variable selection technique, the least absolute shrinkage and selection operator (LASSO) (Tibshirani, 1996). The data-driven selected items were then reviewed by clinical experts, who confirmed the first three items both captured symptoms of CMDs as well as reflected comorbid symptoms with the other disorders. We augmented the original mwTool-12 by adding an additional SUD item (Smith et al., 2010). See Supplementary Annex 2 for MwTool-13 questions and response options, and Table 1 for definitions of a positive screen for each disorder. To ensure proper evaluation of the mwTool-13 performance, participants responded to all 13 questions regardless of their responses to the step one question set.

Table 1.

MwTool-13 questions definitions of a positive screen for each disorder category

Numbering, questions and administration instructions Definition of a positive screen Disorder
Step One Questions 1. In the last 2 weeks, how often have you been feeling down, depressed, or hopeless? ≥ “Several days” to any of the three questions CMD
2. In the last 2 weeks, how often have you been feeling nervous, anxious, or on edge? Does not inform a specific disorder a
3. In the last 2 weeks, how often have you been so restless that it’s hard to sit still? CMD
POSITIVE to questions 1 or 2 or 3, CONTINUE SCREENING.
If NEGATIVE for all three, STOP.
If POSITIVE to question 2, but NEGATIVE to questions 5-11, refer to self-help.
Step Two Questions 4. In the past year, how often do you have a drink containing alcohol? Anyone (regardless of gender): ≥ “Between 2 and 4 times a month” on Q4
Women: “Monthly or less” on Q4 and ≥ “3 or 4” on Q5
Men: “Monthly or less” on Q4 and ≥ “5 or 6” on Q5		 AUD
5. In the past year, how many drinks containing alcohol do you have on a typical day when you are drinking?
6. In the past year, how many times have you used a recreational or illegal drug or used a prescription medication for non-medical reasons? ≥ “Once or twice” SUD
5. In the past year, have you ever felt that your thoughts were being directly interfered with or controlled by some outside force or person in a way that many people would find hard to believe (for instance, through telepathy)? “Yes” to any of the four questions SMD
6. In the past year, have there been times when you felt that a group of people was plotting to cause you serious harm or injury?
7. In the past year, have there been times when you felt that something so strange was going on that other people would find it very hard to believe?
8. In the past year, did you at any time hear voices saying quite a few words or sentences when there was no one around that might account for it?
9. In the past month, have you wished you were dead or wished you could go to sleep and not wake up? “Yes” to any of the three questions SR
10. In the past month, have you had any actual thoughts of killing yourself?
11. In the past 3 months, have you ever done anything, started to do anything, or prepared to do anything to end your life?
Open in a new tab

Abbreviations: AUD, alcohol-use disorder; CMD, common mental disorder; SMD, severe mental disorder; SR, suicide risk; SUD, substance use disorder.

a

While Q1-3 direct continuation to the step two questions for identifying AUD, SUD, and SMD, positive responses to Q1 and/or Q3 are considered indicative of CMD. Of note, endorsing only Q2 and none of the other questions is not indicative of a specific disorder.

Demographic and general health measures

We collected self-reported sociodemographic information (age, gender, marital status, living situation, education, religion, monthly household income, occupation, and ethnicity), physical health history (non-communicable diseases, pregnancy, and parity), and mental health history (prior mental health diagnosis, prior access to mental healthcare).

Data collection & procedures

Over a 2-week period, a team of five research assistants (RAs) were trained to administer the mwTool-13, and eight nurses were trained to administer and confirm psychiatric diagnoses using the MINI. The RAs were all experienced community health workers. The nurses had all received psychiatric training during their nursing formation and had over 10 years of clinical experience. Study staff piloted data collection tools over a 2-week prior to study start. During the pilot period, study staff met with a local psychiatrist to debrief and discuss any concerns or issues with the diagnostic interview. During the review process, challenges were identified in diagnosing SMD, particularly given the overlap between symptoms of psychosis and accepted cultural norms. The nurses were empowered to use their clinical acumen to differentiate between abnormal and culturally normative beliefs.

To reduce recruitment bias and ensure a representative sample of adults at the health care facilities, RAs were instructed to approach every seventh person entering the collaborating health facility. Potential participants were screened for their age and informed about the study. Interested individuals were read the informed consent form in an erected gazebo which provided privacy. The RAs then administered the sociodemographic questionnaire and mwTool-13 to consenting participants. The nurses – blinded to the results of the screening – then administered the MINI in a separate gazebo. Bilingual staff conducted all study activities in a participants preferred language (i.e., English or IsiXhosa). Responses to all measures were recorded using REDCap on a tablet computer (Harris et al., 2009).

Individuals with MINI-diagnosed, psychiatric disorders were managed according to facility policies and South African national guidelines (Petersen et al., 2016). This included referral to existing psychiatric staff; in the absence of specialized psychiatric services in line with the facility Operational Managers purview, the nurses used the South African Adult Primary Care (APC), which is mean to guide clinical decisions, and the Integrated Chronic Disease Management (ICDM) manuals to link individuals to the necessary services (Department of Health, 2014, 2019).

Our target sample included 50 gender-balanced individuals per disorder (depression, anxiety, PTSD, AUD, SUD, Psychosis, Mania (hyper or hypo), SR) and at least 100 individuals without any disorder in order to obtain precise confidence intervals. Given the higher rate in which women attended our study clinics, RAs were instructed to target all available men to ensure gender-balanced sampling,

Statistical analysis

Participants with incomplete responses to the MINI or mwTool-13 were excluded from analysis. We used descriptive statistics to describe the study population and present the prevalence of confirmed MINI-diagnoses. Sensitivities, specificities and 95% confidence intervals (CIs) were calculated for each of the disorder categories using the original two-step administration and the original definition of a positive-case (see Table 1 for Original Definitions). For disorder categories in which the two-step approach and original definition did not yield adequately high sensitivities (>70%), we explored changing the definition of a positive screen and including the questions in the step one questions set. We performed additional analyses restricted to those who received the mwTool-13 in isiXhosa and then stratified analyses with respect to gender, HIV status, and lifetime TB history.

Ethical considerations

The study was approved by the New York State Psychiatric Institute Institutional Review Board (Protocol #8272), the Foundation for Professional Development Research Ethics Committee (8/2021) and the Eastern Cape Department of Health Research Committee (EC_202110_015).

Results

Participant characteristics

The 1885 participants’ socio-demographic characteristics are presented in Table 2. The average age of participants was 39 years, 65% of participants were female, nearly all participants identified as black (97.3%), and 97.4% of participants reported isiXhosa as their dominant language. Nearly all (95.1%) participants received the mwTool-13 in isiXhosa. Of note, 72.4% were seeking health services for themselves, while 27.6% were accompanying someone seeking care.

Table 2.

Participant characteristics (N = 1885), by gender

Mean (SD) or n (%) Total (N = 1885) Men (n = 651) Women (n = 1232)
Age (range: 18–88) 39 (13.1) 39.4 (12.4) 38.9 (13.5)
Perinatal
   Currently pregnant – – 90 (7.3)
   Gave birth in past 12–months – – 127 (10.3)
Marital status
   Single 1114 (59.1) 409 (62.8) 703 (57.1)
   In a relationship/married 624 (33.1) 111 (17.1) 209 (17)
   Separated, divorce, widowed 147 (7.8) 43 (6.6) 104 (8.4)
Dominant language
   isiXhosa 1836 (97.4) 632 (97.1) 1202 (97.6)
   Afrikaans 30 (1.6) 11 (1.7) 19 (1.5)
   English 15 (0.8) 7 (1.1) 8 (0.6)
   Other 4 (0.2) 1 (0.2) 3 (0.2)
Language of mwTool-13 Administration
   English 92 (4.9) 26 (4) 66 (5.4)
   isiXhosa 1792 (95.1) 625 (96) 1165 (94.6)
   Interchangeable 1 (0.1) 0 (0) 1 (0.1)
Race
   White 17 (0.9) 7 (1.1) 10 (0.8)
   Black 1834 (97.3) 634 (97.4) 1198 (97.2)
   Indian 1 (0.1) 0 (0) 1 (0.1)
   Colored 31 (1.6) 10 (1.5) 21 (1.7)
   Other 2 (0.1) 0 (0) 2 (0.2)
Education
   Grade 7 (primary) or less 219 (11.6) 73 (11.2) 146 (11.9)
   Grade 8-11 (before matric) 733 (38.9) 264 (40.6) 468 (38)
   Grade 12 (matric) 745 (39.5) 256 (39.3) 489 (39.7)
   Tertiary 188 (10) 58 (8.9) 129 (10.5)
Residence
   Informal dwelling 551 (29.2) 171 (26.3) 380 (30.8)
   Formal house 1329 (70.5) 478 (73.4) 849 (68.9)
   Other 5 (0.3) 2 (0.3) 3 (0.2)
Income
   < R1000 (~57 USD) 1061 (56.3) 347 (53.3) 714 (58)
   R1000-R5000 (~57-284 USD) 656 (34.8) 227 (34.9) 429 (34.8)
   >5000 (~284 USD) 168 (8.9) 77 (11.8) 89 (7.2)
Employment
   Unemployed 1061 (56.3) 347 (53.3) 714 (58)
   Employed 656 (34.8) 227 (34.9) 429 (34.8)
   Student 168 (8.9) 77 (11.8) 89 (7.2)
   Unpaid volunteer, self-employed, Other 1061 (56.3) 347 (53.3) 714 (58)
Health seeking
   Seeking care 1364 (72.4) 460 (70.7) 902 (73.2)
   Accompanying 521 (27.6) 191 (29.3) 330 (26.8)
Self-reported Health Status
   Non-communicable disease (Any) 577 (30.6) 176 (27) 401 (32.5)
   Mental disorder (Any) 268 (14.2) 135 (20.7) 133 (10.8)
   Ever had TB 284 (15.1) 131 (20.1) 153 (12.4)
   Positive HIV status 487 (25.8) 119 (18.3) 366 (29.7)
Open in a new tab

Note: Two participants reported they were trans/non-binary for “gender.”

At enrollment, participants were asked to self-report current and prior diagnoses of communicable and non-communicable diseases, including TB, HIV and mental disorders (Table 2). Notably, 30.6% reported having a diagnosed non-communicable disease (hypertension, diabetes, epilepsy, asthma or other), 15.1% reported a current or previous bout of TB and 25.8% reported they were living with HIV. Of the 268 participants who reported a previously diagnosed mental health disorder (depression, anxiety, PTSD, bipolar disorder, panic, suicidality, alcohol or substance abuse, or schizophrenia), 85% (n = 228) reported receiving treatment for a mental disorder.

Prevalence of confirmed, MINI-diagnosed mental disorders

The prevalence of confirmed, MINI-diagnosed mental and SUDs is presented in Table 3. Specifically, 36% of participants were diagnosed with at least one disorder: CMD, 24.4%; AUD, 9.5%; SMD, 8.1%; SR (moderate to high), 6%; and SUD, 1.6%. While we achieved our gender-balanced target for CMD, AUD, SMD and SR, we did not for SUD. Of the 673 with at least one diagnosis; 53.8% (n = 362) had more than one diagnosis, of the 458 with a CMD diagnosis, 38.0% (n = 174) had more than one cmd. Of the 205 of with either AUD or SUD, n = 6 had both.

Table 3.

Prevalence of MINI-diagnosed mental and substance use disorders diagnosed, by gender

n(%) Total (N=1885) Men (n=651) Women (n=1232)
Any disorder 673 (35.7) 239 (36.7) 433 (35.2)
CMD 458 (24.3) 107 (16.4) 350 (28.4)
   Major depressive episode 406 (21.5) 93 (14.3) 312 (25.3)
   Generalized anxiety disorder 121 (6.4) 23 (3.5) 97 (7.9)
   PTSD 163 (8.6) 30 (4.6) 132 (10.7)
AUD 180 (9.5) 75 (11.5) 105 (8.5)
SUD 31 (1.6) 26 (4) 5 (0.4)
SMD 152 (8.1) 78 (12.0) 74 (6.0)
   Psychotic disorder 95 (5) 66 (10.1) 29 (2.4)
   Hypomanic episode 47 (2.5) 13 (2.0) 34 (2.7)
   Manic episode 20 (1.1) 7 (1.1) 13 (1.1)
SR 114 (6) 20 (3.1) 94 (7.6)
   Moderate risk 33 (1.8) 7 (1.1) 26 (2.1)
   High risk 81 (4.3) 13 (2) 68 (5.5)
Open in a new tab

Abbreviations: AUD, alcohol-use disorder; CMD, common mental disorder; SMD, severe mental disorder; SR, suicide risk; SUD, substance-use disorder.

Performance of the MwTool-13 (initial 3 questions + 10 questions)

Using the original two-step administration method, the mwTool-13 performed well as evidenced by strong sensitivity for identifying any disorder (83.33%), CMD (91.50%), SMD (71.71%), and SR (86.84%) (Table 4). However, the AUD questions (Q4 and Q5) and the SUD question (Q6) yielded sub-optimal sensitivity using the two-step method (56.67% and 64.52%, respectively). Results restricted to those who received the mwTool-13 in isiXhosa and results stratified by gender, HIV status, and lifetime TB history are available in Supplementary Annex 3. The tool similarly for all sub-groups. There are some differences in performance for identifying any disorder, CMD, depression and AUD by gender, with generally higher sensitivity and lower specificity for women when compared to men.

Table 4.

Performance of the mwTool-13

Disorder Questions Mini prevalence Sensitivity
(95% CIs)		 Specificity
(95% CIs)
Step One Questions
Any disorder Q1-Q3 36% 83.36 (80.32-86.10) 46.45 (43.61-49.31)
CMD Q1-Q3 24% 91.48 (88.54-93.87) 44.57 (41.97-47.19)
   CMD Q1, Q3 24% 88.86 (85.62-91.60) 47.72 (45.10-50.35)
   Major depressive episode Q1, Q3 21% 90.89 (87.66-93.50) 46.99 (44.42-49.57)
   Generalized anxiety disorder Q1, Q3 6% 90.08 (83.32-94.77) 40.82 (38.51-43.15)
   PTSD Q1, Q3 9% 89.57 (83.83-93.81) 41.52 (39.18-43.89)
Step Two Questions
AUD Q4, Q5 a 10% 56.67 (49.09-64.02) 84.5 (82.69-86.19)
SUD Q6 2% 64.52 (45.37-80.77) 93.42 (92.19-94.51)
SMD Q7-Q10 8% 71.71 (63.84-78.71) 61.92 (59.58-64.21)
   Psychotic disorder Q7-Q10 5% 67.37 (56.98-76.64) 60.61 (58.31-62.89)
   Manic or hypomanic episode Q7-Q10 4% 77.61 (65.78-86.89) 60.56 (58.27-62.82)
SR Q11-Q13 6% 86.84 (79.23-92.44) 80.24 (78.3-82.07)
Open in a new tab

Abbreviations: AUD, alcohol-use disorder; CMD, common mental disorder; SMD, severe mental disorder; SR, suicide risk; SUD, substance-use disorder.

a

Those who did not self-identify as male or female are treated as missing as these definitions are gender dependent (n = 2).

Performance of the modified mwTool – SA-mwTool-12

Due to the sub-optimal performance of substance-use questions using the original administration and definitions of a positive screen, we proposed the following modifications. We included the AUD questions in the step one question set, which improved sensitivity. We dropped the drinking frequency question (Q4) as it lengthened the questionnaire without improving the sensitivity. We changed the definition of positive screen for AUD as ≥ 3 or 4 drinks on the drinking amount question (Q5) regardless of gender, which upheld face-validity. A more detailed description of steps considered in arriving at the proposed modification is available in Supplementary Annex 4.

The modified mwTool-13 – henceforth the SA-mwTool-12 – thus included a modified initial set of four questions [Q1–Q3 + Q5 (Drinking Amount, ≥ 3 or 4)], where only those who endorse Q1–Q3 and/or Q5, would receive the remaining eight questions (Q6–Q13). The SA-mwTool-12 yielded strong sensitivity for identifying any disorder (89.66%), CMD (91.48%), AUD (74.44%), SUD (74.19%), SMD (73.02%), and SR (88.60%) (Table 5). Results restricted to those who received the mwTool-13 in isiXhosa and results stratified by gender, HIV status, and lifetime TB history showed similar performance and are available in Supplementary Annex 5. The final SA-mwTool-12, definitions of a positive screen and administration instructions can be found in Supplementary Annex 6.

Table 5.

Performance of the SA-mwTool-12

Disorder Questions Mini prevalence Sensitivity
(95% CIs)		 Specificity
(95% CIs)
Step One Questions
Any disorder Q1-Q3, Q5 36% 89.75 (87.20-91.93) 38.03 (35.29-40.83)
CMD Q1-Q3 24% 91.48 (88.54-93.87) 44.57 (41.97-47.19)
   CMD Q1, Q3 24% 88.86 (85.62-91.60) 47.72 (45.10-50.35)
   Major depressive episode Q1, Q3 21% 90.89 (87.66-93.50) 46.99 (44.42-49.57)
   Generalized anxiety disorder Q1, Q3 6% 90.08 (83.32-94.77) 40.82 (38.51-43.15)
   PTSD Q1, Q3 9% 89.57 (83.83-93.81) 41.52 (39.18-43.89)
AUD Q5 10% 74.44 (67.42-80.64) 75.54 (73.43-77.57)
Step Two Questions
SUD Q6 2% 74.19 (55.39-88.14) 91.69 (90.34-92.91)
SMD Q7-Q10 8% 73.02 (65.24-78.89) 60.99 (58.65-63.29)
   Psychotic disorder Q7-Q10 5% 68.42 (58.08-77.58) 59.66 (57.35-61.95)
   Manic or hypomanic episode Q7-Q10 4% 79.10 (67.43-88.08) 59.63 (57.32-61.89)
SR Q11-Q13 6% 88.60 (81.29-93.79) 79.39 (77.43-81.25)
Open in a new tab

Abbreviations: AUD, alcohol-use disorder; CMD, common mental disorder; SMD, severe mental disorder; SR, suicide risk; SUD, substance-use disorder.

Discussion

In this validation study, the mwTool-13 performed as well as the mwTool-12 screener for identifying any disorder, CMD, SMD, and SR (Lovero et al., 2021), but sub-optimally for AUD and SUD. However, the performance of the mwTool-13 was significantly improved by adding a single AUD question (about drinking amount) to the step one question set and defining of a positive screen for AUD as ≥3–4 drinks. By training RAs to administer the mwTool-13, we have demonstrated an ability to build the mental health screening competency of lay-health workers. This supports evidence that community-level health workers can be trained to provide evidence-based mental health screening in low-resource settings (Wainberg et al., 2017). Furthermore, by validating the SA-mwTool-12, this study adds to the nascent, but growing collection of translated, valid mental health screeners for the South African setting.

As evidenced by findings from this study, there is a large burden of mental health disorders in South Africa and an unmet need for mental health services (Seedat et al., 2008; Herman et al., 2009). Given the limited resources to treat individuals with mental disorders, valid screening tools are needed in low-resource settings such as South Africa (Kagee et al., 2013). One means of achieving valid screeners is to ensure that tools are available in the local languages spoken by people accessing public healthcare services; South Africa has 11 official languages, with isiXhosa spoken primary language of 16% of South Africans and predominately spoken in Eastern Cape Province (Statistics South Africa, 2012). Very few studies have translated and validated mental health screeners into isiXhosa or in South Africa at large. The translated isiXhosa mwTool-13 assists in filling this gap and helps provide the necessary tools for the continued expansion of mental health services in South Africa.

While the SA-mwTool-12 yielded high sensitivities for identifying CMD, the specificities ranged from 41.5% to 47.7%. The SA-mwTool-12 is not designed to function as a diagnostic tool, but to serve as a primary- and community-care level tool to facilitate early detection and intervention. Thus, higher sensitivity is prioritized to ensure those who require further screening and potential intervention are identified. Follow-up assessment to confirm mental health distress or diagnosis would be warranted. Future programing and implementation research using the SA-mwTool-12 should recognize the potential for over-identification of individuals in need of additional assessment and the impact this could have on already strained mental healthcare system.

Hazardous alcohol and drug use are stigmatized behaviors (Sorsdahl and Stein, 2010; Sorsdahl et al., 2012; Van Boekel et al., 2013; Zewdu et al., 2019; Regenauer et al., 2020; Magidson et al., 2022), potentially due to low mental health literacy and criminalization. As such, it is possible that stigma reduced disclosure of alcohol and substance use, as has been documented elsewhere in the region (Hahn et al., 2016). In our study, some participants reported hazardous alcohol and drug use on the mwTool-13 screening, but not when asked about these behaviors during the MINI diagnostic interview (and vice-versa). There is a clear need to understand any reluctance to accurately report substance-use behaviors, as well as why the AUD and SUD screening and MINI questions are being answered or interpreted differently. Cognitive interviewing to systematically understand how participants interpret specific screening items has been identified as a crucial method for reducing measurement error in SUD assessment items (Boness and Sher, 2020). Future research should include cognitive interviewing with the AUD and SUD questions from the mwTool-12 as well as the MINI, particularly when responses are incongruent.

We recommend further validating the SUD question of the mwTool-13 as well as other SUD brief screeners among a population with a higher prevalence of SUD in SA to better understand how that question performs, and explore other appropriate options. The low prevalence (1.6%) of SUD hampered our ability to draw conclusions about the performance of the SUD question (Q6). While limited recent SUD prevalence data exists for South Africa, a behavioral study found the past 3-month prevalence of any drug use was 4.4% and 4.1% in South Africa and the Eastern Cape, respectively (Peltzer and Phaswana-Mafuya, 2018). Another small study conducted among patients receiving care at the psychiatric unit of a hospital in East London, South Africa found that 17.4% of participants reported past year use of psychoactive substances (Tindimwebwa et al., 2021). Recognizing that reporting any use is not equivalent to a SUD diagnosis, it is likely that substance use was under-reported in our sample. Further, research has shown hazardous alcohol use is often comorbid with drug use (Pengpid et al., 2021). Yet, only 0.4% of our total study population (and only 3% of the 180 participants with AUD) were diagnosed with both SUD and AUD. Thus, those with SUD would likely be missed if only AUD screening were offered, demonstrating a need to screen not only for AUD but specifically for SUD as well.

Beyond validating a mental health screening tool, this study yielded many other practical insights for implementing mental health programming in resource-limited settings. First, the modified mwTool-13 offers a practical means to “bundle” screening (a process for simultaneously assessing multiple behavioral health disorders; Mulvaney-Day et al., 2018). In resource-limited settings, most existing validated screening tools identify only a single disorder. As administering a multitude of screeners to identify broad-spectrum mental and SUDs would be impractical and time consuming, the mwTool-13 offers an efficient and effective alternative. The recommended modifications also serve to both shorten the tool and simplify administration. Further, changing the definition of a positive screen for AUD such that it no longer requires a gender-binary classification may facilitate more inclusive, gender-affirming care (Arellano-Anderson and Keuroghlian, 2020). This study also yielded real-world multi-lingual validation data. By allowing participants to choose and vacillate between isiXhosa and English, we were able to validate the mwTool-13 in multiple languages simultaneously. In places that have great ethnic and linguistic diversity, such as South Africa, multi-lingual adaptation and validation of mental health screening is lagging (Kaiser et al., 2019; Kaiser et al., 2022). Ultimately, this study provides both the necessary validation data as well as formative practical evidence for future implementation science research into how best to integrate routine broad-spectrum screening into primary and community care in South Africa.

Limitations

There are some inherent limitations to this study. The original mwTool-12 did not include a question specifically for SUD, and a more robust effort to augment the tool prior to launching the validation exercise may have been warranted. Even though the mwTool-12 is meant to be used in primary and community care settings, the study sample consisted of a targeted sample of adults present at primary and tertiary care facilities. As such, the prevalence data reported in this manuscript should not be interpreted as generalizable or representative of the South African adult population and we did not calculate the positive predictive value or negative predictive value of the tool. We did not have a large enough sample of individuals who were screened in English to separately validate the screener in English. We also administered the mwTool-13 screening prior to the MINI, which may have biased the MINI diagnostic interviews.

Conclusions

The mwTool-13 yielded high sensitivities for identifying CMD and SR. The recommended modifications to the mwTool-13 improved the tool’s performance in identifying AUD, SUD, and SMD while maintaining brevity in the South African setting. However, further research into appropriate screening for harmful substance use is warranted. The resulting SA-mwTool-12 offers a valid, translated and culturally relevant brief screening measure for broad-spectrum disorders in South Africa and other low-resource settings. Findings from this study support the continued expansion of mental health screening in South Africa at the primary- and community-care level, facilitate access to appropriate mental health services, and may inform other validation efforts.

Supporting information

Stockton et al. supplementary material

Stockton et al. supplementary material

Acknowledgements

We express our gratitude to the data collectors and study participants without whom this study would not be possible.

Open peer review

To view the open peer review materials for this article, please visit http://doi.org/10.1017/gmh.2023.89.

Supplementary material

The supplementary material for this article can be found at http://doi.org/10.1017/gmh.2023.89.

Data availability statement

The de-identified data may be made available upon reasonable request.

Author contribution

M.A.S. designed the study, analyzed the results, and drafted the manuscript. E.W.M. and L.M. lead the study staff training and help develop data collection protocols, with support from K.N., N.N., A.C.W., P.N., C.B., and C.G. C.G. lead the clinical and provide clinical oversight throughout the course of the study. C.B. and M.M.W. provide statistical expertise and consultation. C.B. developed the data collection system. G.S., K.L.L., D.O. drafted portions of the manuscript. M.L.W., A.M., and P.N. provided senior leadership and oversight.

All authors contributed to the drafting of this manuscript.

Financial support

This study was supported by National Institute of Mental Health grant U19MH113203, MAS was additionally supported by T32MH096724 and K01MH130226, and KLL was supported by K01MH120258. EWM was supported by the University of California Fogarty GloCal Health Fellowship.

Competing interest

The authors declare there are no conflicts of interest.

Ethics standard

The study was approved by the New York State Psychiatric Institute Institutional Review Board (Protocol #8272), the Foundation for Professional Development Research Ethics Committee (8/2021) and the Eastern Cape Department of Health Research Committee (EC_202110_015).

References

  1. Ali G-C, Ryan G and De Silva MJ (2016) Validated screening tools for common mental disorders in low and middle income countries: A systematic review. PLoS One 11(6), e0156939. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Alloh FT, Regmi P, Onche I, Teijlingen EV and Trenoweth S (2018) Mental health in low-and middle income countries (LMICs): Going beyond the need for funding. Health Prospect 17(1), 12–17. [Google Scholar]
  3. American Psychiatric Association (2000) Diagnostic and Statistical Manual of Mental Disorders. 4th (text revision) edition, Washington, DC.
  4. Arellano-Anderson J and Keuroghlian AS (2020) Screening, counseling, and shared decision making for alcohol use with transgender and gender-diverse populations. LGBT Health 7(8), 402–406. [DOI] [PubMed] [Google Scholar]
  5. Bebbington P and Nayani T (1995) The psychosis screening questionnaire. International Journal of Methods in Psychiatric Research 5, 11–19. [Google Scholar]
  6. Boness CL and Sher KJ (2020) The case for cognitive interviewing in survey item validation: A useful approach for improving the measurement and assessment of substance use disorders. Journal of Studies on Alcohol and Drugs 81(4), 401–404. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Collins PY, Insel TR, Chockalingam A, Daar A and Maddox YT (2013) Grand challenges in global mental health: Integration in research, policy, and practice. PLoS Medicine 10(4), e1001434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Department of Health (2014) Integrated Chronic Disease Managment Manual. Pretoria, South Africa: Department of Health, Republic of South Africa.
  9. Department of Health (2019) Adult Primary Care Guide 2019/2020. South Africa: Department of Health Republic of South Africa.
  10. GBD 2019 and Mental Disorders Collaborators (2022) Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: A systematic analysis for the global burden of disease study 2019. Lancet Psychiatry 9(2), 137–150. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Hahn JA, Emenyonu NI, Fatch R, Muyindike WR, Kekiibina A, Carrico AW, Woolf‐King S and Shiboski S (2016) Declining and rebounding unhealthy alcohol consumption during the first year of HIV care in rural Uganda, using phosphatidylethanol to augment self‐report. Addiction 111(2), 272–279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N and Conde JG (2009) Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics 42(2), 377–381. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Hayward SE, Deal A, Rustage K, Nellums LB, Sweetland AC, Boccia D, Hargreaves S and Friedland JS (2022) The relationship between mental health and risk of active tuberculosis: A systematic review. BMJ Open 12(1), e048945. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Herman AA, Stein DJ, Seedat S, Heeringa SG, Moomal H and Williams DR (2009) The South African stress and health (SASH) study: 12-month and lifetime prevalence of common mental disorders. South African Medical Journal 99(5), 339–344. [PMC free article] [PubMed] [Google Scholar]
  15. Janse Van Rensburg A, Dube A, Curran R, Ambaw F, Murdoch J, Bachmann M, Petersen I and Fairall L (2020) Comorbidities between tuberculosis and common mental disorders: A scoping review of epidemiological patterns and person-centred care interventions from low-to-middle income and BRICS countries. Infectious Diseases of Poverty 9(1), 4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Javadi D, Feldhaus I, Mancuso A and Ghaffar A (2017) Applying systems thinking to task shifting for mental health using lay providers: A review of the evidence. Global Mental Health 4, e14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kagee A, Tsai AC, Lund C and Tomlinson M (2013) Screening for common mental disorders in low resource settings: Reasons for caution and a way forward. International Health 5(1), 11–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Kaiser BN, Ticao C, Anoje C, Boglosa J, Gafaar T, Minto J and Kohrt BA (2022) Challenges in simultaneous validation of mental health screening tools in multiple languages: Adolescent assessments in Hausa and Pidgin in Nigeria. SSM-Mental Health 2, 100168. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Kaiser BN, Ticao C, Anoje C, Minto J, Boglosa J and Kohrt B (2019) Adapting culturally appropriate mental health screening tools for use among conflict-affected and other vulnerable adolescents in Nigeria. Global Mental Health 6, e10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Lovero KL, Basaraba C, Khan S, Suleman A, Mabunda D, Feliciano P, Dos Santos P, Fumo W, Mandlate F and Greene MC (2021) Brief screening tool for stepped-care management of mental and substance use disorders. Psychiatric Services 72(8), 891–897. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Lovero KL, Lammie SL, van Zyl A, Paul SN, Ngwepe P, Mootz JJ, Carlson C, Sweetland AC, Shelton RC and Wainberg ML (2019) Mixed-methods evaluation of mental healthcare integration into tuberculosis and maternal-child healthcare services of four South African districts. BMC Health Services Research 19(1), 1–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Lund C, Kleintjes S, Kakuma R and Flisher AJ (2010) Public sector mental health systems in South Africa: Inter-provincial comparisons and policy implications. Social Psychiatry and Psychiatric Epidemiology 45(3), 393–404. [DOI] [PubMed] [Google Scholar]
  23. Magidson JF, Rose AL, Regenauer KS, Brooke-Sumner C, Anvari MS, Jack HE, Johnson K, Belus JM, Joska J and Bassett IV (2022) “It’s all about asking from those who have walked the path”: Patient and stakeholder perspectives on how peers may shift substance use stigma in HIV care in South Africa. Addiction Science & Clinical Practice 17(1), 1–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Massyn N, Day C, Ndlovu N and Padayachee T (2020) District Health Barometer 2019/20. Durban, South Africa: Health Systems Trust.
  25. Massyn N, Pillay Y and Padarath A (2017) District Health Barometer 2016/17. Durban, South Africa: Health Systems Trust. [Google Scholar]
  26. Massyn N, Pillay Y and Padarath A (2019) District Health Barometer 2017/18. Durban, South Africa: Health Systems Trust.
  27. Mendenhall E, De Silva MJ, Hanlon C, Petersen I, Shidhaye R, Jordans M, Luitel N, Ssebunnya J, Fekadu A and Patel V (2014) Acceptability and feasibility of using non-specialist health workers to deliver mental health care: Stakeholder perceptions from the PRIME district sites in Ethiopia, India, Nepal, South Africa, and Uganda. Social Science & Medicine 118, 33–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Microbiologically Confirmed Pulmonary TB - Centre for Tuberculosis (2019) National Health Laboratory Service M&E Online Dashboards National Institute for Communicable Diseases.
  29. Mulvaney-Day N, Marshall T, Downey Piscopo K, Korsen N, Lynch S, Karnell LH, Moran GE, Daniels AS and Ghose SS (2018) Screening for behavioral health conditions in primary care settings: A systematic review of the literature. Journal of General Internal Medicine 33(3), 335–346. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Murray LK, Skavenski S, Bass J, Wilcox H, Bolton P, Imasiku M and Mayeya J (2014) Implementing evidence-based mental health care in low-resource settings: A focus on safety planning procedures. Journal of Cognitive Psychotherapy 28(3), 168–185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Myer L, Smit J, Roux LL, Parker S, Stein DJ and Seedat S (2008) Common mental disorders among HIV-infected individuals in South Africa: Prevalence, predictors, and validation of brief psychiatric rating scales. AIDS Patient Care and STDs 22(2), 147–158. [DOI] [PubMed] [Google Scholar]
  32. National Department of Health (2018) First National Tuberculosis Prevalence Survey, South Africa. Pretoria, South Africa: National Department of Health. [Google Scholar]
  33. Oh K, Choi H, Kim E, Kim H and Cho S (2017) Depression and risk of tuberculosis: A nationwide population-based cohort study. International Journal of Tuberculosis and Lung Disease 21(7), 804–809. [DOI] [PubMed] [Google Scholar]
  34. Peltzer K, Naidoo P, Matseke G, Louw J, Mchunu G and Tutshana B (2012) Prevalence of psychological distress and associated factors in tuberculosis patients in public primary care clinics in South Africa. BMC Psychiatry 12(1), 1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Peltzer K, Naidoo P, Matseke G, Louw J, Mchunu G and Tutshana B (2013) Prevalence of post-traumatic stress symptoms and associated factors in tuberculosis (TB), TB retreatment and/or TB–HIV co-infected primary public health-care patients in three districts in South Africa. Psychology, Health & Medicine 18(4), 387–397. [DOI] [PubMed] [Google Scholar]
  36. Peltzer K and Phaswana-Mafuya N (2018) Drug use among youth and adults in a population-based survey in South Africa. South African Journal of Psychiatry 24(1), 1–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Pengpid S, Peltzer K and Ramlagan S (2021) Prevalence and correlates of hazardous, harmful or dependent alcohol use and drug use amongst persons 15 years and older in South Africa: Results of a national survey in 2017. African Journal of Primary Health Care & Family Medicine 13(1), 2847. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Petersen I, Fairall L, Bhana A, Kathree T, Selohilwe O, Brooke-Sumner C, Faris G, Breuer E, Sibanyoni N and Lund C (2016) Integrating mental health into chronic care in South Africa: The development of a district mental healthcare plan. British Journal of Psychiatry 208(s56), s29–s39. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Petersen I and Lund C (2011) Mental health service delivery in South Africa from 2000 to 2010: One step forward, one step back. South African Medical Journal 101(10), 751–757. [PubMed] [Google Scholar]
  40. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, Currier GW, Melvin GA, Greenhill L and Shen S (2011) The Columbia–Suicide severity rating scale: Initial validity and internal consistency findings from three multisite studies with adolescents and adults. American Journal of Psychiatry 168(12), 1266–1277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Prince M, Patel V, Saxena S, Maj M, Maselko J, Phillips MR and Rahman A (2007) No health without mental health. Lancet 370(9590), 859–877. [DOI] [PubMed] [Google Scholar]
  42. Prins A, Bovin MJ, Smolenski DJ, Marx BP, Kimerling R, Jenkins-Guarnieri MA, Kaloupek DG, Schnurr PP, Kaiser AP, Leyva YE and Tiet QQ (2016) The primary care PTSD screen for DSM-5 (PC-PTSD-5): Development and evaluation within a veteran primary care sample. Journal of General Internal Medicine 31(10), 1206–1211. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Regenauer KS, Myers B, Batchelder AW and Magidson JF (2020) That person stopped being human: Intersecting HIV and substance use stigma among patients and providers in South Africa. Drug and Alcohol Dependence 216, 108322. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Seedat S, Stein DJ, Herman A, Kessler R, Sonnega J, Heeringa S, Williams S and Williams D (2008) Twelve-month treatment of psychiatric disorders in the South African stress and health study (world mental health survey initiative). Social Psychiatry and Psychiatric Epidemiology 43(11), 889–897. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, Hergueta T, Baker R and Dunbar GC (1998) The MINI-international neuropsychiatric interview (MINI): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of Clinical Psychiatry 59(20), 22–33. [PubMed] [Google Scholar]
  46. Simbayi L, Zuma K, Zungu N, Moyo S, Marinda E, Jooste S, Mabaso M, Ramlagan S, North A and Van Zyl J (2019) South African National HIV prevalence, incidence, behaviour and communication survey, 2017: Towards achieving the UNAIDS 90-90-90 targets.
  47. Smith PC, Schmidt SM, Allensworth-Davies D and Saitz R (2010) A single-question screening test for drug use in primary care. Archives of Internal Medicine 170(13), 1155–1160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Sorsdahl K, Stein DJ and Myers B (2012) Negative attributions towards people with substance use disorders in South Africa: Variation across substances and by gender. BMC Psychiatry 12(1), 1–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Sorsdahl KR and Stein DJ (2010) Knowledge of and stigma associated with mental disorders in a south African community sample. Journal of Nervous and Mental Disease 198(10), 742–747. [DOI] [PubMed] [Google Scholar]
  50. South African National Department of Health (2021) First National Tuberculosis Prevalence Survey, South Africa 2018. Pretoria, South Africa: South African National Department of Health. [Google Scholar]
  51. Spitzer RL, Kroenke K, Williams JB and Löwe B (2006) A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine 166(10), 1092–1097. [DOI] [PubMed] [Google Scholar]
  52. Statistics South Africa (2012) South African National Census of 2011 Report No. 03-01-41. Statistics South Africa, Pretoria, South Africa.
  53. Stein DJ, Seedat S, Herman A, Moomal H, Heeringa SG, Kessler RC and Williams DR (2008) Lifetime prevalence of psychiatric disorders in South Africa. British Journal of Psychiatry 192(2), 112–117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Sweetland AC, Belkin GS and Verdeli H (2014) Measuring depression and anxiety in Sub‐Saharan Africa. Depression and Anxiety 31(3), 223–232. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Sweetland AC, Jaramillo E, Wainberg ML, Chowdhary N, Oquendo MA, Medina-Marino A and Dua T (2018) Tuberculosis: An opportunity to integrate mental health services in primary care in low-resource settings. Lancet Psychiatry 5(12), 952–954. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Tibshirani R (1996) Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) 58(1), 267–288. [Google Scholar]
  57. Tindimwebwa L, Ajayi AI and Adeniyi OV (2021) Prevalence and demographic correlates of substance use among adults with mental illness in eastern cape, South Africa: A cross-sectional study. International Journal of Environmental Research and Public Health 18(10), 5428. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Van Boekel LC, Brouwers EP, Van Weeghel J and Garretsen HF (2013) Stigma among health professionals towards patients with substance use disorders and its consequences for healthcare delivery: Systematic review. Drug and Alcohol Dependence 131(1–2), 23–35. [DOI] [PubMed] [Google Scholar]
  59. Vythilingum B, Field S, Kafaar Z, Baron E, Stein DJ, Sanders L and Honikman S (2013) Screening and pathways to maternal mental health care in a South African antenatal setting. Archives of Women’s Mental Health 16(5), 371–379. [DOI] [PubMed] [Google Scholar]
  60. Wainberg ML, Scorza P, Shultz JM, Helpman L, Mootz JJ, Johnson KA, Neria Y, Bradford J-ME, Oquendo MA and Arbuckle MR (2017) Challenges and opportunities in global mental health: A research-to-practice perspective. Current Psychiatry Reports 19(5), 1–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Walt M and Moyo S (2018) The first national TB prevalence survey. South Africa 2018. [DOI] [PMC free article] [PubMed]
  62. Whiteford HA, Ferrari AJ, Degenhardt L, Feigin V and Vos T (2016) Global Burden of Mental, Neurological, and Substance Use Disorders: An Analysis from the Global Burden of Disease Study 2010. The World Bank, pp. 29–40.
  63. WHO (2020) Global Health Estimates 2019: Disease Burden by Cause, Age, Sex, by Country and by Region, 2000–2019. Geneva: World Health Organization. [Google Scholar]
  64. WONCA (2008) Integrating Mental Health into Primary Care. Geneva: World Health Organization.
  65. Zewdu S, Hanlon C, Fekadu A, Medhin G and Teferra S (2019) Treatment gap, help-seeking, stigma and magnitude of alcohol use disorder in rural Ethiopia. Substance Abuse Treatment, Prevention, and Policy 14(1), 1–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Zuma K, Simbayi L, Zungu N, Moyo S, Marinda E, Jooste S, North A, Nadol P, Aynalem G, Igumbor E, Dietrich C, Sigida S, Chibi B, Makola L, Kondlo L, Porter S and Ramlagan S (2022) The HIV epidemic in South Africa: Key findings from 2017 National Population-Based Survey. International Journal of Environmental Research and Public Health 19(13), 8125. [DOI] [PMC free article] [PubMed] [Google Scholar]
Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr1

Author comment: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R0/PR1

Melissa Ann Stockton 1

May 9th, 2023

Submission of manuscript for review, Global Mental Health

Dear Drs. Bass and Chibanda,

I would be grateful if you would consider the following research article by Stockton et al. “Validation of a Brief Screener for Broad-Spectrum Mental and Substance Use Disorders in South Africa” for publication in Global Mental Health. In this manuscript, we present findings from a validation study evaluating the performance of the Mental Wellness Tool-13 in Eastern Cape, South Africa against a diagnostic gold standard. We believe our manuscript will support the continued expansion of mental health screening in South Africa at the primary- and community-care level, facilitate access to appropriate mental health services, and may inform other validation efforts in low-resource settings. If published, it would add to the nascent, but growing, arsenal of translated and validated mental health screeners for the South African setting.

Finally, we believe that our manuscript complements other papers published in the journal in recent years on mental health screeners and validated tools:

• Abrahams and Lund (2022) “Food insecurity and common mental disorders in perinatal women living in low socio-economic settings in Cape Town, South Africa during the COVID-19 pandemic: a cohort study”

• Bitta et al. (2022) “Validating measures of stigma against those with mental illness among a community sample in Kilifi, Kenya”

• Spedding et al. (2022) “ENhancing Assessment of Common Therapeutic factors (ENACT) tool: adaption and psychometric properties in South Africa”

We confirm that this manuscript has not been published elsewhere and is not under consideration by another journal. All authors have approved the manuscript. Should you have any query, please do not hesitate to contact me via email.

On behalf of all authors, I look forward to hearing from you soon.

Yours Sincerely

Melissa Stockton, PhD

Assistant Professor

Department of Epidemiology

UNC-Chapel Hill

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr2

Review: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R0/PR2

Reviewed by: Arvin Bhana1

This is an interesting and well-written manuscript. Developing validated screening tools for mental health in LMICs is an important research endeavour as it has the potential to significantly contribute to the development of measures that are appropriate for these contexts.

Overall, the screening for mental wellness mw Tool-13 records good sensitivity (true positives), but has unacceptable specificity outcomes (true negatives). The abstract indicates that “The mwTool-13 performed well for CMD, SMD, and SR, but sub-optimally for AUD and SUD (sensitivities of 56.7% and 64.5%, respectively).” Unfortunately, this statement is only partly true and the abstract should more accurately report the low specificity values for CMDs and variable performance of SMDs and low sensitivities for AUDs and SUDs to further develop the screening tool.

Introduction

An important argument in the paper is the need to develop mental health screening tools for low-resource contexts. The paper specifically emphasizes high sensitivity alongside a broad spectrum of psychiatric conditions. How do the authors envisage that this could happen given the focus on sensitivity which leads to an over-identification of patients, potentially overwhelming meagre mental health services, especially in LMIC contexts?

Page 3 (lines 43-52). The argument would logically read better if the paragraph began with lines 47-52, followed by lines 43-46.

Methods

General comment - it would be helpful to identify Tables that are meant to be read in conjunction with the text given that there are additional annexes and tables!

Page 4: Greater detail is needed about the level of training and characteristics of a health care worker who provides “outpatient mental health care services”?

Page 5 (Line 107): “... thorough review by the study investigators, research staff, and local psychiatrist.” Was the purpose of the review to establish the face validity of the measure? Did the procedure adopt a formal review process?

Page 5 (Line 128): It would be useful to have greater clarity on the conceptual basis of the three “filter” questions given that two questions refer to CMDs while a third is deemed to be non-specific, though it could easily be considered to be a measure of an anxiety disorder! The link between these and any substance use or alcohol use disorder or severe mental health disorder is not immediately apparent.

Page 6: (Line 154): Could the authors explain how trained RA’s are equivalent to community health workers? What is the length of training received by the RA’s and nurses and how was fidelity established? What are the demographic characteristics of the RAs and nurses trained in the MINI? For example, did the nurses have any psychiatric training, how long have they been in practice etc? What were the major concerns or issues that emerged in the pilot phase?

Page 7: (Line 172-173). Could the authors please clarify why the ICSM (which is part of an ideal clinic practice) was used instead of Adult Primary Care (APC) given that the former determines policies and procedures of how referrals are to be made at a facility level, while the APC manual is concerned with everyday clinical management of patients as part of DoH policy? The APC is used to further interrogate screening decisions by a primary care nurse.

Page 8 (line 211) and Table 2 (Line 9). While 14% (n=268) of the respondents had a previously diagnosed mental disorder, an astonishing 85% (n=228) of these received treatment. This is unusual in a scare-resource context where the number of mental health service providers is limited. Are there any details about who provided such services and was this at the time of the visit or at some other time?

Results

It would be useful to know about the reliability of the overall or specific elements of the mwTool. If this statistic is deemed to be unimportant, please provide an explanation.

In establishing the utility of a screening tool, an important statement of the accuracy of the screening tool is its relative performance in relation to positive and negative predictive values in identifying those with and without a mental health disorder. Is there an explanation as to why this was not considered in the analysis?

Table 3 shows the prevalence of MINI-diagnosed individuals. Given that the total number exceeds the total recruited, please clarify how many participants had multiple disorders.

Tables 4 and 5 do not indicate a similar prevalence profile making it difficult to understand the relative performance of the mwTool-13/ 12 to that of the MINI. In essence, did the mwTool-13/ 12 identify a similar number of individuals in each of the diagnostic categories as did the MINI?

Tables 2, 3 and 4 all show that AUD questions had better specificity ratios than sensitivity ratios. Could the authors speculate about possible reasons for this change in direction relative to the rest of the mwTool?

Discussion:

Page 10 (line 257). Overall, given the low specificity of the mwTool-13 on CMDS, low sensitivity on AUD and SUD and variable performance on SMDs, the claim that the mwTool as it stands is a valid mental health screening tool will need more substantial justification. Nevertheless, the authors recognise that further work is necessary for the mwTool to be generalizable to other settings and population groups.

Page 10 (Lines 261-268): The authors argue for high specificity in low mental health resource contexts in LMICs. This is confusing as the study deliberately sets out to demonstrate the validity of the mwTool. The argument for specificity then is conflated with making the tool available in isiXhosa. Given the poor performance of the mwTool in relation to specificity, except for AUD and SUD subscales, the intention behind this argument should be clarified.

Page 10 (Lines 270-272). It is argued that hazardous alcohol and drug use are stigmatized and have the impact of reducing disclosure. It is unclear why substance use disorders should be more stigmatizing than having a common mental disorder or a severe mental disorder. Further, the explanation that the low specificity ratios are a function of low numbers is somewhat negated as they are also low for CMDs and SMDs among those who received the tool in isiXhosa, by gender, self-reported HIV status and self-reported lifetime TB history! Those attending the clinic may be aware that reporting SUDs will likely result in a referral. This might explain why the number of SUDs self-reports was low. If participants’ reasons for visiting the facility are available, these might be useful to include in the table on demographics, supporting the argument that most attend such facilities not seeking assistance with AUDs or SUDs.

The discussion about the non-performance of SUD items on the mwTool is disproportionate to the discussion of the performance of subscales on the mwTool, some of which performed poorly as a measure of sensitivity (AUD; Psychotic Disorder).

Conclusions

Page 12 (line 323). The statement that “the mwTool-13 performed well” needs moderation as the mwTool performed well concerning sensitivity on CMDs but not SUDS and AUD which had higher specificity ratios and variable performance on Psychotic Disorders subscale.

Minor Comments:

Page 2 (line 31) sentence needs fixing

Page 2 (line 33) - Reference is incorrect

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr3

Review: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R0/PR3

Reviewed by: Anonymous

Impact statement

The impact statement s written well.

Abstract

The abstract is well structured.

1introduction

Although a rationale is provided for the study, changes that were made to the Mozambique tool are not justified and the rationale of that is not clear. Given the importance of substances use disorders in South Africa and other sub-Saharan Africa, an attempt should have been made to achieve a broader tool. Additional questions were included, but this was not associated with useful psychometric properties for example, a sensitivity of about 77%.

2 Methods

2.1 Study settings

Screening for CMD, SUD etc would ideally be done in primary care settings and so the tool would have been validated in this population. However, the researchers validated the tool in both primary and tertiary settings. this might have affected the outcomes in terms of the achieved psychometric properties as well as reducing its usefulness in the primary care settings.

2.2 Study population

The study population is described well and is appropriate to achieve the aims of the study.

2.3 Measures

The tools used in the study are described well. the choice of the gold standard is justified.

2.6 Ethical considerations

The ethical considerations are well described.

3 Results

All the results are presented well. The sensitivities achieved for the tools are promising and the tool is good enough for assessing CMD, but not helpful for SUD and AUD. This is of concern given the comorbidities of SUD and AUD.

4 Discussion

The discussion is presented well.

Limitations

The study limitations are well presented.

5 Conclusion

Although the conclusions are presented well, the challenges of assessing AUD and SUD need to be attended to well. A tool that will be useful for CMD, Severe mental illness, SUD and AUD are important. This study did not produce such and this is an important limitation.

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr4

Recommendation: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R0/PR4

Editor: Inge Petersen1

Thank you for submitting your manuscript on “Validation of a Brief Screener for Broad-Spectrum Mental and Substance Use Disorders in South Africa”. We can, unfortunately not accept the manuscript in its current form. Kindly attend to the reviewers' comments that are included in this email.

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr5

Decision: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R0/PR5

Editor: Dixon Chibanda1

No accompanying comment.

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr6

Author comment: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R1/PR6

Melissa Ann Stockton 1

Dear Editors,

We greatly appreciate your thoughtful reviews. We have responded to the comments as requested and made all changes to the manuscript in tracked changes. Of note, we noted a discrepancy in how we had captured current severe mental disorders. As such, we made some changes to the tables and result section. Our detailed responses to your comments can be found in the response to reviewers box.

Sincerely,

Authors

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr7

Review: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R1/PR7

Reviewed by: Anonymous

Thank you authors for attending to my concerns.

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr8

Review: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R1/PR8

Reviewed by: Arvin Bhana1

The authors have made significant and appropriate changes to the MS. A concern is the use of the term “performed well” in the MS as a whole. While the measure attained good sensitivities on most of the subscales, it did not do so on all the subscales. The specificities were also poor even though the intention was to obtain high sensitivity indices. On this basis, the use of the phrase “performed well” is misleading. The following issues need to be addressed or modified as appropriate:

Page 1: Line 5: Delete “performed well” in the sentence as it appears to be superfluous

Page 2: Line 32: Impact statement: The term “performed well” should be removed.

Page 3: Line 34: Facilitating access to appropriate mental health services was not tested and therefore should not be part of the impact statement.

Page 7: Line 196-197: It may be useful to indicate that for clinical purposes, it is the APC that has to be used.

Page 9: Line 247: Typo

Page 12: Line 362: Typo

Page 12: Line 363: The term “performed well” does not fit with the description of a measure that was only partially successful regarding sensitivity estimates. It tends to overstate the outcomes of this study which is nevertheless valuable.

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr9

Recommendation: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R1/PR9

Editor: Inge Petersen1

Kindly see reviewers' decisions and attend to the minor revision requests

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr10

Decision: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R1/PR10

Editor: Dixon Chibanda1

No accompanying comment.

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr11

Author comment: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R2/PR11

Melissa Ann Stockton 1

Dear Editors,

We greatly appreciate your thoughtful reviews. We have responded to the comments as requested and made all changes to the manuscript in tracked changes. Our detailed responses to your comments can be found below each comment.

Sincerely,

Authors

Reviewer: 1

Comments to the Author

Thank you authors for attending to my concerns.

Reviewer: 2

Comments to the Author

The authors have made significant and appropriate changes to the MS. A concern is the use of the term “performed well” in the MS as a whole. While the measure attained good sensitivities on most of the subscales, it did not do so on all the subscales. The specificities were also poor even though the intention was to obtain high sensitivity indices. On this basis, the use of the phrase “performed well” is misleading. The following issues need to be addressed or modified as appropriate:

We removed “performed well” from the entire manuscript, and have instead highlighted the high sensitivities

Page 1: Line 5: Delete “performed well” in the sentence as it appears to be superfluous

We removed “performed well” from the entire manuscript.

Page 2: Line 32: Impact statement: The term “performed well” should be removed.

We removed “performed well” from the impact statement.

Page 3: Line 34: Facilitating access to appropriate mental health services was not tested and therefore should not be part of the impact statement.

We have removed this from the impact statement

Page 7: Line 196-197: It may be useful to indicate that for clinical purposes, it is the APC that has to be used.

We have indicated the APC is meant to guide clinical decisions.

Page 9: Line 247: Typo

Corrected

Page 12: Line 362: Typo

Corrected

Page 12: Line 363: The term “performed well” does not fit with the description of a measure that was only partially successful regarding sensitivity estimates. It tends to overstate the outcomes of this study which is nevertheless valuable.

We have removed “performed well.”

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr12

Review: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R2/PR12

Reviewed by: Arvin Bhana1

Page 8: Line 47: The sentence is incomplete

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr13

Recommendation: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R2/PR13

Editor: Inge Petersen1

No accompanying comment.

Glob Ment Health (Camb). doi: 10.1017/gmh.2023.89.pr14

Decision: Validation of a brief screener for broad-spectrum mental and substance-use disorders in South Africa — R2/PR14

Editor: Dixon Chibanda1

No accompanying comment.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    Stockton et al. supplementary material

    Stockton et al. supplementary material

    S2054425123000894sup001.docx (169.1KB, docx)

    Data Availability Statement

    The de-identified data may be made available upon reasonable request.

    Articles from Cambridge Prisms: Global Mental Health are provided here courtesy of Cambridge University Press

    RESOURCES