Validation of a brief mental health screening tool for common mental disorders in primary healthcare

Abstract

Background.

Integrating care for common mental disorders (CMDs) such as depression, anxiety and alcohol abuse into primary healthcare (PHC) should assist in reducing South Africa (SA)’s quadruple burden of disease. CMDs compromise treatment adherence, health behaviour change and self-management of illnesses. Appropriate identification of mental disorders in primary care can be facilitated by brief, easy-to-administer screening that promotes high specificity.

Objectives.

To establish the criterion-based validity of a seven-item Brief Mental Health (BMH) screening tool for assessing positive symptoms of CMDs in primary care patients.

Methods.

A total of 1 214 participants were recruited from all patients aged ≥18 years visiting 10 clinics as part of routine care in the Newcastle subdistrict of Amajuba District in KwaZulu-Natal Province, SA, over a period of 2 weeks. Consenting patients provided basic biographical information prior to screening with the BMH tool. PHC nurses remained blind to this assessment. PHC nurse-initiated assessment using the Adult Primary Care (APC) guidelines was the gold standard against which the performance of the BMH tool was compared. A specificity standard of 80% was used to establish cut-points. Specificity was favoured over sensitivity to ensure that those who did not have CMD symptoms were excluded, as well as to reduce over-referrals.

Results.

Of the participants, 72% were female. The AUD-C (alcohol abuse) performed well (area under the curve (AUC) 0.91 (95% confidence interval (CI) 0.88 - 0.95), cut-point ≥4, Cronbach alpha 0.87); PHQ-2 (depression) performed reasonably well (AUC 0.72 (95% CI 0.65 - 0.78), cut-point ≥3, alpha 0.71); and GAD-2 (anxiety) performance was acceptable (AUC 0.69 (95% CI 0.58 - 0.80), cut-point ≥3, alpha 0.62). Using the higher cut-off scores, patients who truly did not have CMD symptoms had negative predictive values (NPVs) of >90%. Overall, 26% of patients had CMD positive symptoms relative to 8% using the APC guidelines.

Conclusions.

Using a higher specificity index, the positive predictive value and NPV show that at higher cut-point values the BMH not only helps identify individuals with alcohol misuse, depression and anxiety symptoms but also identifies a majority of those who do not have symptoms (true negatives), thus not overburdening nurses with false positives needing assessment. Research is needed to assess whether use of such a short and valid screening tool is generalisable to other clinic contexts as well as how mental health screening should best be introduced into routine clinic functioning and practice.

Close to 14% of the global burden of disease can be attributed to neuropsychiatric disorders primarily related to the disabling nature of common mental disorders (CMDs),^[1,2] which typically include depression, anxiety and psychoactive substance use or alcohol use disorder. A review and meta-analysis of studies between 1980 and 2013 established that 29.2% of individuals globally experienced CMDs at some point in their lifetime.^[3] CMDs have been shown to contribute to the burden of disease in low- and middle-income countries,^[4-6] and can variously compromise adherence to treatment, health behaviour change and self-management efforts.^[7-9] In South Africa (SA), almost a third (30.3%) of the population has experienced a CMD in their lifetime,^[10] with a 12-month prevalence estimate of 16.5% for CMDs (anxiety, mood and substance use disorders).^[11] Although effective treatment for mental disorders is available^[12,13] and can be delivered in routine primary healthcare (PHC),^[14] only about half of patients with a depressive disorder in high-income settings are detected^[15,16] and only 16.5% of all individuals with a 12-month major depressive disorder receive minimally adequate treatment.^[17] In SA this gap is far greater, with only one in four people with a CMD reporting receiving treatment of any kind.^[18] While integrating mental healthcare into existing health systems may be the most effective and cost-efficient approach to improve access to mental health services in SA, it requires addressing major knowledge gaps, inter alia the development and assessment of interventions that integrate mental health screening and treatment into existing health systems^[8,19] as well as training lay counsellors in the identification of mental disorders.^[20] However, screening that is integrated into routine care must use measures that can be administered by non-specialist health staff, are brief and easy to administer, and promote high specificity given the meagre resources available to treat false positives.^[21]

Objectives

This validation study was a substudy of the Southern African Mental Health Integration project on evaluating the scale-up of evidence-based packages for integration of mental healthcare in PHC settings for depression and alcohol use disorders into routine care that is part of the Mental Health Integration Programme (MhINT). Continuous quality improvement strategies^[22] that were being used to drive integration identified a lack of standardised screening tools as well as the complexity and non-implementation of existing tools as bottlenecks in identifying patients with CMDs.

The objective of the study was therefore to establish the criterion-based validity of a mental health screening tool for assessing positive symptoms of CMDs (depression, anxiety and substance abuse) among patients attending PHC facilities. The gold-standard criterion was nurse-initiated assessment using the Adult Primary Care (APC) guidelines. This criterion was chosen given that the MhINT model, which is based on the collaborative care model of the Programme for Improving Mental Health Care (PRIME),^[23] relies on professional nurses trained to use the APC for diagnosis of mental disorders. The APC is an integrated set of algorithmic guidelines that forms part of Integrated Clinical Systems Management.^[24]

Methods

Setting

The study was conducted in the Newcastle subdistrict of the Amajuba District of KwaZulu-Natal Province, SA, over a period of 2 weeks. The Newcastle subdistrict, comprising both urban and rural areas and with a population of 389 117 in 2016, ^[25] is serviced by a district and provincial hospital and 14 PHC facilities. Of these, two clinics were excluded because they were linked to hospitals servicing the subdistrict and a third because its remote location made it difficult to conduct fieldwork.

Measures

The seven-item Brief Mental Health (BMH) screening tool comprises internationally validated tools: Alcohol Use Disorders Identification Test (AUD-C), Patient Health Questionnaire (PHQ-2) and Generalized Anxiety Disorder (GAD-2) measures (Fig. 1).

Fig. 1.

Brief Mental Health screening tool.

AUD-C

The AUD-C comprises the first three items of the 10-item AUDIT, which ask about frequency of drinking alcohol, number of alcoholic drinks and binge drinking. The AUD-C is recommended as a simple and reliable tool for routine assessment of risky drinking and screening for alcohol use disorders.^[26,27]Internationally, a score ≥3 for women or ≥4 for men is considered as screening positive for alcohol abuse. The AUDIT was previously validated for use in SA using trained nurses as a gold standard using the same cut-off points.^[28]

PHQ-2

The PHQ-2 is a two-item self-report questionnaire in which participants are asked to rate how often they felt little interest or pleasure in doing things, and how often they felt down, depressed or hopeless over the past 2 weeks, as a screening measure for depression.^[29,30] Original item responses of 0 - 3 (0 = not at all, 1 = several days, 2 = more than half the days, and 3 = nearly every day) were changed to 0 = not at all, 1 = 1 - 7 days, 2 = 8 - 11 days, and 3 = 12 - 14 days), based on a previous criterion validity study of the PHQ-9 among SA PHC service users with chronic conditions.^[31] A score of ≥2 would be considered as screening positive for depression on the PHQ-2.

GAD-2

The scale comprising the first two items of the GAD-7 scale is recommended for screening for anxiety disorders in clinical practice, with further follow-up for those who screen positive.^[32] A score of ≥3 is considered as screening positive for anxiety on the GAD-2. The GAD-2 has been used in screening for detecting antenatal depression and anxiety disorders in SA women ^[33] and is recommended for screening in primary care settings in the NICE guidelines.^[34]

Criterion standard

The criterion standard for establishing the validity of screening tools would typically use another accepted standard of the construct under consideration, usually a clinician-initiated diagnostic interview. As diagnostic assessments are done by the PHC nurse using the APC guidelines, each of the three scales was compared with an independent assessment done by a professional PHC nurse who had received advanced training in the use of the mental health APC guidelines.

Study procedures

Feasibility of the screening process was initially tested using enrolled nurses who randomly screened 10 patients in the ‘vitals screening room’ where all routine screening is done. An isiZulu version of the BMH was developed from the English version using standard translation and back-translation procedures. To enhance clarity, printed copies of the English and isiZulu versions of the BMH were used to compare meaning and wording changes, which resulted in the isiZulu translation for the word ‘depressed’ being changed from ‘unekhwantalala’ to ‘unengcindezi’ and ‘anxious’ being changed from ‘unexhala’ to ‘unovalo’, as the substituted words were more commonly in use. No other changes were made to the BMH.

Following permission from the operational manager at each of the 10 clinic facilities, fieldworkers introduced the study to all the patients seated in the waiting area as an effort to help improve services. All patients consenting to participate in the research were directed to a private room where they were told that the purpose of the research was to see whether symptoms of depression, anxiety and substance use could be identified using a checklist, prior to being asked the seven mental health screening questions (BMH) by the fieldworker. The patient’s information (date, clinic name, patient name and surname and identity number) was then entered onto a detachable pro forma sheet (professional nurse checklist).

The pro forma sheet reflecting the patient’s scores was detached and kept by the fieldworker for safekeeping and filing. A second part of the pro forma sheet with the patient information pre-filled by the fieldworker accompanied the patient to the PHC nurse, who assessed the patient using APC guidelines and entered the assessment in the patient’s file as well as on the pro forma (checklist). The PHC nurse remained blind to the patient’s scores on the BMH. After the consultation, the patient returned the completed checklist to the fieldworker to link the assessment made by the nurse with the mental health screening questions. All interviews were conducted in either English or isiZulu, depending on the language preference of the patient. On average, the entire process took 5 - 10 minutes without affecting patients’ waiting time.

Sample

Prevalence rates for CMDs vary depending on the study sample. The South African Stress and Health (SASH) community survey provided estimates ranging from 4.5% for alcohol abuse to 4.9% for major depression, ^[4] while clinic-based study populations reflect higher prevalence rates ^[35] and rates may be even higher among those with multimorbid chronic conditions.^[36] Given these variations, the present study used the burden associated with neuropsychiatric disorders in sub-Saharan Africa of 10%,^[37] with power of 80% and an overall significance level of 0.05, for a sample of 1 070 participants.^[38]

All patients aged ≥18 years visiting the 10 clinics as part of routine care over a 2-week period were invited to participate in the study. Any patient in need of acute emergency treatment or unable to give written consent was excluded. Ninety-eight patients refused participation in the study and no further information is available on them; 1 214 participants were finally sampled.

Data analysis

While emphasis was placed on ensuring high specificity values^[39] to establish cut-off scores to ensure that those who did not have CMD symptoms (true negatives) would be excluded and thereby also avoid overburdening the clinic with over-referrals, a solution reflecting optimal sensitivity values (>50%) in relation to high specificity values was favoured.

Descriptive statistics and internal consistency estimates were established for each of the subscales. The percentage correctly classified/likelihood ratio was determined on the basis of optimal cut-off values. The 10% prevalence rate used in this study was used to establish the positive predictive value (PPV) (the probability that people with CMD symptoms do have the condition of interest) and negative predictive value (NPV) (the probability that people without CMD symptoms do not have the condition). Receiver operating characteristics (ROC), the area under the curve (AUC) and the 95% confidence interval (CI) were calculated for each subscale. An ROC curve to determine the overall predictive value of the AUD-C, PHQ-2 and GAD-2 against the criterion standard of the APC was calculated using Stata SE version 14.2 (StataCorp, USA).

Ethical considerations

Ethical approval was obtained from the University of KwaZulu-Natal Biomedical Research Ethics Committee (ref. no. BF190/17).

Results

Of the 1 214 adult patients sampled, 72% were female. This gender distribution is typical of PHC facilities in SA. No other details were collected from the sample, as the focus was on the validity of using the BMH in a real-world PHC setting.

AUD-C

ROC curve analysis showed that the AUD-C performed well, with an AUC of 0.91 (95% CI 0.88 - 0.95) (Fig. 2). A cut-point of ≥4 met the 80% criteria of specificity, and the level of sensitivity was high at 82%. This cut-point correctly classified 82.5% of the population, and the likelihood ratio of a person testing positive for alcohol abuse was more than four times more likely (4.71) (Table 1, Fig. 2). The associated PPV/NPV values were 42.2% and 96.8%, respectively. The Cronbach alpha internal consistency estimate for the AUD-C was 0.87.

Fig. 2.

ROC curvefor the AUD-C. (ROC = receiver operating characteristic; AUD-C = Alcohol Use Disorders Identification Test.)

Table 1.

Performance of the AUD-C in detecting alcohol use disorder among patients in routine care^* (N=1 214)

Cut-point	Sensitivity (%)	Specificity (%)	Correctly classified (%)	LR+	LR−
≥0	100	0	1.41	1.00	0.00
≥1	100	67.62	68.08	3.09	0.00
≥2	100	71.15	71.56	3.47	0.00
≥3	100	76.20	76.53	4.20	0.00
≥4	82.35	82.51	82.50	4.71	0.21
≥5	64.71	88.14	87.81	5.46	0.40
≥6	58.82	91.93	91.46	7.29	0.45
≥7	41.18	94.62	93.86	7.65	0.62
≥8	35.29	96.55	95.69	10.23	0.67
≥9	29.41	97.98	97.01	14.57	0.72
≥10	17.65	98.82	97.68	14.99	0.83
≥11	11.76	99.50	98.26	23.31	0.89
≥12	5.88	99.92	98.59	69.95	0.94
>12	0	100	98.59	1.00	1.00

AUD-C = Alcohol Use Disorders Identification Test; LR+ = likelihood ratio positive; LR− = likelihood ratio negative.

^*Adult Primary Care assessment by the primary healthcare nurse was used as a diagnostic reference standard.

PHQ-2

The PHQ-2 performed moderately well, with an AUC of 0.72 (95% CI 0.65 - 0.78) (Table 2, Fig. 3). While a cut-point of ≥4 met the 80% criteria of specificity, a cut-point of ≥3 was suggested as it also optimised sensitivity at 58%. At this cut-point, 76% of the population was correctly classified and the likelihood ratio of a person testing positive for depressive disorder was two and half times greater at this cut-point (2.56). The associated PPV/NPV values were 73.5% and 92.9%, respectively. The Cronbach internal consistency estimate for the PHQ-2 was 0.71.

Table 2.

Performance of the PHQ-2 in detecting depressive disorder among patients in routine care^* (N=1 206)

Cut-point	Sensitivity (%)	Specificity (%)	Correctly classified (%)	LR+	LR−
≥0	100	0	4.73	1.00	0.00
≥1	92.98	26.28	29.44	1.26	0.27
≥2	78.95	49.26	50.66	1.56	0.43
≥3	57.89	77.37	76.45	2.56	0.54
≥4	42.11	86.86	84.74	3.20	0.67
≥5	17.54	93.56	89.97	2.72	0.88
≥6	12.28	95.74	91.79	2.88	0.92
>6	0	100	95.27	1.00	1.00

PHQ-2 = Patient Health Questionnaire; LR+ = likelihood ratio positive; LR− = likelihood ratio negative.

^*Adult Primary Care assessment by the primary healthcare nurse was used as a diagnostic reference standard.

Fig. 3.

ROC curve for the PHQ-2. (ROC = receiver operating characteristic; PHQ-2 = Patient Health Questionnaire.)

GAD-2

The GAD-2 AUC was 0.69 (95% CI 0.58 - 0.80) (Table 3, Fig. 4). A cut-point of ≥3 was suggested for the same reasons as the PHQ-2, resulting in a sensitivity of 59% and specificity of 76%. At this cut-point, 75% of the population was correctly classified. The likelihood ratio of a person testing positive for anxiety disorder was just over two times greater at this cut-point (2.42). The associated PPV/NPV values were 89.8% and 93.5%, respectively. The Cronbach internal consistency estimate for the GAD-2 was 0.62.

Table 3.

Performance of the GAD-2 in detecting anxiety disorder among patients in routine care^* (N=1 196)

Cut-point	Sensitivity (%)	Specificity (%)	Correctly classified (%)	LR+	LR−
≥0	100	0	1.42	1.00	0,00
≥1	88.24	35.03	35.79	1.36	0.34
≥2	82.35	56.74	57.11	1.90	0.31
≥3	58.82	75.66	75.42	2.42	0.54
≥4	11.76	87.11	86.04	0.91	1.01
≥5	0	94.49	93.14	0.00	1.06
≥6	0	96.10	94.73	0.00	1.04
>6	0	100	98.58	1.00	1.00

GAD-2 = Generalized Anxiety Disorder measure; LR+ = likelihood ratio positive; LR− = likelihood ratio negative.

^*Adult Primary Care assessment by the primary healthcare nurse was used as a diagnostic reference standard.

Fig. 4.

ROC curve for the GAD-2. (ROC = receiver operating characteristic; GAD-2 = Generalized Anxiety Disorder measure.)

Table 4 provides an overall summary of the relative cut-points, sensitivity, specificity and percentage correctly classified for each of the subscales. The PPV and NPV values indicate the likelihood of identifying patients who have the relevant symptoms against an optimised cut-point.

Table 4.

BMH cut-points, sensitivity and specificity, and PPV and NPV values

	Cut-point	Sensitivity (%)	Specificity (%)	Correctly classified (%)	n	PPV (%)	NPV (%)
AUD-C* (N=1 207)	≥4	82.35	82.51	82.50	222	42.2	96.8
	≥5	64.71	88.07	87.74	152
	≥6	58.82	91.93	91.46	106
PHQ-2† (N=1 207)	≥2	78.95	49.26	50.66	628
	≥3	57.89	77.37	76.45	293	73.5	92.9
	≥4	42.11	86.86	84.74	175
GAD-2† (N=1 197)	≥2	82.35	56.74	57.11	525
	≥3	58.82	75.66	75.42	297	89.8	93.5
	≥4	11.76	87.11	86.04	154

BMH = Brief Mental Health; PPV = positive predictive value; NPV = negative predictive value; AUD-C = Alcohol Use Disorders Identification Test; PHQ-2 = Patient Health Questionnaire; GAD-2 = Generalized Anxiety Disorder measure.

^*International scale recommended cut-off: ≥3 for females and ≥4 for males. In this instance, one international cut-off was used: ≥4.

^†International scale recommended cut-off: ≥2.

Discussion

The need for increased focus on CMDs as part of an integrated PHC service is important in the context of the shift in SA’s disease epidemic to multimorbid chronic conditions,^[40] the high rate of comorbid CMDs, and the role that coexisting CMDs play in worsening treatment outcomes in patients with chronic conditions.^[41] An important first step in reducing the treatment gap associated with low levels of identification of those in need of care at PHC level is the ability and capacity to identify CMDs.

Using a higher specificity index, a cut-off score of ≥4 identified alcohol use disorder (AUD-C) symptoms in 18% of patients, while a cut-off score of ≥3 on the PHQ-2 identified depressive symptoms in 24% and a cut-off score ≥3 on the GAD-2 identified anxiety symptoms in 25%. In comparison, APC assessment by PHC nurses showed that 17 patients (2%) had AUD and anxiety symptoms and 57 (6%) had depression symptoms. Employing a higher specificity standard and associated cut-offs, this validation study of the BMH found that between 18% and 25% of patients would need follow-up assessment for diagnosis.

Overall, 26% of patients were identified as having positive symptoms using the BMH screening tool, in contrast to 8% of patients using the APC guidelines. The high NPV values for each of the subscales of 96.8%, 92.9% and 93.5% for AUD-C, PHQ-2 and GAD-2, respectively, confirm that these cut-points will help rule out the probability of diagnosing patients who truly do not have CMD symptoms (Table 4).

In contrast, nurse identification of CMDs using the APC guidelines was low. They also made more dual diagnoses of disorders than individual diagnoses and had a high proportion of cases in which the diagnosis was indeterminate (7%) or missing (15%). It is possible that until recently PHC nurses did not routinely identify patients with CMDs since no clear referral pathways existed for treatment and follow-up. It is also likely that the APC guidelines may have been poorly trained or poorly applied; furthermore, use of an algorithm-based diagnosis is more complex than a simple sum of scores (BMH). These findings are supported by international studies that indicate that training of PHC practitioners in identification of mental disorders does not necessarily improve identification for a number of reasons, including that: (i) visits are time-limited; (ii) the purpose of the PHC visit is generally for physical complaints; and (iii) psychiatric stigma may make it difficult for patients to talk about their emotional difficulties.^[42]

However, the findings from several large-scale studies also indicate that when specific measures are used and little additional computation on the part of the clinician is required, the information from screening may be more readily integrated.^[16] With screening having been found to improve diagnostic rates of mental disorders in PHC settings,^[42] using the BMH in PHC settings should therefore assist PHC nurses in identifying specific patients for further assessment.

Study limitations

Limitations of this study include that we were unable to randomise patients given that the study occurred under real-world conditions, patients with conditions other than CMDs may have been missed or ignored, and the study was limited to one subdistrict in one region of the country.

Conclusions

In the context of low levels of treatment, the BMH screening tool with associated cut-offs favouring optimal sensitivity values (>50%) in relation to high specificity values is likely to minimise over-referrals. Used in this way, the BMH is likely to be useful for use in PHC settings to improve identification of CMDs and potentially increase the number of individuals receiving treatment. Further research is needed to assess whether use of the BMH does indeed improve identification of CMDs by PHC nurses through directing them to initiate assessment of patients for potential CMDs using the APC guidelines as well as to explore reasons for low levels of identification of CMDs when the BMH is not in use. There is a need to establish the generalisability of these findings through evaluation in various other facilities with appropriately powered samples. In addition, there is a need to assess how best to introduce mental health screening into routine clinic functioning and practice, as well as for further evaluation of the BMH when translated into different local languages and regions.