How competent are non-specialists trained to integrate mental health services in primary care? Global health perspectives from Uganda, Liberia, and Nepal

Abstract

Evaluations to objectively assess minimum competency are not routinely implemented for training and supervision in global mental health. Addressing this gap in competency assessment is crucial for safe and effective mental health service integration in primary care. To explore competency, we describe a training and supervision program for 206 health workers in Uganda, Liberia, and Nepal in humanitarian settings impacted by political violence, Ebola, and natural disasters. Health workers were trained in the World Health Organization’s mental health Gap Action Programme (mhGAP). Health workers demonstrated changes in knowledge (mhGAP knowledge, effect size, d=1.14), stigma (Mental Illness: Clinicians’ Attitudes, d=−0.64; Social Distance Scale, d=−0.31), and competence (ENhancing Assessment of Common Therapeutic factors, ENACT, d=1.68). However, health workers were only competent in 65% of skills. Although the majority were competent in communication skills and empathy, they were not competent in assessing physical and mental health, addressing confidentiality, involving family members in care, and assessing suicide risk. Higher competency was associated with lower stigma (social distance), but competency was not associated with knowledge. To promote competency, we recommend (1) structured role-plays as a standard evaluation practice; (2) standardized reporting of competency, knowledge, attitudes, and clinical outcomes; and (3) shifting the field toward competency-based approaches to training and supervision.

BACKGROUND

Persons living with severe mental, neurological, and substance use (MNS) disorders in low-income countries around the world rarely receive adequate care (Degenhardt et al., 2017; Lora et al., 2012; Meyer et al., 2012). Low-income countries have few psychiatrists and other mental health specialists, inpatients services for brief hospitalizations are limited to major cities, and medications are inconsistently available (World Health Organization, 2018). Persons with psychosis may go years or decades without psychiatric or neurological treatment (Meyer, et al., 2012; Mossaheb et al., 2013). Moreover, persons MNS disorders and their families suffer overwhelming stigma leading to isolation and exploitation (Heijnders and Van Der Meij, 2006; Thornicroft et al., 2009). It is not uncommon to be chained, tied, or locked-up within the home to prevent harm by the community or injury resulting from the untreated MNS disorder (Asher et al., 2017). All of these challenges for people living with severe MNS disorders are compounded in the context of humanitarian emergencies when health systems are damaged, medication supplies are disrupted, risks of traumatic exposures are increased, and material resources are scarce (IASC, 2007; Jones et al., 2009).

To address the gap in adequate services for MNS disorders, the World Health Organization (WHO) and other institutions have advocated for “task-sharing”, also known as “task-shifting”, approaches in which health workers, non-health professionals, and lay persons take on the roles traditionally assumed by mental health specialists (World Health Organization, 2016). Integration of mental health services into primary care is one of the hallmarks of task-sharing approaches. The integration of mental health into primary care dates back to the 1970–80s when WHO conducted the multi-country Collaborative Study on Strategies for Extending Mental Health Care (Murthy and Wig, 1983). This initiative focused on training primary care workers to use symptom-based flow charts for the diagnosis and management of MNS disorders. The approach had modest success, but unfortunately did not adequately improve detection and treatment in primary care and ultimately did not become common practice globally (Harding et al., 1983). A decade ago, WHO revisited and updated this approach with the mental health Gap Action Programme (mhGAP), which is an initiative with intervention guidelines, training and supervision materials, and implementation recommendations for primary care workers and health administrators to integrate pharmacological and psychological interventions into routine care (www.who.int/mental_health/mhgap/en/). mhGAP is now used in more than 90 countries for diagnosis and management of depression, psychosis (including bipolar disorder and schizophrenia), epilepsy, substance use disorders, dementia, child and adolescent disorders, and suicidality (World Health Organization, 2016). In 2015, a humanitarian version of mhGAP was released that prioritizes disorders commonly seen in conflict and disaster situations including grief and posttraumatic stress disorder (World Health Organization, 2015).

The impact of integrating psychiatric and neurological treatment in primary care settings through mhGAP has varied widely. Unfortunately, in the majority of published analyses of integrated care, only about one out ten people with MNS disorders were accurately detected by primary care workers trained in the mhGAP Intervention Guide or similar initiatives (Fekadu et al., 2017; Jenkins et al., 2013; Kauye, Jenkins, & Rahman, 2014). To understand the impact or lack thereof related to primary care integration, there is a growing body of literature on training models and outcomes related to knowledge, attitudes, and competency of primary care workers after participating in mhGAP or similar programmes (Keynejad, 2018). WHO reports 12 core competencies that should be achieved by health workers trained in mhGAP (see Table 1). One of the major questions is how best to train and supervise primary care workers given different contexts, resources, health worker educational backgrounds, and accessibility of specialists.

Table 1.

Twelve core competencies for health workers trained in mhGAP (WHO, 2017)

Competency	Description	Domain
1	Promotes respect and dignity for people with MNS conditions	Attitude
2	Knows common presentations of priority MNS conditions	Knowledge
3	Knows assessment principles of MNS conditions	Knowledge
4	Knows management principles of MNS conditions	Knowledge
5	Uses effective communication skills in all interactions with people with MNS conditions	Skill
6	Performs assessment for priority MNS conditions	Skill
7	Assesses and manages physical conditions of people with MNS conditions	Skill
8	Assesses and manages emergency presentations of priority MNS conditions	Skill
9	Provides psychosocial interventions to people with a priority MNS condition and their carer	Skill
10	Delivers pharmacological interventions as needed and appropriate in priority MNS conditions, considering special populations	Skill
11	Plans and performs follow-up for priority MNS conditions	Skill
12	Refers to specialists and links with outside agencies for priority MNS conditions, as appropriate and available	Skill

Central to the question of how best to train and supervise primary care workers is the issue of how to reliably and validly assess adequate competency to safely and effectively provide care for MNS disorders. Competency refers to “the extent to which a therapist [or other health worker] has the knowledge and skill required to deliver a treatment to the standard needed for it to achieve its expected effects,” (Fairburn and Cooper, 2011, p. 373). Competence is near the top of the hierarchy of clinical skills (Miller, 1990), (see Figure 1). At the base of hierarchy are factors such as attitudes toward patient populations (e.g., stigma among primary care workers toward persons with MNS disorders). These attributes are typically evaluated through self-report measures, such as the Social Distance Scale or Depression Attitudes Questionnaire in which health workers report their willingness to interact with persons with MNS disorders or their belief that depression can be treated effectively in primary care settings. The next level is “knows”, which refers to basic knowledge of target conditions and treatments. This is typically assessed with multiple choice assessments or true-false questions. The next level is “knows how”, referring to applied knowledge, which is typically assessed with written clinical vignettes wherein health workers make diagnoses and treatment recommendations. Competence is assessed through structured role-plays in which standardized ‘mock’ patients act out a condition and health workers interact with the standardized patient to make a diagnosis and treatment plan, while being observed (live or through recordings) for clinical performance. Treatment quality is a real-world outcome based on delivery of care with actual patients. This can be assessed through live observations in clinical settings or through clinical or supervision notes.

Figure 1.

Hierarchy of clinical skills (adapted from Miller, 1990) and associated evaluation strategies.

To date, the majority of training and supervision programs for primary care workers in low- and middle-income countries (LMICs) have relied upon self-reported attitude assessments and knowledge tests (Keynejad, et al., 2018). Competence has received limited attention, despite calls for increased documentation of competence using role plays (World Health Organization, 2017) and recommendations skill levels (International Medical Corps, 2018). The issue of minimum competency is especially important when considering severe MNS disorders for which inaccurate diagnosis and management could present greater risk than benefit. For example, treating a patient with antipsychotic or antiepileptic medication who does not actually have the condition is unnecessary exposure to potentially harmful medication, including increasing risk of seizures. Similarly, failure to appropriately manage these medications among women of reproductive age risks harm to pregnant women and their fetuses.

Therefore, our goal was to evaluate competency among primary care workers trained in mhGAP and community-based mental health services for a programme targeting psychosis (including mania) and epilepsy in settings impacted by humanitarian conflict. The program was implemented in a region of northern Uganda impacted by the more than two decades of violence perpetrated by the Lord’s Resistance Army, in southeastern Liberia in communities impacted by a 14-year-civil and subsequently affected by the Ebola virus disease outbreak during the course of the study, and in the middle-hills of midwestern Nepal impacted by a decade-long Maoist revolution (Angdembe, Kohrt, Jordans, Rimal, & Luitel, 2017; Gwaikolo, Kohrt, & Cooper, 2017; Kisa et al., 2016).

METHODS

Intervention package

To extract lessons learned about training and supervision evaluation, we use the implementation of the comprehensive community-based mental health services (CCMHS) package for stigmatized, severe MNS disorders developed for the mental health Beyond Facilities (mhBeF) programme. mhBeF was implemented by Makerere University, Kampala, Uganda, in partnership with Transcultural Psychosocial Organization (TPO) Uganda, The Carter Center Mental Health Programme in Liberia, and TPO Nepal. The CCMHS package integrates three components: (a) strengthening clinical recognition, referral, assessment, and management using the mhGAP Intervention Guide, including an electronic mobile health (mHealth) package; (b) establishing psychosocial and socio-economic services through patient support groups; and (c) conducting stigma reduction targeted activities for health providers, community stakeholders, families, and patients.

The activities within mhBeF included training and supervision for primary care workers, community health workers, and social workers. Psychiatrists and psychosocial experts trained the non-specialist primary care workers to identify and treat epilepsy, psychosis/mania, and depression based on mhGAP (see Table 2 for the full curriculum). The training included mhGAP Intervention Guide modules depression, suicide, psychosis (bipolar disorder and schizophrenia), and epilepsy. In most low-income countries, psychiatrists treat both epilepsy and psychiatric conditions, and, among the general public, epilepsy is more closely linked with psychiatric rather than medical conditions. Within mhGAP, epilepsy is included as one of the target conditions. In addition to mhGAP modules, the training included four other components: (a) reducing stigma and discrimination; (b) community sensitization and detection with referral (pro-active case finding was used in Nepal (Jordans, Kohrt, Luitel, Lund, & Komproe, 2017b)); (c) home-based care; and (d) patient support groups with microfinance activity. Social workers or other non-governmental organization (NGO) workers were trained to supervise the non-medical components and conduct patient-support groups. Our current analyses focus on the attitudes, knowledge, and competence for clinical services delivered by primary care workers.

Table 2.

Curriculum for training health workers in mental health Beyond Facilities (mhBeF) package

Module	Topics	DurationA	Participants
1.  Introduction to mental health and mental illness	1.1 What is mental health and mental illness? 1.2 Local terminology related to mental illness 1.3 What causes mental illness? 1.4 How can mental illness be treated? 1.5 Recovery stories of persons with mental illness?	3.5 hours	All trainees
2.  Stigma and discrimination	2.1 Human rights and disability: Convention on the Rights of Persons with Disability (CRPD) 2.2 What is stigma and discrimination? 2.3 Types of stigma 2.4 Reasons for stigma	4 hours	All trainees
3.  Identification, referral, and collaborative care	3.1 Organization of mental health services in the country and region 3.2 Identification of mental illness in the community 3.3 Referral skills and pathways 3.4 Ethics, confidentiality, and record keeping 3.5 Collaborative care	4.5 hours	All trainees
4.  General mental health skills and emergencies	4.1 Communication skills 4.2 Responding during a seizure 4.3 Verbal de-escalation techniques 4.4 Responding to suicidal behavior (includes mhGAP suicide module)	6 hours	All trainees
5.  Disorder specific modules: Epilepsy, psychosis (and manic episodes), depression	5.1 Epilepsy and seizures 5.2 Psychosis and manic episodes 5.3 Depression 5.4 (Optional: clinic visit and examination of patients)	8 hours (3.5 hours)	Health workers with prescribing rights
6.  Care management and follow-up for patients and families	6.1 Developing a comprehensive care management plan 6.2 Preventing relapse 6.3 Follow-up	3.5 hours	All trainees
7.  Home-base care and care for caregivers	7.1 Home-based care 7.2 Psychosocial wellbeing 7.3 Care for caregivers	5 hours	Community health workers and their supervisors
8.  Community mobilization and advocacy skills	8.1 Community mobilization and awareness raising 8.2 Advocacy skills	3.5 hours	Community health workers and their supervisors
9.  Patient support groups	9.1 Forming groups 9.2 Group facilitation 9.3 Addressing challenges in group facilitation	5 hours	Social workers, NGO workers, or other relevant personnel
10.  Conducting anti-stigma programs	10.1  The purpose of anti-stigma groups 10.2  Conducting health facility anti-stigma groups 10.3  Conducting community anti-stigma groups	5 hours	Social workers, NGO workers, or other relevant personnel

^A.Total duration: Health workers (5-days, 30 hours not including pre- and post-tests and meal/tea breaks); Social Workers included an additional 10 hours for Modules 9 and 10; Nepal employed a 9-day curriculum with additional time devoted to psychosocial communication skills and mhGAP disorder modules.

We developed a training manual “Comprehensive Community-based Mental Health Services Manual” that would be used by resource persons trained under the mhBeF project. The manual was meant to be a resource in each of the three countries to ensure on-going capacity development after the project closure. The process of developing the manual was iterative and involved each component lead developing an outline of the training content. Once all three components (health worker training, anti-stigma and social capital formation) were written, a lead writer was identified to consolidate the components into a single manual and align all the sections based on learning objectives. The first draft was pretested in Liberia in the second year of the project. During the pretest, trainers checked for adequacy of times allocated per session, sequence of sessions, relevance of training content to local health workforce, and comprehension of training content at the end of the training. One of the key outcomes from the pretest was the need to include a module on management of clinical services, including referrals, and collaborative care. Content arising from the pretest workshop was then consolidated and integrated into the second version of the manual, prior to a final review and affirmation by the component leads. Copies of the final manual were developed and disseminated across the three countries.

Settings

In Uganda, the health system is divided into four levels of primary services, then secondary and tertiary services. At the community level, Village Health Teams (VHT) are lay persons volunteering in the government system who are not paid but receive compensation for training and some programmatic activities. They mostly engage in health promotion and prevention activities, but they have been successfully engaged to deliver psychological treatments in other initiatives in Uganda (Mutamba et al., 2018a; Mutamba et al., 2018b). At the Health Center (HC)-II level is primary care services. HC-III has a higher level of services including some mental health services provided by psychiatric nurses. Psychiatric clinical officers, who receive three years of training in the diagnosis and management of MNS disorders, are placed in regional referral/district hospitals. There are 13 regional referral hospitals with 30–40 bed psychiatric units, and there is one National Referral Psychiatric Hospital, Butabika, in the nation’s capital Kampala. The project was implemented in Lira District in central northern Uganda in an area heavily affected by the Lord’s Resistance Army. Lira has a Regional Referral Hospital with a 30 bed capacity mental health ward and five psychiatric clinical officers. Health worker trainees were health facility staff at the HC-II and HC-III level, and the community health volunteers from VHTs. The trainers were Ugandan psychiatrists who had previously received Training of Trainers and Supervisors in mhGAP by WHO. Staff from TPO Uganda also led some sessions. Table 3 provides the details on health workers, trainings by country. Regarding mental health service user involvement, on the first day, a mental health service user and caregiver provided a personal testimonial. Health workers spent a half-day at the nearby regional referral hospital assessing psychiatric patients using the mhGAP Intervention Guide under the supervision of psychiatrists and psychiatric clinical officers. The study sites were Erute South Health Sub District and Erute North Health Sub District.

In Liberia, there was only one psychiatrist at the time of the mhBeF program. In Liberia, one of the solutions to the gap in mental health services has been to train cadres of specialist mental health nurses and physician’s assistants through a six-month training course including didactic and clinical components. This training program to produce Mental Health Clinicians (MHCs) has been conducted twice yearly since 2011 by the Government of Liberia Ministry of Health and Social Welfare in collaboration with The Carter Center Mental Program-Liberia Initiative (www.cartercenter.org/health/mental_health/mh-liberia.html). As of 2015, there were 144 MHCs who were licensed by the government to practice independently to provide mental health services in the primary care setting. The main implementation site was Sinoe county in rural southeast Liberia. Two adjacent counties, River Gee and Grand Kru, were selected as control sites. The control sites share similar geography, healthcare conditions, and demographic characteristics with Sinoe. In Liberia, the training was also delivered in 5 days. The first 2 days included all participants (primary care health workers, community health workers, and service users). In addition, there were eight mental health clinicians (MHCs) who would later serve supervisory roles. Training was conducted by a Ugandan psychiatrist trained in mhGAP. The MHCs also facilitated certain modules, as did members of the Carter Center staff and consultants. Service users who had been trained by the Carter Center also participated as co-facilitators of the stigma component and other sessions, as well as sharing their recovery stories. Local service users, who had previously been treated by MHCs and were in recovery, attended the training as participants and interacted with health worker trainees in break-out sessions and other group activities. After the training, ongoing activities included supervision, conducting health facility and community-based anti-stigma programs, and formation of patient support groups. In Liberia, services and supervision were interrupted during the Ebola virus disease outbreak. Once the Ebola virus disease was under control, a refresher training was conducted and supervision was restarted.

In Nepal, mhBeF was implemented by a Nepali non-governmental organization, TPO Nepal, which was established in 2005 and had extensive experience in psychosocial services for humanitarian affected settings. TPO Nepal was a member of the Programme for Improving Mental Health Care (PRIME) consortium developing primary care and community mental health services (Lund et al., 2012). PRIME also included mhGAP as a base of its implementation. The mhBeF curriculum was modified based on experiences of the PRIME initiative which had locally adapted mhGAP and developed a district mental health plan for treatment of depression, psychosis, alcohol use disorder, and epilepsy (Jordans, Luitel, Pokhrel, & Patel, 2016). The adapted mhBeF curriculum included 9-days training for primary care workers who had prescribing rights for psychiatric medication. Non-prescribing health facility workers (auxiliary nurse midwives) received 5 days of training. Female community health volunteers received 2-days of training focused on pro-active case detection (Jordans, et al., 2017b). Supervision was provided throughout the project and after approximately 1-year of service delivery the prescribers received a 3-day refresher training, and the non-prescribers received a 2-day refresher course. Trainings and supervision were provided by psychiatrists previously trained in mhGAP. Psychosocial counselors provided the training for the communication skills, collaborative care, and community-based components. The psychosocial counselors also provided the training to the Female Community Health Volunteers. After the trainings, the psychosocial counselors conducted the patient support groups. The Female Community Health Volunteers conducted the 1-day health-facility and community anti-stigma programs. In Nepal, TPO Nepal partnered with the Nepal Mental Health Foundation, a mental health service user advocacy group, to train service users who shared recovery stories and co-facilitated the anti-stigma sessions.

Evaluation of Training and Supervision

Primary care workers were evaluated for knowledge, attitude, and competence related to MNS clinical care. Knowledge and attitudes were assessed with standard tools for primary care workers (Hanlon et al., 2018):

• mhGAP Knowledge – True-false and multiple-choice questions were adapted from mhGAP version 1.0 content for PRIME. These questions address general MNS disorders, psychosis, depression, and epilepsy were included, as well two additional questions about mania and bipolar disorder. The full battery of 26 questions were administered to health workers with prescribing rights. For non-prescribing health staff and community health workers, 19 of the questions were administered (i.e., medication related questions were removed).

• mhGAP attitudes – Based on mhGAP Intervention Guide version 1.0 questions, PRIME also adapted a series of attitudinal questions about general MNS disorders, psychosis, depression and epilepsy. A series of similarly constructed questions were added for bipolar disorders. All participants completed these questions.

• Mental Illness: Clinician’s Attitudes (MICA) – The MICA is a 16-item scale with Likert responses that has been developed for health workers and medical students.(Gabbidon et al., 2013; Kassam, Glozier, Leese, Henderson, & Thornicroft, 2010) It specifically assesses attitudes toward learning about mental health and whether providing mental health services is comparable in acceptability and worth to physical health care. The scale range is 16 to 96.

• Social Distance Scale (SDS) – The SDS is a widely used measure to assess willingness to interact with persons from a specific stigmatized group (Bogardus, 1925; Link, Yang, Phelan, & Collins, 2004; Pescosolido, Medina, Martin, & Long, 2013). It has been adapted to assess willingness to interact with persons with mental illness. We used a 10-item SDS with each item scored on a 4-point scale (1–4) for a total score range of 10 to 40.

Competence was evaluated using structured role plays with standardized patients. In each country, specialists in mental healthcare were trained to perform as a standardized patient who had symptoms of depression and epilepsy. The training was conducted in approximately 2 hours during which the specialists were given a vignette describing the patient’s symptoms and life circumstances. Both male and female versions of the scripts were developed. The specialists were instructed to act out symptoms and to have standardized responses to questions about care seeking, treatment goals, and family involvement. Specialists were also trained to share prompts intended to elicit questions about suicide risk and safety planning. The training also included pertinent negatives related to care (e.g., there was no current alcohol use or other psychoactive substances; there was no history of head trauma; the female patient not pregnant). The standardized patients were trained to continue the role plays for 10 minutes or until the health worker trainee ended the session. The standardized patients were trained to score the health worker using a rating tool developed for competency assessment of non-specialists:

• ENhancing Assessment of Common Therapeutic factors (ENACT) – The ENACT tool is used by raters observing standardized role plays of trainees (Kohrt et al., 2015; Kohrt, Ramaiya, Rai, Bhardwaj, & Jordans, 2015). The ENACT includes 18 items plus diagnosis and treatment and originally developed in Nepal. Each of the 18 items is scored at one of three levels. Level 1 refers to not performing the skill or doing it in a potentially harmful manner. Level 2 refers to doing partial elements of the skill. Level 3 refers to completing of the elements in manner consistent with therapeutic benefit. Any items not completed in the 10-minute role play were scored as a 1 (the lowest score) because the standardized role plays were designed to encompass all 18 items. The version of the ENACT used for this study is available as in the Appendix.

One the first day of all trainings, training participants were provided with paper assessments that included the knowledge and attitude questions. In Liberia and Nepal on Day 1, clinical competence role plays were also conducted. On the final day of the trainings, the assessments were repeated. All countries conducted the clinical competence role plays during the post-training assessment. The health workers then provided services for MNS disorders integrated into their regular clinical duties, and they received supervision from a mental health specialist approximately every one to three months. Then after approximately six to 12 months of services, the assessments were completed again prior to a refresher course, with the exception of Liberia where follow-up refreshers and evaluations were delayed by the Ebola virus disease outbreak. Routine technical support supervision was carried out for all trained health workers using mhGAP supervision instruments designed to consolidate training outcomes and strengthen integrated mental healthcare.

Participants and Ethics

Health facilities involved in mhBeF were selected by the government through conversations with the implementing NGOs and national academic institutions. All health workers from selected institutions were invited to attend mhBeF trainings wherein they received per-diem compensation in accord with government rates. All trainees underwent a consent procedure in which the consent forms were read to all participants at the beginning of the training and they had printed copies in the standard language of health education (English in Uganda and Liberia, Nepali in Nepal). They were given opportunities to ask questions one-on-one with research staff about their participation. Any health worker could choose to participate in the training but refuse to take part in the assessments. All countries obtained ethical approvals: Makerere University School of Public Health Higher Degree Research and Ethics Committee and National Council for Science and Technology (HS1552) in Uganda, University of Liberia-Pacific Institute for Research and Evaluation Institutional Review Board in Liberia, and Nepal Health Research Council in Nepal.

Quantitative analysis

Paired t-tests were used to compare measures between two time points. Knowledge was calculated as the percent correct responses (out of 26 items for prescribers and 19 items for non-prescribers). mhGAP attitudes, MICA, and SDS were analyzed as sum scores, with missing values replaced by item means. ENACT sum scores and individual item scores were calculated. Pearson correlations and linear regressions were used to explore associations among treatment outcomes. All analyses were performed with SPSS (SPSS Inc., 2013).

RESULTS

Across the three countries, 206 health workers participated in the first training and at least one assessment point (Uganda, n=71; Liberia, n=91; Nepal, n=44), see Table 4. The majority of health workers were men, with an average age of 37. Few of the health workers had prior training in mental healthcare.

Table 4.

Description of training participants

Participant Characteristics		Uganda	Liberia	Nepal
Health Workers	Prescribers	41	41	29
	Non-prescribers	30	45	15
Gender of Health Workers	Female	38	20	18
	Male	33	71	26
Age	Mean (Standard Deviation)	37.39 (8.14)	36.02 (8.60)	37.82 (8.70)
Treatment ArmA	Comprehensive care	59	29	31
	Primary care only	12	62	13
Training Batches	Number of separate trainings	2	3	3
Health Posts	Number involved in program	12	41	16
Prior mental health training	Any prior training	16	2	11

^AThe trainings were part of a larger study comparing mhGAP-based services limited to primary care settings versus mhGAP plus a community comprehensive care program. Because the training content for mhGAP was the same in both arms and the outcomes did not differ, all subsequent analyses include both treatment arm training participants.

To evaluate the change in knowledge, attitudes, and competency over time, we compared the pre-training and pre-refresher scores, with the exception of Liberia where we used the post-refresher scores because pre-refreshers scores had dropped to baseline levels during the Ebola outbreak (Table 5). Among 206 health workers trained in the three countries, knowledge increased from 50% to 68% (effect size, d=1.14) and stigma and other negative attitudes were reduced (mhGAP attitudes, d=−0.35; MICA, d=−0.64; SDS, d=−0.31). Clinical competency using standardized role plays also demonstrated a strong improvement (ENACT, d=1.68). We found that knowledge significantly improved from pre-training to pre-refresher in Uganda and Nepal with effect sizes greater than 1.00 (Figure 2a). However, in Liberia pre-training to post-refresher knowledge did not significantly increase. Similarly, mhGAP negative attitudes were reduced in Uganda and Nepal, but not Liberia (Figure 2b). The MICA improved in all sites, but Social Distances only improved in Uganda (Figure 2c, 2d). The ENACT demonstrated improved competency from pre-training to follow-up points in Nepal and Liberia (Figure 3). The ENACT was not used at pre-training in Uganda, and it is showed no change in post-training to pre-refresher in that country.

Table 5.

Changes in knowledge, attitudes, and competency

	Pre-training (T0)			Post-training (T1)			Pre-refresher (T2)			Comparison Pre-training (T0) to Pre-refresher (T2)A
	nB	Mean	SD	n	Mean	SD	n	Mean	SD	nC	Paired t-test	p	d
mhGAP Knowledge
Uganda	63	0.53	0.18	62	0.65	0.17	38	0.65	0.16	31	5.88	<.001	1.06
Liberia	82	0.45	0.18	84	0.59	0.16	17	0.51	0.12	16	2.09	.05	.52
Nepal	44	0.55	0.16	44	0.68	0.13	35	0.79	0.09	35	10.37	<.001	1.75
Total	189	0.50	0.18	190	0.63	0.16	90	0.68	0.16	82	10.31	<.001	1.14
mhGAP Attitudes
Uganda	62	48.12	10.63	62	40.04	9.73	38	42.34	10.91	31	−3.58	.001	−.64
Liberia	78	49.33	9.89	86	43.80	10.58	21	49.81	12.23	17	−.58	.57	.14
Nepal	44	44.95	8.04	44	36.69	7.88	35	40.36	7.69	35	−2.46	.02	−.42
Total	184	47.87	9.85	192	40.96	10.11	94	43.28	10.69	83	−3.16	.002	−.35
MICA
Uganda	63	45.59	9.85	62	44.48	13.48	38	42.79	12.02	31	−4.45	<.001	−.80
Liberia	85	49.37	11.53	88	43.87	12.36	21	45.21	9.70	19	−3.08	.007	−.71
Nepal	43	45.10	9.39	44	36.86	7.51	35	38.47	10.50	35	−3.25	.003	−.55
Total	191	47.16	10.67	194	42.47	12.17	94	41.72	11.19	85	−5.89	<.001	−.64
Social Distance
Uganda	62	21.01	5.89	62	18.43	6.26	38	18.36	5.98	30	−4.65	<.001	−.85
Liberia	85	20.49	6.35	88	17.17	6.24	21	19.03	7.97	19	−1.65	.12	−.38
Nepal	29	22.44	6.92	29	26.66	8.76	20	23.57	6.95	20	.60	.56	.13
Total	176	21.00	6.29	179	19.14	7.47	79	19.86	7.06	69	−2.61	.01	−.31
ENACT
Uganda				20	36.50	7.03	17	39.71	6.03	13D	.65	.53	.18
Liberia	23	25.39	3.07	26	34.35	3.74	9	32.22	6.57	7E	4.40	.005	1.66
Nepal	31	23.29	2.78	30	31.53	4.68				30F	7.87	<.001	1.44
Total	54	24.19	3.07	76	33.80	5.46	26	37.12	7.09	9G	1.92	.09	.64

^AAll comparisons are between pre-training (T0) and pre-refresher (T2). An additional assessment was completed in Liberia post-refresher (T3) because of disruption of services and supervision due to Ebola. To standard comparisons, we use T2 for all sites here. Note exceptions for ENACT comparison time points described below.

^BSample size (n) refers to number of participants at the assessment point.

^CSample size (n) for paired t-test refers to participants completing assessments both pre-training (T0) and pre-refresher (T1)

^DENACT comparison for Uganda is between post-training (T1) and pre-refresher (T2).

^EENACT comparison for Liberia is between pre-training (T0) and post-refresher (T2) because this was the final time point of competency assessment.

^FENACT comparison for Nepal is between pre-training (T0) and post-training (T1) because no later assessment was completed.

^GENACT total sample for T0 and T2 paired comparison only includes Liberia because other countries did not assess competency at both of these time points.

Figure 2.

(a) Mental health knowledge among health workers pre-training and post-training and pre-refresher; a post-refresher assessment was conducted in Liberia because of interruption of supervision due to the Ebola virus disease outbreak. Mental health knowledge assessed with an adapted version of mhGAP Knowledge test (mhGAP, version 1). Scores are percentage correct. (b) Attitudes among health workers were assessed with an adapted version of mhGAP Attitudes test (mhGAP, version 1). Higher scores reflect greater stigma. (c) Clinical attitudes related to caring for persons with mental illness were assessed with the Mental Illness: Clinicians’ Attitudes (MICA) scale completed by health workers. Higher scores reflect greater stigma, i.e. more negative attitudes. (d) Willingness to interact with persons with mental illness in personal, communal, and professional settings was assessed with the adapted version of the Social Distance Scale (SDS) completed by health workers. Higher scores reflect greater social distance, i.e. more stigma.

Figure 3.

Clinical competency was assessed using standardized role plays that were scored with the Enhancing Assessment of Common Therapeutic Factors (ENACT). A higher score represents greater competency; scale range ¼ 18–54. Role plays were administered at different times depending on availability of standardized clinical raters. In Uganda, ENACT role plays were conducted post-training and pre-refresher. In Liberia, ENACT role plays were conducted at all assessment time points. In Nepal, ENACT role plays were conducted only pre- and post-training.

In Uganda, at the final assessment when ENACT scores were collected, the mean competency (i.e., achieving 2 or 3 on an item) was 71% (SD=17%, n=24). In Liberia, competency increased from 35% at pre-training (SD=13%, n=22) to 66% (SD=16%, n=28; t=7.36, p<0.001). In Nepal, competency increased from 26% (SD=13%, n=30) to 60% (SD=18%, n=31; t=8.38, p<0.001). In Figure 4, we present the ENACT single item performance. Prior to training, only three items were performed at Level 2 or 3 by at least 60% of primary care workers: non-verbal communication, verbal communication, and eliciting patient’s feedback. For the endline measurements, an additional six items were above the 60% mark. These included rapport building, normalization of feelings, demonstrating empathy, psychoeducation appropriate for the cultural context, providing realist hope, and assessing recent life events. However, half of the 18 items on ENACT were not adequately performed by 60% of the participants. These items were promoting prior coping, assessing functioning, collaborative goal settings, assessing other physical and mental health factors, use of problem-solving skills, explaining confidentiality, involving family, assessing suicidality and conducting safety planning, and incorporating the patient’s explanatory (causal) model.

Figure 4.

Percentage competency on ENACT items at pre-training baseline and endline.

We compared the bivariate correlations among outcomes pre-training and for the endline assessment (see Table 6). At baseline, mhGAP Attitudes and MICA scores were negatively correlated with mhGAP Knowledge. Social distance was positive correlated with mhGAP Attitudes and MICA scores, but not with knowledge. No outcomes were correlated with ENACT competency scores. For endline assessments, knowledge and attitude measures followed the same pattern of associations with the strength of associations generally increasing. At endline, ENACT scores were negatively correlated with Social Distance, i.e., lower Social Distance associated with greater clinical competency in role plays. In a univariate general linear model with all outcomes at endline included, Social Distance remained significantly associated with ENACT scores (β=−0.19, 95% CI=−0.34 to −0.05, p=0.01). When comparing Social Distance with individual items on ENACT at endline and including other outcomes, Social Distance was significantly associated with reduced performance on assessing life events (ENACT #7, F=5.53, p=0.02), assessing other physical and mental health issues (ENACT #8, F=5.45, p=0.02), exploring patient’s explanatory/causal models (ENACT #11, ANOVA F=7.07, p=0.01), and involving family members (ENACT #12, F=12.12, p=0.001).

Table 6.

Correlations among knowledge, attitudes, and competence

Pre-training assessment	mhGAP Attitudes (n=187)	MICA (n=187)	Social Distance (n=175)	ENACT (n=52)
mhGAP Knowledge	−0.36**	−0.49**	−0.13	−0.01
mhGAP Attitudes		0.42**	0.27**	0.06
MICA			0.27**	−0.06
Social Distance				−0.08
Endline assessment	mhGAP Attitudes (n=90)	MICA (n=94)	Social Distance (n=79)	ENACT (n=83)
mhGAP Knowledge	−0.43**	−0.41**	−0.05	0.14
mhGAP Attitudes		0.56**	0.42**	−0.03
MICA			0.46**	−0.07
Social Distance				−0.30**

^**p<0.01

DISCUSSION

In the context of primary care and community-based mental health service integration conducted in three low-income countries affected by humanitarian crises, we evaluated changes in knowledge, attitudes, and competency. We found a general improvement over the course of training and supervision. Among 206 health workers trained in the three countries, knowledge increased from 50% to 68% and stigma and other negative attitudes were reduced. A strength of our evaluation program was that we also evaluated clinical competency using standardized role plays, which also demonstrated a strong improvement.

However, when examining overall performance after training, health workers in Uganda were only competent on 71% of items, and performance was lower in Liberia (66%) and Nepal (60%). Key competencies such as assessing suicidality, involving family members in care, explaining confidentiality, and asking questions about physical health and other mental health issues were not performed by more than half of trained primary care workers. This is despite include the mhGAP suicide module, and sessions on confidentiality, family involvement in care management and follow-up, and disorder specific guidelines on ruling out physical health problems, substance use, and other health conditions. This illustrates the importance of integrating competency evaluations to identify weaknesses in training and supervision curricula and implementation.

Although performance on the mhGAP Knowledge questions was associated at both pre-training and endline with self-reported attitudes on the mhGAP Attitude test and MICA, it was not associated with Social Distance or ENACT. The only measure associated with ENACT scores, was Social Distance at endline. This suggests a relationship between greater willingness to engage with persons with mental illness and clinical performance with standardized patients. In particular, lower overall Social Distance was associated with greater discussion of family engagement, assessment of physical health, discussion of life events, and exploring the patient’s explanatory/causal model. We cannot speak to the direction of causality in this relationship, but it raises questions about whether greater willingness to interact with people with MNS disorders is reflected in greater interest in their personal and family lives, as well as physical health. It is important to note that measures of stigma specific to health professionals (i.e., mhGAP Attitudes and MICA) were not associated with clinical competence, whereas the more general attitudinal measure of willingness to interact, which is typically used with the general public and not specific to health workers, did associate with ENACT scores.

These findings raise questions about what targets and level of improvement are needed when training primary care workers to achieve the greatest likelihood that they will provide ethical and effective care. A modest improvement in knowledge of a few percentage points may be adequate when health workers begin with a strong base of mental health concepts. For example, in other studies with mhGAP evaluations, most health workers started with knowledge on mhGAP assessment between the range of 60–80% with most in the mid-70s, and then showed modest increases of about five to 15%, ultimately achieving mean scores in the 75–90% range (Ekore, Ajuwon, Abdulmalik, & Bella-Awusah, 2016; Humayun et al., 2017; Siriwardhana, Adikari, Jayaweera, Abeyrathna, & Sumathipala, 2016). This modest improvement may be sufficient for safe and effective mental health care delivery. However, our study demonstrated outcomes considerably lower than most of the published studies with average knowledge across Uganda, Liberia, and Nepal of 50% at baseline and 68% at endline. Only one out of four health workers in Uganda and one out of five in Liberia were above 75% knowledge by their final assessments. In Nepal, where the training was nearly twice as long as in the other countries (9 days in Nepal vs 5 days in Uganda and Liberia), baseline knowledge was comparable, but seven out of ten health workers achieved 75% knowledge by their final assessment.

Clinical competency is conceptually the most closely related to actual performance because role plays simulate deployment of knowledge, attitudes, and skills in practice. We observed a lack of connection between multiple-choice knowledge assessment and clinical skills in the ENACT role plays. This may reflect the inadequacy of knowledge tests to predict competency and quality. In Nigeria, although overall knowledge increased, diagnosis of a vignette was only accurate for one out of ten cases in the post-test evaluation (AdebowaleT, Richard Gater, Akinhanmi, & Ogunlesi, 2014). Similarly, in Ethiopia, Kenya, and Malawi, when health workers demonstrated improved knowledge and attitudes, their clinical practice remained inadequate, as evidenced by low rates of detection of MNS disorders (Fekadu, et al., 2017; Jenkins, et al., 2013; Kauye, et al., 2014).

Relationship of training and supervision outcomes with clinical patient outcomes

To accurately determine minimum standards for training and supervision outcomes, it is crucial to connect these evaluations with actual patient outcomes. To date, patient outcomes from mhBeF only have been published for the Nepal site (Jordans, Aldridge, Luitel, Baingana, & Kohrt, 2017a). The Nepal patient cohort included 119 patients diagnosed with epilepsy and 85 diagnosis with psychosis. The number of seizures decreased in the epilepsy cohort. The severity of psychotic symptoms and social functioning improved in the psychosis cohort. Across both cohorts, there were reductions in functional disability, family and caregiver burden, and depressive symptoms. This illustrates that clinical benefits were achieved although providers had shortcomings in knowledge, attitudes, and competency.

Future analyses of mhBeF will need to explore patient outcomes in Uganda and Liberia, as well as compare variation in health worker training and supervision outcomes with clinical outcomes from their health facilities both within and between countries. In addition, we will need to explore other outcomes in mhBeF such as the health facility and community anti-stigma programs and the patient support group benefits.

Limitations

The current study has a number of limitations which should be considered when interpreting the findings or making recommendations for the broader field of mental health service integration in primary care facilities in low-resource and humanitarian settings. First, the study design and analysis were in the context of a proof-of-concept implemention of a comprehensive community-based mental health service in multiple post-humanitarian crisis settings. The study was not powered for training outcomes. Future studies with hypothesis testing should take into account factors such as clustering at the health facility, regional, and country level as that impacts comparisons across countries and general outcome results. More standardized sample sizes and comparison groups would be needed for the three countries. Similarly, the assessment schedules would need to be uniform in the three sites.

Second, although the ability to tailor the training and mhBeF package in each country was a strength, it is also a weakness with regard to identifying the active ingredients in the training and supervision. The duration of training, trainers’ backgrounds, anti-stigma content, and a host of other factors were tailored to the country. In addition, it is difficult to attribute continued improvement or lack thereof to the supervision style, the ongoing anti-stigma activities, the collaboration with other health workers or community stakeholders, or other elements of mhBeF or the local implementation context.

Third, with regard to judging clinical competency, the scores on ENACT are dependent upon the raters who observe the role play. In Nepal, where the ENACT was developed, there was extensive attention to achieving interrater reliability and appropriately scoring with the tool. Similarly, in Liberia, there was practice with raters in Monrovia prior to using the tool Sinoe county. In Uganda, there was less preparation for using the tool, and the raters may have scored the participants higher than if the trainees had been rated by the Nepal or Liberia raters. In the future, for multi-country studies, attempts should be made to evaluate interrater reliability across sites when possible.

Implications and recommendations

Despite these limitations, the current study yields a number of implications for future work in the field of training and supervising primary care health workers and other non-specialists for integration of mental health services. Since the completion of this study in 2015, there have also been two major developments that complement the recommendations we provide below. First, in 2017 WHO released new guidance on the training and competencies for mhGAP (World Health Organization, 2017). A comprehensive 656-page document provides rich details on training, supervision, and issues related to competence. Second, in 2018 the International Medical Corps (IMC), an international NGO globally engaged in mental health and psychosocial support programs in humanitarian settings, released guidance on integration of mental health in primary care in humanitarian settings (International Medical Corps, 2018). We take these two recent developments into account when discussing our recommendations below:

Recommendation 1. Adopt standardized measures of knowledge and attitudes to evaluate training and supervision.

Given that mhGAP provides global guidance for content and implementation, it would be beneficial to have overlap in implementation (including training and supervision) and effectiveness measures. Whereas there are commonly used symptom and functioning tools such as the Patient Health Questionnaire (PHQ-9), Alcohol Use Disorder Identification Test (AUDIT), and WHO Disability Assessment Scale (WHODAS), there is a lack of commonality in training and supervision measures. The mhGAP version 2.0 has a revised series of questions for evaluating each module. The new mhGAP training materials provide guidance on evaluation, as does the IMC Toolkit (International Medical Corps, 2018; World Health Organization, 2017). If this became the standard knowledge assessment tool, then outcomes could be compared within countries across trainings and between countries. For attitudes, there is a range of self-report measures as well as growing incorporation of implicit association measures that do not suffer from social desirability bias in scoring (Kohrt et al., 2018). If more work can be conducted to determine which aspects of stigma and negative attitudes or implicit bias best predict quality of care, then there should be an effort to globally adopt those tools. Ideally, there would be specific and standardized measures for all of the 12 core competencies included in mhGAP training guidance (see Table 1).

Recommendation 2. Include measures of competency using observed structured clinical evaluations, in addition to assessing knowledge and attitudes.

For the field of competency, standardized role plays with observer rating scales are the gold standard in psychological treatments and can be used similarly for combined pharmacological and psychological management. The ENACT tool as described in this study was feasibly implemented at numerous time points and demonstrated improvement over time consistent with what would be excepted. If interrater reliability across raters and sites could be improved, this would be an ideal approach for comparing competency and determining minimum competency standards. WHO training guidance now includes a series of role plays for each disorder and checklists for what competencies should be observed (World Health Organization, 2017). In addition, WHO is currently involved in developing a suite of ENACT-type tools and usage guidance for psychological treatments (Ensuring Quality in Psychological Support, EQUIP), and this may have major benefit for mhGAP trainings as well. The IMC Toolkit advocates a similar approach with a 5-point Likert scale in which minimum competence is defined as scoring 4 or greater on a 7-item scale (International Medical Corps, 2018).

Recommendation 3. Training and supervision should move away from total hours or days as targets to competency-based targets.

Currently, most governments and (I)NGOs work with the generic mhGAP curriculum of approximately 5 days of training. The recent mhGAP training guidelines point out that this is a suggested benchmark and that local implementation should work toward achieving the appropriate skill level through both the training and supervision (World Health Organization, 2017). Similarly, governments and (I)NGOs should explore how much training and supervision is needed to meet competency and quality benchmarks. If 5-days is insufficient to reach these goals, then it is irresponsible and potentially harmful of governments to advocate for a 5-day curriculum. For example, in the current study, we found that a 9-day training in Nepal led to 71% of health workers scoring 75% or above on the knowledge test at endline, but the average clinical competency in role plays was only 60%. However, despite being presented with this and evidence from clinical outcome evaluations (Jordans, et al., 2017a; Jordans et al., 2019), the new mental health guidelines call for a 6-day training that includes the current disorders as well as modules for dementia, child and adolescent disorders, and anxiety (Primary Health Care Revitalization Division Department of Health Services, 2017). Moreover, there limited information on the type and frequency of supervision to address these added training demands in a shorter time. Organizations that generate research on mhGAP and similar initiatives should assure that their findings are clearly communicated to government policy makers to avoid programs that are costly, clinically ineffective, and potentially harmful to patients.

CONCLUSION

In Uganda, Liberia, and Nepal, the mhGAP plus comprehensive community mental health services training led to significant improvements in knowledge, attitudes, and competence sustained 1-year after trainings. However, the primary care workers achieved only 60–70% average competency, with gaps in important areas related to suicide assessment, assessing physical health, and involving family. Potential avenues to implement training and supervision programs assuring minimum competency include adopting standardized measures of knowledge and attitudes to evaluate training and supervision; including measures of competency using standardize role plays to compare participants within trainings and across trainings; and shifting to training and supervision designs based on competency milestones rather than total hours or days of training. These will be important steps to assure effective integration of mental health services in primary care and community settings around the world.

Table 3.

Description of health worker trainings

Program Characteristics	Uganda	Liberia	Nepal
Districts	Lira	Sinoe, Grand Kru, River Gee	Pyuthan
Implementing organizations	TPO Uganda; Makerere School of Public Health; Butabika National Referral Psychiatric Hospital; government health system	The Carter Center; government health system	TPO Nepal; government health system
Trainers	Ugandan psychiatrists; NGO staff	Ugandan psychiatrist; NGO staff	Nepali psychiatrists, NGO staff
Supervisors	Local Ugandan Psychiatric Clinical Officers; Ugandan psychiatrists; NGO staff	Liberian 6-month trained Mental Health Clinicians	Nepali psychiatrist; NGO staff
Trainees: Health Workers-PrescribersA	Primary care health workers	Primary care health workers	Primary care health workers
Trainees: Health Workers-Non-Prescribers	Village Health Team members	General Community Health Volunteers	Auxiliary Nurse Midwives; Auxiliary Health Workers
Participation of Mental Health Service Users	1 local service user and a family member	3 members of Monrovia-based service user group, and 9 local service users	2 members of Kathmandu-based service user advocacy group
Duration of training	5-days combined training for all participants	5-days combined training for all participants	9-days for prescribers; 5 days for non-prescribers

^AHealth worker cadre refers to non-prescribing health workers who may include community health workers, auxiliary nurse midwives and other health community resource persons who do prescribe medications vs. prescribing health workers who are predominantly facility based and have prescribing authority.