Evaluating capacity-building for mental health system strengthening in low- and middle-income countries for service users and caregivers, service planners and researchers

Abstract

Efforts to support the scale-up of integrated mental health care in low- and middle-income countries (LMICs) need to focus on building human resource capacity in health system strengthening, as well as in the direct provision of mental health care. In a companion editorial, we describe a range of capacity-building activities that are being implemented by a multi-country research consortium (Emerald: Emerging mental health systems in low- and middle-income countries) for (1) service users and caregivers, (2) service planners and policy-makers and (3) researchers in six LMICs (Ethiopia, India, Nepal, Nigeria, South Africa and Uganda). In this paper, we focus on the methodology being used to evaluate the impact of capacity-building in these three target groups. We first review the evidence base for approaches to evaluation of capacity-building, highlighting the gaps in this area. We then describe the adaptation of best practice for the Emerald capacity-building evaluation. The resulting mixed method evaluation framework was tailored to each target group and to each country context. We identified a need to expand the evidence base on indicators of successful capacity-building across the different target groups. To address this, we developed an evaluation plan to measure the adequacy and usefulness of quantitative capacity-building indicators when compared with qualitative evaluation. We argue that evaluation needs to be an integral part of capacity-building activities and that expertise needs to be built in methods of evaluation. The Emerald evaluation provides a potential model for capacity-building evaluation across key stakeholder groups and promises to extend understanding of useful indicators of success.

Capacity-building for health system strengthening in global mental health

There has been a strong emphasis on building the capacity of general health workers to deliver mental health care in low- and middle-income countries (LMICs), with the development of evidence-based treatment guidelines (World Health Organization, 2016b) and an expanding evidence base of the effectiveness of such capacity-building approaches (van Ginneken et al. 2013). At the same time, there has been increasing awareness of the need to also strengthen the health system in order to improve access to, and the quality of, mental health care (Petersen et al. 2017). In addition to healthcare providers, three crucially important stakeholder groups for health system strengthening are: service users and caregivers, service planners and policy-makers, and researchers. In a companion editorial we describe a range of activities that are being implemented by the Emerald programme (Emerging mental health systems in LMICs: http://www.emerald-project.eu) (Semrau et al. 2015) to build capacity among these three key target groups to support system strengthening for scale-up of mental health. In this editorial, we discuss approaches to evaluating the impact of these capacity-building activities to ensure that they achieve their intended goals.

Evaluation of capacity-building: what is best practice?

Mental health service users and caregivers

In a systematic review of the literature, we identified several initiatives to increase the involvement of service users and caregivers in activities to strengthen the mental health system in LMICs, for example, in the areas of advocacy, quality control, training of health workers, policy development, service planning and research evaluation (Semrau et al. 2016). However, only four of the identified studies included an explicit evaluation of service user/caregiver involvement, and the methodological quality was considered to be low in most cases. Most evaluations focused on the impact of service user and/or caregiver involvement in the development of national level policies and plans, using qualitative methods or mixed qualitative-quantitative studies with service users, caregivers and service user representatives involved as subjects of research. An ecological study design examined the association between service user and/or caregiver involvement in development of mental health legislation and access to psychotropic medications, but such an approach is vulnerable to ecological bias (McBain et al. 2012). The other main group of evaluations focused on measurement of satisfaction with services, again with service users as subjects of research. Overall, there was a weak evidence base on the most effective models of building capacity of service users and caregivers to support involvement in mental health system strengthening in LMICs (Semrau et al. 2016). This is a missed opportunity because of the current global push to scale up person-centred and co-produced mental health services (World Health Organization, 2016a). With better evidence, powerful arguments can be made to ensure that building the capacity of service users and caregivers to enable greater involvement in system issues (e.g., service development, monitoring and advocacy) become part and parcel of scale-up efforts.

Evidence from high-income country settings supports the use of participatory research with service users and caregivers for studies relating to mental health interventions, services and systems (Rose, 2014), with limited examples from LMICs (Hann et al. 2015). Potential benefits of involvement of service users and caregivers in the process of evaluation of capacity-building include ensuring that the goals of capacity-building are relevant and aligned with priorities, and that service users and caregivers are empowered through the process of co-production of care and knowledge. This leads to a greater chance of longer-term impact and sustainability of capacity-building efforts (Thornicroft & Tansella, 2005; Rose, 2014).

Service planners and policy-makers

A further systematic review was conducted to synthesise knowledge about evaluation of efforts to build the capacity of policy-makers and planners to strengthen the mental health system (Keynejad et al. 2016). Rigorous evaluation was conducted in only a few of the identified studies. Evaluation approaches usually employed mixed methods. Lower quality evaluations were descriptive and not guided by any conceptual framework, whereas higher quality studies sought to combine various data sources, with one study making use of both an evaluator who was external to the programme, as well as an ‘insider’ evaluation (Patton, 1997). Many of the capacity-building interventions for this target group involved long-term engagement and mentoring, with emphasis placed upon the need to develop sustainable, good quality relationships. Evaluation strategies need to incorporate these important indicators of a successful intervention, as well as capturing impacts on the health system.

Mental health researchers

Best practice guidelines for indicators of successful capacity-building for researchers in LMICs have been published (TDR/World Health Organization, 2011, 2016) and are better developed than the frameworks for evaluating capacity-building for other target groups. The ESSENCE framework recommends considering the impact of research capacity-building at the individual, organisational and sub-national or national levels, with emphasis given to understanding the country-specific relationships between these levels (TDR/World Health Organization, 2011). The selected indicators for measuring capacity-building success include a focus on publications and grants. The importance of publication as a hard outcome of capacity-building success has been echoed by others (Zachariah et al. 2013; Kohrt et al. 2014). The range of data sources that may be relevant and acceptable for evaluation of research capacity-building include: annual reports, mid-term and final interviews, publications, citation index, grant agreements, certificates of training and documentation of personal interactions (TDR/World Health Organization, 2011). A need has been identified for existing evaluative frameworks for capacity-building to be applied, tested and adapted for LMIC settings (Thornicroft et al. 2012).

Evaluation framework for the Emerald target groups

The Emerald capacity-building interventions are summarised in Table 1.

Table 1.

The Emerald programme capacity-building interventions

Target groups	Interventions for mental health system strengthening
Mental health service users and caregivers	•Country-specific, multi-faceted approach •Workshops for service users and caregivers to raise awareness and mobilise for greater advocacy and involvement •Workshops for primary care workers and managers to support greater involvement of service users •Ph.D. linked interventions to develop and pilot models of service user involvement in mental health system strengthening
Policy-makers and planners	•Workshops focused on mental health awareness-raising, the chronic care model and mental health system planning •Ongoing dialogue and developing collaborations with the aim of providing technical support and increasing capacity over time
Mental health researchers	•Short-courses in health systems research, implementation science and service user participation in research (co-developed with service user representatives and a non-governmental organisation http://www.basicneeds.org) •Nine Ph.D. students linked to Emerald (Ethiopia n = 3, India n = 2, Nepal n = 1, Nigeria n = 1, South Africa n = 1, UK n = 1), 7 of whom have external supervisors from other Emerald academic institutions. •Development and adaptation of 27 modules on mental health systems for integration within Masters programmes •Fellowships for the MSc in Global Mental Health at King's College London and London School of Hygiene and Tropical Medicine •Mentoring of post-doctoral and mid-level mental health researchers •Participation in the Emerald consortium meetings, with opportunities for mid-level researchers and doctoral students to present their work and participate in writing workshops

The guiding principles of capacity-building in the Emerald consortium were appropriateness, reciprocity and sustainability. These principles were based on recognition of the differing baseline contexts, capabilities and unmet needs of Emerald partners, the bi-directional flow of expertise in north-south partnerships and the imperative to work towards self-sufficiency in LMIC partner organisations.

A mixed quantitative and qualitative evaluation framework was developed for each target group, based on established best practice and the needs assessments conducted for each target group (Semrau et al. 2017), and modified by the capacity-building goals, the nature of the specific interventions and the country context. The cross-country quantitative indicators of Emerald capacity-building success identified for each of the target groups are presented in Table 2. The evaluation framework will now be discussed in relation to the target groups.

Table 2.

Emerald cross-country indicators for capacity-building of mental health service users and caregivers

Cross-country indicators		Evaluation method	Stakeholder group
Process	% of participants starting short course training who completed the full training	Emerald country team records	All
	% of workshop participants who were women and located outside of the capital city	Emerald country team records	All
	Number of workshops run per Emerald LMIC	Emerald country team records	All
	% of participants satisfied with the course	Interviewer-administered satisfaction questionnaire	Service users and caregivers
		Self-completed, semi-structured satisfaction questionnaire	Planners and researchers
	% of workshop participants who were employed in public institutions	Emerald country team records	Researchers
Output	Increase in understanding of service user involvement in workshop participants	Interviewer-administered, pre-post questionnaire developed for Emerald	Service users and caregivers
	Increase in understanding of mental health systems, planning and chronic care models	Self-completed, multi-choice questionnaire developed for Emerald	Planners
	Improvement in attitudes in workshop participants	World Psychiatric Association Stigma Toolkit Attitudes questionnaire (Stuart & Arboleda-Florez, 2001)	Planners
	Number of meetings between policy-makers/planners and the Emerald team post-training	Emerald country team recording of number and nature of meetings	Planners
	Improvement of knowledge in short course participants	Pre-post knowledge tests for each of the short courses	Researchers
	Number (%) of Emerald Ph.D. students graduating within 4 years of registration	Emerald country team records	Researchers
Outcome	Number of workshops run per Emerald LAMIC without external assistance	Emerald country team records	All
	Number of workshops run by service users	Emerald country team records	Service users and caregivers
	Change in unmet needs for capacity-building in mental health systems	Repeat needs assessment amongst key informants	Planners
	Number of applicants per course/number of places available	Emerald country team records	Researchers
	Change in unmet needs at the organisational level for capacity-building in mental health systems research	Baseline and follow-up needs assessment and situation analysis	Researchers
	% of all Emerald publications where Emerald fellow (Ph.D. or MSc) is first author	Searches of Pubmed and Google Scholar	Researchers
	% of all Emerald publications where LMIC partner is first author	Searches of Pubmed and Google Scholar	Researchers
	Number of health system-related publications by short course participants	Searches of Pubmed and Google Scholar	Researchers
	Number of Emerald-linked Ph.D. and mid-level researchers applying for grants	Self-report, online questionnaires	Researchers
	Number of Emerald-linked researchers engaged in mental health research by end of project	Self-report, online questionnaires (Ph.D. and mid-level researchers) Follow-up self-report form for short course participants (12 months post-course)	Researchers
	Number of health systems/implementation science/service user involvement projects undertaken by short course trainees by end of Emerald	Self-report, online questionnaires (Ph.D. and mid-level researchers) Follow-up self-report form for short course participants (12 months post-course)	Researchers
	Number of health system-related conference presentations by short course participants and Ph.D. students	Follow-up self-report form for short course participants (12 months post-course)	Researchers

Emerald evaluation of capacity-building for service users and caregivers

Low levels of literacy in many of the Emerald country sites meant that it was not possible to use self-completed questionnaires to assess the indicators of success. As well as the cost implications of interviewer-administered questionnaires, there was also concern that an interview format would lead to social desirability bias, for example, when measuring satisfaction with the capacity-building workshop. Another important adaptation for this target group was to evaluate change in understanding, e.g., ‘I understand about types of mental health problems’ (strongly disagree/disagree/don't know/agree/strongly agree), rather than change in knowledge.

Qualitative in-depth interviews were conducted at baseline to identify priority goals for service user and caregiver interventions and capacity-building needs (Samudre et al. 2016; Abayneh et al. 2017; Gurung et al. 2017). Follow-up qualitative interviews are planned in order to explore perceptions of the impact of Emerald capacity-building upon service user and caregiver involvement in mental health system strengthening, the level of empowerment and mobilisation, the experience of participation in capacity-building, perceived limitations to the capacity-building approach and recommendations on how to improve capacity-building efforts.

Two of the Emerald-linked Ph.D. students (in Ethiopia and India) focused their research on service user involvement in mental health system strengthening. Although not possible at the cross-country level, the Ph.D. students are using participatory research methods to develop, pilot and evaluate models of service user involvement, with the evaluation of capacity-building as a nested component. See Table 3.

Table 3.

Service user involvement in Ethiopia: participatory research for mental health systems strengthening

Sisay Abayneh: Ph.D. student in Mental Health Epidemiology, Addis Ababa University

This Ph.D. study is located in Sodo district, south central Ethiopia, where the Programme for Improving Mental health carE is supporting the implementation of integrated mental health care (Fekadu et al. 2016). A Theory of Change framework is being employed to engage all relevant stakeholders in identifying the desired outcomes of the programme and the pathways to the achieving these outcomes, as well as indicators of success. Service users, caregivers, service providers and health facility managers will work collaboratively with researchers to oversee the study design and conduct, and the analysis and interpretation of findings. The purpose is to ensure a participatory approach and to facilitate a common understanding amongst the stakeholders.

Emerald evaluation of capacity-building for service planners and policy-makers

The hierarchical nature of relationships with policy-makers or service planners and the research team were apparent across the Emerald partner countries, which affected the nature of the evaluation that was appropriate and possible. As with service users and caregivers, tests of knowledge were considered to be unacceptable for service planners and policy-makers who participated in the short courses and were replaced by pre-post questionnaires examining understanding of mental health systems. Attitudinal change was felt to be of paramount importance in this target group, but also a sensitive area and so attitudinal measures were not used in several of the Emerald countries.

Questions exploring the extent to which organisational capacity-building needs had been met were nested within in-depth interviews being conducted with key informant planners and policy-makers for other objectives of the Emerald project (Petersen et al. 2017).

Emerald evaluation of capacity-building for mental health researchers

The Emerald cross-country indicators for research capacity-building were adapted from the ESSENCE framework (TDR/World Health Organization, 2011). An important indicator of equity and sustainability for research capacity-building was the percentage of course participants who were working in public sector institutions. From the inception of Emerald, each of the country teams emphasised the importance of becoming self-sufficient in delivery of short courses and so an indicator was included to capture the number of courses delivered without external assistance. In the revised ESSENCE framework (TDR/World Health Organization, 2016), corresponding authorship by LMIC partners was emphasised, but for Emerald-linked Ph.D. students first authorship was considered to be essential, and in some of the Emerald LMIC institutions corresponding authorship had no value in terms of professional recognition and promotion opportunities. The indicators for obtaining grants were expanded to measure involvement in system-related projects, which may not have required external funding but provided an indicator of local uptake of the training.

The measurement of quantitative indicators was supplemented by qualitative in-depth interviews with key informants from the Emerald LMIC research partner institutions as well as with participants in the short courses. The interview topic guides explored the extent to which Emerald had contributed to successful capacity-building in mental health systems research, and what could have made the capacity-building efforts more successful. Perspectives on the experience of being part of a multi-country research consortium were obtained from Ph.D. fellows and mid-level researchers through an anonymous online survey. This included feedback on the experience of annual meetings, the extent and usefulness of opportunities to present their work and receive feedback from other consortium members, and the opportunity to be part of a network of Ph.D. researchers working in the area of mental health systems.

Evaluating the adequacy of quantitative indicators of capacity-building success

Within Emerald, an evaluation of the adequacy of quantitative indicators in capturing capacity-building success is being conducted in relation to qualitative exploration. The analysis will include a focus on discrepant cases, for example, apparent low success of capacity-building on the basis of quantitative evaluation but high success identified through the qualitative study, or vice versa. The qualitative study will also probe explicitly around the adequacy of the quantitative indicators in capturing the benefits and limitations of the capacity-building activities from the perspectives of participants and key informants. After reviewing the findings, the Emerald consortium will come to an expert consensus on which indicators can be recommended as capturing important aspects of capacity-building success in this area.

Lessons learned and future directions

Capacity-building to strengthen mental health systems is a complex intervention. In the revised ESSENCE framework for research capacity-building (TDR/World Health Organization, 2016), use of a theoretical framework, for example Theory of Change (De Silva et al. 2014), is recommended to map out the complexity, ensure a participatory approach, guide the choice of indicators and drive evaluation priorities. The Theory of Change approach might be particularly beneficial for evaluation of capacity-building for policy-makers and planners, where policy and service configurations take time to change and upstream indicators of success are needed, as well as for service users and caregivers (Table 3). Although ESSENCE considers evaluation of the system-wide impact of research capacity-building, there is a need to incorporate capacity-building for other target groups to support the attainment of system wide goals.

Although reciprocity was a guiding principle of the Emerald capacity-building activities, this was not measured directly in our evaluation framework, which tended to focus on the capacity built within LMIC partners. Indicators of reciprocity within a research consortium might include measures of the extent to which participating high income country institutions draw on LMIC expertise to develop strategic directions in global health, contributions of LMIC partners to the design and delivery of curricula for Masters programmes in the high-income country partner institutions, student placements in LMIC partner projects and co-supervision of Masters and Ph.D. students. Longer-term outcomes of capacity-building based on reciprocity may include the incorporation of findings from LMIC partner projects into high-income country health systems.

We have tried to capture the potential benefits of participation in a multi-country research consortium per se through the online surveys of Ph.D. fellows and mid-level researchers, as well as through the qualitative interviews with research institution key informants; however, this evaluation approach could be strengthened through examination of the extent of co-authorship of publications between different consortium partners, and use of a satisfaction survey for all consortium members.

Evaluation of capacity-building efforts for mental health system strengthening can learn much from other areas of global health (Amuyunzu-Nyamongo et al. 2013). However, specific competencies are required in the area of mental health (Ng et al. 2016), particularly to achieve genuine participation of mental health service users.

Conclusion

Evidence-based capacity building is needed for mental health system strengthening in LMICs. Evaluation, therefore, needs to be an integral part of capacity-building activities. The field of global mental health is relatively young and there is a need to refine methods for evaluating capacity-building across target groups and to equip researchers to conduct rigorous evaluations. The planned Emerald evaluation described in this editorial provides a potential model for capacity-building evaluation across key stakeholder groups, and promises to extend understanding of useful indicators of success.

Financial support

The research leading to these results is funded by the European Union's Seventh Framework Programme (FP7/2007–2013) under grant agreement no 305968. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Statement of interest

G.T. is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South London at King's College London NHS Foundation Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. G.T. acknowledges financial support from the Department of Health via the National Institute for Health Research (NIHR) Biomedical Research Centre and Dementia Unit awarded to South London and Maudsley NHS Foundation Trust in partnership with King's College London and King's College Hospital NHS Foundation Trust. G.T. is supported by the European Union Seventh Framework Programme (FP7/2007–2013) Emerald project. No other conflicts of interest were declared.