Methods and tools to assess implementation of mental health policies and plans: A systematic review

Abstract

Mental health policies and plans (MHPPs) are important policy instruments and powerful tools to facilitate development of mental health systems and services across the world. We aimed to map and analyse methods and tools used to assess the extent, process and impact of implementing MHPPs. We systematically searched peer-reviewed and grey literature across seven scientific databases. We extracted and analysed the data on a) the characteristics of included studies (e.g., policy areas, region of origin, income setting) and b) the methodology and evaluation tools applied to assess the extent and process of implementation. We included 48 studies in the analyses. Twenty-six of these studies employed only qualitative methods (e.g., semi-structured interviews, focus group discussions, desk review, stakeholder consultations); 12 studies used quantitative methods (e.g., trend analysis, survey) and 10 used mixed-methods approaches. Generally, methods and tools used for assessment were described poorly with less than half of the studies providing partial or full details about them. Only three studies provided assessment of full policies. There is a lack of rigorous research to assess implementation MHPPs. Assessments of the implementation of entire MHPPs are almost non-existent. Strategies to assess the implementation of MHPPs should be an integral part of MHPPs.

Impact statement

This review highlights a lack of both quantity and rigorousness of assessments of implementation of mental health policies and plans as reported across the globe. While studies included in the review often addressed in detail the posed research questions and assessment objectives, they rarely presented clearly their methods and lacked sufficient descriptions of tools used in the evaluation, thus making them hardly interpretable and reproducible in other research contexts. Reports of assessments of entire policies were scarce. Instead, studies largely focused on assessment of certain policy objectives or tried to investigate questions of interest in relation to the implementation process. Thus, the review reveals gaps within implementation science in global mental health and calls for future efforts to better assess the impact of mental health policies and so to enable learning from the lessons made.

Introduction

The need for urgent improvements in mental healthcare systems across the globe has been recognised for a long time (WHO, 2001). The global burden of mental disorders is ever increasing, and the treatment gap still prevails across all income settings. Mental health and well-being have been further compromised by the COVID-19 pandemic, and forthcoming challenges, such as climate change and associated migration and population displacement, are likely to exacerbate the existing burden. Mental health policies and plans (MHPPs) are important policy instruments to spark and concert action for change, yet the methodologies and tools to assess the extent of implementation of MHPPs have not been properly examined.

Since the launch of the World Health Organisation (WHO) Global and European Mental Health Action Plans in 2013, many countries introduced new national MHPPs. Two-thirds of countries in the WHO European region have either developed or updated their national mental health policies or laws since then (WHO, 2018). While this is a welcome development, it is important to ensure that these newly developed or updated MHPPs have a real and important impact on the mental health and well-being of populations.

However, there are a number of challenges that both high-income (HICs) and low- and middle-income countries (LMICs) face in implementing their MHPPs (Zhou et al., 2018), such as limited access to financial and human resources (WHO, 2015) and low public mental health literacy (Campion and Knapp, 2018). In LMICs, challenges including a lack of professional training among healthcare workers, opposition from key stakeholders and resistance to decentralisation of mental health services are reported as more pronounced than in HICs (Saraceno et al., 2007). The processes countries take to implement MHPPs, including identifying the bottlenecks and facilitating factors, are largely unknown due to the lack of implementation and evaluation studies (Zhou et al., 2018). For example, the Strategy of Psychiatric Care Reform from Czechia that was launched in 2017/2018 and has ended in 2022 contains measurable indicators for each of the 10 implementation projects. However, to date no large-scale evaluation has been conducted to assess its implementation. Against this context, and while taking into consideration that many existing national mental health strategies, policies and action plans in the WHO European Region and beyond are now close to their expiration, we aimed to map and analyse tools and methodologies used to assess the extent and process of implementing national or regional MHPPs. This mapping review intends to inform policy development, implementation and evaluation in the WHO European Region.

Methods

We conducted a systematic search of peer-reviewed and grey literature to identify assessments of MHPPS. We followed the protocol recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (https://www.prisma-statement.org/) to report the screening process and findings (Moher et al., 2010; Page et al., 2021) and we registered the protocol in the Prospero Database for systematic reviews (Registration #CRD42022290862).

Search strategy and selection criteria

We searched the following bibliographic databases: Global Health, Medline, Embase, Web of Science, Global Index Medicus, WHO MindBank and Open Grey using four sets of search terms to identify relevant studies. These search terms included i) reform or policy or strategy or plan, ii) mental health or psych* or suicide or dementia, iii) implementation and iv) national or government*. Full search strategy is available in the Appendix. Additionally, we screened the lists of references of studies included from the main search as well as from the systematic review on mental health policies conducted by Zhou et al. (2018). The previous review by Zhou et al. (2018) provided an excellent foundation to understand the challenges in implementing mental health policies; however, to the knowledge of the authors of this systematic review, our study is the first of its kind to map existing methodologies and tools to assess the implementation of MHPPs.

We included studies that covered MHPPs, as well as policies covering specific mental health areas, including child and adolescent mental health, suicide prevention or dementia. These priority areas are included in the WHO European Framework for Action on Mental Health (2021–2025). We used WHO definitions of a mental health policy, which is referred to ‘an organized set of values, principles and objectives for improving mental health and reducing the burden of mental disorders in a population and defines a vision for future action’. A mental health plan is defined as ‘a detailed scheme for implementing strategic actions that addresses the promotion of mental health, the prevention of mental health conditions, and treatment and rehabilitation’.

We excluded studies that were 1) not focused on implementation of policies at a national or regional level; 2) policies that fall outside of the priority areas of the WHO European Framework for Action on Mental Health (e.g., substance use disorders, depression); 3) did not evaluate implementation of mental health policy or 4) did not describe any methods of evaluation. Conference abstracts, study protocols, opinion papers and editorials were also excluded. The study selection was not limited by year of publication or country of origin. Multiple languages (English, French, Spanish and Russian) were searched to ensure relevant studies are identified and captured.

All references identified through databases were imported into Rayyan’s online reference manager. After deleting duplicates, AA and HT independently screened titles and abstracts, following full-text examination of included articles. All disagreements were resolved by discussion.

Data extraction and analysis

We extracted and analysed data relevant to both characteristics of included studies and methods of assessment. Characteristics of the studies included the WHO Region of the studied country; country income classification according to World Bank; study objectives; policy period; policy area (e.g., such as suicide prevention, dementia prevention or mental healthcare development) and scope of evaluation of policy implementation, which we further divided into three categories: 1) progress and 2) process of implementation and impact 3) of MHPPs. We defined

• progress of implementation as a measure of extent to which MHPPs were implemented;

• process of implementation as referring to assessing barriers and facilitators, active ingredients, drivers, cultures, structures, ethics, pace and timing, or other related factors influencing implementation of mental health policies and

• impact of policy as referring to achievements resulting from implementation of MHPPs.

We also extracted information about methods of evaluation including study design (quantitative, qualitative; mixed methods), aspects of evaluation (such as quantitative indicators), theoretical frameworks used for evaluation and tools used to assess the extent and process of implementation of MHPPs (e.g., questionnaire, interview guide).

We distinguished studies that provided description of the tools used in evaluation (e.g., for interviews and focus group discussions, providing an interview guide showing the areas covered by the interviewer) and articles that were limited to simply mentioning the tool. When possible, we contacted authors to provide examples of the tools they had used, but not if these had not been reported in detail. We also highlighted studies that indicated that tools were pretested and those that supplemented detailed instruction or guides for using the methods and tools and interpreting results. Additionally, we wanted to distinguish studies that aimed at extensive assessment of all policy objectives from those that implemented specific programmes or interventions or certain parts of a policy.

In reporting the results, we adopted a structure used in the review of methods and tools to assess food and health policies by Phulkerd et al. (2015). Findings are presented separately by progress, process and impact. Each section includes analysis of policy areas of the studies, measured indicators and finally tools and methods used to assess policy implementation.

Results

The electronic database search identified 7,298 studies. After removing duplicates, 3,120 unique items were left. Unrelated abstracts were excluded based on title/abstract screening leaving 88 full texts for further screening, of which 22 studies were eligible for inclusion. Another 26 studies were selected from either reference lists of included articles or publications known to authors including 7 grey literature publications. In total, 48 studies were included (Figure 1).

Figure 1.

Screening and selection of articles.

Most countries represented were HIC (n = 32) followed by five studies on upper-middle and four on lower-middle-income countries (UMIC and LMIC). Six articles focused on low-income countries (LIC). The European Region was the region with the most publications (n = 22), followed by African (n = 12) and Western Pacific (n = 9) regions; three studies were from the Americas, and one from South-East Asia and Eastern Mediterranean Region each (Table 1).

Table 1.

Characteristics of included studies

Study region
Western Pacific	9
Africa	12
Europe	22
South-East Asia	1
Americas	3
Eastern Mediterranean	1
Income group
High	32
Upper middle	5
Lower middle	4
Low	4
Multinational	3
Scope of implementation assessment
Progress	2
Process	16
Impact	14
Progress, process	3
Progress, impact	5
Process, impact	4
Progress, process and impact	4
Type of methods
Qualitative	26
Quantitative	12
Mixed methods	10

We found 40 studies that assessed implementation of mental health policies, seven studies assessed implementation of suicide prevention strategies in Australia, Japan (n = 2), Northern Ireland, UK Scotland, UK (n = 2) and United States of America and only one study assessed dementia prevention policy in three European countries (Denmark, Germany, Italy). We found no study assessing the implementation of child and adolescent mental health policy. All studies, with the exception of one, focused on policies implemented at the national level. One study assessed the policy at both national and district levels (Doku et al., 2008). Only three studies presented results of assessment of entire MHPPs (Australian Health Ministers Advisory Council, 1997; Department of Health, Social Services and Public Safety, 2012; Loukidou et al., 2013). More details on each study are provided in Tables 2–4.

Table 3.

Summary of identified studies assessing the process of implementation

Author	Study country	Objective of the study	Policy areas	Aspects measured by the study	Design and methods	Tools (scales used)	Tool described
Bikker et al. (2020)	Indonesia	To obtain an insight into how the national-level legislation of the Mental Health Act (MHA) was perceived by psychiatrists in Bali, within their local context, and whether the MHA affected challenges encountered in their clinical practice	Mental health policy	The initial interview topic guide that focused on awareness and understanding of the Mental Health Act, its applicability, the implementation process (challenges to implementation) and views on the mental healthcare system	Qualitative interviews with psychiatrists	Interview topic guide that focused on awareness and under-standing of the MHA, its applicability, the implementation process and views on the mental healthcare system (narrative report)	Fully
Jones (2000)	England; Italy	To compare the implementation of social policy and legislation for mental health reform in Britain and Italy since 1950, with particular attention to implementation at the local scale; to explore the importance of place in the social and spatial restructuring of mental healthcare service provision in two different localities	Mental health policy; Deinstitutionalisation	Spatial patterns of static mental healthcare facilities and how national policy in England and Italy is impacting upon strategies for the planning and development of further facilities	Policy review; service provision maps compilation and analysis; semi-structured interviews with mental health professionals and professional planners	Service provision maps. No details regarding the interview guide were reported (narrative report)	No
Lovell (1986)	Italy	To assess impact of reform and obstacles to its implementation	Deinstitutionalisation	Challenges to implementation and impact of Italy’s Law 180	Interviews with administrators, planners, practitioners and patients	No details regarding the tools were reported (narrative report)	n/a
Mackenzie et al. (2007)	Scotland	To exemplify the challenges inherent in evaluating ‘Choose Life’: The National Strategy and Action Plan to Prevent Suicide in Scotland; to summarise the overarching approach taken to evaluating the first phase of Choose Life; to set out recommendations for framing subsequent evaluation of the initiative as it enters its more mature second phase	Suicide prevention	Challenges to future implementation of the National Strategy and Action Plan to Prevent Suicide	Theory of change workshop	No details regarding the tools were reported (narrative report)	n/a
Abdulmalik et al. (2016)	Nigeria	To explore the mental health system governance situation in Nigeria and to identify the challenges and opportunities for strengthening the mental health system in the country	Mental health policy	Challenges and opportunities for strengthening the mental health system in the country	Interviews with key informants at national, state and district (local government) levels	No details regarding the tools were reported (narrative report)	No
De Vries and Klazinga (2006)	Bosnia; Kosovo	This policy analysis provides insight into the ongoing process of mental health reform and the difficulty of sustaining such reform in post-conflict areas	Mental health policy	Public and political level of support for reform; pace of reform implementation	Documents review; Interviews with key informants	No details regarding the tools were reported (narrative report)	No
Karastergiou et al. (2005)	Greece	To describe the long and laborious process of psychiatric reforms in Greece within the last two decades and the current situation of mental health services	Mental health policy	Challenges and successes of mental health reform	Documents review, communication with stakeholders	No details regarding the tools were reported (narrative report)	No
Aviram and Azary-Viesel (2018)	Israel	To present essential issues and problems that must be addressed; to ensure the success of the reform’s implementation and its realisation	Mental health policy	Issues and challenges arising during implementation of mental health reform task environment and conflict of interests; research and evaluation; transfer of data and protection of privacy	Documents review; qualitative interviews	No details regarding the tools were reported (narrative report)	No
Awenva et al. (2010)	Ghana	To explore the barriers to mental health policy implementation by reporting aspects of a situational analysis of mental health policy development and implementation in Ghana	Mental health policy	Challenges to implementation	In-depth interviews; focus group discussions	Interview and focus group guides (narrative report)	No
Boeree et al. (2021)	Denmark; Germany; Italy	Our multiple-case study was designed to compare how different countries have implemented NDPs; to gain insight into variations in their implementation processes using qualitative data; to determine how the NDP initiatives have been implemented and why their implementation has differed	Dementia policy	Factors influencing the implementation of NDPs challenges	Semi-structured interviews with key stakeholders	Qualitative interview guide based on Perla’s framework of understanding factors that can affect policy (narrative report)	Partially
Doku et al. (2008)	Ghana	To assess the practices of implementing mental health policies and law at national and regional level; to assess the implementation of the mental health policy and law at district level	Mental health policy	Challenges to implementation of mental health policy	Semi-structured interviews and focus group discussions with key stakeholders in mental health (policymakers, programme directors, researchers, academics, health professionals, traditional healers, teachers, journalists, police officers)	Semi-structured interview and focus group discussion guide (narrative report)	Fully
Draper et al. (2009)	South Africa	To describe the process of mental health policy development and the content of this policy in South Africa	Mental health policy	Problems and challenges in mental health policy implementation	Semi-structured interviews with South African academic, government and international partners; literature review; literature review	Semi-structured interview guide (narrative report)	Fully
Fjellfeldt (2020)	Sweden	To explore the implementation process of a new policy framework in terms of regional and local responses to national policy; proposing a substantially broadened understanding of mental health within the Swedish welfare system	Mental health policy	Informants’ thoughts and feelings associated with the implementation process	Semi-structured qualitative interviews with key informants represented by agencies responsible for implementation	No details regarding the tools were reported (narrative report)	No
Grace et al. (2015)	Australia	To analyse the tools and/or policy levers in order to understand the reform process	Mental health policy	Tools or policy levers (e.g., organisation, regulation, community education, finance, payment) that affect the policy implementation	Document review	n/a	No
Park et al. (2015)	Canada	To examine the ethics of mental health policy development and implementation	Services development	Ethical tensions that arise during policy-making and implementation that accentuate the gap between policy, and everyday practice	Document review; qualitative interviews with policymakers and mental health practitioners; ethnography with mental health practitioners; participatory forums with administrators, government officials, persons with mental illness, family members; reflective team meetings	Interview questions (narrative report)	Partially
Toyoma et al. (2017)	Peru	To understand and consolidate the recent developments in mental health policy in Peru	Mental health policy	Challenges for mental health policy implementation and scale-up	Meetings with policymakers; document reviews	No details regarding the tools were reported (narrative report)	n/a
Omar et al. (2010)	Ghana, South Africa, Uganda and Zambia	To explore the factors that underpin the development of appropriate mental health policies and their effective implementation; to report comparative findings on the processes for developing and implementing mental health policies in Ghana, South Africa, Uganda and Zambia	Mental health policy	Factors underpinning the implementation of mental health policy	Semi-structured interviews with policymakers, programme managers, media, medical professional associations, traditional healer unions, mental health user groups and other relevant sectors including prisons, justice, social development, housing and education; documents review	No details regarding the tools were reported (narrative report)	n/a
Mwanza et al. (2008)	Zambia	The study objectives were to explore the current situation in Zambia regarding mental healthcare policy, law and mental health care; understand the general situation regarding mental health needs and priorities; describe the typical wider policy-making processes in the public sector and most importantly the health sector; describe and advance a critical appraisal of the development of the mental health policy and mental health law	Mental health policy	Challenges of mental health policy implementation	In-depth interviews and semi-structured interviews with experts at macro-level, focus groups with nurses, clinical officers and patients; observations	Open-ended questions; semi-structured interview guide (narrative report)	Partially
Stanley-Clarke et al. (2014)	New Zealand	To explore the relationship between government policy and the service development process	Mental health policy	Challenges and motivating factors of policy implementation	Qualitative interviews with nine staff including managers, service providers, a consumer advisor and a cultural representative; document review	No details regarding the tools were reported (narrative report)	n/a
Department of Health, Social Services and Public Safety (2012), Substance Abuse and Mental Health Services Administration (2017)	USA	To understand how the country was implementing the 2012 National Strategy, its challenges to implementation, and recommendations for overcoming those challenges	Suicide prevention	Challenges to implementations of the Strategy	Self-reported qualitative survey, semi-structured discussion with state suicide prevention coordinators	Survey on awareness and use of the National Strategy (narrative report)	Partially
Ryan et al. (2020)	Nigeria	To help inform the utilisation of private–public partnerships for mental health policy implementation in Nigeria and other low-resource settings by documenting a promising example from Benue, a largely rural state in the North Central region with a population larger than many countries	Mental health policy	The strengths and weaknesses and effectiveness of the Comprehensive Community Mental Health Programme as part of implementation of the mental health policy; clients enrolled in programme; referral rates; diagnoses; strengths, weaknesses, opportunities and threats to programme	Field visits involving a combination of desk review and observation of programme activities captured through photographs and field notes	No details regarding the tools were reported (narrative report)	n/a
Abdulmalik (2015)	Nigeria	To provide useful information about the barriers that prevent the successful integration of mental health into primary care from the perspectives of the frontline primary healthcare workers	Mental health policy; integrated services	Barriers to implementation of mental health policy	Free listing; focus group discussion; in-depth interviews with key informants; quantitative survey of primary health workers	Open-ended questions for free listing (% of respondents reporting an answer) Focus group discussions guide (narrative report) Key Informant interview guide (narrative report) Self-reported questionnaire (Yes vs. No or Strongly disagree, Disagree, Don’t know, Agree, Strongly Agree per item)	Fully
Marais and Petersen (2015)	South Africa	To identify systemic factors within institutional and policy contexts that are likely to facilitate or impede the implementation of integrated mental healthcare in South Africa	Mental health policy; integrated services	Barriers and facilitators to mental health implementation	Semi-structured interviews with managers and policymakers	Interview questionnaire addressing adapted Siddiqi’s HSG framework categories (narrative report)	Fully
Ssebunnya et al. (2010)	Uganda	To describe mental health services in Mayuge, a rural district in Uganda from the perspective of mental healthcare as an integrated component of primary healthcare	Mental health policy	Challenges to integration of mental health services into primary care	Semi-structured qualitative interviews and focus group discussions with healthcare managers, primary healthcare workers, health facility managers social workers, teachers	No details regarding the tools were reported (narrative report)	n/a
Petersen et al. (2012)	South Africa	To understand the benefits and challenges of community participation beyond that of scaling up, especially promoting culturally competent mental healthcare and greater community control of mental health	Mental health policy	Facilitators and challenges of mental health policy implementation	Semi-structured qualitative interviews with various stakeholders involved in the delivery and receipt of services and focus group discussions with community health workers	No details regarding the tools were reported (narrative report)	n/a
Department of Health, Social Services and Public Safety (2012)	Northern Ireland	To assess the progress and impact of the Protect Life Strategy against the objectives and targets set for it; to assess the relevance of the Strategy’s objectives, targets, actions and interventions in light of the continuing high rates of suicide and self-harm, and identify any gaps and how they might be addressed; to make recommendations to inform the development of the next phase of the Strategy and Action Plan	Suicide prevention	The governance, cross-departmental working and cooperation with other jurisdictions during the implementation of the Northern Ireland Protect Life Suicide Prevention Strategy and Action Plan 2006–2011	Desk research; stakeholder consultation community and organisations responsible for implementation; focus group discussions with family members, Implementation Groups, survey of Implementation Groups	No details regarding the tools were reported (narrative report)	n/a

Table 2.

Summary of identified studies assessing the progress of implementation

Author	Study country	Objective of the study	Policy areas	Aspects measured by the study	Design and methods	Tools (scales used)	Tool described
Hickie and Groom (2004)	Australia	To provide comment on the latest Australian government commitment to a 5-year plan under the National Mental Health Strategy	Mental health policy	The degree to which the key components of mental health reform have been implemented; experiences of quality of care	Survey with self-administered questionnaire of service providers on perceptions about implementation of reform and on quality of care	No details regarding the tools were reported (narrative report)	No
Sheehan et al. (2015)	Australia	To report a longitudinal evaluation of the implementation of the ‘Managing the Risk of Suicide’: A Suicide Prevention Strategy for using annual surveys; investigate whether the activities were implemented by the agencies that signed up to them	Suicide prevention	The completion of a report on available training programmes or reports of the type and number of activities completed (e.g., the number of training courses delivered)	Survey with self-administered questionnaire of agencies responsible for implementation	A purpose-designed questionnaire to measure progress of strategy implementation (yes vs. no for questions on relevancy of item to agency and then more detailed questions if Yes; percentage of staff employed, number of staff meetings)	Partially
Doku et al. (2008)	Ghana	To assess the practices of implementing mental health policies and law at national and regional level; to assess the implementation of the mental health policy and law at district level	Mental health policy	Progress of implementation of mental health policy	Semi-structured interviews and focus group discussions with key stakeholders in mental health (policymakers, programme directors, researchers, academics, health professionals, traditional healers, teachers, journalists, police officers)	Semi-structured interview and focus group guide (narrative report)	Fully
Draper et al. (2009)	South Africa	To describe the process of mental health policy development and the content of this policy in South Africa	Mental health policy	Existence; problems and challenges in mental health policy implementation	Semi-structured interviews with South African academic, government and international partners.; literature review	Semi-structured interview guide (narrative report)	Fully
Loukidou et al. (2013)	Greece	To determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability of mental health policy	Mental health policy	The level of mental health policy implementation	Semi-structured interviews with clinicians, clinical academics and managers from the public and NGOs sectors; focus group discussions with service providers and service users; specially devised self-administered questionnaires; site visits observations; document and literature review	Semi-structured interview guide (tools were not specified for site observations or questionnaire) (narrative report)	Partially
Dlouhy (2014)	Bulgaria; Czechia	To describe and compare the mental health policies and mental health systems in seven countries of Eastern Europe	Mental health policy	Existence of mental health policy implementation	Qualitative and quantitative survey	The health policy questionnaire. (narrative report)	Partially
Omar et al. (2010)	Ghana, South Africa, Uganda and Zambia	To explore the factors that underpin the development of appropriate mental health policies and their effective implementation; to report comparative findings on the processes for developing and implementing mental health policies in Ghana, South Africa, Uganda and Zambia	Mental health policy	Existence of evaluation and factors underpinning the implementation of mental health policy.	Semi-structured interviews with policymakers, programme managers, media, medical professional associations, traditional healer unions, mental health user groups and other relevant sectors including prisons, justice, social development, housing and education; documents review	No details regarding the tools were reported (narrative report)	No
Nakanishi et al. (2014)	Japan	To investigate the impact of a national suicide prevention strategy on small areas within Japan as well as in other countries	Suicide prevention	Existence of implementation of prevention programmes; patterns of implementation according to categories of prevention programmes	Survey with self-administered questionnaire of agencies responsible for implementation	Survey questionnaire (initiatives implemented vs. not implemented and other items on the presence of suicide-prevention systems)	Partially
Mwanza et al. (2008)	Zambia	The study objectives were to explore the current situation in Zambia regarding mental healthcare policy, law and mental healthcare; understand the general situation regarding mental health needs and priorities; describe the typical wider policy-making processes in the public sector and most importantly the health sector; describe and advance a critical appraisal of the development of the mental health policy and mental health law	Mental health policy	Existence of mental health policy implementation	WHO-AIMS Instrument and Survey Checklist	WHO-AIMS Instrument and Survey Checklist (narrative report)	Partially
Nakanishi and Endo (2017)	Japan	To identify local authorities’ implementation of suicide prevention programmes in terms of local health policies; examine the associations between local health resources and suicide rates in Japan	Suicide prevention	Existence of suicide prevention Strategy implementation; areas addressed in implementation; impact of implementation on suicide incidence	Local authorities were asked to answer what areas they addressed in each programme; statistical data analysis	n/a	No
Department of Health, Social Services and Public Safety (2012)	Northern Ireland	To assess the progress and impact of the Protect Life Strategy against the objectives and targets set for it; to assess the relevance of the Strategy’s objectives, targets, actions and interventions in light of the continuing high rates of suicide and self-harm, and identify any gaps and how they might be addressed; to make recommendations to inform the development of the next phase of the Strategy and Action Plan	Suicide prevention	The level and expenditures of the implementation of the Northern Ireland Protect Life Suicide Prevention Strategy and Action Plan 2006–2011	Desk research; stakeholder consultation with families, community and organisations responsible for implementation; track and analysis of the resources allocated for the implementation of the Strategy	No details regarding the tools were reported (narrative report)	Partially
Substance Abuse and Mental Health Services Administration (2017), U.S. Department of Health and Human Services (2017)	USA	To understand how the country was implementing the 2012 National Strategy, its challenges to implementation and recommendations for overcoming those challenges	Suicide prevention	Existence of implementation; number of suicide prevention activities	Self-reported qualitative survey, semi-structured discussion with state suicide prevention coordinators, mapping of suicide prevention activities	Survey on awareness and use of the National Strategy (narrative report)	Partially
Reid Howie Associates (2006)	Scotland	To reduce suicide by 20% over a 10-year period through the achievement of objectives such as improved early prevention and crisis response, engagement with the media and adoption of an evidence-based approach	Suicide prevention	Extent and level of usage of suicide prevention projects;	Examination of project reports and other existing documentary and background information Interviews with managers and project workers, key staff with a strategic overview of projects. Interviews with service users. Group discussions and postal survey of MDMHTs, with prisoners. Postal survey of MHFA course participants. Discussions with other prison officers	No details regarding the tools were reported (narrative report)	No

Table 4.

Summary of identified studies assessing the impact of mental health policies and plans

Author	Study country	Objective of the study	Policy areas	Aspects measured by the study	Design and methods	Tools (scales used)	Tool described
Hickie and Groom (2004)	Australia	To provide comment on the latest Australian government commitment to a 5-year plan under the National Mental Health Strategy	Mental health policy	Experiences of quality of care	Survey with self-administered questionnaire	Survey questionnaires on perceptions about implementation of reform and on quality of care (narrative report)	No
Lovell (1986)	Italy	To assess impact of reform and obstacles to its implementation	Mental health policy	Impact of Italy’s Law 180	Observation of hospital practices; documents review	No details regarding the tools were reported	No
Munizza et al. (2011)	Italy	To assess adherence to the standards of the National Mental Health plans; to examine the extent to which the third national survey, named PROG-CSM (Progetto Centri di Salute Mentale), that was subsequently undertaken in all Italian CMHCs, in 2005 adhered to the standards defined by the National Mental Health Plan	Mental health policy	Impact of mental health reform on continuity of care; coordination with other community-based services; accessibility; implementation of specific programmes, and provision of care	Survey with self-administered questionnaire	Two survey questionnaires on continuity of care, coordination with other community-based services, accessibility, implementation of specific programmes, and provision of care (narrative report)	Partially
Karastergiou et al. (2005)	Greece	To describe the long and laborious process of psychiatric reforms in Greece within the last two decades and the current situation of mental health services	Mental health policy; Deinstitutionalisation	Impact of mental health reform on deinstitutionalisation and new psychiatric services	Documents review, communication with stakeholders	No details regarding the tools were reported (narrative report)	No
Barbato (1998)	Italy	To illustrate the distinctive features of Italian deinstitutionalisation, showing the factors affecting the decrease in use of psychiatric hospitalisation; to discuss the impact of community care without long-stay beds on patients with severe mental disorders	Mental health policy; Deinstitutionalisation	Impact of mental health reform on deinstitutionalisation and new psychiatric services; effect of these changes on indicators related to four issues: transfer of care, criminalisation of the mentally ill, suicides, and homelessness	Statistical data analysis (psychiatric services before and after the policy change)	n/a	n/a
Bindman et al. (1999)	UK	To describe the application of the Care Programme Approach (policy) and test the hypothesis that the number of people prioritised to receive care under the CPA are predicted by a population-based estimate of need for psychiatric care	Mental health policy	The effect of a Care Programme Approach policy framework	Survey with self-administered questionnaire; statistical data analysis	Survey questionnaire (number of patients recorded on programmes)	No
Boeree et al. (2021)	Denmark; Germany; Italy	Our multiple-case study was designed to compare how different countries have implemented NDPs; to gain insight into variations in their implementation processes using qualitative data; to determine how the NDP initiatives have been implemented and why their implementation has differed	Dementia policy	Effectiveness and ineffectiveness of National Dementia Policies	Semi-structured interviews with key stakeholders	Qualitative interview guide based on Perla’s framework of understanding factors that can affect policy (narrative report)]	Partially
Doku et al. (2008)	Ghana	To assess the practices of implementing mental health policies and law at national and regional level; to assess the implementation of the mental health policy and law at district level	Mental health policy	Impact of implementation of mental health policy	Semi-structured interviews and focus group discussions with key stakeholders in mental health	Semi-structured interview and focus group guide (narrative report)	Fully
Loukidou et al. (2013)	Greece	To determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability of mental health policy	Mental health policy	The effectiveness of mental health policy implementation	Semi-structured interviews with clinicians, clinical academics and managers from the public and NGOs sectors; focus group discussions with service providers and service users; specially devised self-administered questionnaires; site visits observations; document and literature review	Semi-structured interview guide (tools were not specified for site observations or questionnaire) (narrative report)	Partially
Palermo (1991)	Italy	To discuss the socio-psychiatric consequences of the 1978 Italian mental health law; to review the international scientific ideas that led up to it; to describe the socio-political psychiatric views of the late Franco Basaglia, pioneer of the change in the mental health system of the Italian Republic	Mental health policy; Deinstitutionalisation	Consequences; impact of mental health reform	Documents review; personal opinion; statistical reports; critical analyses	No details regarding the tools were reported (narrative report)	n/a
Rey et al. (2004)	Australia	To provide information about changes in service quality from the perspective of psychiatrists	Mental health policy	Changes in the quality of care; psychiatric practice in the past 5 years	Survey with two self-administered questionnaire	Overall opinion about changes in quality of care in the past 5 years (Rating 1–5; where 1 = Improved a lot and 5 = much worse) Opinion on quality of psychiatric care – unspecified rating (narrative report)	Partially
de Girolamo and Cozza (2000)	Italy	To carry out an analysis of the state of application of the law and of its overall effects; to examine to what extent the claims made either by the advocates or by the opponents of the reform have been supported by reliable evidence; to extract a number of general lessons from the Italian experience of mental health reform	Mental health policy; Deinstitutionalisation	The impact of Italy’s mental health reform on deinstitutionalisation and quality of care	Statistical data analysis based on data from national surveys; literature review	No details regarding the tools were reported (narrative report)	n/a
Madianos (2002)	Greece	To explore the possible influence of the 10-year psychiatric reform programme on the achievement of its first strategic objective, deinstitutionalisation of the long-stay patients in the nine public mental hospitals, and the shift to extramural care and rehabilitation	Mental health policy; Deinstitutionalisation	Impact of mental health reform on deinstitutionalisation and new psychiatric services	Quantitative trends assessment	n/a	n/a
Madianos and Christodoulou (2007)	Greece	To examine the first strategic objective of the 20-year psychiatric reform programme: the de-institutionalisation of the long-stay patients in the nine public mental hospitals and the shift to extramural care and rehabilitation	Mental health policy	Impact of mental health reform on decentralisation of mental health services	Quantitative trends (1984–2000)	n/a	n/a
Nakanishi et al. (2015)	Japan	To investigate the impact of a national suicide prevention strategy on small areas within Japan as well as in other countries	Suicide prevention	Impact of the Strategy on suicide cases in various categories of prevention programmes	quantitative trends in suicide deaths 2009–2012	n/a	n/a
Nakanishi and Endo (2017)	Japan	To identify local authorities’ implementation of suicide prevention programmes in terms of local health policies; examine the associations between local health resources and suicide rates in Japan	Suicide prevention	Impact of implementation on suicide incidence	Retrospective longitudinal study: statistical data analysis	n/a	n/a
Australian Health Ministers Advisory Council (1997)	Australia	To bring together complementary views of the National Mental Health Strategy	Mental health policy	Effectiveness and appropriateness of a National Mental Health Reform	Stakeholder Survey; stakeholder consultations; statistical data analysis (change in psychiatric services); expert opinion	Survey instrument on the stakeholders’ perception of changes that have occurred over the period of the National Mental Health Strategy. (narrative report)	No
Department of Health, Social Services and Public Safety (2012)	Northern Ireland	To assess the progress and impact of the Protect Life Strategy against the objectives and targets set for it; to assess the relevance of the Strategy’s objectives, targets, actions and interventions in light of the continuing high rates of suicide and self-harm, and identify any gaps and how they might be addressed; to make recommendations to inform the development of the next phase of the Strategy and Action Plan	Suicide prevention	The effectiveness of the implementation of the Northern Ireland Protect Life Suicide Prevention Strategy and Action Plan 2006–2011; effectiveness of support for families of victims of suicide	Quantitative survey with families; Focus group discussion with communities of interest	What matters survey (0 to 5 importance of support or experience of support, where 0 = not important/not experienced and 5 = really important/excellent)	Fully
Williams et al. (1986)	Italy	To investigate the effect of psychiatric reform on suicide rates	Mental health policy	Impact of mental health reform on suicide, deinstitutionalisation and community mental health services	Quantitative trends in suicide deaths 1973–1978 and 1979–1983	n/a	n/a
Winkler et al. (2021)	Czechia	To assess whether people exposed to the mental healthcare reform or to a nation-wide anti-stigma project report more favourable attitudes and lower desire for social distance towards people with mental health problems than those who were not exposed to such initiatives; to assess the changes in stigmatising attitudes and behaviour in Czechia between 2013/2014 and 2019	Mental health policy	Impact of mental health reform on attitudes towards mental illness (CAMI scale) and intended behaviour towards people with mental illness (RIBS)	Data from three studies were used: the 2013 study containing the RIBS scale; the 2014 study containing the CAMI scale; the 2019 follow-up study containing the RIBS and CAMI scales	Community Attitudes towards Mental Illness; Reported and Intended Behaviour Scale	Fully
Sharkey (2017)	Qatar	To discuss the development of the National Mental Health Strategy for Qatar, Changing Minds, Changing Lives 2013–2018 (General Secretariat of the Supreme Council of Health, 2013), its implementation and the findings from an independent impact evaluation carried out in 2015	Mental health policy	The impact of the National Mental Health Strategy	Literature review; meetings with officials; health professionals from primary; secondary and tertiary health services; representatives from the education sector and other relevant stakeholders; site visits	Data collection instrument based WHO-AIMS items; information specific to the National Mental Health Strategy (narrative report)	No
Department of Health, Social Services and Public Safety (2012), Substance Abuse and Mental Health Services Administration (2017)	USA	To understand how the country was implementing the 2012 National Strategy, its challenges to implementation, and recommendations for overcoming those challenges	Suicide prevention	Relevance of the Strategy goals for their work	Self-reported qualitative survey, semi-structured discussion with state suicide prevention coordinators	Survey on awareness and use of the National Strategy (Narrative report)	Partially
Vázquez-Barquero et al. (2001)	Spain	To describe the impact of Spanish psychiatric reform on the organisation and functioning of mental health services	Mental health policy	Effectiveness of mental health reform	Document review (change in deinstitutionalisation, primary care referrals, community services distribution and utilisation, change in stigma, number of psychiatrists)	No details regarding the tools were reported (narrative report)	n/a
Whiteford et al. (2002)	Australia	To report on the changes to mental health services achieved by 1998	Mental health policy	Impact of mental health reform on service expenditure, public sector service mix, private sector; consumer and carer involvement in services; legislation reform	Annual national survey on progress of policy implementation	No details regarding the tools were reported (narrative report)	n/a
Ryan et al. (2020)	Nigeria	To help inform the utilisation of PPPs for mental health policy implementation in Nigeria and other low-resource settings by documenting a promising example from Benue, a largely rural state in the North Central region with a population larger than many countries	Mental health policy	The effectiveness of the Comprehensive Community Mental Health Programme as part of implementation of the mental health policy	Field visits involving a combination of desk review and observation of programme activities captured through photographs and field notes, statistical analysis of service utilisation data	No details regarding the tools were reported (narrative report)	n/a
Petersen et al. (2011)	South Africa, Uganda	To understand how the use of the common implementation framework assisted in the development of district/sub-district mental health services in the two country contexts with the view to drawing out shared lessons for integrating mental health into PHC in LMICs	Mental health policy	Effectiveness of implementation framework in promotion of integrated care	Qualitative interviews and focus group discussions with managers, service providers and service users	No details regarding the tools were reported (narrative report)	n/a

Progress of policy implementation

Overview

We identified 14 studies that assessed progress of implementation, usually together with process assessment (n = 3) (Mwanza et al., 2008; Draper et al., 2009; Omar et al., 2010) or impact assessment (n = 5) (Australian Health Ministers´ Advisory Council, 1997; Hickie and Groom, 2004; Loukidou et al., 2013; Nakanishi et al., 2015; Nakanishi and Endo, 2017) or both (n = 2) (Reid Howie Associates, 2006; Doku et al., 2008). Two studies assessed solely progress of policy implementation (Dlouhy, 2014; Sheehan et al., 2015). Nine studies assessed policy implementation in HICs, four in UMICs, one in LMICs and three in LICs. Most (86%) of these studies were single-country focused, and two were multinational. Quantitative (e.g., surveys, questionnaires) and qualitative methods (e.g., interviews, focus group discussions) or a combination of both were used, with qualitative methods substantially prevailing.

Policy areas

Five studies, from HICs, focused on suicide prevention policies. The rest focused on mental health policies, which mostly included overarching mental healthcare development strategies. The contents of these overarching policies included mental health promotion and prevention of mental health problems (Hickie and Groom, 2004; Dlouhy, 2014; Nakanishi and Endo, 2017), improving quality of care (Hickie and Groom, 2004; Doku et al., 2008; Nakanishi and Endo, 2017), strengthening research (Hickie and Groom, 2004) and deinstitutionalisation and development of community care (Doku et al., 2008; Loukidou et al., 2013). One study specifically looked at the integration of mental health services into primary healthcare (Draper et al., 2009).

Aspects measured

Progress of policy implementation was expressed as a) existence of policy implementation at all or b) the degree of its implementation. Assessment of existence was expressed in qualitative or survey questions on whether any implementation activities were carried out. Eight studies examined the existence of policy implementation (Doku et al., 2008; Mwanza et al., 2008; Draper et al., 2009; Omar et al., 2010; Dlouhy, 2014; Nakanishi et al., 2015; Nakanishi and Endo, 2017; Substance Abuse and Mental Health Services Administration, 2017). Degree of implementation was measured in a variety of ways in eight studies, seven of which focused on suicide prevention policies. For example, one study assessed perceptions of psychiatrists about degrees to which the key aspects of reform were implemented (Hickie and Groom, 2004), others measured progress by assessing whether a tangible outcome was produced (i.e., specific output) according to an implementation plan (e.g., completion of a report on training programmes) or the number or percentage of activities completed (e.g., number of training programmes provided, number of staff attending an event) (Department of Health, Social Services and Public Safety, 2012; Loukidou et al., 2013; Sheehan et al., 2015; Substance Abuse and Mental Health Services Administration, 2017). One study examined level of usage of projects expressed in number of referrals to the programme or time users spent in the programme (Reid Howie Associates, 2006). Additionally, another study investigated patterns of implementation, that is, the frequency of types of suicide prevention programmes authorities chose to implement in different prefectures (Nakanishi et al., 2015). Yet, other study compared areas of focus (e.g., addictions, unemployment) that were addressed by different authorities across the country in implementation of a national suicide prevention strategy (Nakanishi and Endo, 2017).

Methods and tools

Various methods were employed to measure progress of implementation. Four studies used quantitative methods: surveys or questionnaires (Hickie and Groom, 2004; Dlouhy, 2014; Nakanishi et al., 2015; Sheehan et al., 2015). Three studies used qualitative semi-structured interviews (Doku et al., 2008; Draper et al., 2009; Omar et al., 2010), and one study complemented qualitative interviews with literature review (Draper et al., 2009). Five studies applied mixed-methods approaches (Mwanza et al., 2008; Department of Health, Social Services and Public Safety, 2012; Loukidou et al., 2013; Nakanishi and Endo, 2017; Substance Abuse and Mental Health Services Administration, 2017) with combination of qualitative (semi-structured interviews, focus group discussions, desk review, stakeholder consultations) methods and quantitative surveys or statistical analysis. With the exception of one, all studies used surveys that were specially designed ad hoc self-administered questionnaires, with the WHO-AIMS Instrument and Survey Checklist being the only standardised tool used (Mwanza et al., 2008). Study participants varied across studies and included, depending on the context, users and families, service providers, traditional healers, agencies and organisations involved in the implementation, government and international policymakers.

Five studies used a single method such as survey or semi-structured interviews with key stakeholders, while the rest employed more than one method with a combination of following methods: questionnaires, qualitative interviews and focus group discussions, semi-structured discussions, literature and document review, and quantitative trends for suicide data. Only nine studies provided details on the content of their tools, of which two provided their interview and focus group guide templates in the Supplementary Materials (Doku et al., 2008; Draper et al., 2009).

Process of policy implementation

Overview

Overall, 27 studies assessed the process of policy implementation. Of these, 16 assessed only the implementation process; seven studies provided an assessment of the implementation process combined with an evaluation of the progress or impact of implementation and four studies assessed all three (progress, process, impact). Twenty-three were single-country studies and four evaluated more than one country. Nearly half (n = 13) of the studies were conducted in HICs; five studies by UMICs; four by LMICs and LICs each and one multinational study assessed both UMIC and LIC.

Policy areas

Of all 27 studies, 22 assessed mental health policies where four studies assessed suicide prevention strategies (Reid Howie Associates, 2006; Mackenzie et al., 2007; Department of Health, Social Services and Public Safety, 2012; Substance Abuse and Mental Health Services Administration, 2017) and one study focused on national dementia prevention policies (Boeree et al., 2021). Most of the mental health policies were national development strategies that aimed at decentralisation and deinstitutionalisation of mental healthcare and establishing community-based systems. Other studies investigated policies on mental health system funding change (Aviram and Azary-Viesel, 2018) or looked at the implementation process of multiple consecutive deinstitutionalisation reforms (from 1950 onwards) (Jones, 2000). One study focused on a new policy framework that would expand the understanding of mental ill health within the country’s welfare system and investigated how the process was dealt with by implementing agencies (Fjellfeldt, 2020). Two studies specifically looked at policies centred around the service development process (Stanley-Clarke et al., 2014) and the introduction of a new model of care (Park et al., 2015).

Aspects measured

Most studies assessed challenges or barriers and/or facilitating factors on policy implementation (n = 24). Specific implementation determinants measured included context-dependant features of the policy implementation process such as public and political level of support for mental health reform and pace of its implementation (Ryan et al., 2020); ethical tensions arising during policy implementation (Park et al., 2015); key informants’ thoughts and feelings associated with the implementation process of a new policy framework (Fjellfeldt, 2020) and policy levers through which the mental health system reform was to be implemented (Grace et al., 2015). Two studies used a theoretical framework on drivers and constraints that affect policy development and implementation as a conceptual background to the methodology to guide evaluation. Boeree et al. (2021) described the primary drivers for implementation as follows: 1) planning and infrastructure; 2) individual, group, organisational and systemic factors, as well as contextual factors; 3) the underlying theory and process of change involving all partners and 4) performance measures and evaluation Doku et al. (2008) described three major constraints for effective implementation: 1) lack of strategic planning; 2) inappropriate health system to support the policy and 3) lack of support or resistance to implementation, partly due to stigma associated with mental illness.

Other evaluation frameworks were more comprehensive and included various elements such as context, content and process or stream (such as in Kingdon’s conceptualisation of policy or Walt’s analytical framework), and the various actors involved in mental health policy (De Vries and Klazinga, 2006; Omar et al., 2010; Grace et al., 2015).

One study used a manualised case study methodology to organise and integrate data from various sources across domains of interest. The collection of data was accompanied by the overarching research questions: ‘Is this programme working? Why or why not?’ along with a description of strengths, weaknesses, opportunities and threats (Ryan et al., 2020). Similarly, in a study on ethical tensions that may arise during policy implementation, a specific ethical framework with three analytic levels: i) person-focused, ii) event-focused and iii) discursive practices was developed to capture the experiences of participants involved in the programme (Park et al., 2015).

Methods and tools

Studies used qualitative (n = 22) or mixed-methods (n = 5) techniques for investigation of implementation process. Every study in this review employed at least some form of qualitative methods: key informant interview, focus group discussion, other communications with stakeholders (discussions, meetings, forums, Theory of Change workshop), ethnography or observation or documents review. One study used a free listing technique with broad open-ended questions to elicit a comprehensive list of implementation barriers (Abdulmalik et al., 2016). Quantitative tools included self-reported surveys (e.g., to measure knowledge by health workers about the content of mental health policy). A quantitative survey provided primary healthcare staff with a list of challenges to integrate mental health into primary healthcare, which included dichotomous answers (yes/no or agree/disagree) on potential challenges/barriers that were reported during qualitative interviews (Abdulmalik et al., 2016). Other quantitative methods measured impact of implementation (see below for more details). Participants varied from study to study and included service users, service providers, healthcare managers, media representatives and policymakers at macro level.

Nine studies used single methods such as semi-structured qualitative interview (n = 7), document review (n = 1) or a workshop with stakeholders (n = 1), while the rest used more than one method. These were various combinations of qualitative interviews, focus group discussions, document reviews, meetings with stakeholders, observations and quantitative surveys. Only 10 studies provided rationale or details about the tools they employed of which three were available (Doku et al., 2008; Draper et al., 2009; Bikker et al., 2020) (see Supplementary Material).

Impact

Overview

Of the 27 studies assessing the impact of policy implementation, 16 evaluated only impact, five evaluated impact together with progress, four evaluated impact together with process and two evaluated all three. Twenty-three studies were published in HICs and only four in LMICs. With the exception of two, all studies investigated single countries only. Eight studies applied qualitative methods (e.g., interviews, focus group discussions, document review), 11 studies used quantitative methods (e.g., survey) and eight studies employed a mixed-methods approach.

Policy areas

Most studies assessing the impact of policy implementation focused on general mental health policy (n = 22) where four studies assessed suicide prevention strategies and one study assessed dementia policies. Most mental health policies encompassed the goals of deinstitutionalisation with provision of more community services (n = 20), of which six studies focused on mental health reform in Italy (Lovell, 1986; Williams et al., 1986; Palermo, 1991; Barbato, 1998; De Girolamo and Cozza, 2000; Munizza et al., 2011) and four studies investigated mental health policy in Greece (Madianos, 2002; Karastergiou et al., 2005; Madianos and Christodoulou, 2007; Loukidou et al., 2013). Two other policies included a study where a mental health policy in the United Kingdom policy sought to introduce ‘tiered’ prioritisation of patients (Bindman et al., 1999) and a study in Australia introducing new ‘priority themes’ for mental healthcare development (Hickie and Groom, 2004). Other goals covered by policies included increase of public–private partnerships to deliver mental health services (Ryan et al., 2020), promote integrated primary mental healthcare (Petersen et al., 2011), improve quality of care (Whiteford et al., 2002; Hickie and Groom, 2004; Sharkey, 2017; Winkler et al., 2021), ensure research evidence translation into practice (Hickie and Groom, 2004;Sharkey, 2017; Winkler et al., 2021), raise awareness and reduce mental illness stigma (Sharkey, 2017; Winkler et al., 2021), promote mental health and prevent mental disorders (Australian Health Ministers´ Advisory Council, 1997; Hickie and Groom, 2004) and protect consumer rights (Australian Health Ministers´ Advisory Council, 1997; Winkler et al., 2021).

Aspects measured

The identified studies mainly examined the effectiveness of policies and their appropriateness. Two studies measured appropriateness of policy implementation (Australian Health Ministers´ Advisory Council, 1997; Substance Abuse and Mental Health Services Administration, 2017), which was defined as whether the strategy’s goals and actions remain relevant and suitable to the implementation that is being carried out.

In terms of effectiveness, aspects measured varied substantially between the studies depending on the policy goals and planned deliverables. For example, deinstitutionalisation policies measured changes in psychiatric beds capacities in various settings (Williams et al., 1986; De Girolamo and Cozza, 2000; Vázquez-Barquero et al., 2001; Madianos, 2002; Whiteford et al., 2002; Madianos and Christodoulou, 2007), number of referrals to community services (De Girolamo and Cozza, 2000; Vázquez-Barquero et al., 2001; Sharkey, 2017), number of deinstitutionalised patients (Loukidou et al., 2013), ratio of psychiatric patients identified by a GP (Vázquez-Barquero et al., 2001), transinstitutionalisation of patients to other facilities (Lovell, 1986; Barbato, 1998), prevalence of homeless people with mental disorders and criminalisation of the mentally ill (Barbato, 1998), development of various decentralised services (Lovell, 1986; Williams et al., 1986; De Girolamo and Cozza, 2000; Madianos and Christodoulou, 2007; Munizza et al., 2011), epidemiological data on incidence and treated prevalence (Munizza et al., 2011), change in suicide trends before and after the psychiatric reform (Williams et al., 1986; Barbato, 1998; De Girolamo and Cozza, 2000) and changes in the quality of care (De Girolamo and Cozza, 2000; Rey et al., 2004), in psychiatric practice and in access to care (Rey et al., 2004). Other measures included change in service expenditures (Whiteford et al., 2002), involvement of service users and carers (Whiteford et al., 2002), clinical outcomes in hospital residents and community patients and quality of life of people with mental disorders (De Girolamo and Cozza, 2000); change in public stigma (Vázquez-Barquero et al., 2001; Winkler et al., 2021) and change in number of published scientific articles (to measure the research potential) (Sharkey, 2017). Some studies used subjective measures such as personal opinion from stakeholders about the policy implementation (Boeree et al., 2021). Studies on specific programmes related to policy implementation measured the number of clients enrolled in a programme (Bindman et al., 1999; Ryan et al., 2020).

Studies on suicide prevention strategies measured incidence of suicide across prefectures in Japan where different prevention programmes were implemented (Nakanishi et al., 2015) or the perceived level of support by bereaved families to assess effectiveness of support programmes and resources allocated on implementation of suicide prevention strategy in the Northern Ireland (Department of Health, Social Services and Public Safety, 2012).

An Australian study used a framework with four evaluation focus areas: rights of consumers and carers, mixed services, linkages between mental health services with other sectors and promotion and prevention. Each focus area contained a number of questions to be answered during the evaluation (Australian Health Ministers´ Advisory Council, 1997). Similarly, a Northern Ireland (UK) study, for each predefined evaluation question, defined an evaluation area as well as what to measure and recommended methods and tools for evaluation. For example, effectiveness and impact for individuals and families were defined as an evaluation area and for that particular area evidence base and families service use experiences were investigated by using a pre-developed survey, focus groups with families and reviewing available published evaluations of already completed initiatives (Department of Health, Social Services and Public Safety, 2012).

Methods and tools

Nine studies applied qualitative methods, 11 used quantitative and another seven used mixed-methods approach. Qualitative semi-structured interviews and document reviews were most commonly used with each method mentioned in six and seven studies, respectively. Other qualitative methods included focus group discussions, consultations with experts, meetings with officials and observation.

Among the quantitative methods, statistical analyses of epidemiological data (n = 7) were used most commonly. Other methods included survey questionnaire (n = 9) and using specific scales such as Community Attitudes towards Mental Illness (Taylor and Dear, 1981) and Reported and Intended Behaviour Scale (Evans-Lacko et al., 2011) to measure public stigma (Winkler et al., 2021). Quantitative surveys asked consumers and carers to rate their experience of healthcare (Hickie and Groom, 2004), to choose whether certain aspects of psychiatric practice had increased, remained the same or decreased (e.g., income, satisfaction, patients’ illness severity) or whether the perceived quality of care has improved or deteriorated (Rey et al., 2004).

Twelve studies used only one method while the rest employed a combination of the above-mentioned methods such as survey and statistical analysis, literature review and meeting with officials. Nine studies described the content of their tools, of which only three are available (see Supplementary Material).

Discussion

To our knowledge, this is the first study to systematically map and analyse methods and tools used to assess the implementation of MHPPs. We found no comprehensive, high-quality, peer-reviewed assessment of implementation of MHPPs as such. Given that MHPPs are important instruments to improve mental healthcare and well-being of populations, rigorous peer-reviewed assessment of their implementation is crucial so important lessons can be learned and mental health systems improved. Studies included in our review placed emphasis on the presentation of the results but lacked rigorous methodological description, which makes their tools and methods unclear. Less than half of the included studies provided details about the tools they used for data collection (Munizza et al., 2011; Bikker et al., 2020; Winkler et al., 2021). Only three studies pretested or piloted the tools. Very few publications provided a full description of their tools. For instance, although a substantial majority of studies employed interview and focus group guides and questionnaires that were specifically tailored to the evaluation purposes, they failed to provide samples of the questions they used. Most tools were not commonly standardised, which is likely due to the broad nature of MHPPs and the comprehensiveness of their specific policy area and the diversity of contexts in which they have been implemented.

Only three studies assessed all three categories of implementation (progress, process and impact) and they were all non-academic assessments of suicide prevention strategies. Similarly, assessments of entire MHPPs, as opposed to only certain parts of them, are rare. Clearly, a full and comprehensive assessment of an overarching policy like a mental health reform might be a lengthy (impact can be measured after decades of reform) and resource-demanding process. In contrast, we found that studies focused on specific evaluation questions related to MHPPs implementation (e.g., challenges associated with the reform in various contexts or opinions about the implementation progress; development of public stigma; changes in suicide rates, etc.) were mostly published in academic journals. Such assessment with the primary focus on only one or several aspects of MHPPs provided that rigorousness and transparency of reported methods and results are ensured certainly is a valid alternative strategy. However, these smaller assessments of an MHPP have to be put together into comprehensive reports of MHHPs implementation and made accessible to readers. In any case, evaluation strategy should be an integral part of the MHPPs.

Given the broad nature and complexity of MHPPs, it is likely that there could be a publication bias where studies with narrower research questions get published in academic journals, whereas extensive assessments and evaluations might have been published only as project reports or policy papers. We identified seven such reports, which were extensive national evaluation reports of countries’ MHPPs or suicide prevention strategies (n = 5), one Master’s dissertation on barriers to integration of mental health into primary care in Nigeria (Abdulmalik et al., 2016) and one project report on suicide prevention in prisons in Scotland, UK (Reid Howie Associates, 2006). Usually, the format of such studies allows for situation analysis and a more detailed description of methodology to be included. However, if made available online, such reports tend to be replaced or become inaccessible over years, may not be identifiable through traditional electronic database searches and are usually not peer-reviewed.

We used a broad definition to assess policy implementation focusing on three categories: progress, process and impact. Studies assessing the progress of implementation usually collected data through qualitative and quantitative questionnaires enquiring about progress or level of implementation against policy targets or goals. Findings of evaluations demonstrated that implementation of MHPPs in terms of target achievement or types of programmes adopted was most often partial. For example, in Northern Ireland, UK, only about a fifth of actions were fully progressing to plan, while the rest were in moderate or limited progress. Similarly, in Australia, a study showed that not even two-thirds of activities were measured, of which 42% were fully implemented while 20% were implemented partially. Studies in our review show that some activities are being implemented more effectively than others. In Greece, even though implementation of many activities of national mental health reform was successful, the rate of implementation substantially varied between rural and urban areas. In Japan, where authorities were left to choose the activities for suicide prevention on their own, most preferred to implement ‘public awareness campaign’ and ‘training of community service providers’ over ‘face-to-face counselling’ or introduction of ‘trauma-informed policies and practices’. Such results require a further deeper investigation into the reasons for and effects of certain patterns of implementation. For example, clearly defined one-off projects, activities with specified funding attached to them and having specific agencies responsible for their implementation increase the likelihood of full implementation, whereas activities that are less tangible and thus harder to define, and without a lead agency can be more difficult to implement (Sheehan et al., 2015).

Why certain activities were implemented over others can be understood through evaluation of implementation process in identifying challenges and facilitators as well as views of stakeholders on the process. We found that qualitative research methods, such as Theory of Change workshops, stakeholder meetings, qualitative interviews and focus group discussions, were frequently employed to understand barriers and facilitators of MHPPs’ implementation. Most cited barriers to implementation were poor dissemination of implementation guidelines, inadequate resources (e.g., financial, human or infrastructural) to support the reform process and resistance to changes. Some studies indicated low prioritisation of mental health and stigma as barriers; others reported weak management and poor intersectoral collaboration, difficult political context and the complex nature of interventions as factors hindering policy implementation. In LMICs, these challenges are more and greater than in HICs, especially in terms of funding, human resources and administration (Zhou et al., 2018). In contrast, clear understanding of roles and responsibilities for implementation and ensuring coordination between different stakeholders were identified as facilitating factors.

Context is crucial for appropriate assessment and understanding of the implementation process. Studies largely adapted their evaluation questions to the features of the political, social or economic environment (Petersen et al., 2011; Ryan et al., 2020). In post-conflict areas like Bosnia and Herzegovina, foreign influence was identified as a central theme in implementation of mental health reform, which raised questions on sustainability. In resource-constrained contexts, prioritisation of mental healthcare can be challenging, especially when burden of physical health conditions is high, which hindered implementation of MHPPs (Doku et al., 2008; Draper et al., 2009; Ssebunnya et al., 2010). In decentralised healthcare systems, such as in South Africa, translation of national policies into strategic plans appropriate to the provincial or district level contexts seems to be a key factor for ensuring their successful implementation (Draper et al., 2009).

Assessment of the impact of implementation was largely performed via both quantitative methods, most often pre- and post-policy reform, and qualitative methods, most often by asking stakeholders about their perception on changes brought by MHPPs.

The relatively poor assessment of implementation of MHPPs is in contrast to the more advanced tools used to monitor and guide the implementation of policies at all levels in other public health areas such as in tobacco (WHO, 2013; Cox et al., 2014) and alcohol control (Rekve, 2011), breastfeeding promotion (WHO, 2003; The International Baby Food Action Network Asia, 2008; WHO, 2013) or family planning and reproductive health (Bhuyan et al., 2010). For instance, the Policy Implementation Assessment Tool was developed to guide an assessment of national family planning and reproductive health policy implementation. This tool includes instructions on policy assessment at various levels from stakeholder mapping to organising and analysing data (Bhuyan et al., 2010). It enables to gather information via multifaceted processes and in a systematic, user-friendly manner. The tool consists of an interview guide that is divided into eight sections that focus on assessing context, process of implementation and appropriateness of policy strategies in relation to its objectives. This tool could be potentially adjusted for mental health policies.

Limitations

We recognise that our search strategy was not able to capture all relevant studies, particularly those that focused on the impact of policy implementation. Potentially valuable information could have been missed when studies are published in project reports on certain areas of policy or published in academic journals without mentioning its relation to a specific policy.

There is a lack of information on the tools used in most studies included in our review, as such we were unable to assess the quality of evaluation methods. Instead, we provided information on whether studies described their tools sufficiently and whether they were interpretable.

Due to the broad scope of this review, we were unable to compare tools across contexts and applications. Further research is necessary to determine which tools are optimal for assessing the implementation of MHPPs and to develop recommendations and guidance on evaluation of MHPPs.

Conclusions and recommendations

Our review has highlighted substantial knowledge gaps in assessing the implementation of MHPPs. Our findings should contribute to policy dialogues on the development, implementation and assessment of implementation of national mental health strategies. Efforts should be made to consolidate available methods and tools into clear methodologies that would address various stages and objectives of implementation taking into consideration a variety of possible policy goals. Such a consolidated methodology might result in a checklist that would mirror each objective of MHPPs and that would allow for various contexts to be taken into account as well as for experiences and lessons from implementation and evaluation to be shared.

Based on our review, we recommend the following:

1. Strategy or plan of evaluation of implementation needs to be an integral part of MHPPs and it needs to contain responsibilities and funding.

2. Future evaluations of MHPPs implementation need to be more transparent in reporting details, especially on tools and methodologies used and, where possible, make them accessible to readers.

3. Since the resources are constrained in all settings, partnerships need to be built to ensure high-quality evaluations. Such partnerships might include universities, research institutes and other organisations, both nationally and internationally.

4. Evaluations can be fragmented into smaller studies focused on specific aspects of MHPPs; however, findings from these studies should be put together into complex evaluation reports of MHPPs implementation. Both smaller studies and complex evaluation reports shall be published in peer-reviewed journals to ensure their accessibility and impact.

5. More research needs to be done to understand the current implementation of MHPPs so the lessons made could be learned.

Open peer review

To view the open peer review materials for this article, please visit http://doi.org/10.1017/gmh.2023.3.

Author contributions

A.A.A., P.W. and Y.Y. initiated, planed and designed the study. A.A.A. and H.T. conducted the literature review and prepared the first draft of the paper. A.K. and Z.G. participated in designing the study and contributed to writing and proofreading the draft. P.W., Y.Y. and L.L. supervised the whole study and critically revised the manuscript. M.R., C.R., A.M.T.I., and J.M. critically revised the manuscript and contributed to the final draft of the paper.

Financial support

This work was supported by the World Health Organisation Regional Office for Europe.

Competing interest

The authors declare none.