Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Abstract

Background

Mental health is a critical component of wellness. Public policies present an opportunity for large-scale mental health impact, but policy implementation is complex and can vary significantly across contexts, making it crucial to evaluate implementation. The objective of this study was to (1) identify quantitative measurement tools used to evaluate the implementation of public mental health policies; (2) describe implementation determinants and outcomes assessed in the measures; and (3) assess the pragmatic and psychometric quality of identified measures.

Method

Guided by the Consolidated Framework for Implementation Research, Policy Implementation Determinants Framework, and Implementation Outcomes Framework, we conducted a systematic review of peer-reviewed journal articles published in 1995–2020. Data extracted included study characteristics, measure development and testing, implementation determinants and outcomes, and measure quality using the Psychometric and Pragmatic Evidence Rating Scale.

Results

We identified 34 tools from 25 articles, which were designed for mental health policies or used to evaluate constructs that impact implementation. Many measures lacked information regarding measurement development and testing. The most assessed implementation determinants were readiness for implementation, which encompassed training (n  =  20, 57%) and other resources (n  =  12, 34%), actor relationships/networks (n  =  15, 43%), and organizational culture and climate (n  =  11, 31%). Fidelity was the most prevalent implementation outcome (n  =  9, 26%), followed by penetration (n  =  8, 23%) and acceptability (n  =  7, 20%). Apart from internal consistency and sample norms, psychometric properties were frequently unreported. Most measures were accessible and brief, though minimal information was provided regarding interpreting scores, handling missing data, or training needed to administer tools.

Conclusions

This work contributes to the nascent field of policy-focused implementation science by providing an overview of existing measurement tools used to evaluate mental health policy implementation and recommendations for measure development and refinement. To advance this field, more valid, reliable, and pragmatic measures are needed to evaluate policy implementation and close the policy-to-practice gap.

Plain Language Summary

Mental health is a critical component of wellness, and public policies present an opportunity to improve mental health on a large scale. Policy implementation is complex because it involves action by multiple entities at several levels of society. Policy implementation is also challenging because it can be impacted by many factors, such as political will, stakeholder relationships, and resources available for implementation. Because of these factors, implementation can vary between locations, such as states or countries. It is crucial to evaluate policy implementation, thus we conducted a systematic review to identify and evaluate the quality of measurement tools used in mental health policy implementation studies. Our search and screening procedures resulted in 34 measurement tools. We rated their quality to determine if these tools were practical to use and would yield consistent (i.e., reliable) and accurate (i.e., valid) data. These tools most frequently assessed whether implementing organizations complied with policy mandates and whether organizations had the training and other resources required to implement a policy. Though many were relatively brief and available at little-to-no cost, these findings highlight that more reliable, valid, and practical measurement tools are needed to assess and inform mental health policy implementation. Findings from this review can guide future efforts to select or develop policy implementation measures.

Introduction

Mental health is a critical component of wellness at both the individual- and population-level. A significant proportion of the population is affected by mental illness, including an estimated 20%–40% of the United States (Kessler et al., 2005; Substance Abuse and Mental Health Services Administration, 2018) and international populations (Chisholm et al., 2007; Wittchen et al., 2011). The prevalence of mental illness should also be considered in light of global current events, such as the COVID-19 pandemic (Auerbach & Miller, 2020; Czeisler et al., 2020; Holmes et al., 2020; United Nations, 2020), splintered political systems (Nayak et al., 2021; Yan et al., 2021), and civil unrest (Labott, 2021; Ni et al., 2020; Strohecker, 2021). Unfortunately, access to mental health services is limited for many, due to barriers such as provider shortages, distance from services, and insufficient funding (Kilbourne et al., 2018; Saxena et al., 2007; Stirman et al., 2016; Williams, 2016). These challenges are often exacerbated in developing countries, where resources and mental health infrastructure (e.g., psychiatric hospitals or residential treatment facilities) are even more limited (Ngui et al., 2010). As a result, policy interventions are needed to improve mental health on a population-level.

Public policies present an opportunity to impact mental health education and services on a large scale, reaching, for example, communities, organizations, families, and individuals simultaneously. These “Big P” policies include “laws, regulatory measures, courses of action, and funding priorities concerning a given topic by a governmental entity or its representatives” (Eyler & Brownson, 2016). Within this study, we define “Big P” mental health policies as mandates, guidelines, or recommendations established by governmental entities or legislative bodies for certain practices or services that affect mental health education and/or the provision of mental health services. In the United States public mental health policies have increased access to evidence-based mental health interventions, encouraged changes to mental health service provision using incentives or disincentives, and created preferred treatment lists (Goldman & Azrin, 2003; Raghavan et al., 2008; Wensing et al., 2020). Public policies also have the potential to improve mental health equity by addressing the social determinants of health (SDOH; i.e., economic stability, built environment, health and healthcare, social context, and education) (Dawes, 2016; Rust et al., 2015), concepts such as a stigma and discrimination for those with mental illness, as well as healthcare infrastructure, housing, transportation, and education (Alegría et al., 2003, 2018; Castillo et al., 2018; Hinshaw, 2006; Powell et al., 2016; Purtle et al., 2020).

Despite the potential of public policies to improve health, developing and successfully passing a policy does not guarantee its implementation or sustainment. Policy implementation refers to the process of carrying out a policy mandate (Natesan and Marathe, 2015) and remains understudied compared to earlier stages in the policy process (Bullock et al., 2021). There is an opportunity to apply principles and methods from implementation science, which seeks “to understand not only what is and is not working, but also how and why implementation is going right or wrong, and testing approaches to improve it” (Peters et al., 2013). Despite the potential impact on population health, there is a dearth of implementation science literature directed at the policy sphere, particularly related to mental health, leading to calls for a greater emphasis on policy research (Emmons & Chambers, 2021; Hoagwood et al., 2020; Oh et al., 2021; Purtle et al., 2017, 2016).

Policy implementation is studied by multiple fields, including—but not limited to—implementation science and political science; however, the terminology differs between disciplines. Though both fields emphasize context and its role in helping or hindering implementation, implementation science often distinguishes between internal and external context (commonly called inner setting and outer setting; Damschroder et al., 2009; Nilsen et al., 2013). Additionally, implementation science researchers measure implementation outcomes, frequently citing the Implementation Outcomes Framework (IOF) (Proctor et al., 2011), which encompasses the following: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability. Political science, on the other hand, refers to implementation outcomes as outputs, while implementation outcomes refer to the policy's overall impact on the identified problem (e.g., changes in health outcomes after implementing a nutrition policy; DeGroff & Cargo, 2009). Finally, fidelity to a policy (i.e., the degree to which it was implemented as intended; Proctor et al., 2011) is conceptualized as compliance within political science (Natesan and Marathe, 2015). This review applies an implementation science lens to policy implementation, thus using the following terms throughout: outer setting (see Table 1 for constructs and definitions) and implementation outcomes (see Table 2 for constructs and definitions), which includes fidelity/compliance.

Table 1.

Mental Health Policy Implementation Determinants Assessed in Included Measures (n  =  34)^a.

Domain	Included measures n (%)	Definition
Policy characteristics
Adaptability	5 (14%)	Degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needsc
Complexity	4 (11%)	Perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and number of steps required to implementc
Inner setting
Presence of champions	3 (9%)	Field or practice leaders, people who can facilitate and support practice change among professionalsc,d
Organizational culture and climate	11 (31%)	Culture: Norms, values, and basic assumptions of a given organization; or Climate: Absorptive capacity for change, extent policy compliance will be rewarded, supported, and expected within their organizationc
Policy implementation climate	10 (29%)	Extent to which compliance with the policy mandate will be rewarded, supported, and expected within their organizationc
a. Goals and feedback	3 (9%)	Degree to which [the policy-mandate] goals are clearly communicated, acted upon, and fed back to staff and alignment of that feedback with goalsc
b. Relative priority	6 (17%)	Individuals’ shared perception of importance of the [policy] implementation within the organization, competing prioritiesc
Opinion leaders	1 (3%)	Individuals in an organization who have formal or informal influence on attitudes and beliefs of their colleagues with respect to implementing the policyc
Readiness for implementation	27 (77%)	Tangible and immediate indicators of organizational preparations to implement a policy interventionc
a. Communication of policyb	4 (11%)	Actions taken to disseminate policy requirements and guidelines to implementers
b. Policy awareness or knowledgeb	6 (17%)	Implementing staff/provider awareness the policy mandate exists, or knowledge of policy content
c. Leadership for implementation	5 (14%)	Commitment, involvement, and accountability of leaders and managers with the implementationc,d
d. Trainingb	20 (57%)	Training of staff/providers on how to implement the policy-mandated practices
e. Non-training resourcesb	12 (34%)	Level of resources dedicated for implementation and on-going operations including money…physical space, and time other than training resourcesc
Structure of organization	6 (17%)	The social architecture, age, maturity, and size of an organizationc
Outer setting
Actor relationships and networks	15 (43%)	Presence and characteristics of relationships between parallel organizations that must collaborate for policy implementation to be effectived
Visibility of policy role/actors to others	2 (6%)	Perceived presence and importance of different actors pertinent to implementation of the policyd
Political will for policy implementation	1 (3%)	Societal desire and commitment to generate resources to carry out policiesd
Target population characteristics	4 (11%)	Demographics, norms, neighborhood environments of population groups that affect implementationd

^a Implementation determinants: “Factors believed or empirically shown to influence implementation outcomes” (also called barriers, obstacles, facilitators, etc.) (Nilsen & Bernhardsson, 2019).

^b These constructs were derived from Damschroder et al., 2009 but were sub-divided based on findings during full-text screening.

^c Definition derived from Damschroder et al., 2009. ^d Definition derived from Bullock et al., 2021.

Table 2.

Mental health policy implementation outcomes assessed in included measures (n  =  34)^a.

Domain	Included measures n (%)	Definition
Acceptability	7 (20%)	Perceptions by staff in organizations mandated to implement the policy, or perceptions of other stakeholders, that the policy mandate is agreeable, palatable, or satisfactory (Proctor et al., 2011)
Adoption	4 (11%)	Intention and initial actions of mandated organizations to revise their organizational policies to address policy mandates. (Not policy development or passage of bills into law.) (Proctor et al., 2011)
Appropriateness	7 (20%)	Perceived fit, relevance, or compatibility of the [policy] for a given practice setting, provider, or consumer; and/or perceived fit of the [policy] to address a particular issue or problem; context fit (Proctor et al., 2011)
Cost	5 (14%)	Cost impact of an implementation effort (Proctor et al., 2011)
Feasibility	6 (17%)	Extent to which a new [policy] can be successfully used or carried out within a given agency or setting Level of administration required to implement a policy, often called policy automaticity (Proctor et al., 2011)
Fidelity/compliance	9 (26%)	Degree to which a [policy] was implemented as it was prescribed [mandated] (Proctor et al., 2011)
Penetration	8 (23%)	Integration of a [policy] within a service setting and its subsystems (Proctor et al., 2011)
Sustainability	0 (0%)	Extent [new policy] is maintained or institutionalized within a service setting's ongoing, stable operations (Proctor et al., 2011)
Health equity	6 (17%)	Measure contains items related to health equity or assessing the social determinants of health as they relate to the implementing organization or target population (U.S. Department of Health and Human Services, 2021)

^a Implementation outcomes: “Effects of deliberate and purposeful action to implement new treatments, practices” or policies, which can serve as indicators of the implementation process and overall success (Proctor et al., 2011).

Current Work and Gaps in the Literature

Though policies may be developed and passed by government entities at multiple levels, the burden of implementing policies and sustaining related practices often falls on actors beyond the legislative body (e.g., healthcare settings, community organizations). Additionally, many actors or organizations face additional barriers during policy implementation, including limited financial resources, personnel, or infrastructure (Wright, 2017). To enable comparisons of policy implementation across studies and settings, generalize findings, and ultimately address these barriers, a better understanding of policy-focused implementation constructs is needed, along with high-quality measures of these constructs.

Unfortunately, myriad measurement challenges exist within implementation science. First, many measures fail to operationalize the constructs being assessed (Bruns et al., 2019; McHugh et al., 2020; Watson et al., 2018). There is also an overreliance on single-use measures, which often lack testing and/or reporting of psychometrics, and splicing items from different measures, leading to limited measurement consistency and replicability of findings (Lewis, Weiner, et al., 2015; Martinez et al., 2014; Proctor et al., 2011). Additionally, there is a limited emphasis on pragmatic measures, which are developed to minimize burden and are typically brief, inexpensive, and easy to score and interpret (Glasgow & Riley, 2013; Martinez et al., 2014; Rabin et al., 2015). Despite the importance of psychometric and pragmatic properties, researchers rarely analyze both when developing and testing a new measure (Martinez et al., 2014; Stanick et al., 2018).

Implementation researchers acknowledge the dearth of measures assessing outer setting constructs (Damschroder et al., 2009), which encompass fiscal and policy factors, as well as external networks that can affect implementation (McHugh et al., 2020). These constructs are often difficult to disentangle and assess, which may explain why they remain understudied compared to inner setting constructs (McHugh et al., 2020). Inner setting constructs include factors within an implementing organization that influence implementation (e.g., organizational structure, implementation climate, leadership support) (Damschroder et al., 2009). Policy implementation can be affected by multi-level contextual factors, including outer and inner setting characteristics, and characteristics of the policy itself (Bullock et al., 2021; Eaton et al., 2011; Goldman & Azrin, 2003; Malekinejad et al., 2018). Also known as barriers, obstacles, enablers, or facilitators (Nilsen & Bernhardsson, 2019), we refer to these multi-level constructs as “policy determinants.” Given the limited emphasis on measurement of outer setting constructs and the effect these determinants can have on policy implementation, a better understanding of these constructs is needed.

Additionally, health equity is receiving increased attention within implementation science, underscoring the impact of the SDOH on mental health outcomes (Allen et al., 2014; Compton and Shim, 2015; Shim & Compton, 2020; World Health Organization & Calouste Gulbenkian Foundation, 2014). There are growing calls for improved integration of equity-related constructs into implementation methods, models, and measures (Baumann & Long, 2021; Brownson et al., 2021; Chinman et al., 2017; Douglas et al., 2019; Emmons & Chambers, 2021; Exworthy, 2008; Loper et al., 2021; Metz et al., 2021; Oh et al., 2021; Woodward et al., 2021). However, little is known about the current state of equity-related constructs in mental health policy implementation.

Lastly, while systematic reviews have examined quantitative measures of implementation across public health topics (Chaudoir et al., 2013; Chor et al., 2015; Clinton-McHarg et al., 2016; Emmons et al., 2011; Kaplan et al., 2010; Weiner et al., 2008), less scholarship has been done related to measures of policy implementation. Recent work has examined generalizable measures of health policy implementation (Allen et al., 2020), as well as health policy measures targeting school settings (McLoughlin et al., 2021), healthy food environments (Phulkerd et al., 2016), and chronic disease (Walsh-Bailey et al., 2020). To our knowledge, no such study has been conducted for mental health policies. Measures to evaluate mental health policy implementation are particularly important, as mental health stigma is pervasive and can impact policy development and implementation, as well as mental health outcomes (Conley, 2021; Purtle et al., 2018).

Study Aims

To fill these gaps in the literature, this study sought to (1) identify quantitative measurement tools (i.e., measures, questionnaires, surveys, scales) used to evaluate the implementation of public mental health policies; (2) describe the implementation determinants and outcomes assessed in these measures; and (3) assess the pragmatic and psychometric quality of the identified measures.

Methods

This review builds from a previous review of generalizable implementation measures (i.e., measures transferable across topics and settings) of public policies passed at federal, state/regional, or local levels (Allen et al., 2020). The Allen et al., study identified 70 generalizable measures of health policy implementation and excluded 597 measures, which were policy- or setting-specific (e.g., specific to school settings or mental health policies) (Allen et al., 2020). The current review examined the previously excluded or policy-specific quantitative measurement tools used to evaluate the implementation of governmental mental health policies. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Moher et al., 2009) (see Figure 1 and Appendix A).

Figure 1.

Quantitative studies: PRISMA flow diagram.

Search Strategy

Our search strategy was guided by three implementation science frameworks—the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009), the IOF (Proctor et al., 2011), and a newer, policy-specific framework—the Policy Implementation Determinants Framework (Bullock et al., 2021). The Policy Implementation Determinants Framework and CFIR provide a set of determinants known to impact implementation and were used to create a more thorough representation of policy implementation determinants. While the CFIR is applicable to a variety of implementation levels and settings, the Policy Implementation Determinants Framework contains constructs tailored to policy and the outer setting. The study team operationalized construct definitions from the frameworks to make them applicable to policy (e.g., changed the referent). For example, the IOF describes acceptability of a treatment, service, or innovation; we expanded the definition to include policy or policy-mandated practice. These frameworks guided the identification and selection of implementation determinants and outcomes. Both the CFIR and Policy Implementation Determinants Framework were used to identify determinants of policy implementation, while the IOF was used to identify outcomes of policy implementation.

Allen et al., (2020) conducted the original search, encompassing health policy broadly, in April 2019 in CINAHL Plus, Medline, PsycINFO, ERIC, Worldwide Political, and PAIS databases. The search terms and strategy were constructed with guidance from a research librarian, based on policy and implementation science literature (Bullock et al., 2021; Damschroder et al., 2009; Lewis et al., 2018; Proctor et al., 2011; Rabin et al., 2015; Sabatier & Mazmanian, 1980; Sabatier, 1988; Watson et al., 2018) and expert input. Studies excluded due to policy- or setting-specific wording (e.g., applicable only to mental health policies) were marked in Covidence systematic review software (Covidence Systematic Review Software, 2021).

In the current review, we retrieved studies excluded during full-text screening by Allen et al., due to policy- or setting-specific measures (n  =  597). We then updated the original search strategy, with guidance from a research librarian, to include mental health-specific terms; a similar approach was used in previous reviews (McLoughlin et al., 2021; Walsh-Bailey et al., 2020). We ran the updated search in January 2021, using search strings related to mental health, public policy, implementation, and measurement (see Appendix B; full search strategy available in Appendices C and D).

Inclusion and Exclusion Criteria

The inclusion criteria were: (1) empirical study (including study protocols) of the implementation of public mental health policies (i.e., “Big P” policies) already passed or approved; (2) use of a quantitative measurement tool (i.e., measure, questionnaire, survey, scale); (3) inclusion of at least one implementation determinant or outcome; and (4) published in a peer-reviewed journal between January 1995 through December 2020 in English language. Given faint line between policy enactment and stages of policy implementation, we included studies that did not explicitly name the policy as the independent variable.

To distinguish between “little p” (i.e., organizational-level mandates or guidelines) and “Big P” policies, empirical authors’ descriptions of governing or legislative bodies and policy decision levels (e.g., local, state, national) were used. When information was limited in articles, one reviewer (MP) conducted hand searches to determine the governing body and level responsible for mandating or recommending the policy. Because our coding required copies of each measurement tool, January 1995 was selected as the beginning of the inclusion period as web-based surveying began around that time (Dillman et al., 2009). The onset of web-based data collection was thought to increase the likelihood that instruments had been maintained and were accessible, as compared to paper-based data collection.

The team conducted hand searches to locate measure development articles, technical documents, etc. that contained verbatim items and described the initial development and psychometric testing of the quantitative instruments used in studies that met all inclusion criteria (Lewis et al., 2018). Measure development articles were not included as part of the review, but these were reviewed by the primary (MP) and secondary (EJ) reviewers for relevant information about the measurement tool properties.

We applied the following exclusion criteria: (1) Articles focused solely on agenda setting, policy development, policy content analysis, or policy evaluation were excluded, as this review centered on the implementation of policies once adopted. (2) Empirical studies reported in theses and books were also excluded, as this study included only peer-reviewed journal articles. (3) Because this review aimed to assess implementation outcomes related to policy, studies which assessed only individual-level outcomes (e.g., health behaviors)—also referred to as client outcomes—or service outcomes (e.g., efficiency, treatment effectiveness) were excluded (Proctor et al., 2011). (4) Other health policies implemented in mental health settings or with mental health service users as the target populations (e.g., chronic disease screenings among individuals in residential treatment facilities; housing policies for adults with serious mental illness) were excluded, as this review focused specifically on policies with direct influence on practices or services that affect mental health education and the provision of mental health services (e.g., Affordable Care Act and Medicaid expansion increasing access to mental health care; state-level mandate for implementation of evidence-based mental health interventions). (5) Studies were excluded if full or partial measurement tools could not be located.

Screening Procedures

De-duplication was conducted using Endnote version X9 and Covidence (Covidence Systematic Review Software, 2021), which was also used to manage references and screen studies. We first reviewed all records excluded by Allen et al., (2020) during full-text screening (n  =  597) for policy- and setting-specific measures, then reviewed studies published between April 2019 and December 2020 from the updated search. Two independent reviewers (MP, EJ) screened titles and abstracts to evaluate eligibility. They then independently screened full-text articles to determine inclusion for extraction and resolved disagreements by discussion. When needed, one reviewer (MP) contacted study authors for additional materials or information (e.g., copies of data collection instruments) to clarify eligibility. Three attempts were made to contact each study author for additional materials; those that remained unavailable were excluded.

Data Extraction

Data were extracted from study articles and accompanying materials (e.g., measure development articles, survey instruments) using Microsoft Excel. One reviewer (MP) extracted data for each measure, and a second reviewer (EJ) checked each entry and noted disagreements. The reviewers then resolved disagreements by discussion. A codebook was iteratively developed to aid in extracting and deductively coding study- and measure-related information. For studies using multiple measures, each measure was extracted separately (i.e., reported in separate rows).

Data extraction included information about: (1) general study and policy information; (2) methods used; (3) measure development and testing; (4) implementation determinants and outcomes (see Tables 1 and 2); (5) measure pragmatic and psychometric properties; and (6) policy impact on health equity in included measures. Health equity could be conceptualized as either a policy determinant or outcome, and include references to health disparities, health equity, and the SDOH (U.S. Department of Health and Human Services, 2021). Information about measure development/testing or a measure's pragmatic and psychometric properties was extracted from included empirical articles or, when applicable, from measure development articles.

The Psychometric and Pragmatic Evidence Rating Scales (PAPERS) were used to assess pragmatic and psychometric properties of quantitative measures (see Appendix E) (Lewis et al., 2018, 2021; Powell et al., 2017; Stanick et al., 2019). Psychometric criteria consist of nine categories, each scored on a −1 to 4 scale, with possible total scores of −9 to 36. There are five pragmatic criteria, with possible total scores of −5 to 20. Higher values indicate stronger psychometric properties or more pragmatic characteristics. The psychometric properties evaluated include: (1) internal consistency; (2) convergent validity; (3) discriminant validity; (4) known groups validity; (5) predictive validity; (6) concurrent validity; (7) structural validity; (8) responsiveness; and (9) norms. The pragmatic scale includes five domains: (1) brevity; (2) simplicity of language; (3) cost; (4) training burden; and (5) analysis burden.

Results

General Characteristics of Included Studies

As outlined in Figure 1, the search located 16,015 articles. After removing 4,790 duplicates, we reviewed 11,225 records during title and abstract screening. Two reviewers then screened 105 full-text articles; ultimately, 80 were excluded, leaving 25 articles for inclusion in this review. Within the 25 included studies, there were 34 unique quantitative measures (see Table 3). Many studies administered multiple measures, and two measures—the Evidence-Based Practice Attitude Scale (Aarons, 2004) and the Organizational Social Context Measurement System (Glisson et al., 2008)—were used in two studies.

Table 3.

Quantitative measures identified in studies of mental health policy implementation (n  =  34 unique measures in 25 studies).

Tool name	Number of items	Empirical use Author, year	Country	Description of mental health policy
World Health Organization's Mental Health Policy Checklist	88	(Awenva et al., 2010)	Ghana	12 mental health objectives: Includes de-centralizing mental health services; providing training for mental health professionals; providing transportation for community mental health workers; raising mental health awareness for the family and community; and providing community rehabilitation of the mentally ill
World Health Organization's Mental Health Plan Checklist	159	(Awenva et al., 2010)	Ghana
Organizational Social Context Measurement System	105	(Beidas et al., 2013)	United States	Policy promoting evidence-based treatments in community centers through Philadelphia's Department of Behavioral Health and Intellectual DisAbility Services (DBHIDS)
Evidence-Based Practice Attitude Scale	15	(Beidas et al., 2013)	United States
Infrastructure, Policies and Practices in Child and Adolescents’ Mental Health Questionnaire	160	(Bielsa et al., 2010)	Multiple: 15 European countries	Policies related to mental health promotion and mental disorder prevention for children and adolescents
Higher Education Institution Survey	43	(Callaghan et al., 2012)	England	17 recommendations from the Chief Nursing Officer for England: Includes providing evidence-based care; strengthening relationships with service users and carers; supporting continued professional development for mental health nurses; and improving recruitment and retention for mental health nurses
Mental Health Trust Survey	96	(Callaghan et al., 2012)	England
Mental Health Nursing Survey	128	(Cusack et al., 2017)	Ireland	Policy proposes a holistic view of mental health difficulties, recommends an integrated multidisciplinary approach to mental healthcare, and emphasizes an active partnership between service users, their carers, and family members at every level of service provision
Health Policy Questionnaire	246	(Dlouhy, 2014)	Multiple: 7 European countries	Any national mental health policy initiative across Eastern Europe
Health Education Experts Survey	27	(Eisman et al., 2020)	United States	Michigan Model for Health: A comprehensive, evidence-based K-12 health curriculum, which includes core content areas focused on social and emotional health, nutrition and physical activity, safety, substance use, and personal health and wellness
Primary Care Integration Survey	9	(Gerber, 2018)	South Africa	Requirements for a revised mental health system with supports for the protection of human rights through the Department of Health of South Africa
School Health Policies and Practices Survey (SHPPS)	93	(Guerra et al., 2019)	United States	Recommendations from the Institute of Medicine's 2009 report: Individual states and communities coordinate to apply resources to the promotion and prevention of mental, emotional, and behavior disorders Every Student Succeeds Act: Promoting evidence-based interventions and student supports within the school system
School Health Profiles Survey for Principals	180	(Guerra et al., 2019)	United States
School Health Profiles Survey for Teachers	207	(Guerra et al., 2019)	United States
Cognitive Behavioral Therapy Survey	10	(Kenicer et al., 2012)	Scotland	Recommendations from the National Institute of Clinical Excellence and the Scottish Intercollegiate Guidelines Network: Promoting guided self-help and computerized cognitive behavioral therapy for the treatment of depression, anxiety, and bulimia nervosa
Perceived Characteristics of Intervention Scale	8	(Lau & Brookman-Frazee, 2016)	United States	Los Angeles County Department of Mental Health, Prevention and Early Intervention Program: Providing training and implementation support for six evidence-based/informed practices to address a range of prevalent youth mental health problems
Knowledge and Confidence Survey/Therapists’ Perceptions of EBP Self-Efficacy	2	(Lau & Brookman-Frazee, 2016)	United States
Practice-specific Implementation Support	24	(Lau & Brookman-Frazee, 2016)	United States
Organizational Climate Measure	82	(Lau & Brookman-Frazee, 2016)	United States
Organizational Readiness to Change	97	(Lau & Brookman-Frazee, 2016)	United States
Adaptations of Evidence-based Practices	9	(Lau & Brookman-Frazee, 2016)	United States
Evidence-Based Practice Attitude Scale	15	(Lau & Brookman-Frazee, 2016)	United States
Organizational Social Context Measurement System	105	(Lau & Brookman-Frazee, 2016)	United States
Crisis Resolution Team Survey	91	(Lloyd-Evans et al., 2018a)	England	National Health Service Plan of 2000: National mandate for crisis resolution teams (i.e., multidisciplinary, specialist mental health services that offer brief intensive home treatment to people experiencing a mental health crisis)
Crisis Resolution Team Practice Survey	137	(Lloyd-Evans et al., 2018b)	England
World Health Organization's Assessment Instrument for Mental Health Systems Version 2.2 (WHO-AIMS)	156	(Lund et al., 2010)	South Africa	Mental Health Care Act of 2002: Sets in place mechanisms for decentralization of services, integration of mental health into general health care and the development of community-based care
Implementation Leader Survey	277	(McGinty et al., 2018)	United States	Affordable Care Act: Enables creation of state-level Medicaid health home programs
Illness Management and Recovery Survey	37	(McGuire et al., 2013)	United States	Uniform Mental Health Services Handbook: Mandates the inclusion of “illness management and recovery groups” within Psychosocial Rehabilitation and Recovery Centers
Suicide Prevention Monitor Survey	10	(Mokkenstorm et al., 2018)	Netherlands	National Suicide Prevention Strategy 2014–2017 from the Dutch Ministry of Health, Welfare and Sports: Includes 17 goals across 4 domains—healthcare, education, media and the social-economic sector
Social Network Analysis Questionnaire	N/A	(Park et al., 2014)	Canada	Changing Directions, Changing Lives: The Mental Health Strategy for Canada: Promotes recovery-oriented services in mental health settings
School Mental Health Policy Survey	26	(Patalay et al., 2016)	Multiple: 10 European countries	European Joint Action on Mental Health and Wellbeing (2013–2016): Provides a framework for action in mental health policy and addresses the promotion of mental health in schools
Veteran's Health Administration All Employee Survey	14	(Smith et al., 2017)	United States	Veteran's Health Administration policy, Re-Engage Program: Requires all VA sites to implement a new brief care management program for veterans with serious mental illness
Diagnosis Treatment Combinations Survey	23	(Tummers, 2011)	Netherlands	Health Insurance Law, Diagnosis Related Groups Policy: Stipulates a standard cost for each mental health disorder, thus determining the level of financial exchange for mental health care provision
Diagnosis Treatment Combinations Survey II	23	(Tummers, 2012)	Netherlands
Diagnosis Related Groups Survey	21	Tummers & Van de Walle, 2012)	Netherlands
Diagnosis Related Groups Survey II	15	(Tummers & Bekkers, 2014	Netherlands

The studies were published between 2009 and 2020 and were conducted primarily in the United States (n  =  8), the Netherlands (n  =  5), multiple European countries (n  =  3), England (n  =  3), and South Africa (n  =  2). A single study was also conducted in the following countries: Scotland, Ireland, Ghana, and Canada. A variety of mental health policies were represented. Some policies impacted mental health support services for children and adolescents, while others addressed health insurance or the associated costs of implementation for both consumers and organizations/systems. While some measurement tools were developed specifically to evaluate the implementation of mental health policies (e.g., the World Health Organization's Mental Health Plan/Policy Checklists), others were used to evaluate constructs that impact implementation (e.g., Evidence-Based Practice Attitude Scale, Organizational Readiness to Change Assessment).

While the application of theories, models, and frameworks varied across studies, implementation frameworks such as the Exploration, Preparation, Implementation, Sustainment framework (Moullin et al., 2015), and the CFIR were used in multiple studies. Additional study characteristics, including study design, quantitative methods, setting, and policy level, are highlighted in Table 4, while Table 5 provides details about the measures used in included studies.

Table 4.

Characteristics of included studies with quantitative measures (n  =  25).

Study characteristic	n
Study design
Cross-sectional	18
Protocol	3
Case study	2
Longitudinal	2
Quantitative method used
Survey	24
Secondary data	1
Use of theories, models, or frameworks reported
No	14
Yes	11
Setting
Healthcare	14
Health insurance	4
Broad/Multiple	4
Schools	3
Policy level
National	20
State	1
Local	4

Table 5.

Characteristics of measures utilized in studies of mental health policy implementation (n  =  34 unique measures in 25 studies).

Tool name	Development Author, year	Empirical use Author, year	Implementation determinants assessed	Implementation outcomes assessed	Pragmatic PAPERS score	Psychometric PAPERS score	Psychometric properties assessed
World Health Organization's Mental Health Policy Checklist	(World Health Organization, 2007)	Awenva et al., 2010	RI: Non-training resources; RI: Training; Actor relationships/networks; Visibility of policy role and actors	Feasibility; Fidelity/compliance; Cost; Health equity	11	−1	Norms
World Health Organization's Mental Health Plan Checklist	(World Health Organization, 2007)	Awenva et al., 2010	Readiness to implement; RI: Non-training resources; RI: Training; Visibility of policy role and actors	Fidelity/compliance; Cost; Health equity	10	−1	Norms
Organizational Social Context Measurement System	(Glisson et al., 2008)	Beidas et al., 2013	Organizational culture and climate; RI: Communication of policy;	N/A	0	6	Internal consistency; Structural validity; Norms
Evidence-Based Practice Attitude Scale	(Aarons, 2004)	Beidas et al., 2013	RI: Training	Acceptability; Adoption; Appropriateness	13	8	Internal consistency; Structural validity; Norms
Infrastructure, Policies and Practices in Child and Adolescents’ Mental Health Questionnaire	(Braddick et al., 2009)	Bielsa et al., 2010	Readiness to implement; RI: Communication of policy; RI: Non-training resources; RI: Training; Actor relationships/networks; Target population characteristics	Fidelity/compliance; Penetration; Health equity	9	2	Norms
Higher Education Institution Survey	(Callaghan et al., 2012)	Callaghan et al., 2012	IC: Relative priority; Actor relationships/networks	Penetration	12	−1	Norms
Mental Health Trust Survey	(Callaghan et al., 2012)	Callaghan et al., 2012	Champions; IC: Relative priority; RI: Training; Actor relationships/networks	Penetration; Health equity	10	−1	Norms
Mental Health Nursing Survey	(Cusack et al., 2017)	Cusack et al., 2017	RI: Training; Actor relationships/networks	Penetration	9	4	Norms
Health Policy Questionnaire	Dlouhy, 2014	Dlouhy, 2014	Readiness to implement; RI: Policy awareness/knowledge; RI: Non-training resources; Structure of organization	Cost	7	−1	Norms
Health Education Experts Survey	(Eisman et al., 2020)	Eisman et al., 2020	Organizational culture and climate; Policy implementation climate; IC: Relative priority; RI: Non-training resources; RI: Training	Appropriateness; Fidelity/compliance	12	−1	Norms
Primary Care Integration Survey	(Gerber, 2018)	Gerber, 2018	RI: Policy awareness/knowledge; RI: Leadership for implementation; RI: Training	Acceptability	12	1	Norms
School Health Policies and Practices Survey (SHPPS)	(Centers for Disease Control and Prevention, 2019a)	Guerra et al., 2019	Readiness to implement; RI: Communication of policy; RI: Non-training resources; RI: Training; Actor relationships/networks	Acceptability; Fidelity/compliance	14	−1	Norms
School Health Profiles Survey for Principals	(Centers for Disease Control and Prevention, 2019b)	Guerra et al., 2019	Readiness to implement; RI: Communication of policy; RI: Training	Fidelity/compliance; Penetration	9	4	Norms
School Health Profiles Survey for Teachers	(Centers for Disease Control and Prevention, 2019b)	Guerra et al., 2019	Readiness to implement; RI: Training; Actor relationships/networks	N/A	7	4	Norms
Cognitive Behavioral Therapy Survey	(Kenicer et al., 2012)	Kenicer et al., 2012	Readiness to implement; RI: Non-training resources	N/A	13	−1	Norms
Perceived Characteristics of Intervention Scale	(Cook et al., 2014)	Lau & Brookman-Frazee, 2016	Adaptability; Complexity	Appropriateness; Feasibility	14	−1	Internal consistency; Structural validity; Norms
Knowledge and Confidence Survey/Therapists’ Perceptions of EBP Self-Efficacy	(Lau & Brookman-Frazee, 2016)	Lau & Brookman-Frazee, 2016	RI: Training	N/A	13	5	Internal consistency; Norms
Practice-specific Implementation Support	(Lau & Brookman-Frazee, 2016)	Lau & Brookman-Frazee, 2016	RI: Non-training resources; RI: Training; Actor relationships/networks	N/A	11	0	Not reported
Organizational Climate Measure	(Patterson et al., 2005)	Lau & Brookman-Frazee, 2016	Organizational culture and climate; IC: Goals and feedback; RI: Training	N/A	12	4	Internal consistency; Norms
Organizational Readiness to Change	(Lehman et al., 2002)	Lau & Brookman-Frazee, 2016	Organizational culture and climate; IC: Goals and feedback; RI: Non-training resources; RI: Training	N/A	16	3	Internal consistency; Norms
Adaptations of Evidence-based Practices	(Stirman et al., 2013)	Lau & Brookman-Frazee, 2016	Adaptability	Fidelity/compliance	12	0	Not reported
Evidence-Based Practice Attitude Scale	(Aarons, 2004)	Lau & Brookman-Frazee, 2016	RI: Training	Acceptability; Adoption; Appropriateness	13	8	Internal consistency; Structural validity; Norms
Organizational Social Context Measurement System	(Glisson et al., 2008)	Lau, 2016	Organizational culture and climate; RI: Communication of policy	N/A	0	6	Internal consistency; Structural validity; Norms
Crisis Resolution Team Survey	(Lloyd-Evans et al., 2018a)	Lloyd-Evans, 2018a	Structure of organization; Actor relationships/networks	Penetration; Cost	10	2	Norms
Crisis Resolution Team Practice Survey	(Lloyd-Evans et al., 2018b)	Lloyd-Evans, 2018b	Readiness to implement; RI: Non-training resources; RI: Training; Structure of organization; Actor relationships/networks; Target population characteristics	Fidelity/compliance; Penetration	10	2	Norms
World Health Organization's Assessment Instrument for Mental Health Systems Version 2.2	(Saxena et al., 2007)	Lund et al, 2010	Readiness to implement; RI: Non-training resources; RI: Training; Structure of organization; Actor relationships/networks; Target population characteristics	Cost; Health equity	13	−1	Norms
Implementation Leader Survey	(McGinty et al., 2018)	McGinty et al., 2018	Organizational culture and climate; Policy implementation climate; RI: Policy awareness/knowledge; RI: Leadership for implementation; RI: Non-training resources; RI: Training; Structure of organization; Actor relationships/networks; Target population characteristics	Health equity	8	−1	Norms
Illness Management and Recovery Survey	(McGuire et al., 2013)	McGuire et al., 2013	IC: Goals and feedback; RI: Policy awareness/knowledge; RI: Training	Adoption; Fidelity/compliance; Penetration	12	2	Norms
Suicide Prevention Monitor Survey	(Mokkenstorm et al., 2018)	Mokkenstorm et al., 2018	Readiness to implement; RI: Policy awareness/knowledge; RI: Training; Actor relationships/networks	N/A	13	−1	Norms
Social Network Analysis Questionnaire	(Park et al., 2014)	Park et al., 2014	Opinion leaders	N/A	0	0	Not applicable
School Mental Health Policy Survey	Patalay et al., 2016	Patalay et al., 2016	Organizational culture and climate; Policy implementation climate; IC: Relative priority; Readiness to implement; RI: Policy awareness/knowledge; Structure of organization; Actor relationships/networks	N/A	11	4	Norms
Veterans Health Administration All Employee Survey	(U.S. Department of Veteran Affairs, 2021)	Smith et al., 2017	Organizational culture and climate	N/A	12	4	Internal consistency; Structural validity; Norms
Diagnosis Treatment Combinations Survey	(Tummers, 2009)	Tummers, 2011	Adaptability; Complexity; Organizational culture and climate	Acceptability; Appropriateness; Feasibility	12	9	Internal consistency; Convergent validity; Discriminant validity; Structural validity; Norms
Diagnosis Treatment Combinations Survey II	(Tummers, 2012)	Tummers, 2012	Adaptability; Complexity; Organizational culture and climate; Readiness to implement; RI: Leadership for implementation	Acceptability; Appropriateness; Feasibility	12	9	Internal consistency; Convergent validity; Discriminant validity; Structural validity; Norms
Diagnosis Related Groups Survey	(Tummers & Van de Walle, 2012)	Tummers, 2012	Champions; Organizational culture and climate; Policy implementation climate; IC: Relative priority; Readiness to implement; RI: Leadership for implementation	Acceptability; Adoption; Appropriateness; Feasibility	12	4	Internal consistency; Norms
Diagnosis Related Groups Survey II	(Tummers & Bekkers, 2014)	Tummers & Bekkers, 2014	Adaptability; Complexity; Champions; Organizational culture and climate; Policy implementation climate; IC: Relative priority; Readiness to implement; RI: Leadership for implementation	Acceptability; Adoption; Appropriateness; Feasibility	13	8	Internal consistency; Structural validity; Norms

Note. IC  =  implementation climate; RI  =  readiness to implement. PAPERS pragmatic score range: −5 to 20; PAPERS psychometric score range: −9 to 36.

Measure Development and Testing

Of the 34 unique measurement tools, separate measure development articles were available for 10. The remaining quantitative instruments consisted of a mix of “home-grown” questionnaires, which were developed for use within a single study, and multi-use tools (e.g., The World Health Organization's Mental Health Plan Checklist). Overall, information regarding measure development and testing was limited. Less than half of the measures (n  =  17, 45%) defined the constructs they assessed (i.e., provided definitions of constructs within the empirical use article, development article or other supporting documents, or the instrument itself). Only five measures (13%) reported that items were generated by experts in the field. Finally, there was limited information regarding pilot testing of measures. Six measures (16%) reported pilot testing with a representative sample. Only three measures (8%) reported validity or reliability tests based on pilot testing, and only two (5%) reported that the measure was refined based on pilot testing.

Implementation Determinants and Outcomes

As outlined in Table 1, only five measures (14%) assessed policy characteristics, such as the complexity or adaptability of a policy. Inner setting characteristics were far more common, with 32 measures (91%) assessing at least one organization-level construct. Outer setting characteristics, on the other hand, were assessed in less than half of the included measures (n  =  15, 43%).

The most assessed implementation determinant was readiness for implementation (n  =  27, 77%), which included the following subcomponents: communication efforts related to the policy, policy awareness and knowledge, leadership support for implementation, training, and non-training resources (see Table 1 for definitions). Each of these subcomponents were assessed in multiple measures, with particular emphasis on training (n  =  20, 57%) related to policy implementation and non-training resources (n = 12, 34%). Other commonly measured determinants included actor relationships/networks (n  =  15, 43%), organizational culture and climate (n  =  11, 31%), and the policy implementation climate (n  =  10, 29%).

The most commonly assessed implementation outcomes from the 34 measures were fidelity/compliance to the policy (n  =  9, 26%) and penetration of the policy within the designated setting (n  =  8, 23%) (see Table 2 for definitions). The acceptability and appropriateness of the policy were each assessed by seven measures (20%). The feasibility of implementing a policy was assessed by six measures (17%), while the cost of implementation was assessed by five measures (14%). Only four measures assessed adoption (11%), and no measures addressed sustainability.

Health Equity

Finally, six measures (17%) included at least one item assessing health equity or SDOH. Some measures inquired about the integration or coordination of a mental health policy with existing policy efforts to target SDOH, such as housing, poverty, or antidiscrimination. Other measures assessed factors affecting access to mental health services, such as geographic proximity to providers or availability of transportation.

Psychometric and Pragmatic Evidence Rating Scale (PAPERS) Criteria

As described in Table 5, higher scores on the PAPERS criteria are preferable, indicating that the measures possess better psychometric properties (psychometric score) or are easier to use (pragmatic score). For the 34 unique measures, limited information was available regarding psychometric properties. Possible psychometric scores range from −9 to 36 across nine categories of reliability and validity. PAPERS psychometric scores for included measures ranged from −1 to 17, with a median score of 5. The majority of studies reported only norms, an indication of sample size and distribution (n  =  33, 97%), with a median PAPERS score of 3 (good). Eleven studies reported internal consistency, with Cronbach's alpha scores ranging from 0.49 to 0.97 (median PAPERS score  =  3, good). Seven measures reported structural validity with a median PAPERS score of 3 (good). Lastly, convergent and discriminant validity were reported for two measures, with median PAPERS scores of 4 (excellent) and 2 (adequate), respectively. No information was found regarding known-groups construct validity, predictive validity, concurrent criterion validity, or responsiveness within the 34 included measures.

All available measures but one—the Organizational Social Context Measurement System—were free and accessible to the public either through a web search or contacting the study team (median score  =  4). While the measures varied considerably in length, ranging from 2 to 277 items, 11 measures contained 11 to 50 items (median score  =  3). Most measures were accessible at an 8th–12th grade reading level (median score  =  3). Instructions regarding scoring and interpreting the measures were lacking overall (median score  =  2); minimal information was included to aid in interpreting score ranges, identifying clear cut-off scores, and handling missing data. Similarly, information on training requirements or availability of self-training manuals for those administering the measure was often not reported (median score  =  0, Not reported). Overall, the included measures received a median total score of 12, out of a possible 20, indicating that certain pragmatic criteria were underreported in this sample.

Discussion

This study examined quantitative measurement tools used to evaluate the implementation of mental health policies affecting mental health education and services and assessed measure development, testing (i.e., the extent to which measures are psychometrically tested), and application (i.e., the pragmatic nature of measures). Though there are best practices when developing and testing measures (Boateng et al., 2018), many measures lacked information regarding gold-standard criteria for measurement development and testing. Additionally, there was a number of “home-grown,” single-use measures identified in this sample, which has been reported in previous measures reviews (Dorsey et al., 2021; Mettert et al., 2020; Proctor et al., 2011; Stanick et al., 2021). Unfortunately, the measure development process and psychometric validation of such surveys were rarely reported, presenting a measurement challenge within implementation science and an area for further improvement (Lewis, Weiner, et al., 2015; Martinez et al., 2014; Proctor et al., 2011).

Our findings also highlight the need for a more critical examination of policy implementation determinants and outcomes. Within the included measures, some constructs—such as readiness for implementation, acceptability, or fidelity/compliance—were represented repeatedly. Others were underrepresented (e.g., political will for policy implementation) or absent (e.g., sustainability) from measures, despite the known significance within implementation science (Glasgow & Chambers, 2012). Outer setting implementation determinants were assessed less frequently than inner setting determinants, which has been reported in previous studies (Chaudoir et al., 2013; Chor et al., 2015; Clinton-McHarg et al., 2016; McHugh et al., 2020; Powell et al., 2021; Weiner et al., 2020). The limited attention to outer setting implementation determinants is problematic, as factors like political will can impact mental health outcomes through policy development and resource allocation (Corrigan & Watson, 2003; Shera & Ramon, 2013; Weinberg et al., 2012). Additionally, the overemphasis on compliance as a means of studying implementation has been a critique of policy studies in the past (Thomann & Zhelyazkova, 2017).

The imbalance of constructs within quantitative measures may stem from the length of time that these constructs have received attention. Historically, a large emphasis has been placed on acceptability, while sustainability may appear less often due to its relative “newness” within the implementation literature (Lewis, Fischer, et al., 2015). The absence of sustainability in these studies is perhaps unsurprising, as researchers report similar findings with other assessment models, such as the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework. For example, a 2013 review assessing the use and reporting of RE-AIM in 71 articles found that, of all five RE-AIM dimensions, maintenance (sustainment) was the least frequently reported—at both the individual-level and setting-level (Gaglio et al., 2013). Many of these constructs are theoretically grounded, though, and have a significant impact on policy implementation. Assessing these understudied constructs in future studies may provide researchers with a more complete picture of the processes, contexts, and outcomes of implementation.

Notably, health equity was assessed in few quantitative measures of mental health policy implementation, which has been reported in previous studies of policy measures (Allen et al., 2020, 2021). Moving forward, it will be vital to engage policymakers in this pursuit of equity, as they have considerable potential to impact health, such as allocating resources to and developing appropriate policies (Douglas et al., 2019). Connecting with community members and other stakeholders to identify and address health equity issues during implementation might also prove useful (Douglas et al., 2019). Additional work is needed to bridge health equity research and implementation science, both theoretically (e.g., within frameworks) and practically (e.g., pragmatic equity measures) (Brownson et al., 2021; Emmons & Chambers, 2021; Pollack Porter et al., 2018; Woodward et al., 2021). This presents many opportunities for further evaluation of the presence of equity within policy implementation.

Psychometric and pragmatic qualities of measures were also assessed in this study. With the exception of internal consistency and sample norms, psychometric properties were frequently unreported, as found in previous implementation measures reviews (Allen et al., 2020; Clinton-McHarg et al., 2016; McHugh et al., 2020; Powell et al., 2021; Weiner et al., 2020). For example, information about structural validity was available for seven measures, and convergent and discriminant validity were reported for two measures. However, none of the measures in this sample contained information about known-groups validity, predictive validity, concurrent validity, or responsiveness. These findings suggest that there is significant room for improvement in the development or refining of existing measures to make them more psychometrically sound.

Overall, the observed pragmatic qualities aligned with previous findings (Allen et al., 2020; McLoughlin et al., 2021; Powell et al., 2021; Weiner et al., 2020). Most included measures were freely available (i.e., accessible through supplemental materials with the article or by contacting authors directly), though one was proprietary. Additionally, though some measures were quite lengthy, most were relatively brief (fewer than 50 items). There is substantial need for pragmatic measures to assess implementation (Glasgow & Riley, 2013; Powell et al., 2017; Stanick et al., 2018, 2019). The findings of this work indicate that while practical, accessible measures used to evaluate policy implementation are being developed and utilized, there is still considerable room for improving the availability of brief, user-friendly measures of policy implementation. The development and use of such measures would help organizations to evaluate implementation without placing undue burden on practitioners.

Policy implementation research remains an understudied—but growing—area of implementation science (Emmons & Chambers, 2021; Hoagwood et al., 2020; McGinty et al., 2022; Oh et al., 2021; Purtle et al., 2017, 2016). Policy implementation is complex and can vary significantly across contexts (Berenson et al., 2017), making it crucial to evaluate implementation determinants, processes, and outcomes. Policy implementation also spans multiple levels of government and geopolitical units of epidemiological analyses, furthering the complexity in studying policy implementation (Schnake-Mahl et al., 2022). There are particular challenges in measuring policy implementation across levels of government. For example, policy surveillance systems are often more developed at the national level compared with the local level. The nexus between political science and implementation science offers benefits for measurement efforts. In particular, political science has a rich history of measuring the outer context of policy implementation (the policy and political environments) (Emmons et al., 2021). Valid, reliable, and pragmatic measures are needed to evaluate policy implementation and close the policy-to-practice gap (Martin et al., 2019; Oh et al., 2021). As a subset of the need for valid and reliable measures, there is growing attention on the need for metrics to track indicators of equity (e.g., SDOH, structural racism) (Brownson et al., 2022). Ultimately, this work contributes to the nascent study of policy-focused implementation science by providing an overview of existing measures used to assess mental health policy implementation, as well as guidance and recommendations for future measure development and refinement.

Limitations

While this study fills gaps in the literature, there are several limitations. First, there were likely relevant measures not captured in this study based on our search strategy and inclusion criteria. Our search strategy included only studies and accompanying measures that were published in English language peer-reviewed journals, so measures found exclusively in grey literature, books, or theses were not included. Second, because we used verbatim instrument items in data extraction (e.g., implementation determinants and outcomes), studies were excluded if we could not obtain the measures. Though we reviewed articles and supplemental materials, hand searched peer-reviewed and grey literature, and contacted authors directly, we ultimately excluded 18 studies because measures were inaccessible, introducing the possibility of sampling bias. The missing tools were commonly from older articles (1995–2010) that did not report rigorous measure development and psychometric testing. Third, though we examined the prevalence of implementation determinants and outcomes in the included measures, it is possible that these constructs were assessed qualitatively in the included studies. Finally, based on the study's definition of “Big P” policies, the included studies encompassed both mandated changes in practices or services, as well as the voluntary adoption and implementation of practices or services based on recommendations or guidelines from governmental entities. Despite these limitations, this review contributes to the policy implementation literature by assessing the state of measure development and quality in mental health policy studies.

Future Directions

Based on the findings of this study, there are several areas for further study. First, there is significant room for improvement regarding measure development or refinement of existing measures to make them more pragmatic and psychometrically sound, as well increasing reporting of pragmatic and psychometric properties. Second, there is need for greater transparency and availability of measurement instruments; we recommend that authors make measures available as supplemental materials to publications or add these to web-based measures repositories. Third, this study echoes previous calls for a greater focus on health equity as both a determinant and outcome of policy implementation (Brownson et al., 2021; Emmons & Chambers, 2021; Woodward et al., 2021). Finally, while this project focused on the implementation of mental health policies once adopted, additional work is needed to explore factors involved with policy enactment and the decision-making processes during adoption and implementation.