The Role of Academic-Policy Partnerships in EBP Implementation Barriers and Policymaker Use of Child Mental Health Research

Abstract

Objective:

Strategies are needed to improve policymaker evidence-informed decision-making and the availability of evidence-based, state-supported services. Academic-policy partnerships could be beneficial in promoting these outcomes but are understudied.

Methods:

Using the 2015 NRI survey and a 2019 national survey of child mental health (CMH) directors, states with strong academic-policy partnerships in workforce training or in program implementation/evaluation (IE) were compared to states with no/limited partnerships in these areas on implementation barriers and policymaker CMH research use.

Results:

Strong IE partnerships were associated with more confidence in research use, fewer issues with provider readiness and capacity, but more issues with fidelity. Strong training partnerships were associated with less frequent endorsement of lack of time as a research use barrier.

Conclusions:

Academic-policy partnerships were associated with some benefit across research use and implementation. Because these partnerships may reduce barriers, research should explore characteristics of effective collaborations.

Child mental health (CMH) disorders are prevalent, and although evidence-based practices (EBPs) exist, few youth with CMH disorders receive evidence-based care.¹ According to the Exploration, Preparation, Implementation, Sustainment (EPIS) model, there are several factors that relate to CMH service availability and implementation that should be considered, occurring within inner (e.g., organization and individual adopter characteristics) and outer (e.g., financing, policies, interorganizational networks) contexts.² Research on outer context barriers to mental health (MH) care is limited,^3,4 and funding for the study of policy in particular is low.⁵ This is unfortunate since state policymakers are well-positioned to improve CMH services given their role in allocating resources and enacting regulations that influence EBP use.^3,6 Existing research suggests that policy has not been optimized to encourage improvements in MH care.⁷

A crucial step in improving CMH care is to encourage evidence-informed decision-making by CMH policymakers. This is difficult, as research may not reach a policy audience, be relevant or responsive to policymaker needs, or align with their worldview.^6,8 Because universities are primarily responsible for producing research, establishing partnerships between academic researchers and policymakers could increase evidence-informed decision-making.⁸ The degree of collaboration with researchers, however, varies from state to state across dimensions of EBP implementation support such as training infrastructure and models for the evaluation of state-supported services.⁹ Little is known about the impact collaboration models have on implementing EBPs.

Given these limitations, these preliminary analyses examined whether having strong academic-policy partnerships for EBP training or implementation/evaluation (IE) of state-supported programs were associated with reduced barriers to implementing EBPs and to policymakers’ CMH research use. Because the types of activities performed in partnerships may be associated with outcomes,¹⁰ training partnerships and IE partnerships were examined separately.

Method

Data were collected from the 2015 National Association of State Mental Health Program Directors Research Institute (NRI) survey and a 2019 NIMH-funded web-based survey of state directors of CMH services.^8,11

NRI conducts surveys of state representatives on state mental health agency (SMHA) characteristics. The current study used data collected in 2015 from 46 states and Washington DC. To evaluate type of partnership, we used responses to the NRI questions: “Does the SMHA work with universities/academia to support training of the MH workforce?” and “Is your SMHA working with academic or university partners to implement or evaluate EBPs or promising practices?” States who responded “yes” to either question were then asked to describe their partnership(s) using an open text field. Two authors reviewed all open-text responses, found variation in scope and strength of partnerships and developed a dichotomous variable for training and IE partnerships to categorize strength with 100% agreement. States were categorized as having “no/limited partnerships” if they: responded “no” to working in an academic partnership, responded “yes” but did not describe the partnership, or responded “yes” but described a partnership focused on implementing one or two EBPs. Responses that described partnerships with multiple institutions and/or committed to a range of aims were coded as “strong partnerships.”

We also used the six implementation barrier questions from the NRI survey, rated on a 7-point Likert scale, inquiring about: workforce shortages, financial barriers, provider readiness and capacity, mismatch between community needs and EBP implementation, difficulties maintaining fidelity, and provider/clinician resistance. Responses were dichotomized; ratings of 5–7 indicated a significant barrier and 1–4 indicated no significant barrier.

The SMHA survey was web-based and covered: experiences using CMH research, preferences for receiving research, and perceptions of the extent to which specific CMH issues are priorities for their agency. The survey was sent to 253 SMHA personnel in a position of leadership and was completed by 129 people across 47 states from 12/2019–02/2020. The Drexel University IRB approved this study.

For current analyses, we retained responses from one respondent per state (N=47) according to the following hierarchy: SMHA Heads (N=21; 45%); State CMH Planners (N=15; 32%); respondents from the Children, Youth, and Families Division (N=7; 15%); and Medical Directors (N=3; 6%). One respondent did not indicate their role (2%). Nearly half (N=22) had 10 or more years of experience working at the SMHA, and most (N=42; 89%) had at least a Master’s degree. We analyzed data from three questions rated on 5-point Likert scales covering: confidence using CMH research, the extent to which they used research in different ways in preliminary discussions about developing/changing a policy or program, and perceived barriers in using research. Responses were dichotomized; ratings of 1–3 indicated not confident, rarely, and not a significant barrier and, 4–5 indicated confident, frequently, and a significant barrier, respectively.

Given previous findings that collaboration acts as a facilitator for research use¹² and is associated with policy that supports EBPs,³ Chi-square tests and when assumptions were violated, one-tailed Fisher’s exact tests were used to examine the relationship between type (i.e., training or IE) and strength (i.e., no/limited or strong) of partnerships on EBP implementation barriers and research use, and to evaluate the relationship between implementation and research use barriers. The sample is small, so relationships that were statistically significant and approached significance are reported.

Results

Over 75% of responding states (28/37) reported working with academic institutions to support workforce training. Ten (27%) were categorized as having a strong partnership. Approximately 84% (37/44) reported partnering with academic institutions to implement/evaluate EBPs, of which 21 states (48%) had a strong partnership (Table 1 in Supplemental Material).

In regard to EBP implementation barriers, strong training partnerships were not associated with any barrier. Strong IE partnerships were associated with more frequent endorsement of barriers to attaining/maintaining fidelity, Χ²=4.0, N=44, df=1, p=0.044. Data suggested that strong IE partnerships were marginally associated with fewer endorsements of provider readiness/capacity as a barrier.

Related to CMH research use, strong training partnerships were associated with less frequent endorsement of lack of time (p=0.009) as a research use barrier. Strong IE partnerships were marginally associated with reporting confidence in finding CMH research to inform policy or program development, in interpreting research results, and less frequent reporting of using research because their organization requires them to do so.

Relationships between barriers were then examined. Shortages in the trained workforce was associated with greater endorsement of several research use barriers, including: lack of interaction/collaboration with researchers (p=0.022), questions researchers ask are not relevant to the decisions they make (p=0.018), and unclear presentation/communication of findings (p=0.039). Mismatch between community needs and EBP implementation was associated with greater endorsement of limited agency resources as a research use barrier (p=0.006). Provider readiness/capacity barriers were marginally associated with greater endorsement of limited agency resources as a research use barrier.

Discussion

These preliminary findings suggest associations exist between the presence of academic-policy partnerships and positive outcomes in EBP implementation and policymaker CMH research use. It may be that broadly integrated academic-policy partnerships are more advantageous in some domains compared to no/limited partnerships. Therefore, more research is needed into effective models of collaboration across states for improvements in CMH service availability and quality.

Strong IE partnerships were associated with more issues with EBP fidelity likely because states with strong partnerships were aware of fidelity issues due to monitoring of provider performance. States with no/limited partnerships may not collect these data although maintaining fidelity to EBP implementation is critical for improving outcomes.^13,14 Therefore, identification of this as a barrier is consistent with what are the likely goals of these partnerships. Approaching significance, strong IE partnerships may also be associated with fewer reported barriers related to provider readiness and capacity to implement EBPs, though more research is needed to confirm these findings. The strength of training partnerships was not related to any implementation barriers.

Academic-policy partnerships were important in policymaker CMH research use. Strong training partnerships were associated with less frequent endorsement of lack of time as a research use barrier. This may be because policymakers who champion training may be more effective in finding and applying research evidence to their work, indicating increased organizational expertise. It may also be that states with strong training partnerships have more resources and thus policymakers have more time to allocate toward research use. However, significant differences were not found in endorsement of financing issues or agency resources, suggesting that greater resources may not fully explain this finding. Further, approaching significance, strong IE partnerships may be associated with greater confidence in finding and interpreting CMH research and less frequent endorsement of using research because it is required. Strong IE collaborations may influence the culture of the agency to promote positive attitudes toward research and increase policymaker competence although this needs further investigation.

Several trends in research use and implementation barrier endorsement were similar across states, regardless of academic-policy partnership status. Research use findings were encouraging; most respondents (>75%) reported frequent use and confidence in finding and interpreting CMH research. Further, research use barriers were endorsed less frequently than implementation barriers, with only one item, limited agency resources, reported as a barrier by over 50% of respondents. In contrast, over 70% endorsed barriers in: trained workforce shortages, provider readiness/capacity, and financing issues, indicating widespread systems issues that need to be addressed. Implementation and research use barriers were also related. Endorsement of workforce shortages was associated with more research use barriers, including lack of interaction/collaboration with researchers, limited relevance of research questions, and unclear presentation/communication of findings. Endorsement of a mismatch between community needs and EBP implementation and of issues with provider readiness/capacity were associated with limited agency resources as a research use barrier. These findings suggest that states endorsing more implementation barriers also perceive more barriers to research use. These data also suggest that there is stability in perceived barriers. States that reported implementation barriers in 2015 continued to endorse barriers in the related area of CMH research use in 2019.

There are limitations to this preliminary study. We relied solely on survey responses, and the NRI survey was completed four years prior to the SMHA survey. It may not capture 2019 conditions. The NRI survey is not specific to CMH and may not reflect CMH conditions. Respondents to the SMHA survey were highly educated and experienced, which may not represent all SMHA personnel. Our categorization of partnerships was crude and does not capture the degree of variation in partnerships across states. There were weaknesses in our approach to grouping together states with no partnerships and limited partnerships, as implementing partnerships even for a limited number of EBPs is a major undertaking. Future research should explore differences between levels of partnerships. Finally, other state-level factors (e.g., political party, per capita CMH spending, Medicaid expansion) and individual variables (e.g., respondent interpretation of questions) that were not assessed likely interact in important ways with these outcomes and require attention in future research.

Conclusion

Preliminary results suggest that broadly integrated academic-policy partnerships are associated with CMH research use and EBP implementation. Because these partnerships could improve CMH care, careful examination of varying models of academic-policy partnerships is warranted.

Highlights:

• Although academic-policy partnerships could be effective in reducing both barriers to evidence-based practice (EBP) implementation and to child mental health research use by state policymakers, they are understudied.

• Strong partnerships focused on implementing and evaluating EBP were associated with more confidence in research use, fewer issues related to provider readiness and capacity, but more issues related to fidelity; whereas, strong partnerships focused on training were associated with less frequent endorsement of lack of time as a barrier to research use.

• Barriers to EBP implementation and research use were related to one another.