Simulating the role of knowledge brokers in policy making in state agencies: An agent‐based model

Abstract

Objective

To model children's mental health policy making dynamics and simulate the impacts of knowledge broker interventions.

Data sources

Primary data from surveys (n = 221) and interviews (n = 64) conducted in 2019–2021 with mental health agency (MHA) officials in state agencies.

Study design

A prototype agent‐based model (ABM) was developed using the PARTE (Properties, Actions, Rules, Time, Environment) framework and informed through primary data collection. In each simulation, a policy is randomly generated (salience weights: cost, contextual alignment, and strength of evidence) and discussed among agents. Agents are MHA officials and heterogenous in their properties (policy making power and network influence) and policy preferences (based on salience weights). Knowledge broker interventions add agents to the MHA social network who primarily focus on the policy's research evidence.

Data collection/extraction methods

A sequential explanatory mixed method approach was used. Descriptive and regression analyses were used for the survey data and directed content analysis was used to code interview data. Triangulated results informed ABM development. In the ABM, policy makers with various degrees of decision influence interact in a scale‐free network before and after knowledge broker interventions. Over time, each decides to support or oppose a policy proposal based on policy salience weights and their own properties and interactions. The main outcome is an agency‐level decision based on policy maker support. Each intervention and baseline simulation runs 250 times across 50 timesteps.

Principal findings

Surveys and interviews revealed that barriers to research use could be addressed by knowledge brokers. Simulations indicated that policy decision outcomes varied by policy making context within agencies.

Conclusions

This is the first application of ABM to evidence‐informed mental health policy making. Results suggest that the presence of knowledge brokers can: (1) influence consensus formation in MHAs, (2) accelerate policy decisions, and (3) increase the likelihood of evidence‐informed policy adoption.

What is known on this topic

• Dissemination barriers, such as lacking access to contextually relevant evidence, have been found to inhibit the use of research evidence in public agency policy making.

• Prior research suggests the knowledge brokers could address these barriers.

What this study adds

• When knowledge brokers are present, evidence‐informed policies may be more likely to be adopted.

• The presence of knowledge brokers can influence consensus formation in public agencies.

• The process of simulation model development and the results here serve as a prototype for agent‐based models that employ social network analysis to model policy making within agencies.

1. INTRODUCTION

Policy making is a complex process. ¹ , ² Complex processes are characterized by behaviors that dynamically interact over time through feedback loops and evolving, nonlinear relationships to “create … a larger whole.” ³ For example, policy makers must not only consider the best available research evidence to inform what decision alternatives might have the most efficient, positive impact, but also assess the potential impact of decision alternatives given other relevant decisions being made and their knowledge of the context in which they are intervening. ⁴ , ⁵ , ⁶ , ⁷ , ⁸ Ornstein and colleagues clarified how complexity shapes policy making through three aspects: (1) dimensionality (the need for multiple, interconnected decisions to be made); (2) ruggedness (the effect of one decision is difficult to disentangle from prior or future decisions); and (3) context‐specificity (the impact of decisions is dependent upon their context). ⁴ The behaviors that emerge from complexity, such as policy making decisions, are often difficult to anticipate or accurately identify without formally modeling and simulating the interactions between system components. ⁹ , ¹⁰

The current study focuses on the role of research evidence in policy making related to children in state mental health agencies (MHAs). ¹¹ MHAs fund approximately 8500 providers and serve about 7.3 million patients, of whom about one‐third are under age 20. ¹¹ , ¹² MHAs have discretion in terms of which children's mental health policies and programs to adopt in their systems and how they are implemented. ¹² Although the complexities of research use in policy making apply to a range of issues, the current study focuses on children's mental health policy because rates of mental health problems (e.g., anxiety, depression, suicide) have been increasing in the United States ¹³ , ¹⁴ , ¹⁵ , ¹⁶ and are being exacerbated by the COVID‐19 pandemic. ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ There is a growing evidence base about policy and programmatic interventions—such as coordinated specialty care for first‐episode psychosis ²⁷ trauma‐focused cognitive behavioral theory for youth ²⁸ —that can improve children's mental health and also increase access to services. ²⁹ , ³⁰ , ³¹ By simulating the use of children's mental health research in policy making within MHAs, the current study seeks to identify strategies to accelerate the translation of this evidence into practice via policy.

1.1. Policy making occurs in a social system

Policy making is an inherently social process. Even when an individual policy maker or small group is tasked with adopting a particular policy, networks of peers, advocates, and practitioners can influence policy makers. ³² , ³³ , ³⁴ Policy making thus requires decision‐making processes that have been conceptualized as a function of collaborative social networks and cooperative agenda setting. ³⁵ Social network analysis (SNA), which offers a systems‐based methodology for modeling group dynamics, ³⁶ , ³⁷ , ³⁸ may prove useful for modeling policy making through identifying network‐based targets for policy making process interventions. ⁹ , ³⁷ , ³⁹ , ⁴⁰

Previous SNAs of policy making processes have primarily been developed for fields such as environmental science and resource management. ⁷ , ⁴¹ , ⁴² Some have focused on how health‐related conditions or behaviors (e.g., obesity, ⁴³ tobacco use ³⁷ ) can spread through social networks. While system dynamics models and agent‐based models (ABMs) have been used to study policy making in other fields such as agriculture and environmental policy, ⁴⁴ none have done so in mental health ⁴⁵ The current study seeks to address this knowledge gap.

1.2. Knowledge brokers, complex systems, and evidence‐informed policy making

One way to influence evidence‐informed policy making is the use of knowledge brokers. ⁴⁶ Knowledge brokers can act as knowledge managers who compile and assess evidence for a particular policy, intermediary agents who facilitate connections between policy makers and researchers, and capacity builders who provide research training to policy makers. ⁴⁶ Knowledge brokers have been studied in a range of policy and practice settings, including clinical care, ⁴⁷ schools, ⁴⁸ , ⁴⁹ , ⁵⁰ and health departments. ⁵¹ , ⁵² They may be consultants or longtime university partners. ⁵¹ , ⁵² , ⁵³ Findings suggest that knowledge brokers may be particularly effective when functioning as intermediary agents. ⁴⁸ Moreover, knowledge broker interventions have increased evidence‐informed decision‐making among clinical practitioners ⁵¹ and improved patient outcomes. ⁴⁷

Knowledge broker interventions remain understudied in mental health policy making. Recent evidence shows that stronger partnerships between researchers and MHA officials may improve confidence in research use. ⁵⁴ A survey of MHA officials found that dissemination barriers—such as lacking access to contextually relevant evidence—were inversely and independently associated with the frequency of research use to inform). ⁵⁵ Dissemination barriers may be reduced by the introduction of knowledge brokers into MHAs.

This manuscript aims to advance the use of systems science in health policy research by using mixed‐methods primary data collection and analysis to inform the development of a prototype ABM. The ABM employs SNA to model policy making dynamics and explores how the following may influence evidence‐informed policy adoption decisions related to children's mental health in MHAs: (1) policy characteristics (i.e., policy cost, contextual alignment, and strength of evidence), and (2) interventions that leverage knowledge brokers to increase policy makers' awareness of supporting evidence.

2. METHODS

A sequential explanatory (QUAN → qual) mixed‐methods research design was used to collect and analyze primary data from state MHAs. ⁵⁶ , ⁵⁷ Surveys were analyzed using quantitative methods, and findings informed qualitative interview data collection and analysis. Quantitative and qualitative results were then triangulated to directly inform development and design of the ABM.

2.1. Surveys

2.1.1. Data

Web‐based surveys of state MHA officials were fielded between November 2019 and February 2020. Complete details about the survey methodology are published elsewhere. ⁵⁵ The survey was completed by 221 officials (response rate: 33.7%). The response rate was significantly higher among respondents from states in the Midwest than Northeast US Census region (44.9% vs. 25.2%, p = 0.004). There was not a significant difference in response rate by gender (p = 0.11, as predicted by first name using Gender API Estimator, used in prior work with MHA officials). ⁵⁸ All survey items were explicitly related to policy‐ and decision‐making and research use related to children's mental health. Research was defined in the survey as “information produced by using reliable data and systematic methods—such as findings reported in peer‐reviewed publications or from analyses of local, state, or national data,” a definition consistent with that of the US Commission on evidence‐informed policy making. ⁵⁹ This manuscript focuses on survey variables that were influential in shaping ABM design decisions.

Frequency of using children's mental health research for instrumental purposes: Respondents separately indicated the frequency with which they “Use research to decide about content or direction of a policy or program” during three policy making process phases: Agenda setting, policy development, and policy implementation (1 = “Very rarely,” 5 = “Very frequently”). Aggregate scores of the frequency of instrumental research use in children's mental health policy making ranged from three to 15, α = 0.82.

Dissemination‐related barriers to using children's mental health research: Respondents were asked “To what extent do you perceive each issue as a barrier to using children's mental health research in your work?” and rated the extent to which four issues were barriers on 5‐point Likert scales (1 = “Not a barrier,” 5 = “Major barrier”). Aggregate scores of dissemination‐related barriers to using children's mental health research ranged from four to 20, α = 0.71.

Primary sources when seeking children's mental health research: Respondents were asked “If you were going to seek out children's mental health research to make a policy decision, who would you turn to?” and instructed to select up to three sources from a list of eight options (e.g., internal staff, partner agencies, university researchers).

Important features of children's mental health research: Respondents were asked “If you were to receive children's mental health research, how important would it be that the research have each of the following characteristics?” and rated the importance of five factors on 5‐point Likert scales (1 = “Not important,” 5 = “Very important”).

Factors that influence MHA children's mental health policy priorities: Respondents were presented with a list of nine factors and, for each, instructed to rate “How much influence you think it currently has on your agency's children's mental health priorities in general” on 5‐point Likert scales (1 = “No influence,” 5 = “Major influence”).

2.1.2. Analysis

Analyses were conducted with respondents stratified according to the relative amount of control they could exercise within their MHA's organizational structure. Respondents in roles such as MHA director, deputy director, and medical director were coded as high‐control, those in roles such as children's division director were coded as moderate‐control, and SAMHSA program administrators were coded as low‐control. One‐way ANOVAs and chi‐square tests compared the means and proportions across these three levels of organizational hierarchy.

2.2. Interviews

2.2.1. Data

Semistructured Zoom‐based interviews were conducted with 64 state and county mental health (n = 33) and substance use (n = 31) agency officials in between January and June 2021. The officials were from agencies in eight purposely selected states that varied in the terms of aggregate scores for frequency of research use (high/low) and barriers to research use (high/low)—classified as such through analysis of the MHA survey dataset and a similar survey of substance use agency officials. ⁶⁰

Interviews focused on agency officials' experiences with children's policy making and factors that influence policy making processes. In particular, questions focused on the role of research evidence and dissemination‐related barriers, and questions around policy making were added to inform the ABM. Interview questions were informed by prior descriptive and linear regression analyses of the survey datasets. ⁵⁴ , ⁵⁵ , ⁶¹ The interviews were approximately 45 min in duration, audio recorded, transcribed, and imported into NVivo 11 (QSR International Pty Ltd, Melbourne, Australia) for analysis.

2.2.2. Analysis

To inform the design of the ABM, directed content analysis ⁶² was conducted using an approach adapted from the “rigorous and accelerated data reduction (RADaR)” technique. ⁶³ Two interviewers reviewed transcripts and assigned text to broad coding categories with definitions that represented three factors that influenced officials' support for a new policy: cost, contextual alignment, and strength of evidence. The selection of these three factors was informed by survey data and reviews of research use in policy making. ⁸ , ⁶¹ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ , ⁶⁸ The three factors are hereafter referred to as salience weights, because, as explained further below, different agents in the simulation model prioritized each differently and assigned different levels of salience to each. The two coders then reviewed text coded to these salience weights and assigned text a priori sub‐codes with definitions that captured variation in the extent to which each salience weight influenced support for a new policy. The two coders and a third member of a project team then collaboratively developed concise declarative statements that captured main findings related to each factor.

2.3. ABM development

2.3.1. Agents and environment

Agents in the model were of three types: MHA policy makers, MHA staff, and knowledge brokers. MHA officials were represented as policy makers, the only agents who had final voting power to adopt or reject a proposed policy, or other staff. Knowledge brokers were modeled as agents initially external to MHAs.

Dekker's ⁶⁹ extension of Kawachi et al.'s ⁷⁰ network rewiring method was used to construct MHA social networks. This method allows for the network structures to approximate characteristics of real‐world social networks (i.e., a low mean distance between network members, a moderate prevalence of local clustering between members, and an approximation of the power‐law degree distribution of nodes). ⁶⁹ , ⁷¹ Ties in the model represent regular communications between members (agents or nodes). During baseline and intervention simulations, each MHA had a fixed population of 25 agents (5 policy makers, 20 staff). Results were analyzed across 50 timesteps in which agents interacted. Policy makers and staff were randomly assigned the three salience weights, informed from interview results, for strength of evidence, cost, and contextual alignment. Knowledge brokers in the intervention runs were also assigned salience weights, however, their salience for strength of evidence was always set higher than the other two. The 50 timesteps represented 50 weeks, which was intended to approximate a MHA's annual budget cycle. Each set of 50 timesteps represents a model run, and each baseline and intervention condition contained 250 runs.

The following simulation outcomes were assessed: Agent policy support level, the likelihood of policy adoption or rejection, and time to decision for both those agents in high‐control positions (i.e., policy makers) and all MHA agents. Agents' policy support levels ranged from 0 (no support) to 1 (complete support); at the end of each model run, agents' policy support levels translated to policy decisions as follows: adoption (policy maker support >0.85), rejection (policy maker support <0.15), or indecision (policy maker support 0.15–0.85). Time to decision was defined as the timestep in which the final policy decision stabilized, or when agents' preferences stopped adjusting.

A unique policy was generated for each simulation run consisting of three salience weights: cost, contextual alignment, and strength of evidence. Each salience weight was randomly assigned scores ranging from −1 (worst) to 1 (best), with 0 representing moderate. Policy quality was the average of these three scores. Policy preference scores, ranging from 0 (not important) to 1 (very important), were also randomly generated at baseline to model an agent's perceived importance of the three policy characteristics for their individual policy making decision.

2.3.2. Baseline simulation

Adapted from Friedkin and Johnsen ⁷² and Namatame and Chen, ⁷³ each MHA agent was randomly assigned an alpha level (mean = 0.5, SD = 0.15, range = 0–1) that captured the degree to which an MHA agent was open to the influence of other agents relative to being anchored to their own initial opinion. We created a vector in which agents consulted with their connected colleagues in the network at each timestep. MHA agents thus updated their policy support level at each timestep through a weighted average of their current policy support level and the policy support level of agents with whom they interacted. The three baseline conditions were distinguished by the degree to which policy makers were connected to other network members: high‐degree (highly connected to other members), low‐degree (few connections to other members), and random‐degree (randomly connected to other members).

2.3.3. Tested interventions

Based on prior research ⁴⁶ and the current study's survey and interview results, we identified the introduction of external knowledge brokers to MHAs as an evidence‐informed policy making intervention and assessed the degree to which it impacted MHA policy making. Knowledge brokers were modeled as initially external to MHAs who develop relationships with policy makers and aimed to build organizational capacity for evidence‐informed policy making. We tested three intervention conditions in which we introduced four knowledge brokers to MHA social networks. Policy makers and staff interacted with one another, while knowledge brokers were directly connected only to policy makers and had a fixed degree of four (i.e., interacted with four policy makers). More information including the parameters and equations for the model are included in Appendix A (Table A1 and Equations).

2.3.4. Example: A week in the life of an agent

Angela is a policy maker in the MHA. She starts out the budget year (model run) in week one considering a policy to implement. Her initial support level for the policy is determined by her preferences for cost, strength of evidence, and contextual alignment (i.e., her randomly assigned salience weights) and how closely those preferences match the policy under consideration. During each week (or timestep) in the model, Angela discusses the policy under consideration with her colleagues—those she is connected to in the agency network—and updates her support level for the policy based on her interactions, previous preferences, and level of influence/preference malleability within the network. If knowledge brokers are present (i.e., under the intervention conditions), she is likely to consult with them as well. Angela's support for the policy is updated as an average of her support and that of her connected colleagues, weighted by the level of influence/preference malleability (i.e., alpha) of each agent. Her support for the policy stabilizes at some point at which her preferences cease to change after continued interactions. At this point, Angela has decided to adopt, reject, or not make a decision on whether to adopt the policy. If she and the majority of her policy making colleagues support the policy, the agency does so.

2.3.5. ABM dashboard

A dashboard was created to show the inner workings of the ABM for the research team to observe the mechanisms, processes, outcomes, and other model metrics. Figure 1 shows a screenshot of the dashboard for additional information on the model.

FIGURE 1.

Knowledge broker ABM dashboard. The dashboard tracks model metrics, such as the maximum number of connections and other network statistics in the agency, as well as time to decision and policy characteristics [Color figure can be viewed at wileyonlinelibrary.com]

3. RESULTS

3.1. Survey results

We selectively present results from new analyses of survey data that were particularly influential in shaping the ABM. Research was used frequently for instrumental purposes (aggregate instrumental research use score = 12.0 on 15‐point scale) and dissemination‐related barriers to using children's mental health research were prevalent but not pervasive (aggregate dissemination‐related barrier score = 11.9 on 20‐point scale) (Table 1). ANOVAs revealed that the US Census Region of a respondent's state was not significantly associated with frequency of research use for any of the purposes (p ≥ 0.79).

TABLE 1.

Key results from survey of state mental health agency (MHA) officials that informed decisions about the design of the agent‐based model (ABM) (N = 221)

		Position within the agency organizational structure
	All (N = 221)	High‐control (n = 62)	Moderate‐control (n = 66)	Low‐control (n = 93)	F/x 2	p
Research use
Aggregate frequency of instrumental research use score (range: 3–15)	12.01	12.50	11.69	11.96	1.81	0.17
Frequency of instrumental research use during agenda setting phase	4.04	4.16	4.02	3.99	0.62	0.54
Frequency of instrumental research use during policy development phase	3.96	4.13	3.80	3.99	2.09	0.13
Frequency of instrumental research use during policy implementation phase	3.93	4.04	3.86	3.90	0.56	0.57
Dissemination‐related barriers to using children's MH research in policy making
Aggregate dissemination‐related barrier score (range: 4–20)	11.86	11.78	11.55	12.23	0.68	0.51
Lack of actionable messages/recommendations in summaries of research	3.16	3.09	3.11	3.25	0.41	0.66
Lack of interaction or collaboration with researchers	3.06	3.02	2.90	3.24	1.34	0.26
Questions that researchers ask are not relevant to decisions	2.90	3.02	2.77	2.94	0.83	0.44
Unclear presentation/communication of research findings	2.74	2.80	2.72	2.70	0.16	0.86
Where turn to for children's mental health research to inform policy decision
University researchers	57.0%	61.3%	59.1%	52.7%	1.29	0.53
Staff within my agency	38.0%	37.1%	36.4%	39.8%	0.22	0.90
Importance of features of disseminated children's MH research evidence
Be relevant to residents of my state	4.47	4.45	4.53	4.43	0.46	0.63
Provides data on cost‐effectiveness/budget impact	4.41	4.42	4.48	4.35	0.57	0.57
Be delivered by someone I know or respect	3.27	3.29	3.2	3.31	0.21	0.81
Factors that influence agency children's mental health policy priorities
Budget issues	4.34	4.48	4.50	4.13	4.66	0.01
State resident demand	3.55	3.53	3.56	3.55	0.01	0.99
Research evidence	3.29	3.32	3.41	3.17	1.15	0.32

Over half (57.0%) of MHA officials identified university researchers as a primary source they turn to when seeking out children's mental health research to inform policy decisions. Just over one‐third (38.0%) of MHA officials reported that they would turn to staff within their agency for this research. This indicates that research evidence is exchanged between officials within agencies and, along with the policy making literature described above, contributed to the decision for the ABM to have a network design.

The features of children's mental health research perceived as most important by MHA officials were research being relevant to state residents (mean = 4.47) and research providing data on cost‐effectiveness/budget impact (mean = 4.41). The finding regarding relevance to state residents signals the importance of research evidence being aligned with the contexts in which MHA officials work, contributing to the decision to include a “contextual alignment” salience weight in the ABM. The finding that data about cost‐effectiveness/budget impact was perceived as a very important feature contributed to the decision to include a “cost” salience weight in the ABM. The source of research being someone that the MHA “know[s] or respect[s]” (mean = 3.27) was only rated as a moderately important feature of children's mental health research. Given this was not perceived as a very important feature, a knowledge broker intervention may be feasible since it entails an external agent (who may not be initially known or respected by MHA officials) joining the MHA to facilitate linkages to research.

Budget issues (mean = 4.34) were rated as factors that have substantial influence on children's mental health priorities of MHAs. This is consistent with “data about cost‐effectiveness/budget impact” being perceived as a very important feature of research and further supported the inclusion of “cost” salience weight in the ABM. State resident demand (mean = 3.55) and research evidence (mean = 3.29) were perceived as having moderate influences on children's mental health priorities of MHAs, further supporting the inclusion of “contextual alignment” and “strength of evidence” salience weights in the ABM.

3.2. Interview results

Table A2 presents key findings and illustrative quotes that informed the development and inclusion of salience weights in the ABM. These findings are also concisely summarized below.

3.2.1. Cost salience weight

The overwhelming majority of agency officials expressed that cost is the single factor with the most influence on agency decisions related to children's behavioral health policies. These officials also noted, however, that cost did not have as much influence on their personal opinions about policies and that cost was only one of many factors (e.g., along with research evidence, politics, community needs) they consider when deciding to support or oppose a new policy.

3.2.2. Strength of evidence salience weight

Most MHA officials and about half of substance use agency officials indicated that research evidence was equally important as cost and other factors in children's behavioral health policy making. Opinions about the influence of research evidence varied considerably across respondents. Factors frequently identified as affecting the impact of research evidence on children's behavioral health policy making include agency culture, time and staff resources, personal interest in research, access to locally relevant research, and payer and legislature requirements to use research evidence and fund evidence‐informed practices.

3.2.3. Contextual alignment salience weight

Agency mission/vision and local needs were identified as contextual factors that affected children's behavioral health policy making as well as the impact of research evidence. When asked about the influence of agency mission/vision on policy making process, the majority of MHA officials felt that it often had substantial influence on their personal support for a new policy as well as the agency decision to adopt or reject it. This feeling was not universal, however, as some respondents believed that their agency mission/vision was too vague to guide the child‐specific policy making. Most respondents also expressed that policy making processes were strongly influenced by the needs of the populations they serve. For example, many respondents described how they used local data to inform their children's policy making priorities. Some respondents emphasized the importance of meeting basic unmet service needs before considering the widespread adoption of new evidence‐informed practices, and other expressed that there was often misalignment between the characteristics of research evidence and local needs.

3.3. ABM results

Informed by the interview and survey results along with the extant literature on policy making, ¹ , ² the prototype ABM simulated knowledge broker interventions in MHAs to increase the uptake of evidence‐informed policies. We focus on two main outcomes from the model: policy decisions and time to decisions in baseline and intervention model runs.

In the 1500 runs of the model (250 each for three baseline and the three intervention conditions), most policies resulted in indecision, meaning that support for the policy stabilized somewhere between the adoption and rejection thresholds. The top panel of Figure 2 shows that policies were adopted in 13.2% of the conditions, rejected in 13.4%, and that no decision was reached in 73.4% of the conditions. The bottom panel in Figure 2 shows that when policies were relatively high or low quality based on the three salience weights, decisions were reached more quickly than middle quality policies under all conditions. The lowest‐quality policies resulted in a decision after about 20 weeks (or timesteps), as did the highest‐quality ones. Policies that were not relatively poor or high quality—those scaled around zero and rated moderate—took the longest amount of time to reach either a decision or stabilize between the two thresholds.

FIGURE 2.

Likelihood of policy adoption (top) and time to decision (bottom), based on overall policy quality. In each baseline and intervention run policy makers were either highly connected in the agency network (high‐degree), had only a few connections (low‐degree), or were initialized with a random number of connections (random‐degree). Intervention runs added knowledge brokers to the agency network [Color figure can be viewed at wileyonlinelibrary.com]

Figure 3 shows the time to decision (agency‐wide and policy makers only, respectively) under each baseline and intervention condition. The largest differences are seen under the knowledge broker intervention when policy makers had only a few ties, or low degrees, in the agency network. This indicates that sparsely connected policy makers reached consensus quickly relative to conditions where they were more connected to other network members. Under the two other intervention conditions (high‐ and random‐degree policy makers), the differences from the baseline runs were minimal.

FIGURE 3.

Time to decision for entire agency compared to that for policy makers under each model condition, based on overall policy quality. In each intervention run, policy makers were either highly connected in the agency network (high‐degree), had only a few connections (low‐degree), or were initialized with a random number of connections. Intervention runs added knowledge brokers to the agency network [Color figure can be viewed at wileyonlinelibrary.com]

Finally, we created a two‐way analysis of variance interaction plot to investigate how the introduction of knowledge brokers influenced the time to policy decisions. Figure 4 shows the interaction between the policy maker network degree condition and the presence of knowledge brokers. We see that the introduction of knowledge brokers in both the high‐ and random‐degree policy maker conditions potentially increases the average time to decision, however, for relatively isolated, or low‐degree policy makers, knowledge brokers are associated with a decrease in time to decision.

FIGURE 4.

Interaction between policy maker context and presence of knowledge brokers [Color figure can be viewed at wileyonlinelibrary.com]

4. DISCUSSION

This study is first application of system science methods to understand policy making in MHAs, and one of a growing number of applications of these methods to understand evidence‐informed policy making more broadly. ⁴⁵ , ⁷⁴ , ⁷⁵ , ⁷⁶ The mixed‐methods approach used here serves as an example of how empirical quantitative and qualitative data can be analyzed to inform development of an SNA‐based ABM. The survey and interview findings were largely consistent with prior research about mental health policy making in the United States. ⁶¹ , ⁶⁴ Considerations related to costs and community and political context were perceived major drivers of decision‐making, and research evidence was largely perceived as valuable, regularly used, and having influence in certain contexts. The ABM findings, however, shed interesting new light on how the policy making dynamics could be impacted by knowledge brokers to promote evidence‐informed decision‐making. Over half (57.0%) of MHA officials identified university researchers as a primary source they turn to when seeking out children's mental health research to inform policy decisions. This finding suggests a desire for academic research among MHA officials and supports the notion that a knowledge broker intervention may be acceptable and desirable. ⁵⁴

Employing a sequential explanatory mixed‐methods research design to inform computational model development could prove to be a useful approach to address complex issues in public health and beyond. Simulation results suggest that the introduction of knowledge brokers into MHA social networks changes the dynamics of the policy making process. While there were virtually no differences in the quantity of normatively good policies adopted between the baseline (no knowledge brokers) and intervention (knowledge brokers present) conditions, the policy making process was qualitatively different when knowledge brokers consulted policy makers. Though knowledge brokers may not have the power to change the level of resources (i.e., time, budget) in an agency, it is likely they can position MHAs to take advantage of policy windows and adopt evidence‐informed policies more quickly by facilitating and expediting the process of consensus formation within agencies around policy adoption. ⁷⁷ , ⁷⁸ , ⁷⁹ While many simulation models focus solely on outcomes, for example, policy adoption or rejection, ⁴⁵ examining process variables such as time to decision is also a useful endeavor.

The prototype ABM results also suggest that when knowledge brokers are present and policy options with strong research evidence support are present—and the cost of implementation is aligned with an agency's resources as well as mission or scope (i.e., context) —such policies are more likely to be adopted. Knowledge brokers act as a mechanism or tipping point, moving agencies across the threshold of policy adoption (or rejection, in the case of policies where costs are high, or the evidence base is weak).

4.1. Limitations, implications, and future directions

A limitation of the study is the MHA official survey response rate of 33.7%. Although higher than other recent surveys of similar policy makers, ⁵⁸ , ⁶⁰ the response rate raises questions about whether the research use practices of survey respondents are representative of all MHA officials. While we observed significant differences in response rate by US Census region, this is unlikely to have biased the results because we did not find that US Census Region was significantly associated with the frequency of research use.

The model development process and the results here serve as a prototype for ABMs that employ SNA to model health policy making within various types of agencies. The steps of: (1) quantitative data collection and analysis, to inform (2) qualitative data collection and analysis, then (3) triangulation of these results and extant literature on consensus formation, policy‐ and decision‐making, to finally (4) development and design of computational simulation models that take advantage of innovative social network techniques and strategies, can ostensibly be reproduced in other substantive, complex challenges in public health and health care service utilization.

Though the prototype model presented here may have caveats, such as equally weighting three policy characteristics (i.e., salience weights of cost, contextual alignment, and strength of evidence) that are likely not considered equally in day‐to‐day policy making processes, the results presented here demonstrate that final outcomes, that is, policy adoption, rejection, or indecision, are not the only metrics by which to judge the quality or effectiveness of a particular intervention. Time to decision, as well as the potential qualitative influence of knowledge brokers, are also important as they might influence the degree to which policy makers can capitalize upon often brief policy windows and build consensus within the agency, which might lead to more effective implementation and sustainment of policy decisions.

Next steps for the development of this model include varying: (1) the number of knowledge brokers introduced into the agency context, (2) agency size, (3) the number and proportion of policy makers within the agency, and (4) the salience weights given to each of cost, contextual alignment, and strength of evidence as policy characteristics. Furthermore, the research team will augment the model with additional variables to uncover other mechanisms promoting evidence‐informed policy making and other emergent phenomena, including inter‐agency collaboration as well as state executive and legislative influence over agency policy making.

5. CONCLUSION

This is the first application of ABM to evidence‐informed decision‐making in health policy. Results suggest that the presence of knowledge brokers can influence consensus formation in MHAs, accelerate policy decisions, and increase the likelihood of evidence‐informed policy adoption.

APPENDIX A.

TABLE A1.

Key agent‐based model parameters

Parameter	Description	Values
Policy
Cost	The cost of the proposed policy	Range: −1 to 1; Higher value indicates less expensive policy.
Contextual alignment	The degree to which the proposed policy is aligned with the unique context of the agency	Range: −1 to 1; Higher values indicates greater contextual alignment.
Strength of evidence	The degree to which the evidence base supports the proposed policy	Range: −1 to 1; Higher values indicate greater strength of evidence.
Policy quality	The average value of the policy's cost, contextual alignment, and strength of evidence	Range: −1 to 1; Simple average of the above three scores.
Individual agents
Type	The type of agent	Policy maker, staff, knowledge broker.
Policy preference scores
Cost	An agent's preference for considering costs of the proposed policy	Range: 0–1.
Contextual alignment	An agent's preference for considering the contextual alignment of the proposed policy	Range: 0–1.
Strength of evidence	An agent's preference for considering the strength of evidence of the proposed policy	Range: 0–1; (the policy preferences are scaled so that the three preference values add up to 1).
Alpha (influence)	The degree to which an agent is open to policy opinion influence from other agents	Range: 0–1; normally distributed with mean = 0.15; std. deviation = 0.15. Values close to 1 imply that agents will mostly value their own policy opinion; values close to 0 imply that agents will mostly value the policy opinions of others.
Policy support level	The degree to which an agent supports or opposes the proposed policy	Policy support starts at zero, can then become more positive or negative after policy evaluations and interactions with other agency members.
Scaled policy support level	The degree to which an agent supports or opposes the proposed policy	Range: 0–1; logistic transform of the policy support score. Scaled positive support approaches 1.0; scaled negative support approaches 0.0.
Organizational structure
Agency size	Number of agency members	25; 5 policy makers and 20 other agency members (note that for interventions, four knowledge brokers are added).
Agent degree	How many ties each agent has to other agents	Average degree = 4.
Model outcomes
Policy decision (all MHA agents)	The final decision on the proposed policy reached by all agency members	Adopt, reject, no decision. A policy is adopted when the average scaled support exceeds 0.85; a policy is rejected when the average scaled support dips below 0.15.
Policy decision (policy makers)	The final decision on the proposed policy reached by agency policy makers	Same as above, but just for decision makers.
Time to policy decision (all MHA agents)	The time step (tick) in which the policy decision is reached (i.e., the average scaled support first exceeds 0.85 or dips below 0.15)	Range: 1–50 or NA (NA when there is no policy decision at the end of the model run).
Time to policy decision (policy makers)	The time step (tick) in which the policy decision is reached (i.e., the average scaled support first exceeds 0.85 or dips below 0.15)	Same as above, but just for policy makers.

A.1. Model equations

Equations for two core elements of REDMOD:

1. Initial policy evaluation (calculated once for every actor in the agency at the beginning of the model run) is the weighted sum of the preferences of an actor for cost, feasibility, and research multiplied by the actual policies cost, feasibility, and amount of research support.

$$V_{\text{eval}}=\mathit{pre}{f}_{\text{cost}}*\mathit{\text{polic}}{y}_{\text{cost}}+\mathit{pre}{f}_{\text{feas}}*\mathit{\text{polic}}{y}_{\text{feas}}+\mathit{pre}{f}_{\text{rsch}}*\mathit{\text{polic}}{y}_{\mathit{\text{rsch}}}.$$

• 2. Policy support updating (calculated every tick for every actor in the agency)

$${\mathit{\text{Support}}}_t={\mathit{\text{Support}}}_{t-1}+\mathit{Eva}{l}_{\mathrm{Avg}}+\mathit{TH},$$

where Support _t = Updated support for policy at tick t; Support _t−1 = Support for policy at previous tick; TH = equals a random “trembling hand” adjustment that adds random noise to the support calculation; Eval _Avg = The weighted average of the actor's policy evaluation with the policy evaluation from the actor's connected neighbors.

Eval _Avg is calculated as follows:

$${\mathit{\text{Eval}}}_{\mathrm{Avg}}=\alpha *V_{\text{eval}}+\left(1-\alpha \right)*\mathit{\text{mean}}\left(N_{\text{eval}}\right),$$

where $V_{\text{eval}}$ is the actor's initial policy evaluation; $N_{\text{eval}}$ is a connected neighbor's initial policy evaluation; $\alpha$ is a salience weight that determines how much each actor pays attention to her own policy evaluations, versus how much they take into account the policy evaluations of her connected neighbors.

TABLE A2.

Key findings from rapid analysis of interviews with state and county mental health and substance use agency officials that informed salience weights included in agent‐based model (N = 64)

Salience weight	Main supporting finding	Illustrative quotes
Cost	Cost considerations have major influence policy decisions, but other factors also have substantial influence	“‘How much would cost function into the decision?’ I would say, having years ago lost my total commitment to idealism, significant impact. If I can't fund it, it doesn't make sense to do it. I just frustrate everybody.” “‘The cost?’ Well, it's an aspect. But it's one dimension. I mean, you can't evaluate that one dimension on its own. I mean, there's like also the side of, ‘What are the alternatives that makes it actually relevant to what the cost is?’ ‘What are the benefits?’ ‘What else is already being offered?’ ‘What are competing priorities or options in the space?’.”
Strength of evidence	Research evidence is valued by agency officials and has influence on policy decisions, but other factors often have greater influence and barriers often inhibit the impact of research on decision making	“I think [research is] pretty important. I mean, because those are the things that we then have to go back and have to tell our executive director or our county executive why: why we are or why we are not going to move forward with a project or continue to fund a project. So if I don't have actual research or action steps that support that decision, we can definitely get jammed up for either funding or defunding a program, so I would say it's pretty important.” “[Decision making] happens really fast. We don't have the time or feel like we should make the time to slow down, look at research, make decisions based on the data and what's happening in the real world. So, I think we feel a lot of pressure to produce things before we've had a chance to actually get enough time to analyze them.” “And we do have [Data] Workgroup that really helps inform our efforts … So that's really been helpful to us because we believe prevention should be data driven. If you're not looking at your data and you're—I like to use the term “prevention confetti”—you're just like throwing things up in the air and hoping they work.”
Contextual alignment	When agency mission/vision is clear and relevant to child‐specific issues, it is one of many factors that influences agency decision making processes a	“I have the mission taped right behind my computer, and I have since the day I started. I was raised under leadership that really stressed being mission‐driven. And so, it's so important because we're in such a tight budget situation. I was raised under a boss that walked around the office and would randomly ask people to recite the mission … I get asked on a daily basis to make a decision to fund a program or to contract with a provider and—I can't say enough—on a daily basis we've got to go back to our mission to give me any sort of backbone with making these decisions. Because otherwise, we would be everywhere and doing nothing.” “Our vision is pretty focused on serving kids and families in [the state] … It's good to know that's where the agency is at, but I don't think it necessarily impacts my decision‐making. There are other variables that come into play in terms of feasibility and cost.” “I don't think our agency really, frankly, has a mission or vision for children's services.” “Our vision is pretty focused on serving kids and families in [the state] … It's good to know that's where the agency is at, but I don't think it necessarily impacts my decision‐making. There are other variables that come into play in terms of feasibility and cost.” “I don't think our agency really, frankly, has a mission or vision for children's services.”
	The extent to which research evidence is aligned with local context and needs affects the extent to which it influences agency decision‐making processes	“And so, it's not just about cost or data. I don't want to just do a program because people think it's a good idea. I think it has to also meet a need. It has to, you know, meet the local or statewide use patterns.” “Research can be so rigid in terms of, you know, like clinical trials. Research can be so rigid that it can only work in this tiny vacuum … So, if you are doing research around Black youth—that's who it's gonna be in [my area], just because of our demographics—and thinking about risk factors and protective factors involved in sort of supporting youth. The way that I see it is when we're thinking about substance use disorder, you know, what we're treating is not [only the substance use issue] what we're treating is trauma, right? We're treating trauma and we're treating pain. So if we're not thinking about where that pain originates from, I mean, of course it's not going to be effective, right?”

^a This finding mainly reflects interviews with mental health agency, not substance use agency, officials because a question explicitly about agency mission/vision was only asked in the mental health agency official interviews.

Combs T, Nelson KL, Luke D, et al. Simulating the role of knowledge brokers in policy making in state agencies: An agent‐based model. Health Serv Res. 2022;57(Suppl. 1):122‐136. doi: 10.1111/1475-6773.13916