A Concept Mapping Approach to Guide and Understand Dissemination and Implementation

Abstract

Many efforts to implement evidence-based programs do not reach their full potential or fail due to the variety of challenges inherent in dissemination and implementation. This article describes the use of concept mapping—a mixed method strategy—to study implementation of behavioral health innovations and evidence-based practice (EBP). The application of concept mapping to implementation research represents a practical and concise way to identify and quantify factors affecting implementation, develop conceptual models of implementation, target areas to address as part of implementation readiness and active implementation, and foster communication among stakeholders. Concept mapping is described and a case example is provided to illustrate its use in an implementation study. Implications for the use of concept mapping methods in both research and applied settings towards the dissemination and implementation of behavioral health services are discussed.

Many efforts to implement programs designed to improve the quality and outcomes of behavioral health services have not reached their full potential due to a variety of challenges inherent in the implementation process. Concern about this “knowledge–practice” gap has focused attention on identifying and testing processes and mechanisms that facilitate or inhibit dissemination and implementation of efficacious practices.^1–4 Consistent with the current National Institute for Mental Health (NIMH) Strategic Plan⁵ and broader National Institutes of Health initiatives, the dissemination and implementation of effective health, mental health, and social service strategies should be a priority for service systems and organizations providing behavioral health services.

A number of conceptual models of dissemination and implementation exist, typically representing several phases including pre-implementation (i.e., preparation, adoption), implementation (i.e., initial, active, full), and sustainment as well as multiple levels such as system, organizational, and individual.^6–8 One such model is the Exploration, Preparation, Implementation, and Sustainment (EPIS) model, which highlights multiple implementation phases and both outer and inner context in public sector services.⁶ Successful dissemination and implementation must take into account these various phases and levels at which change must occur.^9–11 Consensus exists that multicomponent or blended implementation strategies are needed in order to confront implementation challenges, as many different factors need to be addressed in sequence or in tandem for effective implementation and sustainment that retains impact in the target community.^7,12–15

Among the largest challenges in translating research into real-world settings are multiple, at times divergent, stakeholder priorities and perceptions in the implementation process and the diversity of practice settings. Addressing implementation issues at multiple system levels requires consideration of various stakeholder groups and the different ways they may act and be impacted by behavioral health practice change. Stakeholders viewing service change from the policy, system, and organizational perspectives may have different views and priorities relative to those from clinical and consumer groups.^16,17 Differing perceptions and priorities during implementation may lead to variability in buy-in and willingness to proactively participate in implementation processes and problem solving. Additionally, while many commonalities exist with regard to implementation issues across practice settings, there are likely unique differences that require identification and attention. In order to better identify what factors may be important during implementation, it is essential to understand the perspectives of different stakeholder groups including areas of convergence and divergence.

Concept mapping (described below) provides a structured approach to collecting qualitative and quantitative data while engaging stakeholders in the research process. Using concept mapping to consider multiple stakeholder perspectives and foster communication across group, may better facilitate the implementation and use of effective practices in behavioral health⁷ consistent with the goals of the NIMH Strategic Plan.⁵

Concept Mapping

Concept mapping is a powerful mixed method approach because it provides an efficient way to generate an interpretable conceptual framework that is developed in the language of participating stakeholders.^18,19 Mixed methods research involves the integration of qualitative and quantitative method philosophies, designs, strategies, analytic approaches, and interpretations.^20–22 As such, mixed method studies rely on the combined strengths of both qualitative and quantitative approaches to offer a more comprehensive, more detailed, and richer understanding of research issues than either approach alone.^23–25 Concept mapping moves beyond focus group or survey methodology to integrate qualitative and quantitative methods in a logical and progressive way that engages stakeholders in the research and theory generation process. It is also a useful tool for understanding implicit dimensions of a phenomenon that may not be readily apparent with more typical survey methods and thus, can be used for theory development and refinement. Concept mapping has grown from an interest in using theory-driven evaluation,²⁶ being successfully employed largely in planning and evaluation studies,¹⁸ to be generalized for other types of studies including the implementation of innovations.^16,17

Concept mapping capitalizes on the knowledge of individuals but uses a group “brainstorming” format to generate ideas and develop a conceptual framework of implementation that can guide theory development, planning, and organizational development. Interactions and discussion of topics in a group of relatively homogeneous participants is intended to activate knowledge structures about implementation, and to deepen the level of issues brought forth in the concept mapping session. Diverse stakeholders identify issues in an implementation, prioritize those issues, identify where consensus exists, and determine whether a common vision can enable the implementation. The resulting products of concept mapping are visual maps that illustrate group ideas and concerns, how the ideas are related to one another in a multidimensional concept space, how the ideas are organized/clustered into general concepts, and how concepts are rated in terms of criteria relevant to stakeholders, often importance or changeability. The concept mapping process and results can be used during multiple phases of an implementation. For example, as suggested by the EPIS model, in the exploration phase, concept mapping can be used to identify critical needs for behavioral health system, organization, or practice change. In the preparation phase, it can be used to identify community strengths and issues needing attention to promote effective implementation. In the active implementation phase, it can be used to develop an understanding of the implementation process and related emergent issues. In the sustainment phase, concept mapping can be used to identify and support continued use of behavioral health innovations or evidence-based practices (EBPs) in the community.

Concept mapping is comprised of six main phases: (1) preparation—stakeholders are identified, the focus question is developed, and a schedule is set, (2) generation—stakeholders (consumers, providers, organization directors, county representatives) participate in groups and brainstorm a set of statements related to the focus question created in the preparation phase, (3) structuring—each participant sorts the statements into piles based on similarity and each participant rates each statement on importance and changeability, (4) representation—data are used to conduct multidimensional scaling (MDS) wherein each statement is a point on a “concept map,” with statements grouped together by more people being visually closer to each other, and cluster analysis is then used to aggregate similar groups of statements into clusters, (5) interpretation—the investigators work with stakeholders to help them develop their own labels and interpretations for the clusters, and (6) utilization—using the maps to help address and understand the original focus. The final concept map can be used as a conceptual framework for operationalizing the constructs of interest and/or as the basis for developing measures and displaying results.²⁷

Several concept mapping studies have been conducted in the field of implementation and dissemination. Examples of projects include the examination of factors impacting the implementation of new programs in public sectors of care;²⁸ the identification of factors affecting the adoption of new technology by faculty, staff, and administrators in higher education;²⁹ the development of a conceptual model of factors that influence the translation research into practice in tobacco prevention and control;³⁰ supporting a statewide action plan for translating public health research into practice;³¹ assessing fidelity during model transfer of a psychiatric rehabilitation model;³² and the examination of barriers and facilitating factors to implementation of EBP in a community mental health system.^16,17 Common elements of all studies include the reliance on the input of multiple stakeholders and the development of a visual framework to help understand dissemination and implementation.

A Concept Mapping Study of Evidence-Based Practice Implementation in Child and Adolescent Mental Health

For this paper, the use of concept mapping is illustrated through an investigation of factors affecting the implementation of EBPs in a large county children’s mental health service system.^16,17 The goal of the larger study was to advance both a conceptual model and practical understanding of barriers and facilitating factors that influence EBP implementation. While this article discusses some of the findings from the study, the main purpose here is not to communicate findings in detail as more comprehensive discussion of the study findings is reported elsewhere.^16,17 The purpose of the current article is to demonstrate how concept mapping strategies can be used to understand complex and often divergent stakeholder perceptions regarding implementation. This article uses the study described below to illustrate how concept mapping can be employed as a strategy: (1) to understand and develop a model of factors affecting dissemination and implementation, (2) to increase stakeholder communication and collaboration, and (3) to assess implementation readiness including areas that may need to be addressed before and during implementation.

Study Methods and Analysis

Participants

Working with county children’s mental health officials, public agency directors, and program managers, the research team recruited 31 individuals who represented a diversity of organizational levels and a broad range of mental health agencies and programs including mental outpatient, day treatment, case management, and residential. Six stakeholder groups were recruited comprised of county mental health officials (n=5), agency directors (n=5), program managers (n=6), clinicians (n=7), administrative support staff (n=3), and mental health service consumers with children receiving services (n=5).

Procedures

Preparation stage

Concept mapping is a multistaged procedure that elicits input and feedback from stakeholders throughout the process.²⁷ Once stakeholders have been identified and recruited, the next task is to develop the focal statement that will guide the brainstorming sessions. For this study, the investigators met with a mixed (across levels) group of stakeholder participants and explained that the goal of the project was to identify barriers and facilitators of EBP implementation in public sector child and adolescent mental health settings. They then cited and described three specific examples of EBPs representing the most common types of interventions that might be implemented; participants were provided a written summary of training requirements, intervention duration and frequency, therapist experience/education requirements, cost estimates, and cost/ benefit estimates. The investigative team then worked with the study participants to develop the following “focus statement” to guide the brainstorming sessions: “What are the factors that influence the acceptance and use of evidence-based practices in publicly funded mental health programs for families and children?”

Generation stage

Once the focus statement is created, brainstorming sessions are conducted. In this study, separate brainstorming sessions were conducted with each stakeholder group (county officials, agency directors, program managers, clinicians, administrative staff, and consumers) in order to promote candid response and reduce potential desirability effects. In response to the focus statement, participants were asked to yield concise statements that described a single concern related to implementing EBP in the youth mental health service system. Participants were also provided with the three examples of EBPs and the associated handouts described above to provide them with easily accessible information about common types of EBPs and their features. A total of 230 items were generated across all six stakeholder groups. The investigative team then distilled these into 105 distinct statements indicating specific barriers and facilitators by eliminating duplicate statements or combining similar statements. Statements were randomly reordered to minimize priming effects.

Structuring stage

The next step in the concept mapping process involves sorting of the statements derived from the brainstorming groups. Researchers met individually with each study participant, gave them a stack of 105 cards (one statement per card), and asked each participant to sort statements into piles based on similarity, yielding as many piles as the participant deemed appropriate. The final step of the structuring stage, prior to preparing the data for analysis, is to obtain participant ratings for each statement. Ratings are specific to the intended project, but generally include some dimension of importance, impact, or changeability. For this study, each participant was asked to rate each statement describing what influences the acceptance and use of EBPs in publicly funded mental health programs on a zero- to four-point scale on “importance” (from 0 “not at all important” to 4 “extremely important”) and “changeability” (from 0 “not at all changeable” to 4 “extremely changeable”) based on the questions, “How important is this factor to the implementation of evidence-based practice?” and “How changeable is this factor?”

Representation stage

In the representation stage, data are analyzed using a combination MDS and hierarchical cluster analysis statistical procedures. While an individual interested in producing a concept mapping project could feasibly use these traditional procedures alone to create the maps of statements and clusters, the use of the Concept Systems³³ software greatly enhances the ease of analyses, including the creation and interpretation of the resulting maps representing the data. The Concept Systems software allows a user to input the card sort piles and ratings of up to 50 participants. This input is then analyzed using MDS and hierarchical cluster analysis, resulting in the production of interactive concept maps and pattern matches, described in greater detail below. Data from the card sort created during the structuring stage are entered into the Concept Systems software as a similarity matrix. A similarity matrix is created by arranging each participant’s card sort data in rows and columns denoting whether or not they placed each pair of statements in the same category. For example, a “1” is placed in row 3, column 1 if someone put statements 1 and 3 in the same pile, indicating that those cards were judged as similar. Cards not sorted together received a “0.” Matrices for all subjects are then summed, yielding an overall square symmetric similarity matrix for the entire sample. The square symmetric similarity matrix is analyzed using MDS analysis, which creates a two-dimensional “point map” or a visual representation of each statement and the distance between them based on the square symmetric similarity matrix. Each statement is represented as a numbered point, with points closest together more conceptually similar. The point map can illustrate in one representation all of the ideas and interrelationships generated by the group process. It also allows for the creation of one overall solution for all participants as well as individual configurations for each stakeholder group. The “stress value” of the point map is a measure of how well the MDS solution maps the original data. The stress value is derived from normalized residual variance for a perfect relationship of a regression of the distance of dissimilarity or similarity, with lower stress values reflecting better fit of the MDS point map to the original data. The range for concept mapping has been reported as 0.21–0.37.³⁴

The MDS values are then used as the input for hierarchical cluster analysis using Ward’s algorithm to partition the MDS point coordinates into nonoverlapping clusters. This cluster analysis method used in concept mapping is equivalent to that used in the traditional quantitative data reduction method, and is used to delineate clusters of statements that are conceptually similar. An associated cluster map using the grouping of statements is created based on the MDS point map. To determine the final cluster solution, investigators evaluate potential cluster solutions (e.g., 12 clusters, 15 clusters) and then agree on the final model based on interpretability. As part of the collaborative process, participants are invited to collaborate with the research team in defining the meaning of each cluster and identifying an appropriate name for each of the final clusters. For the study described here, 14 clusters were identified as barriers of facilitators to the acceptance and use of EBPs (stress value=0.26). The labels assigned by the stakeholder team included: “consumer values and marketing,” “consumer concerns,” “impact on clinical practices,” “clinical perceptions,” “evidence-based practices limitations,” “staff development and support,” “staffing resources,” “agency compatibility,” “costs of evidence-based practices,” “funding,” “political dynamics,” “system readiness and compatibility,” “beneficial features (of evidence-based practices),” and “research and outcomes supporting evidence-based practices.”

Additional quantitative data are added to the map, resulting in a “cluster rating map.” Cluster rating maps contain both the clustering of the points (statements) and the incorporation of the statement rating information on importance and changeability. Overall cluster ratings are averages of the ratings for each item in each cluster and are represented by layers on a cluster rating map. These maps can be created for the overall group as well as for separate stakeholder groups. For example, once the final map has been produced, comparisons among different stakeholder groups can be made by aggregating importance ratings for each group so that there is a map that represents the entire population of interest while also allowing differences among participants to be identified within the clusters.

Figure 1 presents the cluster rating map of importance for the described study. The number of layers in each cluster’s stack indicates the relative level of importance participants on average ascribed to factors within that cluster, with more layers indicating a higher mean importance rating for the statements making up that cluster. A smaller cluster indicates that statements were more frequently sorted into the same piles by participants (indicating a higher degree of similarity). Proximity of clusters to each other indicates that clusters are more related than to clusters further away. This likely explains, for instance, why “system readiness and compatibility” and “agency compatibility” are near each other but distant from “consumer concerns.”

Figure 1.

Cluster rating map of importance

A complementary adjunct to the creation of the concept map involves a procedure called pattern matching (see Fig. 2). Pattern matching allows for the comparison of different dimensions (e.g., importance of factors affecting EBP implementation vs. perceived changeability of factors affecting EBP implementation) among all stakeholders or the comparison of one dimension among different groups of stakeholders (e.g., importance of factors affecting EBP implementation among those in direct practice vs. importance of factors affecting EBP implementation among those in policy and administration). Pattern matching also allows for the quantification of the relationship between these two sets of interval level ratings, aggregated at the cluster level by providing a Pearson product–moment correlation coefficient, with higher correlations indicating greater congruence.

Figure 2.

Pattern matching of importance and changeability ratings

When comparing importance and changeability ratings in the current study, the correlation was r=0.25. Importance and changeability ratings for the highest-rated statements in the top eight clusters are presented in Table 1. Participants rated funding as both the most important and the least changeable of all clusters. Staffing resources were also rated as important and as more changeable than funding. Staff development and support were perceived as equal in importance to staff resources, but even more changeable. Perhaps most promising, participants also saw clinical perceptions as both of “very great” importance and “high” in changeability. Areas indicated as high in importance and changeability, such as clinical perceptions, represent important initial target areas to address as part of implementation readiness. By presenting information in a manner which may improve clinical perceptions of EBP, stakeholders may be able to improve buy-in and increase the successfulness of implementation.

Table 1.

Importance and changeability ratings for the highest-rated statements in the top eight clusters

Clusters and statements	Importance	Changeability
Funding	3.17	1.95
Available funding for EBP	3.77	1.94
Willingness of funding sources to adjust requirements	3.39	1.58
Lower long-term cost if future need for treatment is reduced	3.10	2.26
Staff development and support	3.16	2.54
Openness and adaptability of staff, clinicians, and managers	3.71	2.65
Buy-in and commitment of staff and trainees	3.68	2.58
Staff abilities and potential to develop required EBP skills	3.42	3.06
Staffing resources	3.16	2.26
Time for training, supervision, and ongoing training in EBP model	3.74	2.19
Time and resources available for supervision and training	3.48	2.13
Training of current staff to be EBP trainers	3.32	2.65
Costs of EBP	3.13	2.16
Having clear knowledge of the exact costs	3.48	2.55
Cost of training	3.35	2.00
Potential benefit for agency/program (revenues, competitive advantage)	3.06	2.45
Research and outcomes supporting EBP	3.09	2.40
EBP proven effective in real-world settings	3.48	2.61
Validity and reliability of evidence	3.42	2.26
Ongoing effectiveness, proof that the EBP is still working	3.32	2.39
Impact on clinical practice	2.81	2.55
Confusion about what an EBP is	2.74	3.16
EBP implementation effect on quality of therapeutic relationship	3.03	2.77
Ability to individualize treatment plans	3.19	2.48
Consumer values and marketing	2.87	2.56
EBP fit with system of care values (e.g., family involvement)	3.32	2.65
Communicating and marketing EBP to consumers	2.65	2.71
Empowered consumers demanding measurable outcomes	2.65	2.32
Clinical perceptions	2.88	2.70
Clinician knowledge and perceptions of EBP	3.35	3.00
Interest, openness, curiosity of clinicians, and managers	3.23	2.94
EBP provides new skills and clinical perspectives	3.23	2.61

Interpretation stage

Results interpretation is typically a real-time, participatory process where stakeholders interact with the totality of all of the group’s ideas. The interpretation stage involves gathering stakeholder participants to explain and discuss the results of the concept maps and pattern matching. This includes examining the point map to understand which statements are most related to each other, examining the cluster rating map to determine which clusters of statements were rated most important to the focus statement, and examining the pattern matching to determine key areas to target based on high ratings. In the current project, the investigative team assembled the stakeholder team to review the process that had transpired, present the maps, explain the findings detailed above, and facilitate discussion of the meaning of the results to the stakeholders.

Utilization stage

The utilization stage involves working with the stakeholder team to determine the best ways to use the maps and reports produced as part of the concept mapping procedures. Possible uses of the output from concept mapping include creating a framework for a strategic implementation plan, serving as the basis for developing an evaluation design, and tracking implementation progress over time. In the current project, the results from the concept mapping procedure were used to develop a conceptual model of EBP implementation in children’s mental health, as well as to develop a measure for assessing the implementation readiness of stakeholders at the policy and practice levels. This measure has been used to help agencies target areas to address prior to implementation, such as staffing resources and consumer concerns as well as to help bolster strengths already present such as agency and system compatibility. Additionally, the utilization stage provided an opportunity for greater collaboration and understanding among the various stakeholder groups. Activities in the utilization stage also promoted continued academic–public collaborations between the research team and county mental health, provider organizations, and consumer advocates. Such interactions are imperative to the success of implementation and dissemination efforts which require coordination and collaboration among stakeholders at multiple system and organizational levels.⁶

Discussion

Information gleaned from the concept mapping procedure and results have proven useful for both researchers and community stakeholders. For researchers, such data can serve to elaborate conceptual models of implementation, facilitate an understanding of the roles and needs of various stakeholders in implementation, and serve as the basis for developing measurement tools. Results from the study described above have provided valuable data regarding important factors to consider at multiple levels as part of implementation readiness. These results have been published in widely cited journals, attesting to the methodological rigor in using concept mapping to examine dissemination and implementation. This data has also served as the foundation for the development of an implementation readiness questionnaire and implementation readiness interview guide currently being utilized with stakeholders across multiple implementation efforts.

The use of concept mapping in implementation studies also provides an array of applied benefits to the community stakeholders. The resulting maps provide an easily interpretable visual framework for understanding and conceptualizing the barriers, facilitating factors present in the community as part of implementation readiness, and for keeping participants on task for implementation planning. The concept rating map and pattern matching also provide useful information including target areas for change (high importance, high changeability ratings) and areas of convergence and divergence for moving forward with an implementation while addressing the barriers and needs of multiple stakeholders. Finally, the concept mapping process itself serves a number of beneficial features, including facilitating communication among stakeholders and improving group cohesiveness. It allows participants to describe their ideas and thoughts in a language familiar to them rather than the language of the evaluator or researcher, hence providing a sense of ownership over the results and the implications they may have for the organization.

While the study and related products described here focused on barriers and facilitating factors as a part of implementation readiness for EBP, other potential uses of concept mapping may include the development of measurement tools to assess implementation progress throughout the stages of implementation.^6,7 Future studies in different service systems with different stakeholders will be useful for determining similarities and differences in barriers and facilitating factors across systems. For example, one current project utilizes concept mapping to examine barriers and facilitators in early and later phases of implementation of EBPs in Assertive Community Treatment teams. Further, EBP implementation represents just one domain of implementation for which concept mapping holds utility. The general principles of concept mapping, along with its ability to adapt to the needs of the target stakeholder groups, apply to other dissemination and implementation efforts, such as those focused on chronic disease prevention, substance abuse prevention and treatment, and other important physical and behavioral health initiatives.

Implications for Behavioral Health

Barriers and facilitators of implementation of innovation and change are present at multiple levels of health care delivery: the patient level, the provider team or group level, the organizational level, or the market/policy level.¹² Indeed, empirical research increasingly demonstrates that effective dissemination and implementation must take into account the complexity of the service system, and multiple levels of stakeholders and processes involved at each stage of implementation.^6,35,36 The case study described above provides support for the use of concept mapping as an efficacious method to examine and improve implementation of behavioral health services. The strengths of concept mapping, above traditional methods, allow for scholars to address health initiatives across a spectrum of behavioral health issues consistent with the goals of the NIH.

Concept mapping provides a framework to not only gather crucial information regarding implementation, but to do so in a manner that facilitates cohesion, communication, and collaboration across stakeholder groups. Such processes for eliciting community stakeholder input may have promise for facilitating more reciprocal exchanges between cultures of research and practice in behavioral health, while also improving the ability to implement and sustain innovations. In this sense, concept mapping has the potential to help bridge gaps between groups, with divergent goals and needs, in a manner that will improve behavioral service system outcomes for the long-term, with specific awareness of population needs. Implementation science reinforces the importance of considering not only summative end point health outcomes, but also of performing formative evaluations to assess the extent to which implementation is effective in a specific context to optimize intervention benefits, prolong sustainability of the intervention in that context, and promote dissemination of findings into other contexts.³⁷ Concept mapping is a tool that continues to be useful to address challenges of effective dissemination and implementation across a spectrum of behavioral health services.