Mind maps and network analysis to evaluate conceptualization of complex issues: A case example evaluating systems science workshops for childhood obesity prevention

Abstract

Across disciplines, it is common practice to bring together groups to solve complex problems. Facilitators are often asked to help groups organize information about and better understand the problem in order to develop and prioritize solutions. However, despite existence of several methods to elicit and characterize how individuals and groups think about and conceptualize an issue, many are difficult to implement in practice-based settings where resources such as technology and participant time are limited and research questions shift over time. This paper describes an easy-to-implement diagramming technique for eliciting conceptualization and a flexible network analysis method for characterizing changes in both individual and group conceptualization. We use a case example to illustrate how we used the methods to evaluate African American adolescent’s conceptual understanding of obesity before and after participating in a series of four systems thinking workshops. The methods produced results that were sensitive to changes in conceptualization that were likely driven by the specific activities employed during the workshop sessions. The methods appear strong for capturing salient levels of conceptualization at both individual and collective levels. The paper concludes with a critical examination of strengths and weaknesses of the methods and implications for future practice and research.

1. Introduction

It is common practice to bring together groups to collaboratively decide upon solutions for complex problems. These groups and their facilitators have a challenging task. It is human nature to simplify complex problems to cope with difficult decision-making tasks (Sterman, 1994). If simplifications are biased or do not accurately account for complexity, solutions are often ineffective or can even worsen the problem (Sterman, 2006). Furthermore, individuals often have different tacit, unspecified perceptions about complex problems that create disagreement about solutions and paralyze decision-making (Vennix, 1999; Vennix, 1996). Thus, facilitators of collaborative groups must help participants conceptualize and appreciate the complexity of the issue, and use that understanding as the foundation for making decisions. Yet, facilitators typically have little information about how participants conceptualize an issue or the extent that their conceptualization incorporates and organizes complexity over time.

Conceptualization involves forming an internal representation of something by mentally combining its characteristics and parts (Craik, 1943; Johnson-Laird, 1995). Internal representations are the basic building blocks for learning and retention and believed to guide how individuals make decisions (Ifentaler, Masduki, & Seel, 2011; Shavelson, 1974; Snow & Lohman, 1989). For complex problems, a simple representation is not likely a complete depiction of the situation and can lead to ineffective solutions (Sterman, 2006). Conversely, a complex but disorganized representation can also hinder learning or paralyze decision-making (Ausubel, 1963). At the collective level, groups with more complex representations have also been shown to be more successful and perform better (e.g., producing higher quality products) (Carley, 1997; Curseu, Schalk, & Schruijer, 2010), and groups who have similar internal representations are more likely to have a shared view of the problem and solutions (Cannon-Bowers, Salas, & Converse, 1993; Rouwette, Vennix, & Mullekom, 2002; Scott, Cavana, & Cameron, 2013). Evaluation of participants’ conceptualizations can offer facilitators information about specific characteristics or parts of an issue that are becoming more or less clear to participants or becoming more or less agreed upon among a group. This information is useful to guide learning and improve shared decision-making.

Evaluating conceptualization involves two main steps: elicitation and characterization. First, one must elicit individuals’ internal representations (Jones, Ross, Lynam, Perez, & Leitch, 2011). Indirect elicitation approaches ask participants to write or provide verbal responses to prompts (e.g., “what do you think caused the rise in obesity?”). Direct elicitation approaches guide participants to represent their internal representations as a diagram or concept map. Both direct and indirect elicitation can either ask for open-ended responses (ideographic) or constrain responses to pre-determined concepts (nomothetic) (Curseu et al., 2010). Second, one must characterize conceptualization by looking for patterns or regularities in the characteristics and parts of the elicited responses.

A wide range of methodologies have been developed and used to evaluate individual or group conceptualization (Al-Diban & Ifenthaler, 2011; Fokkinga, Bleijenbergh, & Vennix, 2009; Johnson, O’Connor, Spector, Ifenthaler, & Pirnay-Dummer, 2006; Kane & Trochim, 2007; Kim, 2013; Novak & Cañas, 2006; Pirnay-Dummer, Ifenthaler, & Spector, 2010), and each have varying strengths and limitations. All elicitation methods distort one’s internal representation to varying degrees (Doyle & Ford, 1998; Scott et al., 2013). Elicitation asks individuals to create a representation of an internal representation (whether in verbal or written descriptions), which in itself changes conceptualization and even simple prompts introduce confounding effects. Many direct elicitation approaches that were originally designed as tools to help individuals or groups organize information and learn new concepts (Kane & Trochim, 2007; Novak & Cañas, 2006) likely distort representations more than indirect elicitation methods such as interview techniques that were designed specifically to reduce such confounding influences (Bougon, Baird, Komocar, & Ross, 1990; Nicolini, 1999). Ideographic elicitation with open-ended responses also distorts representations less than nomothetic because it does not constrain participants to certain concepts. By constraining choices, nomothetic methods make assumptions about the need to include and exclude different concepts, and may fail to capture stakeholders’ perceptions that are important for group discussion about the nature of the problem and solutions.

There are also numerous methods to characterize conceptualization and each method answers different questions. The methods generally focus on one or a combination of characterizations including: (1) assessment of accuracy (e.g., comparisons of concept maps with an expert map (Koszalka & Epling, 2010)), (2) assessment of the similarity of concept maps across a group (Johnson et al., 2006; Pirnay-Dummer et al., 2010), and (3) assessment of patterns of structure in the concepts and their connectivity (Ifenthaler, Pirnay-Dummer, & Seel, 2007). Each characterization method requires a coding process to track the presence (or absence) or concepts and structures in the participants’ elicited representations (e.g., counting the frequency of specific concepts). Choices about how closely to code to participant’s exact ‘words’ (e.g., apple = apple, orange = orange) or to generalize (e.g., apple = fruit, orange = fruit) create tradeoffs between capturing similarity across maps and masking differences (Carley, 1997).

Ultimately, different elicitation and characterization methods have different strengths and answer different questions. Researchers recommend the choice of methods should be consistent with the research questions of interest (Daniels & Johnson, 2002). However, evaluation in real-world practice is qualitatively different than in research settings. First, time and technological resources for data collection are more limited in practice than research settings, and a method that is fast and easy to implement is often a top priority that eclipses other considerations. Also a resource consideration, indirect elicitation methods such as interviews are intensive, especially when collaborative groups involve participants with less-developed or discordant language skills (e.g., youth, individuals with limited formal education or limited English proficiency) making them less acceptable or feasible. Second, evaluation in practice is ongoing and the specific research questions shift over time. Thus, a flexible approach that allows for different methods of characterization over time is needed.

Using established methods as a foundation, we designed an elicitation and characterization method with practice-based considerations in mind. We designed the elicitation method with the aim to prioritize feasibility in practice while limiting distortion of participants’ representations. We designed the characterization method with the goal to prioritize flexibility while allowing measurement of complex thinking. The purpose of this paper is to document and share the method. We first briefly review major existing methods that informed our research. Second, we will use a case example to illustrate the data collection and analysis methods we developed to evaluate how conceptualization of obesity changed among a small group of African American youth participating in a series of systems thinking workshops. Specifically, we will describe how we collected and used mind maps to evaluate changes in the breadth, depth and structural complexity of the youths’ conceptual understanding of obesity and assess their collective conceptualization of obesity. Finally, drawing on this case study, we will describe our evaluation strengths and weaknesses.

2. Review of individual and group conceptualization assessment methods

There are numerous methods that can be used to elicit and characterize conceptualization, many of which have been used as both facilitation and assessment methods. Our goal is not to provide a comprehensive review, but describe key methods and major concerns of each that informed our approach. Chiefly, we highlight their strengths and weaknesses related to feasibility in practice, ability to limit distortion, and flexibility in characterization of complexity.

2.1. Group-based methods

Group concept mapping, as described by Kane & Trochim, 2007, is a mixed method approach to create a visual representation of thoughts, ideas, and plans that arise from a group of people. The group concept mapping process involves several steps. First, groups of individuals brainstorm and sort ideas on a topic of interest. Next, the sorted ideas are analyzed using multidimensional scaling and cluster analysis to generate a map of ideas that is partitioned into clusters based on sorting patterns. Finally, stakeholders review the map and interpret clusters and relationships between them.

Group model building, from the field of system dynamics, involves a group of stakeholders in the process of building and using qualitative or computational simulation models. Group model building generally involves small groups of 10–15 participants with specific knowledge of or stake in an issue. The participants meet in-person with a team of facilitators over several months to years (Hovmand, 2013; Vennix, 1996). Some of the facilitation techniques (many of which that have been described in detail elsewhere (Hovmand et al., 2012)) involve diagramming activities that elicit participants’ internal representations of an issue of interest. As one example, a facilitator will work with participants to develop a causal loop diagram, which visualizes how different variables influencing an issue are interrelated. The facilitator will guide participants to: (1) identify the problem variable, (2) add causes, including chains of cause and effect, that lead to the problem variable, (3) add consequences, including chain reactions of consequences, that result from the problem variable, and (4) identify feedback loops wherein consequences result in changes in the identified causes (Vennix, 1996).

Both group concept mapping and group model building were originally designed as problem-structuring methods that facilitators have applied across a variety of disciplines to help groups conceptualize many complex issues, e.g., trust in community-academic partnerships (Frerichs, Kim, et al., 2017), childhood obesity (Keita et al., 2016; Nelson et al., 2015), and domestic violence (Hovmand & Ford, 2009; O’Campo et al., 2015). Group concept mapping is especially useful for strategic planning (Trochim & Kane, 2005; Trochim, 1989). It is a strong option when the goal is to break a broad issue into its smaller component parts, but is less suited to understand conceptualization of causal connections and pathways. In contrast, group model building is well suited to elicit causal pathways. However, the elicitation processes of both methods are relatively time-consuming, often taking place over several sessions, making it difficult to track changes in conceptualization across discrete time-points. Although there are situations in which these methods are appropriate to assess conceptualization, often it is inappropriate to use the method to both ‘intervene’ on and assess conceptualization. Furthermore, group model building helps collaborative groups create models that combine participants’ knowledge with objective data and evidence; thus, models intertwine participants’ conceptualization with other information, which is problematic for evaluation.

2.2. Individual-based methods

Researchers and practitioners have developed numerous methods to help individuals outline relationships between concepts in the form of concept maps or diagrams (Buzan, 1982; Cañas, Bunch, Novak, & Reiska, 2013; Johnson et al., 2006; Pirnay-Dummer et al., 2010). Buzan’s (1982) mind mapping approach guides individuals to visually map information in a hierarchical format – major ideas are connected directly to a central concept and other ideas branch out. Another methodology created by Novak & Cañas, 2006 provides structured steps to help individuals diagram distinct concepts related to an issue, draw links between concepts, and label the links with meaningful statements or propositions (e.g., ‘causes’, ‘prevents’, ‘determines’). Others have developed similar methods that induce different types of constraints to the elicitation process (Cañas et al., 2013; Ifenthaler, 2008; Johnson & O'Connor, 2008; Johnson et al., 2006). For example, Analysis-Constructed Shared Mental Model methodology fully or semi-constrains individuals to create diagrams from a set of concepts pre-determined by experts on the issue (Johnson & O'Connor, 2008). Finally, several open-ended and structured interview methods have been developed such as the ‘Self Q Test’, which attempts to limit researcher influence and distortion by providing the participant with a series of non-directive, self-exploration questions (Bougon, 1983; Nicolini, 1999).

Over the past three decades, Novak’s and other’s individual concept mapping methods have been used extensively in education and instruction domains with child and adult learners (Novak & Cañas, 2006) and for educational assessment (França et al., 2004; Reiska, Soika, Möllits, Rannikmäe, & Soobard, 2015; Rice, Ryan, & Samson, 1998; West, Park, Pomeroy, & Sandoval, 2002). Buzan’s mind mapping is a relatively unstructured approach that is useful for brainstorming. It is well suited to understand how individuals perceive hierarchical relationships, but weaker for assessing conceptualization of causal connections and pathways. In contrast, approaches such as Novak’s concept mapping are stronger for assessing conceptualization of causal connections. However, similar to group concept mapping and model building, these individual-based approaches were designed as problem-structuring methods that give participants new ways to think about and formulate conceptual relationships (Cañas et al., 2003). This is especially concerning when the tools introduce more structure to the elicitation process that is not only time-consuming, e.g., requiring 45 min or more (Cañas et al., 2003), but provides considerable opportunity for conceptualization to change in the process. More rigorously designed interview techniques are least subject to distortion, but these are often not feasible to implement in practice.

Different methods for characterization vary in terms of their flexibility for measuring patterns and structure. Some methods have developed computer-based platforms to help with analysis. For example, CmapTools, a computer-based platform for creating and analyzing concept maps based on Novak’s methodology, calculates measures such as the average correct propositions (compared to an expert map) and average concept centrality, i.e., the number of connections in and out of a given concept (Cañas et al., 2013). This type of automated platform can be valuable to practitioners who may not have required analytical skills, but can also constrain the options for measures. Several researchers have drawn upon network analysis (Carley, 1997; Curseu et al., 2010; Ifentaler et al., 2011), which provides a flexible foundation to operationalize measures to assess patterns in concepts and their connections from individual concept maps including complex associations such as “cycles”, i.e., concepts connected in a circular pattern.

3. Case example

3.1. Engaging youth with systems science to understand childhood obesity

To motivate the need for feasible elicitation methods in practice and a characterization method that is flexible and captures conceptualization of complexity, we present a case example. The broader goal of our case example was to pilot test facilitation techniques adapted from group model building (an earlier noted group-based problem structuring method) and evaluate their influence on rural African American youths’ understanding and support for addressing childhood obesity. Evaluating the impact of group model building on conceptualization is of particular relevance since this approach has received increased attention for its potential to help collaborative groups better understand and develop solutions for complex problems (Frerichs, Lich, Dave, & Corbie-Smith, 2016; Hovmand, 2013; Vennix, 1996). A key premise of group model building is that the activities (e.g., visual diagramming) improve the participants’ ability to understand an issue’s complexity (Vennix, 1999). Evaluation of group model building, however, has largely relied on qualitative and self-reported measures such as participants’ perceived level of insight and mental model alignment (Rouwette et al., 2002; Scott et al., 2013) or used approaches subject to distortion and difficult to implement in practice (Doyle & Ford, 1998; Fokkinga et al., 2009; Scott et al., 2013; Vennix, Gubbels, Post, & Poppen, 1990).

For our case example, we did not have the financial, human, and time resources to conduct separate interviews or extensive elicitation processes with the youth, but recognized the importance of evaluating their conceptualization. Thus, we used existing methods, but adapted them for our needs. The goal of our case example is to illustrate our feasible elicitation method in practice and the types of insights our characterization methods can provide about changes in conceptualization of complexity at individual and collective levels. We describe the group facilitation strategies here briefly. Additional details of the curricula and overall evaluation findings from the workshops are described elsewhere (Frerichs, Hassmiller Lich, et al., 2017). The institutional review board of the University of North Carolina at Chapel Hill approved this study.

Over the course of one month, we held four workshop sessions with 21 African American adolescents ages 13–17. Each of the four workshop sessions lasted approximately 4 h and sought to: (1) introduce youth to leadership and complex systems problems, (2) introduce the use of system dynamics diagramming to understand problems affected by complex systems, (3) advance youth identification of effective intervention strategies within complex systems, and (4) guide youth to communicate about complex systems. The first session involved an introduction to concepts including didactic information on obesity and major properties of complex systems (e.g., non-linearity, feedback loops). The second session, relative to all other sessions, involved extensive diagramming activities that were designed to elicit participants’ perceptions of concepts related to obesity and help them make connections between the concepts (including potential complex interrelationships). During this session participants were guided to create their own graphs and causal loop diagrams about obesity independently, which were then discussed as a group. Several themes arose from their diagrams and discussions. The facilitator used these themes to create four seed diagrams focused on the following: (1) self-esteem and body image, (2) advertising of sugar-sweetened beverages, (3) school and parent influences, and (4) fast food. The seed diagrams were presented during the third session, and the youth divided into small groups to advance the seed diagrams and identify potential solutions. The fourth session focused on communication and youth developed short presentations about the solutions they identified in the prior session.

4. Evaluation methods for individual and group conceptualization

We will use our case example to introduce our methods for evaluating individual and group conceptualization. First, we describe a relatively unstructured diagramming technique that combined adapted versions of Buzan’s mind mapping and group model building diagramming techniques to elicit conceptualization. Second, we describe the network analysis methods we applied to the mind maps to operationalize measures of and characterize conceptualization with a focus on measures of complexity.

4.1. Elicitation procedures

We used a diagramming technique that combined group model building diagramming (Vennix, 1996) and Buzan’s mind mapping techniques (Buzan, 1982). We choose Buzan’s mind mapping because it allowed for a free-flow of associations with a central concept (in our case, obesity); thereby, reducing confounding effects of more structured diagramming methods. However, we added simple directions and a guided example that were more consistent with group model building diagramming techniques that aim to elicit causal (as opposed to hierarchical) relationships.

At the beginning of the first session and after each session, participants provided a mind map “about obesity in your community”. Before creating their first mind map, a facilitator provided an overview of mind maps and guided the group to create a mind map on the unrelated issue of academic performance. The facilitator placed a flip chart with “grades” written in the center in front of the group. The facilitator asked participants to volunteer factors that influenced their grades, which were wrote on the flip chart (to the left of center) with an arrow connecting each concept to “grades”. Next, the facilitator asked the participants to volunteer potential outcomes that result from their grades, and similarly wrote these on the flip chart (to the right of center) with a connecting arrow to grades. Finally, the facilitator asked participants to volunteer either additional factors or arrows (to illustrate additional relationships between existing concepts), which were added to the flip chart. The facilitator reviewed the diagram with the participants, summarized the relationships, and gave the participants an opportunity to ask questions.

After the facilitated example, participants were provided a 2-page worksheet with written instructions on the first page. The second page was blank with the exception of the word “obesity” written in the center. The facilitator verbally read the written instructions to participants, which were as follows: “on the next page, create a mind map about obesity in your community. The issue of ‘obesity’ is already written on the page for you. Brainstorm and add what you think leads to or happens as a result of the issue.” Participants were instructed there were no right or wrong answers, and provided 15 min to complete their mind maps.

4.2. Preparing the mind maps for analysis

The mind maps were entered into an excel spreadsheet. The first and second spreadsheet columns were used to enter each pair of factors that participants connected with a line or arrow on their mind maps(e.g., if a participant wrote “fast food” with a line connected to “obesity,” it was entered as “fast food” in column one and “obesity” in column two). We used content analysis to code the factors the participants provided on their mind maps. We identified, a priori, potential categorical themes based on major domains identified in the literature (i.e., physiology, food and eating behaviors, physical activity, psychosocial factors, food environment, physical activity environment) (Finegood, Merth, & Rutter, 2010). In order to reduce bias, two research team members (one who facilitated sessions and another who did not) independently coded the mind maps. First, they independently coded a random selection of 10 maps, met and discussed findings and discrepancies, and defined a total of 19 subthemes of the a priori categories. The two research team members then independently coded the remaining maps with the concept subthemes. Coding at the level of subtheme was determined to be the best approach to enable generalization of similarities across individual maps with sufficient detail (i.e., we did not want to mask differences). Inter-rater reliability was found to be high for all categories (Krippendorf’s alpha > .85), and the few differences were reviewed and reconciled by discussing and reaching consensus.

4.3. Data structure and measures

We used network analysis as the basis for operationalizing measures and characterizing individual and collective conceptualization. Network analysis is a structured technique to study ‘something’ in terms of relationships between its component parts. For example, social network analysis focuses on studying groups of individuals in terms of relationships between each individual (Wasserman & Faust, 1994). We studied the mind maps in terms of the relationships between the factors participants provided (coded into concept sub-themes). Network analysis provided a vocabulary to denote structural properties (i.e., patterns and regularities of relationships) (Barabási, 2016) and a set of mathematical operations to quantify properties of conceptualization (Ifentaler et al., 2011).

4.3.1. Data structure for individual measures

We applied network analysis to the mind maps in order to derive measures of individual conceptualization as follows:

• A graph G(N,E) is composed of two sets of information: a set of nodes, N = n₁, n₂, n₃…n_g, and a set of edges, E = e₁, e₂, e₃, …, e_L, between pairs of nodes. There are g nodes and l edges. In application to the mind maps, we created a graph, G, for each participant’s mind map, wherein the concept subthemes were nodes (e.g., n₁ = “obesity”, n₂ = “unhealthy food”, n₃ = “physical activity”) and the lines drawn between were edges. By nature of our diagramming procedure all participant’s mind maps included the node “obesity” and up to 19 of the identified concept subthemes. An edge, E_k = (n_i,n_j), was included in the set for a participant’s mind map when a line was present between two concept subthemes re-presented by n_i and n_j in the graph. For example, if a line connected “obesity” and “unhealthy food”, then e₁ = (“obesity”, “unhealthy food”).

• Positions of nodes, N, and edges, E, are examined with regard to their proximity to one another. If the edge e_k = (n_i,n_j) is in the set of edges, then the two nodes, n_i and n_j, are described as adjacent and the edge, e_k, is incident with nodes n_i and n_j. In application to the mind maps, if there was a line drawn between two concept sub- themes (e.g., n_i = “obesity” and n_j = “unhealthy food”), they were joined by an edge, e_k, and the two nodes n_i, n_j of a graph G were considered adjacent.

• The degree of a node, d(n_i), is the number of edges that are incident with it. In application to the mind maps, we defined a concept subtheme’s degree as the number of incident lines connecting it to other concept subthemes (e.g., if a line was drawn between “obesity” and “unhealthy food” and another line was drawn between “obesity” and “diabetes”, the degree of the concept subtheme “obesity” was equal to 2).

• A path is defined by a sequence of distinct nodes and edges connecting any two nodes. If a path begins and ends on the same node, it is called a cycle. In application to the mind maps, chains of concept subthemes connected with lines were considered paths (e.g., a participant includes concept subthemes and lines on their mind map as follows: “low self-esteem”–- “over-eating” –- “obesity”). Chains that started and ended with the same concept subtheme were called cycles (e.g., extending the example, a participant also connects “obesity” back to “low self-esteem” with a line). Youth used a combination of both lines and arrows to connect factors, and we defined cycles only as those that had arrows to specify directionality in a circle.

• The density of a graph is the proportion of possible edges that are actually present. In application to the mind maps, the density was defined as the number of lines present, l, among the concept sub-themes, g, divided by the maximum possible lines (a function of the number of concept subthemes).

4.3.2. Individual measures of conceptualization

We used and extended several measures based on network analyses that were previously developed (Calori, Johnson, & Sarnin, 1994; Curseu et al., 2010; Ifentaler et al., 2011) to assess changes in the breadth, depth, and structural complexity of conceptualization (Table 1). Breadth of Concepts measures breadth; the number of concept subthemes (nodes) on a mind map. Shallowness of Concepts measures depth; the sum of concept subthemes (nodes) without connections (edges) to others (i.e., isolated nodes) on a mind map. Surface structure also measures depth and is the number of adjacent concept subthemes (i.e., node-edge-node) on a mind map. Connectedness is a measure of complex thinking that is the proportion of lines on the mind map divided by the total number of possible lines (i.e., density). Cyclic structures is a measure of deeper and more organized complex thinking that is the number of cycles on each mind map.

Table 1.

Mean or median measures of breadth, depth, and structural complexity of youths’ conceptualization of obesity.

Measure	Indicator	Operationalization	Pre-Session (n = 19)	Post- Diagramming (n = 19)	Post- Final Session (n = 13)	p-value
Breadth of Conceptualization - Concepts	The ‘size’ of the underlying cognitive structure indicating a breadth of understanding of the phenomenon	Mean (SD) of the sum of all nodes	6.0 (2.0)	6.9 (2.2)	6.8 (2.1)	0.08
Depth of conceptualization - Shallowness	The number of isolated concepts without connections indicating limited understanding (if high) of a phenomenon in question	Median (Range) of the number of nodes without edges to any other nodes	0 (0–5)	0 (0–5)	0 (0–3)	0.36
Depth of conceptualization - Surface Structure	The number of connections between concepts indicating understanding of how concepts fit together	Mean (SD) of the total number of adjacent nodes (i.e., node-edge-node)	6.5 (3.7)	12.3 (7.6)	8.6 (3.1)	0.0008
Structural complexity of conceptualization - Connectedness	Overall measure of connectedness indicating a deeper understanding of complexity of underlying subject matter	Mean (SD) of the number of actual edges divided by the total number of possible edges based on the number of vertices	0.36 (0.28)	0.44 (0.24)	0.36 (0.14)	0.96
Structural complexity of conceptualization - Cyclic structures	A measure of understanding complex associations and relationships	Median (Range) of the number of paths (undirected or directed) that return back to the starting node of the starting edge	0 (0–3)	0.5 (0–7)	0 (0–3)	0.91

4.3.3. Data structure for collective measures

We applied several concepts of weighted graphs in order to visualize and derive quantitative measures of collective conceptualization as follows:

• Weighted graphs are graphs that assign a value to each edge, adding a third set of information to the graph, W = (w₁,w₂,w₃, … w_L) (Newman, 2004). Each edge of a weighted graph is associated with a value from a set of real numbers. In application to the mind maps, we aggregated the participants mind maps and created a weighted graph for each session. Each edge of the graph is associated with a weight equal to the number of times participants connected a pair of concept subthemes.

• Node weighted degree centrality is the extent that a node is adjacent to other nodes and defined as: C_D = d(n_i) ^(1-α) * s(n_i)^α where, d(n_i) is the node’s degree (as previously defined), s(n_i) is the node’s strength calculated as the sum of the weights associated with the node’s incident lines, and α is a tuning parameter that determines the relative importance of degree compared to strength (Opsahl, Agneessens, & Skvoretz, 2010). Adjusting α lower than 1 places more importance on degree (number of ties), and adjusting α greater than 1 places more importance on strength (weight of ties). For example, in application to the mind maps, consider two hypothetical examples: (1) five participants connect “unhealthy food” to “individual attributions” and five participants connect “unhealthy food” to “socioeconomic status” or (2) ten participants connect “unhealthy food” to other concepts, but each participant connects it to a different concept. If α = 1, then the weighted degree centrality of “unhealthy food” in both cases is equal to 10. In contrast, with α = 1.5, the weighted degree centrality equals 22 in the first and 9.9 in the second example.

• Graph degree centrality is an aggregated measure that characterizes variation in degree centrality of the graph’s nodes. The measure is the sum of the differences between the maximum degree centrality observed in a graph and each node’s degree centrality, which is divided by the maximum theoretically possible summed difference of a given graph (Wasserman & Faust, 1994). The measure ranges between 0 and 1, where 0 is attained when all nodes have the same degrees and 1 is attained when the variation between the nodes’ degrees is the largest possible (essentially when one node is connected to all other nodes, and the other nodes are not connected with each other like a central hub with spokes). In application to the mind maps, graph degree centrality was calculated for each session’s weighted graph.

• Partitioning refers to the segmentation of a graph, G, into subsets of nodes (Wasserman & Faust, 1994). The goal is to find subsets of nodes that demonstrate ‘cohesiveness’ with respect to a graph’s relational patterns. Cohesive subsets generally are well connected among themselves and relatively well-separated from remaining nodes. In application to the mind maps, we use a partitioning algorithm (Clauset, Newman, & Moore, 2004) to identify subsets of cohesive concept subthemes for each session’s weighted graph.

4.3.4. Collective measures of conceptualization

We visualized the aggregated mind maps for each session (using only participants who consistently attended all sessions). We used the thickness of the lines to represent the relative weight assigned to each edge and color-coded nodes to represent subsets of concept subthemes identified by partitioning. To assess collective conceptual understanding, we qualitatively compared the visualized aggregated mind maps across sessions. In addition, we defined quantitative measures of collective conceptualization for each session. Weighted concept connection centrality. A measure of the extent that many participants are connecting a concept subtheme to many of the same concept sub-themes. We used a tuning parameter of α = 1.5 (to place more importance on concept subthemes that were collectively linked to many of the same concept subthemes across participants) and calculated each concept subtheme’s weighted degree centrality for each session’s weighted graph. Collective degree centrality. A measure of the extent of variation that participants emphasize one (or a few) concepts as central hubs (when the value approaches 1) or that participants disperse connections across concepts equally (when the value approaches 0). We calculated the collective degree centrality for each session to assess changes in the overall structure of connections among concept sub-themes.

4.3.5. Mind map analysis

The excel spreadsheet was imported into R (R Core Team, 2016) and the packages igraph and tnet were used to visualize the mind maps and calculate network measures. We also used SAS (SAS Institute, Cary, NC) for descriptive (frequencies, means, medians) and regression analyses. We examined distributions and calculated the appropriate descriptive measures (i.e., mean, median) for all concept subthemes and individual-level measures of conceptualization. To assess changes in individual-level measures, we used generalized repeated measures regression models with appropriate distributions based on response variables (i.e., linear for normal, poisson for count, and logistic for binary measures).

5. Results

For our case example, a total of 21 participants completed mind maps of obesity across five time-points, which resulted in a total of 80 mind maps (n = 19 at baseline, n = 21 post session one, n = 14 post session two, n = 13 post session three, and n = 13 post session four). The analysis method provided insight about changes in youths’ conceptual understanding of obesity at multiple levels of detail. Table 1 provides quantified measures that synthesize the breadth, depth and structural complexity of participants’ conceptualizations, and Table 2 and Fig. 1 provide context about how specific factors and connections among factors were changing. The analysis also allowed for assessment of collective conceptualization through visualization (Figs. 2–4) and quantified measures of patterns and structural dimensions (Table 3).

Table 2.

Number of mind maps with and median number of concept subthemes per mind map pre- and post- systems thinking sessions.

	Definition	Pre-Session (n = 19)		Post-Diagramming Session (n = 14)		Post-Final Session (n = 13)
		# of mind maps (%) with sub- theme	Median (Range) of sub- theme occurrences per mind map	# of mind maps (%) with sub- theme	Median (Range) of sub- theme occurrences per mind map	# of mind maps (%) with sub- theme	Median (Range) of sub- theme occurrences per mind map
Physiology
Biological, genetic influences	Words or phrases that relate to physiological or genetic influences such as genes/genetics, family heritage, ‘run in the family’, biology, metabolism	2 (10.5%)	0 (0–1)	2 (14.3%)	0 (0)	0	0 (0)
Physical, health, disease outcomes	Words or phrases that relate to physical or health-related outcomes such as heart disease, diabetes, cholesterol, pain.	10 (52.6%)	1 (0–2)	10 (71.4%)	3 (0–8)	8 (61.5%)	1 (1–4)
Food and Physical Activity
Unhealthy foods	Foods or food properties that are generally considered less healthful such as fast food, soda, sugar, fats	13 (68.4%)	1 (0–4)	10 (71.4%)	1 (0–4)	10 (76.9%)	2 (0–5)
Physical Inactivity	Words or phrases related to physical activity (e.g., exercise) qualified as low or not existing	14 (73.7%)	1 (1–2)	9 (64.3%)	1 (0–3)	5 (38.5%)*	0 (0–1)
Psycho-social
Individual attributions	Words or phrases that relate to individual responsibility for health behaviors such as laziness, self-control, overeating, etc.	11 (57.9%)	1 (0–4)	5 (35.7%)	0 (0–2)	7 (53.8%)	1 (0–3)
Individual emotional, psychological responses	Words or phrases that relate to individual emotional or psychological issues such as stress, self-esteem, frustration, emotional problems, lack of confidence, self- doubt	14 (73.7%)	1 (0–4)	12 (85.7%)	1 (0–4)	9 (69.2%)	1 (0–3)
Parental influences	Words or phrases that relate to parental influences such as unhealthy parents, parenting	0	0 (0)	1 (7.1%)	0 (0–1)	1 (7.7%)	0 (0–1)
Peer influences	Words or phrases that relate to influences from peers such as peer pressure, relationship problems, bullying	2 (10.5%)	0 (0–2)	5 (35.7%)	0 (0–3)	1 (7.7%)	0 (0–1)
Food Environment
Food appeal	Words or phrases that relate to the appeal of foods such as enticing, tasty	1 (5.3%)	0 (0–2)	1 (7.1%)	0 (0–1)	3 (23.1%)*	0 (0–1)
Food price/affoordability	Words or phrases that relate to the cost of foods such as fast food prices, price of food, healthy foods cost too much	6 (31.6%)	0 (0–1)	3 (21.4%)	0 (0–3)	2 (15.4%)	0 (0–2)
Food access/convenience	Words or phrases that relate to the ease of accessing or preparing foods such as ‘more fast food restaurants’, ‘not time to cook’	2 (10.5%)	0 (0–1)	0	0 (0)	1 (7.7%)	0 (0–1)
Food advertising	Words or phrases that relate to the advertisement or promotion of foods such as commercials, food ads	2 (10.5%)	0 (0–1)	3 (21.4%)	0 (0–1)	2 (15.4%)	0 (0–1)
Physical Activity Environment
Physical activity access/convenience	Words or phrases that relate to the ability to access places to be physically active such as ‘less sidewalks’, ‘no gyms’	2 (10.5%)	0 (0–2)	0	0 (0)	1 (7.7%)	0 (0–1)
Physical activity affordability	Words or phrases that relate to the cost of accessing places to be physically active such as ‘affoordable prices for gyms’	2 (10.5%)	0 (0–1)	0	0 (0)	2 (15.4%)	0 (0–1)
Other
Socioeconomic status	Words or phrases that denote one’s socioeconomic status such as education, income, ‘lack of money’	2 (10.5%)	0 (0–4)	6 (42.6%)	0 (0–2)	8 (61.5%)*	1 (0–3)
Solutions	Words or phrases that are an explicit indication of a potential solution or strategy to address obesity such as ‘use celebrities’	0	0 (0)	0	0 (0)	1 (7.7%)*	0 (0–4)

^*Odds of mind map having the node significantly increased across sessions (p < 0.01).

Fig. 1.

Most commonly connected pairs of concept subthemes on participants’ mind maps pre- and post- systems thinking sessions. This figure presents the pairs of concept subthemes that the most participants connected at pre-, post-diagramming, and post-final systems thinking workshop session. The three columns represent the three sessions and the pair of concept subthemes are listed from most to least common from top to bottom. Colors are used to highlight the changes in the ranking of the pairs across sessions.

Fig. 2.

Visualization of youths’ aggregated mind maps before systems thinking sessions. This figure provides the visualization of the youths’ aggregated mind maps at baseline. Each node is a concept subtheme with obesity provided in the center. The concept colors represent cohesive subsets identified by partitioning algorithm. White indicates the concept was not written on any of the participants’ mind maps. Line thickness represents the number of participants who connected the pair of concepts.

Fig. 4.

Visualization of youths’ aggregated mind maps after final systems thinking session. This figure provides the visualization of the youths’ aggregated mind maps after the final systems thinking workshop session. Each node is a concept subtheme with obesity provided in the center. The concept colors represent cohesive subsets identified by partitioning algorithm. White indicates the concept was not written on any of the participants’ mind maps. Line thickness represents the number of participants who connected the pair of concepts.

Table 3.

Weighted node degree and collective degree centrality of the mind maps across sessions.

	Concept sub-theme (Weighted Degree)
	Pre-Session	Post Diagramming Session	Post Final Session
	Obesity (80)	Obesity (101)	Obesity (76)
	Unhealthy food (21)	Negative health outcome (47)	Unhealthy food (32)
	Low physical activity (20)	Unhealthy food (36)	Negative health outcome (17)
	Negative health outcome (19)	Psychological/Emotional factor (29)	Psychological/Emotional factor (16)
	Psychological/Emotional factor (18)	SES (16)	SES (15)
	Individual attribution (18)	Low physical activity (13)	Individual attribution (10)
	Other (8)	Death (12)	Other (9)
	Eating too much (8)	Peer influence (11)	Low physical activity (8)
	Food price (8)	Other (9)	Eating too much (7)
	SES (7)	Individual attribution (5)	Food price (7)
	Physical activity access (5)	Eating too much (5)	Death (6)
	Advertising (5)	Food price (5)	Food appeal (4)
	Biology/genetics (4)	Biology/genetics (3)	Physical activity price (3)
	Food access/convenience (4)	Advertising (3)	Food access/convenience (2)
	Peer influence (3)	Food access/convenience (2)	Solutions (2)
	Death (3)	Parent influence (1)	Advertising (2)
	Food appeal (2)	Physical activity access (0)	Physical activity affordability (1)
	Physical activity affordability (2)	Solutions (0)	Parent influence (1)
	Solutions (1)	Food appeal (0)	Biology/genetics (0)
	Parent influence (0)	Physical activity affordability (0)	Peer influence (0)
	Collective Graph Centrality Measure
Degree Centrality	0.72	0.54	0.69

5.1. Breadth, depth, and structural complexity of conceptualization

Table 1 provides the mean (standard deviation (SD)) or median (range) for breadth, depth, and structural complexity of conceptualization measures at three time-points: (1) baseline, (2) post-diagramming session (session two), and (3) post-final session, which were chosen based on their significance to illustrate trends. The number of concept subthemes that participants placed on their mind maps increased with marginal statistical significance from a mean (SD) of 6.0 (2.0) to 6.8 (2.1) (p = 0.08). Surface structure significantly increased across the sessions, beginning with a mean of 6.5 (3.7) at baseline, increasing to 12.3 (7.6) following the second session, and then dropping to 8.6 (3.1) after the final session (p < 0.001). The number of isolated subthemes and the measures of structural complexity (i.e., connectedness and cyclic structures) did not significantly change across the sessions. The marginal and significant increase in the number of concepts and surface structure, respectively, meant participants added more concepts and more connections between concepts across sessions. This finding indicated participants were moving from a simple to a more complex understanding of obesity. However, lack of change in structural complexity highlighted deeper levels of complex thinking, i.e., identification of cycles among concepts, was not achieved.

Table 2 provides details of specific changes in concept subthemes observed on the mind maps across sessions, and Fig. 1 provides details of specific changes in surface structure, i.e., pairs of concept subthemes joined with a line. The odds the mind maps included the concept of: (1) “physical inactivity” significantly decreased, (2) “food appeal” significantly increased, and (3) “socioeconomic status” significantly increased. At baseline, the third most frequently connected pair was “obesity” and “individual attributions,”; post-diagramming session, this pair was not made by many participants, but it returned to the third most frequently connected pair after the final session. “Obesity” and “socioeconomic status” was not a frequently connected pair at baseline, but the odds this pair of concept subthemes was connected increased significantly post-diagramming and post-final sessions, to the fifth and sixth most frequent connection, respectively. There was also a significant increase in the odds that mind maps contained a connection between “socioeconomic status” and “unhealthy food” across sessions. These results provided context to specific areas of conceptualization that widened and narrowed, and aligned with participants’ discussion during the diagramming session and of their seed diagrams. For example, participants discussed how the cost of fast versus healthy foods influenced choices for families of low socioeconomic status in their community.

5.2. Collective conceptualization

Figs. 2–4 present the visualization of the aggregated weighted mind maps at baseline, post-diagramming, and post-final session, respectively. The weight of the lines in the figures provides visualization of the extent participants were connecting specific pairs of concept sub-themes and the color of the nodes denote groups of cohesive concept subthemes identified by the partitioning algorithm. At baseline, there were many thin lines connecting concept subthemes and there were no strong visual patterns (Fig. 2). In comparison, after the diagramming session, there were thick lines connecting a triad of concept subthemes: “obesity”, “unhealthy foods”, and “negative health outcomes” (Fig. 3). After the final session, there were lines of relatively medium thickness found among four concept subthemes: “obesity”, “socioeconomic status”, “unhealthy food”, and “psychological or emotional factors”. The lack of clear patterns in the baseline figure suggested little cohesiveness in how participants were conceptualizing the issue. The visible patterns detected after the second and final session highlighted participants were not only individually improving in their appreciation of complexity, but that there were specific areas of agreement. This finding likely resulted from the group exercises that guided participants to collectively discuss and diagram issues related to obesity.

Fig. 3.

Visualization of youths’ aggregated mind maps after systems diagramming session. This figure provides the visualization of the youths’ aggregated mind maps after the second and diagramming intensive workshop session. Each node is a concept subtheme with obesity provided in the center. The concept colors represent cohesive subsets identified by partitioning algorithm. White indicates the concept was not written on any of the participants’ mind maps. Line thickness represents the number of participants who connected the pair of concepts.

Table 3 outlines quantitative collective measures across sessions. The concept subthemes of “obesity”, “unhealthy food”, “negative health outcomes”, and “psychological or emotional factors” had the highest weighted concept degree centralities across all sessions. The weighted concept degree centrality of “individual attributions” was high at baseline (18) and was markedly lower after the diagramming and final session (5 and 10, respectively). Conversely, the weighted concept degree centrality of “socioeconomic status” was low at baseline (7) and then increased after the diagramming and final session (to 16 and 15, respectively). Weighted concept degree centrality assessed which concept subthemes participants were collectively connecting to the most other concepts, and because of the tuning parameter, emphasized subthemes that participants were connecting to the same subthemes. For example, “socioeconomic status” increased not only because participants were connecting it to more concepts that could be different for each individual, but were connecting it to the specifically the same concept subthemes (i.e., “unhealthy food” and obesity).

Collective degree centrality was moderately high (0.72) in the baseline collective mind map, decreased post-diagramming session (0.54), and regressed (increased) to nearly the initial baseline value after the final session (0.69). This measure provided a quantified indicator of overall patterns of connections. The moderately high collective degree centrality at baseline indicated that most participants made connections in a hub-and-spoke structure; wherein, “obesity” was a central hub that was connected to different concept subthemes (but subthemes were not largely connected to each other). The measure decreased after the second (diagramming intensive) session, which indicated participants were not simply adding subthemes to the central issue, but also made more links among concept subthemes. The links indicate a level of organization of the complex connections. However, the measure regressed after the final session. The findings suggested the measure was sensitive to changes in the youths’ conceptualization, which was likely more thoughtful following the second session’s diagramming exercises that guided them to explicitly and visually connect concepts.

6. Discussion

Our mind mapping and network analysis method provided a flexible process for eliciting and characterizing individual and group conceptualization. As shown by our case example, we used the methods to evaluate changes in the breadth, depth and structural complexity of youths’ conceptual understanding of obesity and graphically represent and assess changes in the youths’ collective conceptualization of obesity. It appeared the process yielded measures that were sensitive to changes in conceptualization likely driven by the specific activities employed across the sessions from our case example. The methods have potential for application in practice to strengthen our understanding of how conceptualization shifts over time and in response to different facilitation strategies.

The mind map elicitation method was quick to implement and did not require extensive resources (i.e., only pencil and paper were required). Most youth finished their mind maps within 10–15 min. The only materials required were blank paper and a writing utensil. The mind map elicitation method is of value for groups such as community coalitions, cross-sector collaboratives, or community-academic research teams that bring together individuals who have limited time to work together and limited time outside of meetings to complete assessments. Another strength of the mind map elicitation method is the lower likelihood of confounding influence that is a noted challenge with more structured elicitation methods (Scott et al., 2013). In our case example, we found several of our measures changed following a session that used structured diagramming facilitation techniques and then subsequently regressed towards the baseline. If our elicitation process had involved more structured diagramming, there is a risk that evaluation would not have been as sensitive to these changes.

An important contribution of our method is the use of network analysis as a flexible approach to assess individual and collective conceptualization. For example, we found an increase in measures that indicated more complex cognitive structures were forming on an individual level, and we found an emphasis and clustering of connections among food access, socioeconomic status, unhealthy foods, and negative health outcomes at the collective level (Fig. 3). From group-based methods, researchers have applied network analysis to data from group concept mapping to better understand relationships among constructs elicited (Goldman & Kane, 2014; McLinden, 2013). From individual-based methods, researchers have assessed “sharedness” among a group of individuals or constructed scores that assessed how individuals or groups of individuals compare to experts’ conceptualizations (Johnson et al., 2006). We focused operationalizing measures of complexity without imposing a “right” solution for comparison; however, the flexibility of our approach would also allow us to construct measures such as individual map centrality or comparisons to expert maps as evaluation questions shift.

6.1. Lessons learned and implications

Our methods have weaknesses. First, although our elicitation method was brief and addressed potential confounding influence, the limited structure yielded wide variation that made it challenging to assess certain patterns. For example, we did not constrain participants to use arrows or explain their connections. Without such constraints, participants used both lines and arrows to connect concepts and we found it difficult to determine if and when connections were intended as causal pathways or other types of relationships (e.g., hierarchical, temporal). It may be desirable to provide additional and reinforced instructions about use of arrows for causal relationships, but there is a delicate balance in order to avoid introducing confounding influences via the elicitation process. Second, although our method appears strong for capturing the most salient levels and structural elements of conceptualization, it may be limited in measuring constructs that involve deeper consideration such as delays between cause and effect and accumulation that are of potential interest in evaluation studies regarding conceptualization of complex issues (Grösser & Schaffernicht, 2012). In order to measure such constructs, different elicitation processes or subsequent cognitive interviews with participants would be required. Unfortunately, such additions can be burdensome for participants and may not be feasible in practice.

Our current study is limited by a lack of formal validation. Future studies should ask participants for their agreement with map coding to establish credibility of results. Also, the method should be compared with results from more time-intensive interview techniques in order to assess convergent validity. Specific measures of characterization can also be assessed for their ability to predict other survey-based measures. For example, our conceptual framework for the systems thinking workshops hypothesized that as participants’ perceptions about obesity grew in complexity, they would increase in their support of more programs and policies (i.e., the more complex an issue is, it requires multi-pronged, multi-level solutions) (Frerichs, Hassmiller Lich, et al., 2017). We found promising evidence in a post-hoc analysis that ‘connectedness’, a measure of structural complexity indicative of systems thinking, had a moderate correlation with a measure of participant’s support for obesity prevention policies ( = 0.42, p < 0.001). This and other conceptually consistent predictive measures such as perceived level of insight should be tested with larger sample sizes.

The visualization of the collective mind maps revealed interesting patterns that are potentially valuable to guide facilitation strategies. For example, in our case study, the workshop facilitators were not fully aware of the connections being made by participants among “obesity”, “unhealthy foods”, and “negative health outcomes,” – particularly, that so many participants were making the connections to “negative health outcomes.” –The collective visualization made these connections apparent (Fig. 3). If this had been available during the workshops, facilitators could have shown the aggregated weighted mind maps (e.g., Figs. 2–4) to participants and use them to elicit discussion about why connections were being made and their interpretations of these connections. This process would be similar to the interpretation sessions employed within group concept mapping where the concept map is shown to participants who discuss and interpret (Trochim & Kane, 2005). Group concept mapping typically presents the concept map as a final culmination of the process; our method would allow for presentation of ongoing iterations throughout a group’s work together.

In conclusion, our method provides a new way to describe individual and collective conceptualization of issues, and uses an elicitation approach conducive to practice-based settings. Across disciplines, groups come together to solve complex problems; yet, due to evaluation challenges, we do not fully understand how conceptualization changes over time. Our method can help us learn more about conceptualization and improve evidence on the impact of different group facilitation methods.

Biographies

Leah Frerichs is an Assistant Professor in the Department of Health Policy and Management in the School of Public Health at the University of North Carolina at Chapel Hill. Her research is focused on the intersection of community-based participatory research and systems science to address health disparities, largely in the area of chronic disease prevention. Her work has had a major focus on understanding and developing interventions that target social and physical environment influences on youth’s health behaviors in underserved communities.

Tiffany Young is a Postdoctoral Research Associate at the University of North Carolina at Chapel Hill. Dr. Young completed her doctorate at the University of South Florida (USF), School of Aging Studies. Her research focuses on developing and evaluating community programs for older adults. More specifically, her dissertation research utilized mixed methods and the community-based participatory research (CBPR) approach to develop, implement, and evaluate an intergenerational physical activity program for kinship families. Her current interest is in utilizing CBPR as a framework for engaging community stakeholders in the design, implementation, and evaluation of programs to address broad community health issues.

Gaurav Dave is affiliated with The American Evaluation Association, American Public Health Association, and The University of North Carolina Center for Health Equity Research. Dr. Dave is the Administrative Director of the Community Academic Resources for Engaged Scholarship Services at the NC TraCS Institute, UNC-Chapel Hill funded by the NIH’s Clinical and Translational Sciences Award (CTSA). He is an expert in the field of formative and summative evaluation of multi-level, multi-component and multi-contextual health programs and interventions.

Doris Stith is the Executive Director of the Community Enrichment Organization (CEO) in Tarboro, NC. She has been Executive Director of CEO for 23 years. Ms. Stith has considerable expertise in community outreach and program development, and her organization provides after-school and mentoring programs to hundreds of youth and families.

Giselle Corbie-Smith is Professor of Social Medicine and Medicine at the School of Medicine, and nationally recognized for her scholarly work on the practical and ethical issues regarding involvement of minorities in research. She has effectively developed and conducted research across systems to address the health needs of vulnerable populations with the goal of eliminating health disparities, while providing support to pre-doctoral, post-doctoral and junior faculty trainees. Her recent line of inquiry has focused on developing methods and interventions to engage minority and underserved communities in research.

Kristen Hassmiller Lich is an Assistant Professor of health policy and management at the UNC Gillings School of Global Public Health. She specializes in operations engineering and complex systems and has significant experience in the practical applications of systems science to prevention science research and maternal and child health.