Concept Mapping as a Method to Engage Patients in Clinical Quality Improvement

Abstract

Patient engagement has become a primary care research and practice priority. Little guidance exists, however, on how best to engage patients in primary care practice improvement, or how to measure the impact of their engagement. We present an overview of group concept mapping as a method for engaging patients in primary care practice improvement. We detail the group concept mapping process as a tool for use in primary care practice improvement, research, and evaluation, and we present resources to enable researchers and practice leaders to use this tool in practice improvement. To illustrate the method, we present a practice-based quality improvement project conducted with patients and staff at a large urban academic primary care practice.

INTRODUCTION

Patient engagement has been described as a “coming revolution”¹ and as a “blockbuster prescription”² for improving health care. It is prioritized in research funding at the Patient-Centered Outcomes Research Institute (PCORI) and the Agency for Healthcare Research and Quality (AHRQ), and it is mandated by the National Committee for Quality Assurance for patient-centered medical home (PCMH) accreditation. Patient engagement may occur across the dimensions of direct care, organizational design and governance, and policy making according to a framework proposed by Carman et al.³ This framework posits that a continuum of engagement exists within each of these dimensions: at the highest level of the organizational design dimension, patients co-lead initiatives, including quality improvement initiatives.³ Research supports positive outcomes associated with patient engagement in quality improvement: a systematic review by Crawford et al⁴ showed that patient involvement in quality improvement efforts led to improved patient information resources and access to care. Furthermore, a recent AHRQ guide for patient and family advisory programs noted that patient involvement in improvement efforts can lead to better quality, safety, and patient and staff satisfaction.⁵

Increasingly, primary care practices, including PCMHs, attempt to engage patients across a continuum that includes research and practice improvement. Early evidence suggests that practices that have involved patients in medical home transformation efforts view the engagement positively.^6,7 Attempts to engage patients in quality improvement efforts remain uncommon, however. A recent survey of 123 PCMHs found that fewer than one-third engaged patients in these efforts, although most of them reported soliciting patient input in other areas.⁶ Another survey of 249 small PCMHs found that only 15% included patients on quality improvement teams.⁸ One barrier to directly engaging patients may be the lack of guidance on a clear method of doing so.

Medical practices typically solicit patient input in practice improvement through suggestion boxes and surveys, and increasingly, with patient advisory boards.^6,5 Interviews and focus groups are more common in research settings, but they are also used to solicit and organize patient involvement in practice settings.^9,10 These approaches are time and labor intensive, however, and qualitative data collection and analysis require training. Furthermore, the products of these qualitative techniques are often text-heavy documents that do not translate easily to visual displays of complex ideas and patterns that may be more readily understood.¹¹ Because of the clear importance of patient engagement, methods are needed to formalize engagement and measure its outcomes. In this article, we present a method of “structured conceptualization” called group concept mapping, a promising method to engage patients in primary care practice improvement. We illustrate its application through an example from our work involving patients in quality improvement in a large family medicine practice.

CONTEXT AND RESOURCES

The term concept mapping is not new. It describes an entire set of knowledge representation theories and applications in educational and cognitive psychology,¹² where the concepts in a particular subject area and their interrelationships are visually represented in a concept or knowledge map. In these maps, the proximity of the concepts to each other serves as an index of how related they are, and the lines or arrows between the concepts describe or imply the nature of their relationships.

The group concept mapping method that we describe here was developed in the 1980s by William Trochim. Trochim extended the ideas and theories of knowledge representation and concept maps by fully conceiving of them as a group process: participants define the conceptual space through their responses to a question prompt, and define the relatedness of the ideas by sorting the responses into cluster groupings based on perceived similarity. This process yields a visual depiction, in the form of a concept map, based on the ideas’ relatedness. All the participants brainstorm and organize knowledge in a domain, and then interpret and use the resultant maps. An early article detailing the development of the method and its statistical techniques is available online.¹³ Many (if not most) group concept mapping research and evaluation projects use the software designed by Trochim (Supplemental Appendix, available at http://annfammed.org/content/14/4/370/suppl/DC1).

As a patient/consumer engagement method, Trochim’s concept map has been used primarily in public health community-based participatory research.^14–16 A small number of medical studies have used this method to elicit patient and caregiver experiences with chronic illness management.^17–20 In a review, we found only 3 articles using concept mapping to engage patients in practice improvement,^21–23 only 1 of which was conducted in primary care.²³ None of these works provides a methodologic treatment of concept mapping for use in primary care practice improvement.

METHODOLOGIC BASIS

Group concept mapping brings together qualitative techniques (brainstorming and idea or theme sorting) and quantitative techniques (multidimensional scaling and cluster analysis). In short, multidimensional scaling is a statistical method that uses aggregated similarity ratings derived in the sorting task to generate a visual representation of the data—called a point map—on which each point is an idea from brainstorming and the distances between points serves as an index of their relatedness. Cluster analysis is applied to the scaling results to produce clusters of these points, which minimize the distances of the points in each cluster from one another and to produce a visual cluster of related ideas. Statistical details to perform these analyses, including programming language for SAS (SAS Institute, Inc), are given by Trochim and Linton.²⁴ Supplemental Figures 1 and 2, available at http://www.annfammed.org/content/4/4/370/suppl/DC1, show the point and cluster maps, respectively, that resulted from our quality improvement study. Although all of these techniques are analytically robust in themselves, direct research on the group concept mapping method has only recently been published. Rosas and Kane²⁵ reviewed 69 group concept mapping studies and concluded that the reproducibility reliability of sorting and rating is very high.

MAPPING PROCESS

The concept mapping process we present here for practice improvement is a hybrid process mixing features of both a research application and a program planning one. In this section, we present the details of our process and the lessons that we learned from our quality improvement project (Table 1). A total of 41 individuals—16 patients and 25 clinic participants (physicians, a nurse practitioner, and several clinic and quality staff)—participated; patients were recruited from the clinic waiting area by asking all patients who entered the clinic during 6 half-days of data collection between May and July 2014 if they would participate in the study. Patients who agreed to participate were compensated for their time with a $10 gift card to a local retailer. Clinic participants were recruited from the practice at a monthly clinical research meeting and from the clinical quality improvement committee. Both patient and clinic participants gave verbal consent to participate in the study. No demographic or identifying information was collected. The university’s institutional review board approved the study.

Table 1.

Concept Mapping Steps and Lessons Learned

Step	Description	Lessons Learned
Preparation	Identifying and recruiting stakeholders; creating and testing the focus prompt	Build support for project with leadership; include administrative stakeholders in process Use a third-person prompt Extensively pilot-test focus prompt Recruit many participants; allow for attrition between steps
Generation of statements (brainstorming)	Group, individual, or online brainstorming can be conducted simultaneously or sequentially	Allow brainstorming group process to reach saturation; use a mix of written and group brainstorming to encourage responsiveness by all members Carefully screen output for redundancies and clear wording before using it for the sorting task Solicit responses from a wide range of stakeholders, even if they may not be involved in further tasks Brainstorming output can be used for other visual displays such as word clouds
Structuring of statements (sorting and rating)	Individual sorting of statements into categories online or in person Rating statements in 1–2 domains (impact, importance, feasibility, etc)	Allow ample time and consider remuneration for sorting task For practice-based implementation, provide a large table and quiet space for sorting Consider wrapping process in a practice “engagement day”
Representation (maps and other visual displays)	Creating a point map and cluster maps using multidimensional scaling and cluster analysis Visual depiction of rated statements in clusters “go/no-go” and pattern matching graphical displays	Not all projects will require all visual output types: choose output that satisfies task purpose Point map as an intermediate step is not generally useful: cluster maps are more intuitive displays Some projects may find cluster maps not useful and can move to the other visual displays Pattern matching is especially useful for identifying differences in ratings between patients and clinicians/clinical staff
Interpretation	Sharing output with participants for interpretation as a group with facilitation	Can generate maps with interpretative guidance from some stakeholders but not necessarily all For our practice improvement purposes, interpretation step not emphasized
Use	Brainstormed solutions, visual conception of “problem space,” graphical displays of concordance between groups in ideas can drive practice change	Consider presentation at staff meetings and faculty meetings, and across multiple stakeholders, including those not involved in the other steps

Our group concept mapping process consisted of 6 steps: preparation, generation of statements, structuring of statements, representation of statements, interpretation, and use of concept maps. We describe each below.

Preparation

The preparation step consists first of identifying stakeholders/participants. This group should consist of members who are related through their involvement in a specific organization or process, or around a specific problem. The number of participants can vary but should include a “wide variety of relevant stakeholders.”²⁶ Although it is possible to engage a large number of participants, typically 10 to 40 are needed to achieve saturation in the brainstorming phase.²⁶ For a project that will compare groups with each other in terms of differences in their statement ratings, 10 is likely to be the smallest appropriate size per group.

Next, the participant group and leaders create a focus prompt. This statement or question will serve to encourage (or constrain) the breadth of ideas generated by brainstorming. Careful attention should therefore be paid to this step, and should include a logical analysis of possible responses. The focus prompt should be pilot-tested before it is used, and principles of question design typical to survey design are appropriate. Such principles include avoiding double-barreled questions (those questions phrased in such a way that 2 questions are being asked simultaneously) and ambiguous questions. We used the following focus prompt: “What are some ways that patients can be involved at our practice in order to improve the care we provide?” We later realized that our focus prompt was double-barreled and therefore tended to generate responses in domains of general practice improvement (eg, increasing access) and of involving patients in practice improvement (eg, patients could develop educational materials). Use of a third-person stem (such as, “What are some reasons people might not want to be involved…”) can be especially helpful in allowing participants to offer ideas and responses that are generalizable to “most people” or “other people,” thereby easing any concerns that may arise for participants about sharing private opinions in a group setting.

Generation of Statements

At this stage, participants/stakeholders brainstorm responses to the focus prompt. They can participate in person individually or in groups, or via the Internet individually. Although there appears to be some benefit to group-based brainstorming in terms of idea generation, there are also potential benefits to allowing members to brainstorm individually, such as minimizing the possibility that strong voices will overrun the group process. The literature on brainstorming suggests that placing emphasis on the volume of ideas while deferring judgment on their quality maximizes creative idea generation.²⁷

When we conduct brainstorming with staff at our practice, we ask first for written (private) responses to allow for the expression of unpopular ideas, and then proceed to group brainstorming. A skilled facilitator should conduct the brainstorming and should attend to the group dynamic (if in the group setting) to maximize responses and to determine that brainstorming is complete. In our experience, the facilitator should look for something similar to “saturation” in qualitative research—the point at which no new themes are emerging.

In our project, we invited patients to come to a conference room in the practice after their appointments, which resulted in a mix of individuals or small groups. Participants were asked to brainstorm in response to the focus prompt in a session that was facilitated by a member of the study team. We wrote all responses on a whiteboard and left them on the board for the day. This process produced a type of hybrid group brainstorming as everyone could see the responses others had made.

Two separate brainstorming sessions occurred with primary care clinicians and clinical staff in 2 separate group meeting settings.

Table 2 shows examples of the ideas that were brainstormed by both patients and staff, organized by cluster. We collected data from 16 patients and 25 clinic participants, with an approximately equal number of responses generated in each group, despite the different methods of elicitation used in brainstorming. We diverged from the fully participatory process of engaging the exact same participants for all phases; instead, subsets of clinic participants, and different groups of patients completed the tasks sequentially. This flexibility increased our ability to complete all the tasks and, in our experience, did not detract from the engagement of the participants at each step. General consensus suggests that it is acceptable to brainstorm with a larger group first, and then perform the sorting and rating tasks with increasingly smaller groups or subsets.²⁸

Table 2.

Results of Statement Cluster Analysis With Examples of Idea Statements

Cluster	Examples of Idea Statements
Access and navigation (12 statements)	Assist patients with navigating interactions with other departments Identify patient transportation needs and concerns Give more time for self-management support to the medical assistants Access texting program to help remind patients about appointments, studies, medications, and referralsa
Empowering patient proactivity and self-care (14 statements)	Patients could participate in creating an action plan for improving their health carea Encourage patients to bring their medications to their visits Examination rooms could have computers with health-related or disease-specific education modules for patients to view while they wait Patients should have high expectations of our health care system, but they must also be patient: modern health care is complicated and frustrating for everyone at times
Formalize patient involvement in the practice (14 statements)	(Patients could) work on developing standards of care in the practice Provide more opportunities for patient feedback, committees, questionnaires, rating evaluationsa Patients could participate in a “run-through” of practice redesign ideas Patients could draft materials like brochures for procedures
Patient-provider communication (23 statements)	Allow patients to give feedback to their providers Patients could write down questions before their appointmenta Provide a way for patients to give feedback to providers about how they are doing Train the doctors to ask sensitive questions, for example, about sex or addiction
Community resources (20 statements)	(Patients could) assist the practice in identifying community resources that can support health (The practice could) provide help for patient caregivers Offer more interactive health education opportunitiesa Provide opportunities in the waiting room for things that patients could participate in
Technology (18 statements)	Have a laptop or iPad at check in for patient health surveys and entering patient information Increase opportunities to communicate with health professionals using portal or website Have a method to help patients track their progress in managing their chronic diseasesa Help patients with computers in the waiting room

^aAn idea that appeared in our go-zone diagram as having both high feasibility and high impact.

Structuring of Statements

The next step consists of sorting and rating, tasks that can also be conducted either in person or on the Internet. If done in person, these tasks can also take place individually or in groups, although the tasks are individual ones, so there is no benefit to a group other than economy of time. For sorting, participants read each statement from the combined brainstorming session(s) and create groupings for the statements. The groupings should adhere to the following guidelines: (1) at least 2 groupings of statements are required, (2) “miscellaneous” or “other” groupings are not allowed, (3) each grouping must contain at least 2 idea statements, and (4) each idea statement can be placed into only 1 grouping. Depending on the volume of ideas generated, this task can be time-consuming for participants; sorting the approximately 100 ideas in our study took our stakeholders 20 to 50 minutes. In our project, all patients completed the sorting task in person, whereas clinic participants performed the task using an online interface for the concept mapping software. If performed by hand, sorting information can be input directly into the proprietary software or into a spreadsheet program for analysis (Supplemental Appendix, available at http://annfammed.org/content/14/4/370/suppl/DC1).

Because patients in our practice waiting room were consenting into the study and performing the task at the same time, we had a high completion rate. An analysis of group concept mapping in research studies found a completion rate for sorting and rating tasks of slightly more than 50%.²⁷

At this step, there is an opportunity for participants to rate each statement. For our study, we chose 2 rating dimensions: “How much of an impact would (this idea) have in the practice if it were implemented?” and “How feasible would (this idea) be to implement?” Each had a scale ranging from 1 (least impact or lowest feasibility) to 5 (most impact or highest feasibility). Using the concept mapping software, the rating data can be visualized as an average across a cluster, an average within each statement, or an average within clusters or within items, across a group. Collecting demographic data on participants allows for a comparison of ratings between groups, such as patients vs clinic participants. In our project, patients and clinic participants prioritized some clusters of ideas differently: for example, clinic participants believed that “Technology” and “Empowering Patient Proactivity and Self-Care” were most feasible as practice improvements, whereas patients rated “Better Patient-Provider Communication” and “Access and Navigation” as more feasible. The rating data can be displayed in a “go-zone” diagram showing the rating dimensions along the x and y axis and producing a quadrant map of statements; 1 quadrant contains all statements rated as having both high feasibility and high impact. Finally, the degree of agreement between groups can be quantified and visualized in a pattern match diagram (Supplemental Figures 2 and 3).

Representation of Statements

Once participants have individually sorted the statements, the results are aggregated across all individuals. First, a point map is created by aggregating all participants’ sorting data and analyzing it using multidimensional scaling. Next, a suggested set of boundaries is drawn on the point map to form the cluster map. Statements in the point map and cluster analysis are numbered so that participants can easily identify each point as a statement.

Many different clustering strategies are possible within a single project; user judgment is required to decide on meaningful clusters. Near outliers to clusters can be moved to adjacent clusters; this type of visual adjustment can partition the points into more interpretable clusters and is an important opportunity to engage participants in the group concept mapping process. A numerical diagnostic can also be used to guide the choice of final cluster map solution—the stress index—which indicates the degree to which the map accurately represents the grouping data. The program uses repetition of words in each cluster grouping to suggest names for the clusters, and then users or participants can modify the names. The cluster maps can be decided within the group in a fully participatory manner, or by a smaller group, or by an individual doing the analyses. In our study, we decided on a 6-cluster solution (Supplemental Figure 1).

We used the concept mapping software to create rating maps within each cluster by overlaying participants’ impact and feasibility ratings of ideas onto the cluster maps. In addition, we created go-zone diagrams to show the ideas that participants perceived to have both high feasibility and high impact (Supplemental Figure 2). Finally, the software can generate pattern matches, which allow comparisons of the patients’ and clinic participants’ feasibility and impact ratings (Supplemental Figure 3).

For a practice improvement project, the maps may be secondary to the visualization of the rating data. For example, in our case, finding that a cluster of ideas corresponded to “increasing technologic capacity in the practice” was of less interest than the feasibility and impact rating of the practice change ideas in this cluster, including “Increase opportunities to communicate with health professionals using portal or website.”

Interpretation and Use of Maps

In a fully participatory process, interpreting the maps takes place within the group. Final decisions about the use of the maps and other data would be driven by the initial reasons for the process and its desired outcomes. For program planning applications, having feasibility and impact ratings of ideas for program implementation would help with decision making. In our project, the research team generated a final cluster solution, rating data (separately for clinicians and patients), pattern matching, and go-zone diagrams showing ideas rated as having both high feasibility and high impact.

We shared these data with our practice’s Patient and Family Advisory Council. The council’s response was unanimously positive, and the members expressed that it would be a reasonable method for them to use for generating and prioritizing their own group activities, a process that is under way. (Our council also participated in another group elicitation and mapping project as participants, responding to the prompt “What does whole-person orientation in primary care mean?”) A draft version of this article was shared with our department chairperson, who forwarded it to faculty. Simultaneously, we used a modified version of the group concept mapping process for brainstorming and mapping both faculty and staff transformation ideas when our practice transitioned into an accountable care organization. These data were presented at a department-wide “town-hall” meeting. We also used a modified concept mapping process in our departmental strategic planning. Each of these examples attests to the power of concept mapping as an efficient method of eliciting, organizing, and visualizing ideas among stakeholders.

DISCUSSION

To summarize, group concept mapping is a promising method with several potential applications in primary care. As more primary care practices formally integrate patients into practice improvement, concept mapping offers a feasible patient engagement technique that can also illustrate and quantify the convergence and divergence of patient and other stakeholder ideas, highlight the unique perspective that patients bring to practice improvement, and provide a basis for patient-centered practice improvements. Concept mapping may also be a powerful method for stakeholder engagement in other types of clinical research because it demonstrates and quantifies the effect of patient involvement in the process. Our experience demonstrated the feasibility and utility of concept mapping to engage patients and clinicians in a busy urban primary care practice in practice improvement research. Although multiple sessions at the practice were needed to recruit a sufficient number of patient-participants, the brainstorming sessions ultimately yielded more than 100 quality improvement ideas from both patients and clinic participants at our practice. The pattern matching and go-zone diagrams will aid in prioritizing improvement ideas.

Group concept mapping is adaptable to many practice improvement and research purposes. It is flexible enough to allow users to pick the steps and outputs that are most important and feasible. The cost of using proprietary concept mapping software or the services of a statistician to program SPSS (IBM Corporation) or SAS may be a barrier for some practices or practice groups. If this is the case, a practice may choose just to brainstorm, have participants rank statements on paper, and use a spreadsheet program to generate average ranking order, to compare groups of participants, or both, if needed (Supplemental Appendix, available at http://annfammed.org/content/14/4/370/suppl/DC1).

We suggest maintaining the participatory spirit of the methodology, regardless of how it is used, and we recommend considering the visual cluster analysis for organizing the group concept mapping results into a meaningful action plan.

More research is needed on the concept mapping process and outcomes. Future studies could compare group concept mapping with other patient engagement techniques for efficacy and efficiency. Understanding patient acceptability of group concept mapping is also important—it may necessitate a greater patient time commitment than interviews or focus groups, but it may also allow for more active patient involvement in multiple steps of the process. The group concept mapping process, as we have presented it here, appears to be a viable method of engaging stakeholders, and we suggest that research is needed that measures engagement as an outcome of group concept mapping.