Not of One Mind: Mental Models of Clinical Practice Guidelines in the Veterans Health Administration

Abstract

Objective

The purpose of this paper is to present differences in mental models of clinical practice guidelines (CPGs) among 15 Veterans Health Administration (VHA) facilities throughout the United States.

Data Sources

Two hundred and forty-four employees from 15 different VHA facilities across four service networks around the country were invited to participate. Participants were selected from different levels throughout each service setting from primary care personnel to facility leadership.

Study Design

This qualitative study used purposive sampling, a semistructured interview process for data collection, and grounded theory techniques for analysis.

Data Collection

A semistructured interview was used to collect information on participants' mental models of CPGs, as well as implementation strategies and barriers in their facility.

Findings

Analysis of these interviews using grounded theory techniques indicated that there was wide variability in employees' mental models of CPGs. Findings also indicated that high-performing facilities exhibited both (a) a clear, focused shared mental model of guidelines and (b) a tendency to use performance feedback as a learning opportunity, thus suggesting that a shared mental model is a necessary but not sufficient step toward successful guideline implementation.

Conclusions

We conclude that a clear shared mental model of guidelines, in combination with a learning orientation toward feedback are important components for successful guideline implementation and improved quality of care.

Health systems are focusing major resources to improve the quality of health care delivery and to create improvements that are rapid, measurable, and sustainable (Committee on Quality of Health Care in America 2001). Clinical practice guidelines (CPGs) help achieve these objectives by standardizing diagnosis and treatment procedures based on the latest evidence. The Veterans' Health Administration (VHA), the largest integrated health care system in the United States, mandated the implementation of CPGs throughout all its facilities starting in the mid-1990s as a way to systematize the delivery of evidence-based care (Kizer 1996). VHA's performance in 2000 outperformed Medicare on 12 of 13 indicators (Jha et al. 2003). VHA's data monitoring system, the External Peer Review Program (EPRP), shows continuing variability among VHA facilities in their performance (U.S. Department of Veterans Affairs Office of Quality and Performance, Programs and Services 2004). We were interested in understanding the sources of this variability and propose herein that it may be in part attributable to the presence or absence of shared mental models of guidelines among facility personnel. We define shared mental models as cognitive representations of situations and concepts that can be shared by members of a group.

The need to turn to organizational level variables to explain existing patterns in quality of care efforts such as guideline implementation is an emerging theme in the health services research literature. Rubenstein et al. (2000) pointed out that a majority of research investigating the implementation of evidence into practice has focused on changing provider behavior to the exclusion of the external environment, characteristics of the organization, and characteristics of the clinical practice. In addition, research on changing provider behavior confirms the lack of efficacy of didactic interventions aimed solely at the provider (Bero 1998). Davies, Spears, and Pugh (2004) found that although VHA providers had overall positive beliefs about CPGs, they identified system level barriers to their use such as insufficient time, resource constraints, and inadequate access to guideline materials. Other studies have now documented that more of the variance in guideline adherence can be explained at the facility level than at the provider level (Krein et al. 2002). Taken together, these streams of literature highlight the importance of facility or system level influences on provider beliefs and behavior towards guidelines and guideline implementation.

BEYOND THE INDIVIDUAL PROVIDER: INSIGHTS FROM THEORY

There are various theoretical models that concur with the empirical findings discussed above, and may provide additional explanation. One useful perspective comes from the recent application of complexity science to health care organizations. Some theorists have described health care organizations as exemplars of complex adaptive systems (CASs) (McDaniel and Driebe 2001). CASs are characterized by (a) a diversity of agents that (b) interact with each other and (c) are capable of undergoing spontaneous self-organization (Cilliers 1998). In health care, the multiple teams of professionals often required to accomplish the goals of health care organizations, and their respective interactions, comprise such a system (McDaniel and Driebe 2001). One of the critical tenets of CASs that has been supported by health care research is that attention must be paid to patterns and richness of interaction, not just amount or frequency of interaction (Ashmos and McDaniel 1991; Anderson and McDaniel 1999; McDaniel and Driebe 2001). Thus, changes to the individual agents in the system are insufficient to explain the observed changes to the system as a whole, as was found in the aforementioned research (Bero 1998; Rubenstein et al. 2000; Krein et al. 2002; Davies, Spears, and Pugh 2004).

One important set of interactions in complexity science concerns how agents share information and knowledge. With his theory of organizational knowledge creation, Nonaka (1994) provides a framework for understanding some of these connectivity patterns (Nonaka and Takeuchi 1995). According to Nonaka, “organizational knowledge creation should be understood as a process that amplifies the knowledge created by individuals and crystallizes it at the group level through dialogue, discussion, experience sharing or observation” (Nonaka and Takeuchi 1995, p. 239). Nonaka reminds us that knowledge creation is a cyclical process, a spiral with no end point. Individual justified true beliefs are shaped and influenced by social interaction and ultimately internalized as shared mental conceptions. Thus, the origination of shared knowledge begins within the individual, and through the knowledge creation process becomes a shared resource to the group and organization. Shared knowledge can be explicit, as illustrated by computerized knowledge bases such as the National Guideline Clearinghouse (Agency for Healthcare Research and Quality 2004), which houses CPGs from multiple agencies for public use; alternatively, shared knowledge can be tacit, as with cultural norms or group beliefs about various phenomena. It is in this latter group that shared mental models play a role.

SHARED MENTAL MODELS AND GROUP BEHAVIOR

Mental models, whether shared or held individually, are “psychological representations of real, hypothetical, or imaginary situations” (Johnson-Laird, Girotto, and Legrenzi 1998). Mental models share several characteristics: (a) mental models are based on a person's (or group's) belief of the truth, not necessarily on the truth itself (i.e., mental models of a phenomenon can be inaccurate), (b) mental models are simpler than the phenomenon they represent, as they are often heuristically based, (c) they are composed of knowledge, behaviors, and attitudes, and (d) they are formed from interactions with the environment and other people (Kraiger and Wenzel 1997; Johnson-Laird, Girotto, and Legrenzi 1998). They are thought to form the basis of reasoning, and have been shown in many different fields of research to influence behavior. Although conceptually similar, they distinguish themselves from attitudes in that attitudes, i.e., positive or negative evaluations of a situation, are considered an evaluative subcomponent of mental models; attitudes, along with knowledge and previous behaviors, are the ingredients with which mental models are formed (Lord 1997). From a complexity science and knowledge creation perspective, shared mental models are evidence for the existence of “connectivity” among the agents in the CAS; in the case of the present study, shared mental models about guideline can be construed as a marker for effective guideline implementation.

A growing literature on shared mental models has examined its role in team performance and effectiveness. Research has demonstrated that teams with a shared mental model of their work environment exhibit better planning and coordination than teams without such a mental model (Stout et al. 1999). In follow-up studies, teams with a shared mental model performed better than teams without a shared mental model (Mathieu et al. 2000; Fiore, Salas, and Cannon-Bowers 2001). Interestingly, research has also demonstrated that the accuracy of the shared mental model is not as important as its mere presence (Edelson 2000); this suggests that having a common vision of CPGs could play an important part in their effective implementation in a facility. To the extent that members in a facility do not share a mental model of guidelines (whether because of the absence of a mental model, or the presence of conflicting mental models), we believe that this could pose a barrier to effective CPG implementation. Thus, the primary focus of this research is to investigate the influence of mental models of CPGs on guideline implementation efforts within VHA health care systems, concentrating on two research questions:

1. To what extent is there variability in employees' mental models of CPGs?

2. Do high- and low-performing facilities have different mental models of CPGs?

METHOD

Site Selection

The research questions and data presented in this study were part of a larger data collection effort designed to examine barriers and facilitators to CPG implementation. Fifteen facilities nested within four Veterans Integrated Service Networks (VISNs) were selected based on their adherence to the practice guidelines during fiscal year 2001, as documented in the EPRP, a random chart abstraction process conducted by an external contractor to audit performance on numerous quality of care indicators, including those related to compliance with CPG recommended care. A VISN constitutes a collection of medical centers within a geographic area designed to serve a veteran population; each VISN is managed by a central office, which is charged with the efficient delivery of services across its member facilities. Facilities representing a range of performance were included to solicit a more comprehensive understanding of the process of implementing the practice guidelines. Thus, four facilities from each VISN were selected on the basis of their EPRP scores: one high performer, one low performer, and two facilities who showed significant improvement over the previous 2 years. Because of administrative difficulties (lack of local IRB), however, one of the facilities was unable to participate in the study, leaving the final sample at 15 facilities.

Participants

Two hundred forty-four employees from various levels within the 15 facilities were interviewed for the study. Within each facility, personnel at three different organizational levels were invited to participate: facility leadership (e.g., facility director, chief of staff), middle management and support management (e.g., quality assurance manager, chief of primary care, IT manager), and primary care personnel (e.g., physicians, nurses, nurse practitioners, and physicians' assistants). All three organizational levels were adequately represented in the sample: facility leadership (18 percent), middle and support management (52 percent), and primary care personnel (30 percent). Facility leadership assisted in identifying clinical and managerial personnel with requisite knowledge, experience, and involvement in guideline implementation. These employees were then invited to participate in a 1-hour semistructured interview to be conducted at their local facility. These interviews were conducted by pairs of interviewers, all of whom formed part of the research team.

Procedure

Six interviewers were deployed in pairs in Spring 2002 into the participating sites. The interviewers were all research investigators of various backgrounds (three organizational psychologists, a sociologist, a physician, and a nurse), with in-depth knowledge of the project; most were involved with the project since its inception. Each pair traveled to a given site, where together they interviewed participants either individually or in small groups, depending on the participants' schedule and availability. Interviewers took turns leading the interview; one interviewer led the conversation, while the other concentrated on active listening and note-taking, with the opportunity to ask for clarification or follow-up questions; the interviewers then reversed roles for the following interview. After a visit to a site, interviewers were split and paired with different partners for their following site visit. All interviewers were trained a priori on interviewing and field note protocol.

Participants were asked how CPGs were implemented at their facility, including strategies, barriers, and facilitators. To tap their mental models of CPGs, we asked participants what the term “clinical practice guidelines” meant to them, and their opinions on the effectiveness of CPGs as a means to improving the quality of care. Although we used prepared questions to guide the interview process, participants were invited to (and often did) offer additional relevant information not explicitly solicited by the interview questions. The interviews were audio recorded with the participants' consent for transcription and analysis. The interviewing team also took field notes during the interviews as an additional source of data.

The interviewers conducted the interviews over a span of two days at each facility. At the end of each trip, the interviewers compared field notes compiled during the site visit and crafted an initial profile of the facility's climate toward guidelines (their “story”), based on the participants' remarks as well as the interviewers' own informal observations. These profiles were instrumental in better understanding the emerging patterns in the data, particularly for the second research question.

Data Analysis

EPRP Rankings

To ensure an accurate representation of performance on CPG adherence, EPRP data were obtained for each of the participating facilities concurrently with our interview data collection. Facility scores were computed as a composite of performance on five CPGs: diabetes, depression, smoking cessation, ischemic heart disease, and hypertension; this composite was then used to rank the 15 facilities in order from the highest to the lowest performer.

Interview Transcripts

Interview transcripts were analyzed using techniques adapted from grounded theory (Strauss and Corbin 1998). The lead interviewer for each interview reviewed the original transcript in search of responses to the question “what does the term ‘clinical practice guidelines’ mean to you?”; in addition, he or she also searched for additional instances of CPG mental models that were not in direct response to this question. Although most of the passages identified were in direct response to the mental models question, a number of relevant comments emerged elsewhere in the conversation with the respondents. The inclusion criterion for these passages was simply that their content could have been an appropriate answer to the mental models question. Following this initial pass, the secondary interviewer reviewed the coded transcripts for corroboration.

After all the relevant passages were identified, we began the process of open coding, where the text passages were examined for recurring themes and ideas (Strauss and Corbin 1998). The passages identified in the process described above were classified into themes by the principal author, and reviewed by a second investigator for corroboration. This resulted in 19 themes that captured the universe of mental models reported by the study participants. Definitions were crafted for each of the 19 themes, and were reviewed by two subject matter experts (SMEs) on CPGs for corroboration; where appropriate, themes were either merged or split to most meaningfully capture range of mental models in the data. The second step consisted of axial coding, where the 19 themes identified in the open coding phase were compared and thematically related to each other; this resulted in three general classes of themes about CPGs.

A third step, selective coding, is often present in many grounded theory studies; this involves selecting one of the categories developed during axial coding as the core category around which the theory that best fits the data is to be built. Selective coding is not appropriate here, however, because the present study's main focus is on the variability in mental models; reducing this variability to a core category would be counterproductive (Dey 1999). Consequently, no selective coding was done, as the research question in this study was most appropriately answered at the axial coding level.

FINDINGS

Universe of Mental Models

The open coding phase revealed 19 different themes indicative of mental models across employees in the 15 facilities visited. During the axial coding phase these themes were categorized into three overarching thematic classes, or meta-themes: quality of care, administrative/pragmatic (A/P), and resistance. Table 1 presents these themes and their respective classes.

Table 1.

Mental Model Themes of Clinical Practice Guidelines, Organized by Class

Theme	Congruence	Frequency
Quality of care
1. Best practices	7	10
2. Evidence-based recommendations	7	25
3. Guideline emphasis	6.5	18
4. Standard/quality of care	6.5	37
5. Consistency of care	5	11
6. Minimum standard	4.5	8
Administrative/pragmatic
7. Published document from VHA	6	8
8. Checklist/recipe	5	30
9. Keeping people current	5	5
10. Efficiency system	3.5	1
11. Performance measures	3.5	30
12. Documenting provider actions	3	5
13. Clinical reminders	2.5	12
14. Financial algorithm	1.5	1
15. Ignorance of guidelines	1	7
Resistance
16. Only providers worry about guidelines	4	1
17. One more thing to do	1.5	2
18. Unreasonable expectations	1	2
19. We do not need guidelines!	1	6

Quality of Care

Half of the participants who reported on their mental models associated the use of CPGs with quality of care. The most common theme in this thematic class centered on guidelines as either best practices or a standard for delivering consistent, high-quality care. Some focused on the standardization aspect of the guidelines, that is, using the guidelines as a way to ensure that everyone receives the same level of care. Others focused on the benchmarking aspect of the guidelines, using the guidelines as a way to ensure that care was being delivered at a certain level of quality, or as a goal to meet or exceed.

A recurring concern reported by many participants was the need for clinical judgment over and above the algorithms in the guideline. Although an appreciable number of respondents believed that guidelines should be followed to the letter, there was an equally vocal subset of respondents who perceived guidelines as a decision-making tool in helping deliver high-quality care. More specifically, they believed that the guidelines were not necessarily meant to be followed verbatim, and that as long as there was proper documentation and justification, it was acceptable to exercise clinical judgment and deviate from the prescribed course of action as outlined by the guideline.

Administrative/Pragmatic

Themes falling under the A/P class of mental models tended to center on the idea that guidelines are largely system or bureaucratic dicta, serving mainly as a means to document, monitor, and evaluate behavior, and are not necessarily related to improving patient outcomes per se. Most respondents whose mental models fell in this classification equated guidelines with either some sort of algorithm or checklist, or with the performance measures used to monitor compliance with guideline implementation. Respondents who perceived the guidelines as performance measures tended to demonstrate minimal understanding of where the guidelines came from, or the body of evidence that supported them—guidelines were simply a mandated, numerical performance standard to be accomplished, in many cases “because that's what the VISN said we were doing,” as illustrated in the following quote from a nurse manager:

… And basically was told, well these are guidelines. … certain things were being pulled out and monitored and reported on an ELC (sic: executive leadership council) and whatever was being measured at the VISN level was where the efforts were placed at the local level to get the numbers in line.

As mentioned above, the second most common interpretation of guidelines in this class was that they were some sort of checklist or algorithm to be followed. Interestingly, respondents were close to evenly split on the perceived value of such an algorithm: many equated the algorithm with cookbook medicine, and a certain degree of inflexibility; slightly more respondents, however, did not view guidelines as a hindrance; and, some of them actually expressed value in having an algorithm for their use in daily practice.

Other themes in this class were far less frequent. One recurring theme equated guidelines with the clinical reminders in the computerized patient record system. Another recurring theme equated the guidelines with a published document sent from headquarters, whose contents are immutable and inflexible. Finally, some respondents saw the guidelines as a tracking tool to document provider actions, while others saw guidelines as a sort of continuing education, a way of keeping people current with medical advances.

Resistance

Although they were in the minority, a small number of the participants we interviewed expressed resistance to the idea of a mandated set of CPGs. Some expressed their concerns that the guidelines placed an undue and unreasonable burden on an already overtaxed workforce, suggesting that the process of going to the guidelines and documenting their compliance with the guidelines simply took too much time, that they were “yet another thing that they had to do.” Finally, a small contingent of respondents expressed that the guidelines were an unnecessary step, that there was no need for CPGs to even exist, and that they hindered, rather than facilitated the work of the provider and other personnel, sometimes suggesting that there were errors in the evidence and recommendations contained in some of the guidelines.

As can be seen from the number of different themes in Table 1, there is considerable variability in the way employees perceive CPGs, thus providing support for our first research question. Descriptions and detailed examples of the nineteen themes are available in Appendix A at http://www.blackwellpublishing.com/products/journals/suppmat/HESR/HESR00387/HESR00387sm.htm.

Congruence of Mental Models with VHA Directive

During the coding process, one question that resulted from the variability of the mental models exhibited in the sample centered on the congruence of these mental models with the intended vision of the VHA. Were the differences observed the result of minor disagreements with the spirit of CPGs? Or were they significant deviations from the original intent, suggesting the possibility of some kind of miscommunication? Follow-up analyses were conducted to explore this question in more depth. The 19 themes developed during the open coding phase were reviewed by the SMEs mentioned earlier for congruence with the VHA's intended vision of CPGs. The SMEs compared each of these empirically derived themes and their definitions to the expressly stated definition of CPGs found in VHA Directive 96-053, Roles and Definitions for CPGs and Clinical Pathways (U.S. Department of Veterans Affairs. Veterans Health Administration 1996), and assigned a numerical congruence rating from 1 to 7 (1 being least congruent, 7 being most congruent). The themes were then tabulated by frequency and congruence with the VHA Directive (Table 1).

In general, those themes that were judged as most congruent with the VHA directive tended to be the most frequently reported themes (r=0.54, p<.05). Two-thirds of the statements (66 percent) reflected a mental model of guidelines that was moderately high to highly congruent with the VHA directive (i.e., a congruence score of 5 or higher). In addition, all of the themes in the quality of care class (which constituted 50 percent of the responses) received a congruence score of five or better, with one exception (minimum standard, which received a score of 4.5). Taken together, these findings suggest that for a majority of the sample, differences in the way respondents perceived guidelines are still within the general spirit of CPGs as conceptualized by the VHA. Despite these results, a significant proportion of the statements (34 percent) reflected ideas that were either only moderately or not congruent with the directive, which could in part explain some of the variability observed by previous researchers in guideline implementation.

Mental Models in High- and Low-Performing Facilities

This final research question asked, is there a difference between high- and low-performing facilities in the CPG mental model patterns reported by their respective members? To address this question, responses to the mental model question were examined in the high- (n=5)1 and low-performing facilities (n=4). Patterns in the high-performing facilities were compared among each other searching for potential commonalities, as were patterns in the low-performing facilities. Once patterns were identified we relied on the corpus of field notes, facility profiles, and informal observations from interviewers to validate the observed patterns and to provide interpretive context. For example, a facility that exhibited a shared conception of guidelines as “quality of care” highlighted visual displays in the clinic setting advertising importance of guideline-related activities to high-quality care (e.g., posters, brochures, screen savers, etc.). This was taken as support for the designation of that particular clinic as emphasizing “quality of care.”

Three of the high performers shared what appeared to be a performance-driven mental model of CPGs: placing a heavy emphasis on performance measures. Reports from these facilities indicated that the guidelines were not simply tied to performance, but in many ways inextricably linked. A fourth facility exhibited a solid quality of care mental model throughout the facility; some respondents suggested that the guidelines were insufficient to deliver high-quality care, and that providers were held to a higher standard than simple guideline compliance:

I've determined that they're minimal standards, that you would do this for any patient given this disease. These are just the screening tools to get you started to go on a little further. We ask our physicians, and this is a big bone of contention, but we ask them to use their clinical judgment. Sometimes they feel like they may be stuck in the guidelines. Then we try to have a lot of discussion and dialogue with them locally as well as at our VISN level … And we hammer it out for two days. Are these guidelines useful? Do they make any sense? Can we implement them? What should we change? I mean we've been very instrumental in having some of the guidelines changed nationally because that's how strongly we feel.

The fifth facility exhibited somewhat more variability than the others, however, there was a clear mental model of the guidelines as focusing on quality of care among the physicians and nurses. Thus, although the high-performing facilities each demonstrated somewhat different mental models of guidelines, all five facilities seem to have a reasonably clear mental model, regardless of what the model itself might be. The low-performing facilities, however, suggested a different story: Analysis of these facilities indicated that the clear, shared mental models exhibited by these facilities were a necessary but insufficient step toward performance.

In three of the four low-performing facilities, the dominating feature was a lack of a clear, dominant mental model. One of the facilities suffered from a lack of direction—no dominant mental model emerged from the interviewees' responses, as most had different conceptions of what CPGs were and how they were implemented in the facility. In the other two facilities, there was a clear split in mental models between upper administration and primary care personnel—while the upper levels of administration seemed to show an understanding of the evidentiary and quality of care nature of the guidelines, the primary care personnel perceived the guidelines as a more bureaucratic phenomenon, often equating the guidelines with the performance measures:

All this are coming down as a directive performance, division performance, the national performance, you know, issues. And if you're not careful if you lose part of it, lose the emphasis on patient care; you mechanically do these things, not actually delivering the care. [sic]

Facility leaders reported that they took it upon themselves to inform the providers and other clinical staff that the guidelines were a priority, and that the performance measures were being used to monitor their performance, and that a certain standard on the performance measures was expected. In turn, primary care personnel reported that management placed too much weight on EPRP data.

Taken together with the high-performing facilities, these data would suggest that the presence (or absence) of a clear shared mental model seems to be associated with successful CPG implementation. One alternative explanation for the observed findings is that the mental model patterns exhibited by the high-performing facilities are more positive than those exhibited by the low-performing facilities, and that it is the mental models' positive nature rather than their shared nature that accounts for the observed differences among facilities. To examine this possibility, frequency counts of each of the themes in the A/P class were tabulated for each facility, as the themes in this class can be categorized into positive and negative (in general, the quality of care themes can all be construed as positive, while the resistance themes can all be construed as negative). The proportion of positive to negative theme frequencies was then compared across facilities. Facilities consistently had a higher proportion of positive themes than negative themes, and this pattern held throughout the facilities. As there were no appreciable differences in the distribution of A/P themes across facilities (i.e., a low performer had appreciably more positive than negative A/P themes, just like the high performers did), there appears to be more support for the shared mental model explanation than for the positive mental model explanation.

Despite the findings discussed above, one facility exhibited a strong, clear mental model of CPGs, yet was one of the lowest performing facilities in the sample. This facility exhibited a clear quality of care and evidence-based medicine mental model of CPGs. However, its performance was highly variable across measures, ranking from as low as 2 to as high as 10 (higher rankings are better) among the 15 facilities sampled in this study, for any given performance measure. To try to account for this anomaly, we turned to reports of the facilities' “story” as captured by the investigators' field notes and facility profiles; specifically, we compared reports from the high-performing facilities with reports from the anomalous facility.

In addition to exhibiting clear mental models, the high-performing facilities also displayed a strong commitment to data-driven performance improvement: all five facilities used EPRP data as a bona fide source of feedback, using the information from the EPRP reports to educate providers, identify problem areas, and allocate appropriate resources to improve them, and in some cases as a motivational tool for primary care personnel.

One facility placed great confidence in the leadership of their guideline steering committee, who emphasized guideline education, rather than guideline implementation.

Q: If I were going to ask you as a committee “what's the best process, as concise a description that you can tell me, to implement clinical practice guidelines?” How would you respond?

A: I would say educating first. . .[getting] everybody to listen … from the beginning to see how important it is. … Once you get buy-in you have to fine tune your processes so you have some, everybody agrees how we're going to do it and what's the easiest way to do it, who's the lowest level person that can do it. You know that type of thing. I mean we're to that level now. We're past the buy-in issues.

This same facility also relied heavily on internal monitoring as a diagnostic tool, and used a 100 percent sampling rate for their performance measures. This practice minimized perceptions of injustice or unrepresentativeness among employees, as they tended to perceive the practice as a data-driven feedback process for all, rather than a selective monitoring device. In another facility, the clinical executive board reviewed EPRP reports, identified areas needing attention, and assembled process improvement teams to address these areas. Process improvement teams were assembled depending on the nature of the problem, populating the teams with the personnel most knowledgeable about the area in question.

In sum, all of the high-performing facilities reported using the performance information as a diagnostic tool, which was reportedly used to (a) educate employees on the value of guidelines, (b) identify problems/opportunities for improvement, (c) allocate resources as necessary, and (d) motivate employees to change procedures and behaviors in order to improve their performance. This, however, was not the case with the anomalous facility. Examination of the data for this facility indicated that its mental model was the result of a strict top-down approach to information dissemination. Management informed (not educated) the lower echelons about the mandated CPGs, as well as the way in which they would be implemented; clinicians reported having very little, if any, input about the implementation process, and if so, only after the fact. The result was that although clinicians understood the value and intent of CPGs, the guidelines were nevertheless perceived as a blunt instrument that afforded little flexibility in cases that did not fit the guidelines. Thus, although a shared mental model may be a necessary first step toward better performance, it is not sufficient to guarantee implementation success; a strong commitment to genuine feedback and improvement behaviors seems to be a second important ingredient.

To further support the findings observed in the high- and low-performing facilities, similar analyses were conducted on the remaining facilities, those whose EPRP performance had improved over a 2-year period: Responses for each facility were sorted by overarching class, to observe the pattern of responses in a particular facility; patterns were then compared among each other searching for potential commonalities. We predicted that facilities whose EPRP scores had improved over time would exhibit characteristics that are more congruent with the high-performing facilities than with the low-performing facilities. Facilities with higher EPRP rankings, i.e., those that were closer in score to the high-performer facilities, did indeed exhibit characteristics that were more similar to the high performers than to the low performers. Such facilities tended to exhibit clear mental models, usually focused on quality of care. As facilities moved farther away in ranking from the high-performing facilities they tended to exhibit more instances of resistance toward guidelines, more variability in mental models, and more evidence of a split between senior management and front line personnel. Thus, high scoring improvers resembled high-performing facilities more than low-performing facilities, and vice versa for low scoring improvers; this pattern provides additional support to the findings observed in the high- and low-performing facilities.

DISCUSSION

This qualitative study examined the relationship between health care personnel's mental models of CPGs and guideline implementation. The findings indicated that (a) variability existed in respondents' mental models of CPGs; (b) although a majority of the various mental models was congruent with the VHA's intent, a noteworthy proportion was not; and (c) high- and low-performing facilities exhibited differing patterns of both shared mental models and the use of performance data as feedback for improvement. In particular, high-performing facilities exhibited a shared conception of guidelines while lower performing facilities did not; this is consistent with previous research on shared mental models, which links the presence of shared mental models to improved coordination and task performance (Stout et al. 1999; Edelson 2000; Mathieu et al. 2000; Fiore, Salas, and Cannon-Bowers 2001).

Of particular interest in this research, however, was the finding that simply having a shared mental model may not be sufficient to influence performance. In one of the low-performing facilities, for example, a shared conception of practice guidelines as evidence-based did not seem to be associated with good performance. Closer scrutiny of the interview data revealed that the high-performing facilities not only promoted a shared mental model of guidelines, but also used performance data as a bona fide feedback mechanism that gave them the necessary information to identify problems/opportunities for improvement, educate employees about guidelines, and allocate appropriate resources to address the problem/opportunity. In contrast, implementation efforts in the poor performing facility took the form of leadership mandates to embrace practice guidelines, without feedback or resource support. The behavior exhibited by the high performers is consistent with the knowledge creation and CAS theories enumerated earlier in this paper.

One of the properties of CASs is iteration, the idea that “small changes in the initial conditions of the system can have significant effects after they have passed through the emergence-feedback loop a few times (often referred to as the butterfly effect)” (Milleret al. 2001; Fryer 2004, para 4). The use of performance measure data as evidence with which to make decisions about how to implement guidelines more effectively, in effect, as a learning opportunity, is an excellent example of the emergence-feedback loop that facilitates iteration in a CAS. From a knowledge perspective, Nonaka's “knowledge spiral” posits that iteration is central to organizational knowledge creation (Nonaka 1994; Nonaka and Takeuchi 1995), especially as members of the system begin to internalize explicit knowledge and make it tacit, part of the shared group conception. The feedback-learning loop exhibited by the high-performing facilities is an example of this internalization process; the shared mental models could be considered markers of tacit knowledge internalized by the members of the system during the knowledge creation process.

As can be seen from the previous discussion, various theoretical streams, including shared mental models, knowledge creation, and CAS, all shed light on the findings in this study. Consistent with these streams, and adding new evidence to the mental models literature, this study found that the combination of shared mental models and an active feedback mechanism, rather than the mere presence of shared mental models, were the common characteristics in higher performing facilities. This is a significant contribution that has not been addressed by the aforementioned research streams, and is worthy of more controlled testing. One possibility is that this feedback learning loop mediates the relationship between shared mental models and performance observed in the literature; this, however, is an empirical question.

Limitations

As with all research, this study has several limitations. First, mental models data were not available from every respondent. This was in part because of the format of the interview; in several cases, more than one participant was interviewed at a time—depending on the flow of the interview, not every participant in the interview may have individually voiced his or her mental model of CPGs. Nevertheless, there were sufficient data available to show both the diversity of mental models of CPGs across the VHA, and the role of shared mental models in CPG implementation.

Second, participants within sites were not randomly selected—local contacts at each facility nominated potential participants based on their involvement in guideline implementation efforts at the facility. This limits generalizability, as the sample is not necessarily representative of the entire facility. However, the purposive sample in this study comprised a “test of limits” or proof of concept—it captured the perspective of those most likely to champion implementation efforts at their facilities. One would expect that such a participant sample would exhibit very little variability in their mental models of guidelines—yet this sample showed exactly the opposite for the low-performing facilities. Thus one could expect at least as much variability from a random sample.

Future Directions

The current study opens doors for several areas of future research. First, the current study presented the possibility that mental models and feedback mechanisms interact in systematic ways that lead to specific outcomes; that shared mental models are a necessary but insufficient step to organization-level change, that how feedback is interpreted and used plays a significant role in how mental models can shape behavior, playing possibly a mediating role. Future research should concentrate on better understanding this relationship; other system agents and interconnections, such as leadership and existing infrastructure, could also be studied via the lens of CAS, which could shed new insight on effectuating positive system-level changes in health care organizations.