A Mixed Methods Approach to Assess the Impact of an Interprofessional Education Medical Error Simulation

Abstract

Objective. To evaluate the impact of a single, half-day interprofessional education (IPE) simulation on disclosing medical errors, and to compare the impacts on pharmacy students with those on students from other health professional programs.

Methods. A mixed methods approach was used to provide a comprehensive understanding of the immediate and persistent outcomes of a realistic medical error disclosure simulation. Anonymous pre- and post-simulation quantitative data were collected using a validated attitudinal survey instrument administered at the time of the simulation. To assess more sustained impacts of the simulation, uni-professional focus groups were held several months following the simulation.

Results. The survey analysis showed that a significant positive change in attitudes towards teamwork, roles and responsibilities occurred in students in most of the professions represented, with pharmacy students experiencing positive changes across a wider range of interprofessional attitudes. The focus group results showed that there were persistent impacts across all professions on learners’ knowledge, skills, attitudes, and confidence in disclosing medical errors in interprofessional teams.

Conclusion. Mixed methods analysis of a high-fidelity IPE error disclosure simulation demonstrated that single IPE activities, if realistic, can have significant positive impacts on students’ interprofessional attitudes and competencies, and increase confidence in conducting team-based error disclosures.

INTRODUCTION

In an effort to improve the quality of health care delivery, training health professionals to work effectively in interprofessional (IP) teams has become a high priority of many educational institutions, accrediting bodies, and the health professional community. Giving students the opportunity to learn about and interact with students and faculty members from other health professional programs is the purpose of interprofessional education (IPE). Furthermore, IPE is a required element in the curricula of most foundational health professions, including medicine, pharmacy, and nursing. Interprofessional curricula is not standardized across educational settings and is delivered using a variety of methods.^1-4 Assessing the impact of IPE activities on learners’ ability to work effectively in IP teams is difficult because of the lack of established assessment methods. However, there is accumulating evidence that many IPE activities result in a positive impact on learner outcomes, including IP attitudes, knowledge, skills, and behaviors.^1,4-6

The University of Utah Interprofessional Education Program developed an IPE curriculum including a number of simulations and activities to provide students across the health sciences with opportunities to work in IP teams to find solutions to common problems encountered in IP health care practice. One such IPE simulation focusses on the disclosure of medical errors as an IP team and includes students from pharmacy, medicine, graduate-level nursing, and physician assistant (PA) programs. The learning outcomes for the medical error IPE activity map to the four Interprofessional Education Collaborative (IPEC) Core Domain Competencies (Values and Ethics for IP Practice, Roles and Responsibilities, IP Communication, and Teams and Teamwork).⁷ The simulation that is the subject of this study was conducted in several sessions during the summer of 2016. Prior to the simulation, students were required to complete online assignments, including several readings (eg, a handout on recommended stages of disclosure and the institutional risk management disclosure policy) and two videos on medical error disclosure (Why Do Errors Happen?,⁸ and an in-house production on the value of disclosing medical errors). Students participating in the simulation were divided into IP teams of four to six members and given the details of a fictional medical error scenario wherein an eight-year old patient was inadvertently given potentially toxic doses of an ototoxic antibiotic. Students were instructed to act as the patient’s health care team and inform the patient’s parent or grandparent (played by an actor) about the medical error as an IP team. The teams interacted with the actor three times during the simulation, and the actor adopted a different demeanor during each encounter, progressing from disbelief, to anger, to sadness. Following the simulation, each team was debriefed by a faculty facilitator who had watched the simulation remotely.

We hypothesized that this simulation would have different impacts on students based on respective professions. Because the simulation involved a medication error, it might be expected to have a greater impact on pharmacy students compared to other health professional students. To evaluate the impact of this IPE simulation and to compare the impact on pharmacy students with that on students from other programs, we implemented a mixed methods research approach using a validated survey instrument and student focus groups. This study also sought to develop qualitative assessment methods that were cost-effective and generalizable to other IPE activities at the University of Utah and other educational institutions.

METHODS

This study employed a mixed methods approach, using pre- and post-IPE survey data and student focus groups. By obtaining student perspectives through focus group interviews, a broader understanding of the impacts of the simulation on the students was obtained, thus enhancing the potential for IPE program improvement.⁹ The mixed methodology, which was given an exempt status by the University of Utah Institutional Review Board, was chosen based on recommendations in a white paper by the Institute of Medicine Committee on Measuring the Impact of Interprofessional Education on Collaborative Practice and Patient Outcomes.⁵ This white paper emphasized that mixed methods research would “yield insight into the ‘what’ and ‘how’ of an IPE intervention/activity and its outcomes.”⁵ A mixed methods approach enables the research team to benefit from the complementary strengths of quantitative and qualitative methods to provide an in-depth exploration of the impacts of IPE simulations.^5,10

All students participating in IPE simulations are required to complete the quantitative surveys within one week before (pre-IPE) and one week after the simulation (post-IPE). These surveys contain Likert-scale multiple-choice questions on students’ attitudes towards IPE and medical error disclosure, several open-ended questions, basic demographic information such as the student’s professional program, and six questions that were used to generate a unique anonymous code for each student to allow pairing of pre- and post-IPE survey responses. This code allowed the use of paired t test for statistical analysis of pre- and post-IPE survey data for each student. When the anonymous codes for either pre- or post-IPE survey data could not be paired because the student had generated different codes, the survey data were redacted and not included in the analysis. The pre-IPE and post-IPE survey instruments also incorporated the Interprofessional Attitudes Scale (IPAS),¹¹ a tool developed and validated at the University of Utah that was designed to assess IP attitudes based on the 2011 IPEC Core Competencies for Interprofessional Collaborative Practice.⁷ Two additional questions directly related to the importance of medical error disclosure as an IP team were also included in the pre- and post-IPE surveys. Paired t test statistical analysis was performed using GraphPad Prism, version 7.0, (GraphPad Software, La Jolla, CA, www.graphpad.com).

Student focus groups provided complementary qualitative data on the impacts of the medical error disclosure simulation on IP competencies. This facilitated a group dynamic that Bradley and colleagues believe “can serve as a catalyst to generate unique insights into understanding shared experiences and social norm.”¹² The focus group discussions served to illuminate the similarities and differences in opinion among the various groups of participants, eg, pharmacy students compared to nursing students.¹² Focus groups were conducted eight to 10 months after the simulation (March through May of 2017) so that students would have sufficient time to assimilate and apply the knowledge, skills, and behaviors gained from the simulation in their professional practice experiences.

All 186 students who had participated in the IPE disclosing medical errors simulation during the summer of 2016 were invited to attend a focus group. The demographics of the focus groups and simulation participants are presented in Table 1. The simulation learners included fourth-year medical students, third-year pharmacy students, second-year students in the physician assistant (PA) program, and students at various stages in graduate (masters- or doctorate-level) nursing programs (Table 1). Participants were recruited primarily through email invitations from the research team to the students’ university email addresses, by announcements at the end of classes, and by student representatives and program administrative staff members. Incentives to participate in the focus group (a $5 gift card to the campus store and lunch during the focus group) were explained in all communications. All students that responded to the focus group invitation were given the opportunity to participate in a focus group.

Table 1.

Demographics of Health Professions Students Who Participated in a Simulation Involving Team-based Medical Error Disclosure

A total of 36 students participated in the eight focus groups (Table 1). Each focus group consisted of a single profession in order to obtain profession-specific information regarding the impact of the IPE simulation. There were at least two focus groups for each profession except for graduate nursing students, for which only one focus group was held. Focus groups were conducted in small conference rooms on campus. These rooms were chosen because of their convenience for students and because they were “neutral” space to the participants, ie, they held no significance to any student’s professional program, thus ensuring that the space used did not impact the study.¹³ Prior to each focus group, a member of the research team obtained consent from all of the participants.

Trained volunteers moderated the focus groups using a script with seven open-ended questions (Table 2). The research team developed these questions to assess the impact of the IPE activity on students’ attitudes, knowledge, skills, and behaviors. The progression of questions started with students’ experience observing or participating in medical error disclosure prior to the IPE simulation, then proceeded through increasingly higher levels of the Kirkpatrick/Barr IP learning outcomes model (Table 2), with questions about their experiences during the IPE simulation working in IP teams (Levels 1 and 2), and finally ending with their experiences since the IPE simulation and how the simulation had impacted them (Level 3).^14,15 Prior to using the script in a focus group setting, the questions were tested for effectiveness with several simulation facilitators. In addition to a focus group moderator, each focus group also had a trained volunteer to act as an observer who took notes, started the audio recorder, and provided a summary to all participants at the end of the session to ensure the correct understanding and accuracy of the notes.¹⁶ The volunteers, who were trained and supervised by experienced members of the research team, ensured that the focus groups ran effectively and consistently from one session to the next.¹⁶ Several of the volunteers were librarians with the Spencer S. Eccles Health Sciences Library. Other volunteers included master’s degree students from a variety of non-health professional degree programs on campus. None of the volunteers had any educational affiliation with the specific health professions included in the simulation. Moreover, the one faculty member on the research team with an affiliation to a health profession obtained consent from students prior to discussions and was not present during the discussions. By selecting neutral moderators, professional bias was minimized and students could feel comfortable to speak openly, without fear of having their comments impact their progress in their program.¹³

Table 2.

Questions and Corresponding Kirkpatrick/Barr Interprofessional Outcomes From Focus Group Sessions Conducted Following Completion of an Interprofessional Education Simulation Involving Team-based Medical Error Disclosure

Digital audio files recorded during the focus groups were transcribed by a third-party company (rev.com, Austin, TX) and checked by the research team for accuracy. Handwritten notes by the observer of the focus group were also transcribed. These notes acted as a back-up data source in case of technical difficulties with the recordings. Removal of all names and any identifying information from all transcripts protected subject anonymity.

Transcribed data from the interview audio files were uploaded to qualitative analysis software (NVivo, QSR International, Burlington, MA), where an initial deductive content analysis took place using the four IPEC core competency domains⁷ and the five Interprofessional Attitudes Scale (IPAS) subscales as codes.¹¹ Each transcript was coded by two researchers, and each researcher coded their assigned interviews individually. Each pair of researchers included one who attended the focus group represented in the transcript and a member who did not attend. Following coding, the research team met to discuss outcomes and resolve coding discrepancies. Upon completion of the deductive analysis, the entire team revisited the data to conduct a basic inductive thematic analysis to identify emerging themes related to the initial research questions. This recursive process explored the entire data set, separately and jointly, to identify themes of interest and offer a way to cross-check the entire data set for patterns.¹⁷ The team compared focus group results within and between disciplines in order to understand similarities and differences among focus groups. Extensive team discussions assisted in consolidating codes and reaching consensus on key themes related to IPEC competencies and IPAS subscales. The literature on the beneficial impacts of IPE supported an additional framework for analysis of the focus group data based on the Kirkpatrick/Barr model of IP learning outcomes.^14,15 Responses to the focus group questions were thus also mapped to achievement of increasing levels of IP learning outcomes as described in Table 2.

RESULTS

As expected from the design of the IPE simulation, which focused on a group of students disclosing a medical error together as an IP team, coding of the student focus group transcripts from across all cohorts were predominantly associated with the IPAS subscale of teamwork, roles, and responsibilities (TRR).¹¹ Other IPAS subscales (patient-centeredness [PC], diversity and ethics [DE], and community-centeredness [CC]) were minor themes.¹¹ The IPEC core competency domains⁷ of teams and teamwork (TT) and IP communication (IC) were the most common themes. Other IPEC domains, including values and ethics (VE) and roles and responsibilities (RR), were also important themes. Differences among professions were primarily seen as emerging themes or sub-domain themes that provided a richer understanding of the IP attitudes and competencies impacted by the simulation. In addition, the focus group questions provided information regarding levels of Kirkpatrick/Barr IP outcomes achieved (Table 2).^14,15 For example, a major theme voiced during the pharmacy student focus groups was the unanticipated value that other professions place on the role of pharmacists on an IP team (Level 2a, modification of perceptions and attitudes). This theme did not come up in focus groups of the other professions. Pharmacy students also indicated, more than those from other professions, that they learned valuable patient communication skills from other professions on the team, especially nurses (Level 2b, acquisition of IP knowledge and skills).

The student quotes provided in Appendix 1 reflect important themes from each cohort. Many were annotated according to how the quotes were coded during the deductive analysis based on IPAS subscales and IPEC core competencies. Unannotated quotes are emergent themes that were identified subsequent to the deductive analysis. Representative focus group discussions are presented here according to profession, beginning with pharmacy students and followed by medical, physician assistant, and graduate nursing students. The responses of students are organized according to the sequence of questions asked during the focus group (Table 2), with annotations corresponding to levels of the Kirkpatrick/Barr IPE learning outcomes model.

Prior to the IPE experience, most of the pharmacy students had minimal if any experience with medical error disclosure, although some students had been in retail and hospital pharmacy situations where minor mistakes with prescriptions had occurred. These events were typically handled by the pharmacist in charge. Based on what they had heard from pharmacists they knew or had experienced during rounding on rotations, many of the pharmacy students came into the simulation with the expectation that other professions would not value the knowledge of pharmacists related to medication use, drug interactions, and adverse effects. As one student explained, “Among pharmacy professionals I had talked to before is [the impression] that the rest of medicine doesn't see pharmacy as really medical providers. They [the pharmacist] are not there to really help them out and give knowledge.” (Abstract 1, Quote 2; TRR, IB) They were surprised that students from other professions highly valued pharmacists on the IP team because of their extensive drug knowledge. As one student shared, “So, when I was speaking, they were listening. And that's just something that I really wasn't expecting.” (Abstract 1, Quote 2; TRR, IB, IC, RR) Another student commented, “It was definitely beneficial hearing all these other medical professionals say that actually they do really want to have pharmacy incorporated into all of the different decision-making, how beneficial it was for them.” (Abstract 2, Quote 1, TRR, IB). These statements from the pharmacy students reflect their increased confidence to participate actively on an IP team to provide their pharmacy expertise (Level 2a).

The majority of the pharmacy students indicated that disclosing a medical error as part of an IP team was valuable (Level 1), and each team member brought a different professional perspective (Level 2b; Quotes 3 and 4, TRR, TT, RR). One pharmacy student commented, “things can be missed very easily, but when you have multiprofessional teams, specialized in many different areas, it gives a very holistic approach to patient care to make sure that nothing was missed.” (Abstract 1, Quote 4, TRR,TT, RR).

Some of the pharmacy students shared that they learned important aspects of a good disclosure from students from other professions, including how to disclose an error honestly and candidly, being supportive of the patient and family, and not promising too much (Level 2b). One student shared, “A big take away I had was the way the medical students articulated the error, the way they interacted with the patient, and how they were really strategic on the way they worded things, being very honest and open with what happened, while also not saying, “Yes, your child will be okay.” Because really they didn't know.” (Quote 5, DE, IC, VE). A related lesson learned was how well the nursing students were able to develop rapport with the family member (Level 2a; Quote 6, PC, IC).

Several pharmacy students had an experience in their IPE scenario where the medical students took over most of the conversation with the actor during the simulation, making it difficult for students representing other professions to contribute (Quotes 7 and 8, TRR, PC, IB, TT, RR, IC). This might have provided more continuity to the disclosure, but left the other students feeling “snubbed” and not part of an IP team (Level 2a, but negative).

Subsequent to the IPE simulation, most of the pharmacy students had not been in situations where they could observe a disclosure or use the skills they had learned during the IPE simulation. However, there were several students who had either directly or indirectly experienced a disclosure, or who had been in situations where they could apply some of the skills they had acquired during the medical error disclosure (Level 3). One student shared an experience where they had to contact a prescribing doctor to postpone a chemotherapy treatment to prevent possible harm to the patient, “knowing that other medical team members will respect me gave me the confidence. I called the doctor up, and the doctor was totally supportive of our decision.” (Abstract 1, Quote 9, PC, IC, VE, RR).

Another student described how he had applied interprofessional communication skills learned during the simulation (Level 3), specifically “how we can discuss between nurses, doctors, and pharmacists in such a way that it doesn't look like you're trying to be superior in any sort of knowledge, has definitely been beneficial (Abstract 1, Quote 10, TRR, IC, RR).

Although many of the pharmacy students had not yet applied the skills and concepts they learned in the simulation, most of the students felt that they had gained confidence in their ability to engage in a difficult conversation (Level 3). The following quote reflects comments made by nearly every pharmacy student: “I'm not sure where my confidence is right now, but it has definitely increased considerably since before the IPE. Undoubtedly. Just the confidence that I'll be able to address the problem appropriately, and how I will be able to talk with the patient has increased. There's probably a lot that I still need to learn. And hopefully I don't get too much experience with it, but I'm definitely more confident in understanding how one should go about it (Quote 11, PC).

While most of the medical students participating in the focus groups had not observed medical error disclosures prior to the IPE simulation, most recalled having had some didactic exposure to the principles of error disclosure and difficult patient-doctor discussions. Several students had observed a disclosure on rotations, ranging from a minor prescription error to errors that were more significant (eg, one that included a head injury and one that resulted in death of a patient). In the case of the error that led to the patient’s death, the student indicated they had learned valuable techniques and the importance of medical error disclosure.

The medical students conveyed a full range of responses regarding their experiences during the error disclosure simulation and the value of the IPE experience, in particular with regard to developing IP collaborative team competencies (Level 1). However, a small number of medical students in one focus group said they learned nothing from the simulation about disclosing errors as an IP team (Level 1, negative; Quotes 12 and 13). One of these students related, “I think the learning to disclose errors was good knowledge to do, seeing the case was good, but as for interacting with a nurse, or a PA, or the pharmacist, I've done two years of clinical experience where I've worked with them for eight or nine hours a day for four weeks straight. I've learned what they do, I've gotten to know their personality (Abstract 1, Quote 13).

However, not all of the medical students felt this way. One medical student in this same focus group had positive things to say about the IPE simulation (Level 1, positive; Level 2a). He said, “I find that simulations where we're actually trying to work as a team and act out a scenario is very beneficial. And I have taken these scenarios to the clinic and used them.” (Abstract 1, Quote 14, IC). He went on to say, “on my team I had a PA student, and a nurse, and another medical student. I think that it gave us an opportunity to really think about what we bring to the table and what our different skills are.” (Abstract 2, Quote 14, IC).

When asked about team roles during the simulation, many of the medical students said that they were chosen to lead the team or took the lead as others on the team acquiesced because it was more realistic for the physician to take the lead in a disclosure (Level 2a, negative; Quote 15, TRR, TT, RR). Some of the medical students who took a lead role indicated that they wished that other students would have been more actively engaged with the patient during the simulation. However, some commented that they would have preferred less engagement by other team members (Level 2a, negative), as exemplified by this student’s comment, “A lot of times there's just someone else that jumps in. If someone else did jump in, sometimes you'd cringe at what they said, and then you're back at square one.” (Abstract 1, Quote 16, TRR, TT, IC, RR).

With other teams, the professions balanced the roles more evenly, and these teams seemed to function better. Several medical students commented on how team roles were determined and that having a variety of disciplines strengthened the team (Level 2a, positive). One medical student commented about the team dynamics in disclosing the error, “I was the leader and there was a nursing student, but we all contributed so when one person got stuck, someone else would take over. And I had people who I thought said really great things, and when I got to a point where I was thinking, “I don't know how to answer this question,” someone else would jump in and answer it really well.” (Abstract 1, Quote 17, TRR, TT, IC). Another medical student said this about their team’s IP dynamics, “we were saved by having a team. We just happen to have one person who knew the doses, the correct doses. I was the attending and I didn't know the correct doses so when I started getting a little confused, this other person on our team stepped in and then started describing, “Here's the correct doses and this is what we gave” and she knew side effects and everything. She was a pharmacist.” (Abstract 1, Quote 18, TRR, TT, IC).

Several of the medical students thought it was very important for the team to have a huddle and discussion before meeting with the patient and another between each patient encounter (Level 2b). One student summarized well the important elements of a good IP team disclosure as, “Talking over what happened with the team and figuring out [the plan] first. Keeping the patient apprised that you are looking into something. Once everything's figured out, talking with the team about what you're going to talk about and what roles you hope each of them will take in, presenting that [information] to the patient. Going to the patient, being open and honest, not trying to hold anything back. And then mentioning that not only was this error made, but this is what we're doing to try to fix it (if it is fixable) and also what we're doing going forward, to try to investigate and improve this in the future.” (Abstract 1, Quote 19, TT, IC, VE).

The design of the simulation scenario revealed critical information to the health care team in stages, and this frustrated some of the medical students. However, the importance of early disclosure, perhaps before all the information regarding the cause and consequences of the error was available to the health care team, was mentioned by some medical students as being an important lesson learned from the simulation (Level 2b; Quote 20, VE). Medical students who had observed a medical error disclosure prior to the simulation appreciated the opportunity to practice a disclosure, and to learn how they might react to patients who are extremely angry and upset (Level 2b; Quotes 21, 22, 23). Several of the medical students had observed an actual error disclosure subsequent to the simulation and reflected on the similarities and differences between the simulation and real life (Level 3). One student commented, “when a disclosure happens, neither party is satisfied and I think that's okay because both parties are upset that it happened.” (Abstract 1, Quote 24, TRR, PC, DE, IC, VE). He went on to say, “I think people behave differently in a non-simulation environment where you're more willing to give and take with what has happen[ed] and what will happen in the future.” (Abstract 2, Quote 24, TRR, PC, DE, IC, VE). This student also was able to see how this simulation might translate to other important conversations such as informed consent prior to surgery (Quote 24). Other medical students were able to relate how they might apply the skills learned in the medical error disclosure simulation in encounters where patients were angry or upset or in difficult conversations such as end of life discussions (Level 3; Quotes 25, 26, 27, 28, PC, IC, VE). One student summarized this well in saying, “Disclosing medical errors is just one of many difficult conversations that you have to have with patients and their families. And so similar skills can be applied to whether it's a goal, end of life type goals-of-care discussion, or whatever it may be. But doing the same thing [as the simulation] where you have everybody involved in that patient's care and the patient and their family all together having that discussion (Quote 27, PC, IC, VE).

When asked about their confidence in being able to disclose an error, all of the medical students felt at least somewhat more comfortable with the process (Level 3; Quotes 29 and 30). One student related, “I would rate myself a three [on a scale of five] and I say that because I feel that I know, somewhere in my mind, the tools that I need to disclose an error. But I'm so inexperienced in disclosing errors that I feel I would fumble for words and struggle to convey accurately how we feel, or what we'll do differently in the future.” (Quote 29).

Several medical students in one focus group said they did not see the value of practicing error disclosure in an IP team (Level 3, negative). However, other medical students found the simulation to be a valuable experience in interprofessional collaboration, and the simulation increased their confidence in terms of taking care of patients (Level 3, positive) as conveyed by this student’s comment: “it's given me more confidence in the healthcare system as a whole, because when you think about everything that goes into patient care, as the physician, you feel like you’re responsible for everything and it's very overwhelming. Then to have these other professions, whether it's PT, OT, nursing, pharmacy, or whatever, to have them all step in and play a role, you realize, “Oh. They are so much better at this part of patient care than I am.” It's so nice to be able to have them come in and take charge of that [part of patient care], to have that support (Abstract 1, Quote 31, TRR, TT, VE, RR).

The physician assistant (PA) students in general had little experience or training in error disclosure prior to the IPE simulation, although several students had worked in settings where there were protocols for submitting errors to the institutional administration and team-based root cause analysis of medical errors.

Several PA students had worked previously with pharmacists on IP teams, but the error disclosure simulation provided a different kind of interaction (Level 1). Participants in both PA focus groups indicated that pharmacists were valuable members of their simulation teams because of their knowledge of toxic drug effects and their ability to explain these effects to the rest of the team (Level 2a). Several PA students noted that it was also useful to have a pharmacist as part of the team during the patient interaction because of their ability to answer the patient’s questions knowledgeably. The PA students in general felt it was useful to have an IP team to plan the disclosure and prepare for possible questions that might come up during the error disclosure, what they could promise the patient, and to provide support for one another during the disclosure (Level 2b; Quote 32, TRR, TT, IC, VE, RR, Quote 33, TRR, PC, DE, IC, VE). This comment is representative of what many PA students said, “I thought it was awesome that we agreed as a team before we talked to the parent or the patient's family member that we were going to back each other up and we were going to support each other. We weren’t going to throw anybody under the bus. I thought it was a really good experience to have that sense of unity and teamwork rather than one person just taking the fall for everything. As a group, we accepted the [responsibility for the] error and dealt with the consequences of that and tried to make it better (Quote 32, TRR, TT, IC, VE, RR).

The PA students related different accounts of team leadership during the IPE simulation. In some of the teams, the medical students assumed a leadership role during the disclosure because of their provider role, but in other teams, the pharmacist or other profession that felt the most comfortable disclosing was the primary person interacting with the patient on behalf of the team. Regardless of who took the lead, most of the PA students agreed that it was important to establish continuity during multiple patient encounters (Level 2b; Quote 34, TRR, PC, TT, IC, RR).

The PA students described several important lessons learned from the error disclosure simulation including disclosing early, clearly, and honestly; providing empathy and emotional support for the patient/family member; maintaining good continuous communication among the team members; rebuilding trust with the patient/family member; and avoiding the use of medical jargon and terminology (Level 2b).

Among the students in the two PA focus groups, only one student described observing a potential medical error after the disclosure simulation (Level 3). Other PA students noted how they might apply the techniques learned in the simulation to difficult conversations (Quotes 35-38), such as this comment made by one student: “I've had plenty of angry patients when you say you're not going to prescribe narcotics, [and it’s important] not mirroring their behaviors, not becoming frustrated, remaining calm. (Quote 35, PC, IC, VE) (Level 3). An important aspect to success in difficult conversations is having a prepared response, as conveyed by this PA student’s comment, “the biggest impact for me is just realizing how important it is to have all our preparation done before you have these kinds of conversations. Preparation for yourself, but also preparation with whoever's going to be part of that conversation (Abstract 1, Quote 38, TRR, PC, IC, VE, RR).

When asked whether their confidence in being able to disclose an error increased because of the simulation, most of the PA students indicated that they felt more confident but that they would not be comfortable with disclosing a serious error, especially by themselves (Level 3). In one PA focus group, the confidence level of the students was still at the lowest level, and the confidence level in the other PA group was only slightly higher (2 or 3 on a scale of 1 to 4). Inexperience seemed to be the most important factor affecting the students’ confidence level (Quote 39).

Unfortunately, because of an audio equipment malfunction, there was no recording of the focus group involving graduate nursing students. The observer’s notes from the session indicate the themes expressed by this group were very similar to those brought up in the other cohorts.

Quantitative Data

The IPAS tool for assessing students’ IP attitudes was incorporated into anonymous pre- and post-IPE surveys. In addition to the IPAS items, the surveys included two questions regarding the importance of team planning (Q1: It is important that healthcare professionals plan together prior to disclosure of a medical error) and disclosure as part of a team (Q2: It is important that healthcare professionals disclose medical errors as a team). The results of the statistical analysis of survey data according to the profession students represented are presented in Table 3. A significant positive increase in IP attitudes towards the IPAS subscale of TRR was seen in students from all professions except graduate nursing students. Pharmacy students were the only group in which a significant improvement in IP attitudes in additional IPAS subscales (specifically, PC and CC) were seen. Although we had hypothesized that the medical error IPE simulation would affect pharmacy students differently from other professions, we were surprised to see a significant impact on their IP attitudes in three of the five IPAS domains. The impact of this simulation on the PC subscale is not unexpected, but we cannot readily explain the impact of the simulation on CC as this IPE simulation had little to do with community-centered IP health care.

Table 3.

Comparison of Survey Results of Health Professions Students Who Participated in an Interprofessional Education Simulation Involving Team-based Medical Error Disclosure

The only cohort for which a significant effect was not seen on any IPAS subscale was the graduate nursing student cohort. However, the discussions during the one focus group session with graduate nursing students indicated that these students (representing only three of the 36 students who participated in the simulation and not recorded because of a technical malfunction) appreciated the importance of IP teamwork in disclosing medical errors. One explanation for the lack of an effect on any IP attitudes of this cohort as measured by the IPAS tool is that the graduate nursing students had extensive clinical experience prior to the medical error disclosure. Thus, their IP attitudes may have already reached a maximum level for their profession. Additional explanations for whether an IP activity might have an impact on any particular cohort of students would include whether the student’s profession plays a significant role in the IP activity, the novelty of the role played by a student’s profession in the activity compared to what might commonly occur in clinical practice, and whether the students have had similar training in a didactic course or other component of their curriculum. The systematic reviews by Hammick and colleagues¹ and Reeves and colleagues⁴ provide a rich discussion of the importance of learner characteristics and process factors on the effects of interprofessional education.

Medical students were the only students who demonstrated a significant positive increase in response to the survey question regarding IP planning prior to disclosure of a medical error (Q1). Pharmacy students were the only students to demonstrate a positive increase in response to the survey question regarding disclosing medical errors as a team (Q2).

The results of the quantitative analyses provided by the IPAS surveys conducted at the time of the IPE activity were consistent with the results of the qualitative analysis of focus group discussions as determined using a deductive coding approach. Both the qualitative and quantitative analyses indicated that students in most professions demonstrated an immediate positive effect of the IPE simulation on IP attitudes related to the IPAS subscale of TRR, and on IP competencies across all four domains of IPEC core competencies. In addition, analysis of the focus group discussions using the Kirkpatrick/Barr learning outcomes model indicated persistent profession-specific impacts of the simulation on IP skills (Level 2b), attitudes (Level 2a), and behaviors (Level 3).

DISCUSSION

Systematic reviews of the effects of IPE using the Kirkpatrick/Barr model of IP outcomes have demonstrated that IPE can have a positive effect on learners’ attitudes (Level 2a), knowledge and skills (Level 2b), and behavioral change that translate to professional practice (Level 3), and that lead to changes in organizational practice (Level 4a) and improvements in patient care (Level 4b).^14,15 Our goal in the present study was to assess the differential short-term and longer-term impacts on IP attitudes, knowledge, skills, and behavioral change of a short, high-fidelity medical error disclosure simulation on learners in different undergraduate professional degree programs using a mixed methods approach. Although there are several studies in the literature that have assessed the impact of uniprofessional training in medical error disclosure,¹⁸ very few have dealt with assessing the immediate and long-term impact of training to disclose medical errors in IP teams. The study by Ragucci and colleagues describes the evaluation of a low-fidelity simulation involving IP team disclosure of a medical error with students from medicine, PA, nursing, and pharmacy.¹⁹ The IP teams that had a pharmacy student trained in error disclosure performed significantly better in the simulation in several respects, including explicit disclosure, apologies, and forthright responses to the patient (Level 2b). Poirier and colleagues describe a high-fidelity IP error disclosure simulation with dental, nursing, and pharmacy students using actors who portrayed three different affects (relieved, angry, sad/distrustful).⁶ Students self-assessed and were assessed by faculty members using instruments to measure disclosure and IPEC competencies. The authors concluded that IP error disclosure simulations are effective learning environments that helped students develop IPEC teamwork and roles and responsibilities competencies (Level 2b). At the University of Washington, where low-fidelity IP medical error disclosure training has been part of the IPE curriculum for several years, the immediate impact on learners (medical, nursing, pharmacy, dental, and PA students) included thinking about error disclosure from a team perspective (Level 2b), learning how to disclose medical errors (Level 2b), and learning about other professionals’ roles and perspectives (Level 2a).^20,21 These three reports on the immediate impact of error disclosure simulations on learner attitudes and competencies is thus qualitatively similar to the impact we found in our study. These studies did not report long-term impacts of the IP disclosure training (ie, Levels 3 and 4).

Our study, in addition to supporting much existing evidence concerning the immediate impact of a high-fidelity simulation on student attitudes towards working in IP teams, also explored longer-term impacts (five to eight months later) on student confidence in disclosing errors and disclosing errors in IP teams. The focus group discussions showed that most students across all professions retained the knowledge of the essential elements of effective error disclosure (Level 2b) and gained an increased confidence in disclosing medical errors and engaging in difficult conversations with patients (Level 3). The overall impact of this realistic simulation on IP attitudes and competencies may thus be comparable to that seen in longer-term experiential IPE courses²² and underscores the importance of authenticity for positive IPE outcomes.^1,4

Another valuable finding from the present study is that the simulation increased pharmacy students’ awareness concerning the high regard that other professions hold of pharmacists as members of IP teams. This theme likely contributed to positive changes in the TRR subscale, and perhaps the positive impact on pharmacy students across a wider range of IP attitudes compared to other healthcare professions. Knowing that other professions value your profession and professional role is an aspect of professional socialization important for effective IP team collaboration.²³ Realistic IPE simulations can thus contribute to students developing confidence in their role as an appreciated member of an interprofessional team.

CONCLUSION

The combination of quantitative and qualitative data provided complementary and corroborative evidence that a short, but realistic IPE medical error disclosure simulation increased student confidence in disclosing medical errors, reinforced the importance of IP communication in disclosing errors effectively, and had an overall positive and persistent impact on IP attitudes related to teamwork, roles, and responsibilities, particularly among pharmacy students. Future goals are to assess the impact of this and other IPE activities on higher levels of the Kirkpatrick/Barr IPE learning outcomes model including transfer of IP learning of graduates to individuals’ professional practice after graduation, changes in organizational culture, and improvements in patient care.

Appendix 1. Representative Quotations From Health Professions Students who Participated in Focus Group Sessions After Completing an Interprofessional Education Simulation Involving Team-based Medical Error Disclosure (Abbreviations Following Each Quote Refer to IPAS Subscales and IPEC Core Competencies)

Quote 1: Pharmacy student: Among pharmacy professionals I had talked to before is [the impression] that the rest of medicine doesn't see pharmacy as really medical providers. They [the pharmacist] are not there to really help them out and give knowledge. So, when I was talking about that [during the simulation], the medical, and the nursing students said, “Well that definitely couldn't be further than from what we experience ourselves in the hospital. We always want to have the pharmacist there so that we can talk to them about the medications.” It was definitely beneficial hearing all these other medical professionals say that actually they do really want to have pharmacy incorporated into all of the different decision-making, how beneficial it was for them. (TRR, IB)

Quote 2: Pharmacy student: I really thought that the medical student, and the other PA [student] that was there as [they were going to say], “We’re going to take over” and I was just going to sit there and not really say anything. But it was funny because they did show an appreciation for the pharmacists there. So, when I was speaking, they were listening. And that's just something that I really wasn't expecting. I was just expecting them to try to handle everything. That's just what I've seen typically even when I was doing rounds on my rotation over the summer. (TRR, IB, IC, RR)

Quote 3: Pharmacy student: It's nice to have someone that kind of reins you back like, “Okay this is how maybe we should go about it. What do you think about us presenting it in this fashion.” Or bringing up information on something else, [to] help broaden that view so that you don't just stay too focused, and you miss other aspects. So it's definitely good working in a team, and with different professionals because they take so many different outlooks on even the same problem that you may not have. It helps you be more rounded and actually address the issue more properly. (TRR, TT, RR)

Quote 4: Pharmacy student: I think the benefit of having multiprofessional teams is that it prevents things from falling through the cracks. We might miss one piece of information and then the med student catches it. Or maybe it falls through both of us, but then the nurse catches it. Medicine, there's so much going on that things can be missed very easily, but when you have multiprofessional teams, specialized in many different areas, it gives a very holistic approach to patient care to make sure that nothing was missed. Which I thought was really neat to see in the simulation. I think that should be the future of medicine. (TRR, TT, RR)

Quote 5: Pharmacy student: A big take away I had was the way the medical students articulated the error, the way they interacted with the patient, and how they were really strategic on the way they worded things, being very honest and open with what happened, while also not saying, “Yes, your child will be okay.” Because really they didn't know. (DE, IC, VE)

Quote 6: Pharmacy student: The two nurse practitioners that we had there, they bounced off each other very well [and did] not overwhelm the patient. They also were the ones that came up with, “Hey, we need to make sure that we're not sitting down in such a way that we appear to be surrounding the patient.” … In their experience… it's more comfortable for the patient to be closer to the door and also directly in front of the medical professionals, and not surrounded. Which never even occurred to me as something to think about. (PC, IC)

Quote 7: Pharmacy student: Maybe part of it is that they [medical students] feel like [they have] more of a responsibility, like there's more weight added to the medical provider, the doctor[-patient] relationship. In [other] IPEs, I had med students who were not like this, but I felt like he [the medical student in this simulation] was pretty resistant to letting people onto his turf, of taking responsibility as a team... His attitude was, “Oh, that nurse is really good at talking to this patient, but I'm immediately going to forget that.” Maybe that's a little harsh, but that was my impression. I was just left feeling snubbed by this med student, then just wowed by the nurses. The nurses were so good. (TRR, PC, IB, TT, RR)

Quote 8: Pharmacy student: We would deliberate every time the grandmother [the actor] would leave the room about what we’d feel comfortable saying, how to say it, what we’d want to tell her, and we all agreed on making some certain decisions. But we did have two medical students. One of the medical students was a little bit more passive about making decisions and talking to the parent, whereas the other one tried to take more control over anybody else that was talking. I'm personally used to that, just because I see it a lot in those types of situations. But still I think we were able to express what everybody wanted to say, versus just only having that one person continuously talking, and talking. Not that that's a bad thing, it's just that if you want to say something, it's a little irritating. (TRR, TT, IC)

Quote 9: Pharmacy student: It's boosted my confidence of what I can do. The doctor had ordered that [chemotherapy], and then the pharmacist caught it, and they said, “Okay, you are going to call them up [the prescribing doctor], and present what you know and your recommendation”, which still made me a little nervous, but at the same time I wasn't thinking about the [IPE] simulation at the time, but knowing that other medical team members will respect me gave me the confidence. I called the doctor up, and the doctor was totally supportive of our decision. They said, “Oh, good catch. I didn't realize they [the patient] had done that, had dental work done.” So I think situations like that just serve as a reminder that, yes, other medical professionals will listen to us, and respect what we know. (PC, IC, VE, RR)

Quote 10: Pharmacy student: Having that conversation over the phone in such a way that it doesn't sound like I'm telling them [the prescriber] that they don't know anything, because that's a great way to get shut down really, really, quick. And I've seen that happen to one of my pharmacists. So just talking with other professionals, not necessarily even with the patient at this particular IPE, but just how we can discuss between nurses, doctors, and pharmacists in such a way that it doesn't look like you're trying to be superior in any sort of knowledge, has definitely been beneficial. (TRR, IC, RR)

Quote 11: Pharmacy student: I'm not sure where my confidence is right now, but it has definitely increased considerably since before the IPE. Undoubtedly. Just the confidence that I'll be able to address the problem appropriately, and how I will be able to talk with the patient has increased. There's probably a lot that I still need to learn. And hopefully I don't get too much experience with it, but I'm definitely more confident in understanding how one should go about it. (PC)

Quote 12: Medical student: Okay. I think that the idea of IPE is excellent, phenomenal, great, conceptually necessary. The implementation of it, I think it provides nothing. I can't say I came away having gained anything from it that I wouldn't otherwise have had. (IC)

Moderator: Were you able to learn anything from the various professions that you were interacting with?

Medical Student: It was fun to interact with them and I enjoyed getting to know them as people and talking to them. You know, making those connections, but did I learn anything educationally? No.

Quote 13: Another medical student in the same focus group: I think it's a great idea [IPE simulation], but it really doesn't work that much. I think the learning to disclose errors was good knowledge to do, seeing the case was good, but as for interacting with a nurse, or a PA, or the pharmacist, I've done two years of clinical experience where I've worked with them for eight or nine hours a day for four weeks straight. I've learned what they do, I've gotten to know their personality.

Quote 14: Medical student: I have the opposite opinion [as the other medical students quoted above]. I feel like the simulations, including IPE, are very beneficial. I don't get a whole lot out of sitting in class and listening to lectures. I find that simulations where we're actually trying to work as a team and act out a scenario is very beneficial. And I have taken these scenarios to the clinic and used them. So, on my team I had a PA student, and a nurse, and another medical student. I think that it gave us an opportunity to really think about what we bring to the table and what our different skills are. I felt like I was the least informed on my team. The PA student, especially our nursing student, had a lot more experience, a lot more clinical experience than me and also had actually experienced disclosing medical errors before. It felt like our team, we did a good job of working as a team and it went smoothly because we identified the strengths between the people on our team. (IC)

Quote 15: Medical student (the one who earlier commented that he hadn’t learned anything from the simulation): I feel like a lot of times, the medical student ends up taking the lead role just because it is somewhat realistic, the doctors lead the team. I remember taking the lead basically just because of my being a medical student. Good, because again, I think it's realistic, being a leader of the care team a lot of times. (TRR, TT, RR)

Quote 16: Medical student: A lot of times there's just someone else that jumps in. If someone else did jump in, sometimes you'd cringe at what they said, and then you're back at square one, “Well, now I have to start completely over because now you've just ruined any rapport we might have built in the past five minutes of talking.” (TRR, TT, IC, RR)

Quote 17: Medical student: We selected based on your typical roles, but which professions are on your team is just luck of the draw. Our group was pretty balanced. For one of them [referring to one of the patient encounters during the simulation], I was the leader and there was a nursing student, but we all contributed so when one person got stuck, someone else would take over. And I had people who I thought said really great things, and when I got to a point where I was thinking, “I don't know how to answer this question”, someone else would jump in and answer it really well. So it's not even necessarily what professions are involved, it's just I had really great people in my group who did a really good job communicating. (TRR, TT, IC)

Quote 18: Another medical student added: I had some PAs that were really excellent, as well. And then a fellow med student that wasn't. In our scenario, we were saved by having a team. We just happen to have one person who knew the doses, the correct doses. I was the attending and I didn't know the correct doses so when I started getting a little confused, this other person on our team stepped in and then started describing, “Here's the correct doses and this is what we gave” and she knew side effects and everything. She was a pharmacist. (TRR, TT, IC)

Quote 19: Medical student (summarizing the important elements of a good disclosure): Talking over what happened with the team and figuring out [the plan] first. Keeping the patient apprised that you are looking into something. Once everything's figured out, talking with the team about what you're going to talk about and what roles you hope each of them will take in, presenting that [information] to the patient. Going to the patient, being open and honest, not trying to hold anything back. And then mentioning that not only was this error made, but this is what we're doing to try to fix it (if it is fixable) and also what we're doing going forward, to try to investigate and improve this in the future. (TT, IC, VE)

Quote 20: Medical student: The early disclosure stood out to me. That's something that I feel like I took away. It's okay to go to them [the patient or family member] having prepared with as much as you know, but also being willing to say, “Look, we're coming to you right now because we just found out about this. We don't know all the information, but here's our plan. We plan to do this, this, this, and this to investigate it, and we're already putting things in place right now to prevent this from happening to your loved one or anyone else, but we'll keep you updated.” And I think that's better than, “Four days, we've been working on this and sorry, we forgot to tell you” or “We did not tell you because we were trying to figure it out without you guys knowing.” (VE)

Quote 21: Medical student: I think it will definitely help me in the future. When I observed it for the first time, when I was on Peds [rotation], I was so uncomfortable watching this. And when I did the simulation, I was so uncomfortable doing it, but afterwards I thought, “I'm glad I did that because I feel more prepared for the next time that I have to do it.”

Quote 22: Another medical student: I would agree. I think it's not something that you want to have to do cold without having practiced and so, at least that aspect of it was very useful.

Quote 23: A third medical student: It was pretty useful for me because I didn't realize when I talk to an angry patient, apparently my voice gets a little bit more soothing, more calming, and apparently some people found that insulting. The actor playing the family member told me [during the debrief], “I felt like you were treating me like a kid the more slowly your voice went” and I was thinking, “Because you were getting more and more angry at me.” I learned, personally, what I do in reaction to an angry person because I usually don't get an angry person in my face. So it was a great controlled environment for me to learn that in.

Quote 24: Medical student: I have [seen disclosures since the IPE simulation] and I think the disclosure went well in light of a very terrible situation. The thing that stood out to me is that when a disclosure happens, neither party is satisfied and I think that's okay because both parties are upset that it happened. Things that I appreciated were what other students have already mentioned about being honest and forthright about what happened, and talking about what you'll do to prevent it from happening in the future. I was also impressed with, in real life, it's a little more realistic to have a give and take conversation with somebody, that was at least helpful for me to see. I don't know that you can necessarily simulate that but I think people behave differently in a non-simulation environment where you're more willing to give and take with what has happen[ed] and what will happen in the future. I'm going into general surgery and getting consent for surgery is common and I've seen people do a very good job with informed consent. And some people do a very, very poor job. Some of the skills that you learn in this medical disclosure simulation are helpful in communicating well with a patient when you're giving them informed consent. Because patients value different things and expect different things out of an operation. Skills [from this simulation] that you learn in communication, and team-based care, and setting expectations, and things like that, translate well. (TRR, PC, DE, IC, VE)

Quote 25: Medical student: For me, it [the simulation experience] did help, getting yelled at by the patient [actor] because I've been yelled at by two patients since then, in the clinic, and I think that maybe I handled it differently. I was more calm. I was shocked when the IPE patient [the actor] started yelling at me. It became very real for everyone in the room. But I do think that I may have been more calm, more patient with my real patients. And less shocked.

Quote 26: Another medical student: It wasn't necessarily having the experience of being yelled at but having the experience of saying, “I messed up.”

Quote 27: Medical student: Disclosing medical errors is just one of many difficult conversations that you have to have with patients and their families. And so similar skills can be applied to whether it's a goal, end of life type goals-of-care discussion, or whatever it may be. But doing the same thing [as the simulation] where you have everybody involved in that patient's care and the patient and their family all together having that discussion. (PC, IC, VE)

Quote 28: Another medical student added: I completely agree. I can think of multiple circumstances, with end of life or those kind of situations, where there was a very similar scheme where we met together as a team. We had certain things we wanted to talk about. We talked to the patient and got their understanding, and had a plan moving forward. It was the very kind of structure as any kind of difficult conversation. (PC, IC, VE)

Quote 29: Medical student: I would rate myself a three [on a scale of five] and I say that because I feel that I know, somewhere in my mind, the tools that I need to disclose an error. But I'm so inexperienced in disclosing errors that I feel I would fumble for words and struggle to convey accurately how we feel, or what we'll do differently in the future. I think that the inexperience portion makes me feel less confident.

Quote 30: Medical student: I don't care how long you've been practicing, you could be a 50-, 60-year old doctor. It's never going to be a comfortable conversation. To walk into the room and say, “Hey, I screwed up. Your life is going to be worse because of something I did.” It's true, that's never going to be an easy, well, I hope it's never going to become, a comfortable conversation.

Quote 31: Medical student: I feel like it's given me more confidence in the healthcare system as a whole, because when you think about everything that goes into patient care, as the physician, you feel like you’re responsible for everything and it's very overwhelming. Then to have these other professions, whether it's PT, OT, nursing, pharmacy, or whatever, to have them all step in and play a role, you realize, “Oh. They are so much better at this part of patient care than I am.” It's so nice to be able to have them come in and take charge of that [part of patient care], to have that support. (TRR, TT, VE, RR)

Quote 32: PA student: I thought it was awesome that we agreed as a team before we talked to the parent or the patient's family member that we were going to back each other up and we were going to support each other. We weren’t going to throw anybody under the bus. I thought it was a really good experience to have that sense of unity and teamwork rather than one person just taking the fall for everything. As a group, we accepted the [responsibility for the] error and dealt with the consequences of that and tried to make it better. (TRR, TT, IC, VE, RR)

Quote 33: PA student: We also had an agreement before discussing with the patient about not promising things, because it could be very easy to say “Well, we're going [to] do this and this and this and this” before you know whether you can even do that, like monetary [compensation] things. And to be careful in stepping lightly on what you say initially before you know all the facts. (TRR, PC, DE, IC, VE)

Quote 34: PA student: We felt like continuity was important, so we had at least one person the same each time [for each patient encounter], I think it was the pharmacist, and then the rest of us would trade off, because we felt like that same person, a familiar face, would be good. How we came to that decision I don't recall, but we each took a turn [to trade off]. We got more information each time [we met with the patient] and then we decided “I'll do this next part” or “You can do the next part.” It wasn't really one person [who] took the lead, per se, in my group. (TRR, PC, TT, IC, RR)

Quote 35: PA Student: I've had plenty of angry patients when you say you're not going to prescribe narcotics, [and it’s important] not mirroring their behaviors, not becoming frustrated, remaining calm. (PC, IC, VE)

Quote 36: PA student: If you escalate, things only escalate, so staying calm as the provider or as the team can control the situation. (IB, PC, DE, VE, IC)

Quote 37: PA student: And you can use explanations for [why you can’t prescribe]. You can say “It's the policy of our clinic that in prescribing these medications ...,” so you take a team approach [by citing policy]. (TRR, TT, VE)

Quote 38: PA student: I think the biggest impact for me is just realizing how important it is to have all our preparation done before you have these kinds of conversations. Preparation for yourself, but also preparation with whoever's going to be part of that conversation. That was my big take-away, really, how important that is to have a successful [difficult] conversation. (TRR, PC, IC, VE, RR)

Quote 39: PA student: While I logistically now know how to go about it in the team aspect, I still have fear of just word-vomiting something terrible and not great towards the patient. I don't know that there's really any way to simulate that, to practice it more than what the simulation was. I think just having difficult conversations or conversations where I may be placed where I don't necessarily know all the information, but still have to relay the events to a patient, will help with the comfort level of going into an uncomfortable situation.