Impact of a Paper vs Virtual Simulated Patient Case on Student-Perceived Confidence and Engagement

Abstract

Objective. To evaluate online case simulation vs a paper case on student confidence and engagement.

Design. Students enrolled in a pharmacotherapy laboratory course completed a patient case scenario as a component of an osteoarthritis laboratory module. Two laboratory sections used a paper case (n=53); three sections used an online virtual case simulation (n=81). Student module performance was assessed through a submitted subjective objective assessment plan (SOAP) note. Students completed pre/post surveys to measure self-perceived confidence in providing medication management. The simulation group completed postmodule questions related to realism and engagement of the online virtual case simulation. Group assessments were performed using chi-square and Mann Whitney tests.

Assessment. A significant increase in all 13 confidence items was seen in both student groups following completion of the laboratory module. The simulation group had an increased change of confidence compared to the paper group in assessing medication efficacy and documenting a thorough assessment. Comparing the online virtual simulation to a paper case, students agreed the learning experience increased interest, enjoyment, relevance, and realism. The simulation group performed better on the subjective SOAP note domain though no differences in total SOAP note scores was found between the two groups.

Conclusion. Virtual case simulations result in increased student engagement and may lead to improved documentation performance in the subjective domain of SOAP notes. However, virtual patient cases may offer limited benefit over paper cases in improving overall student self-confidence to provide medication management.

INTRODUCTION

Situated learning is a teaching strategy that places the learner in a simulated but realistic environment where they assume a specific role and set of tasks or challenges designed to achieve prespecified learning objectives.¹ Through situated learning, connections between complex real-world situations and classroom experiences can be developed.² One category of situated learning used in pharmacy education is online case scenarios that use virtual patients. Virtual patients may be defined as “computer programs that simulate lifelike clinical scenarios in which the learner becomes the health care professional making therapeutic decisions.”^3,4 Common scenarios ask pharmacy students to assume the role of the pharmacist and responsibility for drug therapy decisions and outcomes. The use of simulated patients in the doctor of pharmacy (PharmD) curricula aligns with the Accreditation Council for Pharmacy Education’s (ACPE) Guidance for Standards 2016, which encourage employment of patient simulations and other active-learning strategies to facilitate and enhance student learning experiences that may be challenging to achieve in real-life practice settings.⁵

While use of virtual patient technology is established in medical education, less has been published in the area of pharmacy education.³ To date, literature is available describing software or web-based, telephone-based, and e-mail-based virtual patient programs in PharmD curricula.^4,6-16 Despite a lack of comparison of virtual scenarios to an alternative method of learning, virtual patient scenarios are innovative, realistic, and engaging learning activities by student report and improve student didactic knowledge.^4,7,9-14

Benedict and colleagues surveyed students at the conclusion of a therapeutics course that had implemented patient scenario simulations to augment traditional lectures and found that students either agreed or strongly agreed the simulations were enjoyable (92%), stimulated interest in course content (82%), and should be further incorporated into the curriculum (86%).¹² In a disease state management course, Douglas et al assessed the impact of virtual patient case scenarios on pharmacy student clinical competence skills through administration of four short-answer questions before and after exposure to patient case simulations. Evaluation of student performance determined students performed significantly better on three of four competency-based short-answer questions (p<0.001) after completion of the scenarios.¹⁴

In the literature, a deficit of studies compare virtual patient case scenarios to traditional instruction or paper-based patient cases, and few studies evaluate the impact of virtual patient learning activities on student performance. Benedict et al assessed the effectiveness of virtual patient cases to promote self-directed learning vs traditional lecture-based instruction in an advanced therapeutics course.¹³ Students assumed the role of a health care provider to make recommendations in a realistic clinical scenario. Evaluation of knowledge acquisition and retention using examination performance was equivalent between students who received lecture instruction one year and students who received the combination of lecture and virtual patient cases in the subsequent year. In another study by Ray and colleagues, pharmacy students were randomized to a simulated patient case with a mannequin or a paper-based patient case involving management of a narcotic and acetaminophen overdose.¹⁵ Student knowledge acquisition and retention were comparable, with no statistical differences noted. It remains unclear if virtual patient case scenarios offer any benefit over traditional paper-based patient cases in terms of student engagement, perception, confidence, attitude, and skill performance.

The current study builds on literature that supports the use of simulated patients to increase student engagement and knowledge and contributes a unique perspective by directly comparing an online virtual simulated patient case to a traditional paper case with a cohort of students simultaneously enrolled in a course. Specifically, the current study investigated an osteoarthritis interactive virtual case scenario to determine if a virtual simulated patient format has educational benefit over the historically used paper-based patient case format in terms of student performance, engagement, and perception of confidence in patient management. The aim was to determine if additional paper-based patient cases should be transitioned over to the more resource intensive online case scenarios at the University of Wisconsin-Madison School of Pharmacy, and to inform other colleges and schools of pharmacy considering use of similar learning activities.

DESIGN

The University of Wisconsin-Madison School of Pharmacy requires students to complete a 4-semester pharmacotherapy sequence in the second and third years of the traditional PharmD program. Each of the four courses is composed of four credits and includes a lecture (three lecture periods and one discussion period per week) and a skills-based laboratory component (one 2-hour laboratory session per week). Based on student preference, availability, and elective course schedules, students enroll in one of five weekly laboratory sessions offered Monday through Friday. The laboratory courses use active learning to teach skills necessary to improve patient outcomes related to medication use. A traditional laboratory learning activity involves using paper patient cases to instruct and evaluate students in assessing a patient profile to provide appropriate medication management.

In summer 2012, laboratory coordinators applied for and were awarded an educational innovation grant to adapt and expand a pre-existing paper patient case into an interactive virtual case scenario. In virtual case scenarios, instructors put course content into compelling story-like narratives, and students assume an active role in applying what they learn. Often used in a blended learning environment, these activities build connections between complex, real-world situations and classroom experiences. Instructors framed course activities using the “7 Cs” methodology, which provides a framework for the design process when creating authentic virtual, interactive cases. The seven core design elements (content, context, challenges, characters, choices, consequences, and connections) allow instructors to create robust, real-world challenges and decisions that engage students and facilitate learning.¹⁷

The osteoarthritis paper case was adapted to a virtual simulated case through the Case Scenario/Critical Reader (CSCR) Builder authoring tool created at the University of Wisconsin. The CSCR tool allows students to interact with multiple media elements and web resources. Examples of the CSCR Builder can be viewed on the University of Wisconsin Engage website.¹⁸ Case scenarios vary in complexity, but instructors can assume a minimum of 20 hours to design, develop, and implement them.

Adaptation of the osteoarthritis patient case was designed to simulate the real-life practice situation of rounding in an ambulatory clinic setting with a pharmacy preceptor and attending physician. Goals for adapting the paper patient case to a virtual simulated activity were to increase engagement of students in caring for a patient and to increase relevance to and realism of future clinical practice. Objectives of the learning activity were to allow students to assume responsibility for assessing medication use, identify medication related problems through a simulated patient interview and navigation of a medical record, and provide appropriate drug therapy recommendations to optimize medication outcomes.

Through the osteoarthritis simulated patient case, students immersed themselves in an ambulatory clinic scenario by reading questions, comments, and feedback from a pharmacy preceptor, attending physician, and patient. Students examined the scenario by reading text and clicking on forward/backward navigation buttons. To provide a more interactive environment, students were required to select answers to multiple-choice questions and/or chose appropriate patient-specific questions from a list of prepopulated questions. After selecting a patient question, the patient response was visible on the screen along with feedback from the pharmacy preceptor regarding the quality and appropriateness of the question. A link was provided to a simulated electronic medical record (EMR) at the bottom of each screen. The EMR represented a realistic ambulatory care chart environment.

After attending a traditional 75-minute osteoarthritis lecture, third-year PharmD students enrolled in the fall 2012 Pharmacotherapy III course (n=134) completed an osteoarthritis laboratory module, which included required pre/postlaboratory activities. Figure 1 depicts the order of prelaboratory, laboratory, and postlaboratory activities completed by students. In activities before the laboratory, students completed an online tutorial on how to present a patient case to a provider, assumed the role of a pharmacist working in an ambulatory clinic and assessing a patient to provide drug therapy recommendations to the patient’s provider following an initial office visit through completion of a patient case, and documented pharmacotherapy recommendations through submission of an electronic subjective-objective-assessment-plan (SOAP) note in the online course management system.

Figure 1.

Flow diagrams depicting the activities and assessments of paper and simulation student groups.

Two laboratory sections (Tuesday and Thursday, n=53) were assigned to the traditional paper case for completion of prelaboratory activities; three sections (Monday, Wednesday, and Friday, n=81) used the virtual case simulation. Students’ SOAP notes were randomly assigned to one of seven postgraduate year-one (PGY1) residents participating in instruction of the laboratory as a part of a longitudinal teaching rotation for grading. The SOAP notes were manually graded within one week of submission using a standardized rubric where each of the four SOAP note domains (subjective, objective, assessment, and plan) earned a score of likely harmful (0), needs improvement (1), satisfactory (2), or excellent (3). A score of “Credit” was earned if a student received less than three needs improvements and did not receive a likely harmful in the category domains.¹⁹ On average, 20 minutes were spent grading each SOAP note. Grading instructors were blinded to section-assigned differences in prelaboratory activities (completion of paper vs simulated patient case). With the exception of prelaboratory case assignment, all other laboratory activities were identical; therefore, residents were blinded to case type when assessing SOAP notes.

During the 2-hour laboratory session, all students participated in groups of 4-5 in a live, simulated clinic activity in which they assumed the role of clinical pharmacists and conducted a follow-up patient interview. Students were instructed to imagine 14 days had passed since the patient’s initial (prelaboratory) office visit. During the visit, students conducted the following activities: (1) conducted a patient interview to gather subjective data necessary to assess the patient and determine the safety and efficacy of the patient’s osteoarthritis medication; (2) discussed and interpreted laboratory values with a resident physician; (3) used patients’ subjective and objective information to identify their drug-related problems and formulate a plan for resolution; (4) presented the patient case and final medication-related recommendations to an attending physician; and (5) provided patient education on the agreed-upon therapeutic plan. All students completed similar in-laboratory activities.

Pharmacy residents played multiple roles on any given laboratory day, but only one role within a given student group. Roles included the patient, resident, and attending physician. Residents also provided students informal verbal feedback on interactions with the attending physician and patient. Activities for which students received feedback included student assessment of medication efficacy and safety through a patient interview, presentation of the patient case and therapy recommendations to an attending physician, and patient education on the final agreed-upon therapeutic plan.

In this study, a quasi-experimental pretest/posttest design was used. The primary research question evaluated whether the use of an online patient case simulation resulted in increased student self-perceived confidence levels and engagement in managing medications compared with the use of a traditional paper patient case for an osteoarthritis patient. A secondary research question evaluated if students assigned to the online patient case simulation performed better on submitted SOAP notes compared to students who completed the traditional paper case.

Students were invited to complete voluntary, electronic surveys before and after the experience and were informed participation would not affect their score in the laboratory module and/or course. Students in the paper and simulation groups completed a 13-item pre-experience confidence survey prior to beginning prelaboratory activities (Figure 1). After pre-experience surveys closed, students completed prelaboratory activities (including completion of a patient case and submission of a SOAP note) and laboratory activities. After completion of all prelaboratory and laboratory activities, students were invited to participate in the 13-item postexperience confidence survey. Students assigned to the simulation group completed an additional 14-item engagement and realism survey, which compared the simulation to a previously completed paper case.

After all students completed the laboratory module and postassessments, the virtual case simulation was made available to students in the paper group as an optional activity so that no student had an advantage over another. Similarly, the traditional paper case was made available to students who were assigned to the virtual case simulation. This study was approved by the University of Wisconsin-Madison Education and Social/Behavioral Science Institutional Review Board.

EVALUATION AND ASSESSMENT

Of the 81 students assigned to the simulation group, 86% responded to the pre/postexperience surveys. Eighty-three percent and 85% of the 53 students assigned to the paper group completed the pre/postexperience surveys, respectively. Pre-experience and postexperience responses were matched using student identification numbers. After the matching process had been completed, all identifying data was deleted and unmatched data was excluded from analysis. Matched response rates for the simulation and paper groups were 78% (n=63) and 77% (n=41), respectively. A chi-square test of independence or Fisher’s exact test was used to analyze individual SOAP note domain scores and baseline confidence scores between groups (4 point scales). The effect of paper vs simulated patient case exposure on the change in individual (7- point scale) and summed (0-39-point scale) confidence scores, and total SOAP note scores (12-point scale) was assessed using Mann-Whitney tests. Statistical analyses were performed using Stata, v13.1 (StataCorp, College Station, TX). A p value of ≤0.05 was considered significant.

To evaluate for potential confounding factors, the pre-experience survey included questions estimating student professional experience at baseline rounding on patients in an ambulatory clinic setting and accessing a patient chart to monitor therapy (Table 1). Baseline group comparisons revealed similar exposure (p>0.05) between the paper and online simulation groups.

Table 1.

Baseline Experience Questions and Group Comparisons Using Chi-Square Test of Independence

The 13-item confidence survey administered before and after the experience was designed to evaluate the impact of an online patient case simulation on student self-perceived confidence levels. It asked students to rate their confidence at that moment in effectively performing a task related to the management of a patient with osteoarthritis using a 4-point unipolar scale of 0 (not at all confident) to 3 (very confident) (Table 2). Use of the 4-point unipolar scale was based on consumer confidence measures and was developed with reliability and participant discrimination of choices in mind.²⁰ The baseline and postexperience confidence scale exhibited a Cronbach alpha estimate of internal consistency of 0.95. Baseline student confidence for all 13 survey items was similar between the student groups (p>0.05).

Table 2.

Results of Self-Reported Student Confidence Measures^a

Analysis of confidence in both student groups (postexperience vs pre-experience) revealed a significant increase in all confidence items (p≤0.02, Table 2) and in individual group confidence summed scores following module completion (p<0.0001, Table 3). No difference was found in change of overall confidence (postmodule minus premodule confidence summed scores) between the groups (p=0.13, Table 3). Comparison of the degree of change in individual confidence items between groups revealed the online simulation group had a significantly increased degree of confidence in assessing medication efficacy (p=0.03) and documenting a thorough assessment (p=0.02) compared to the paper group (Figure 2). Additionally, while not significant, students in the simulation group trended towards an increased change in confidence compared to those in the paper group in management of the overall care of a patient with arthritis in clinical practice after module completion (p=0.05).

Table 3.

Mean Pre/Post Confidence Survey Summed Scores Comparison^a

Figure 2.

Degree of change trend analysis of student self-perceived confidence in managing an arthritis patient^a.

An immunization laboratory module took place four weeks prior to the osteoarthritis module in which all students used a paper case to assess a patient’s immunization status and provide immunization recommendations in the form of a SOAP note. Students in the osteoarthritis simulation group answered postexperience questions related to realism and engagement of the simulation compared to the previously completed paper immunization patient case [14-item scale; 5-point, bipolar scale of 0 (strongly disagree) to 4 (strongly agree)].

The 14-item scale was designed to evaluate student-perceived realism and engagement of the online patient simulation compared to a traditional paper case. These were measures adopted from social judgment literature that (in general) ask participants to assess the perceived realism of television shows.²¹ Mean engagement and realism scores are reported in Table 4. Students agreed or strongly agreed that completion of the simulated patient case precipitated increased interest (75%), enjoyment (71%), relevance (68%), and realism (75%) of the learning experience when compared to the previously completed paper immunization patient case. Additionally, students agreed or strongly agreed that the completed simulation assisted them in learning new content (60%) and that it was realistic to clinical practice (73%).

Table 4.

Results of Online Patient Simulation Group Engagement and Realism Scores^a

The SOAP note submitted as a prelaboratory assignment was used to evaluate student performance in the osteoarthritis laboratory module. Comparison of SOAP note performance between the simulated and paper case groups was a secondary research question. Students in the simulation group performed better in documenting the subjective domain of the SOAP note, as evidenced by significantly higher scores in this domain compared to those in the paper group (p=0.008, Table 5). No differences were found between the paper and simulations groups in the objective, assessment, or plan domains (p>0.05). The four domain scores were summed to determine a student’s final SOAP note score (potential score of 0-12). When comparing the simulation and paper groups, student performance on the entire SOAP note was not different [mean=5.9 (SD=2.1); 5.5 (1.8), respectively, p<0.11).

Table 5.

Mean SOAP Note Domain and Total Scores and p Values^a,b,c

DISCUSSION

This research went beyond the reporting of aggregate data evaluating and describing a single virtual patient experience. The objective and design of the study directly compared two educational modalities (a virtual simulated patient case to a traditional paper case), and limited confounding factors by running the comparison within the same group of pharmacy students simultaneously enrolled in a Pharmacotherapy III course, who, with the exception of the simulated patient case, completed identical course activities. The results of this study build on literature that suggests pharmacy students find virtual simulated patients more engaging, enjoyable, and realistic than traditional paper cases.

To our knowledge, no previously published studies have evaluated possible added benefit from use of a virtual simulated case over a traditional paper case on self-perceived student confidence in appropriately managing a patient’s medication. This study demonstrated a similar increase in overall student confidence managing a patient’s medication regimen in the paper and simulated patient groups following completion of a patient case. Two areas in which student-perceived confidence in the simulation group increased to a greater degree than in the paper group were the ability to assess the efficacy of medications to treat a patient and to document a thorough assessment. The simulation activity required students to choose appropriate patient-directed medication assessment questions from a list to simulate a patient interview, and students were provided standardized electronic feedback on their choice of questions. This activity may have influenced student confidence in medication assessment and documentation to a greater degree than completion of the traditional paper case, where simulated patient-directed questions and subsequent feedback on direction of medication assessment prior to SOAP note completion was not feasible.

Although the simulation was not associated with a significant increase in overall confidence or performance compared to the osteoarthritis paper case, it did precipitate better student engagement, enjoyment, and relevance. This suggests students preferred the patient simulation, so transition of additional paper cases to online patient simulations may be a teaching strategy to consider. These findings add evidence to the body of literature supporting pharmacy students’ preference for virtual simulated patient cases over traditional paper cases.^12-13

One potentially novel finding was that students in the simulation group performed statistically better on the subjective section of submitted SOAP notes compared to those in the paper group, which may be attributed to the format in which students were presented subjective patient information. The simulated patient interview activity may have influenced students to more thoroughly consider and include pertinent subjective information in the subjective section of the SOAP note, thereby improving scores in this domain over those of students in the paper group.

In the paper case, students received the same data but in a passive, single-page document they simply had to read. Active collection of data in a more true-to-life format may have inspired students in the simulation group to better assimilate the information and select only relevant information to include in their submitted SOAP notes vs indiscriminately cutting and pasting in the majority of information provided in the paper case document. The nonsignificant differences noted in objective, assessment, and overall SOAP notes scores in the simulation group may be a result of the study being underpowered. A power analysis determined 550 students (225 in each group) would be needed to have an 80% power to detect a significant difference between the two patient case groups.

The ability of this study to show an increase in student SOAP note performance in the subjective domain in the simulation group compared to a control differs from results previously published evaluating student performance after completion of a simulated patient case. Benedict and colleagues reported no difference in student performance on assessment questions scores between students who completed virtual patient simulations in an advanced therapeutics course and students who had completed the course the year before without patient case simulations.¹³ Additional research with a control group should be performed with an increased sample size to evaluate student knowledge, knowledge retention, and skills obtained from completion of virtual simulations compared to paper cases.

Limitations of this study include the use of students from one class at a single university and nonrandomization of students to simulation and paper case groups. Students were assigned to paper and simulation groups based upon their laboratory section enrollment, which was chosen by the student with consideration of elective classes, work schedules, and preference. Therefore, students with like interests may have been grouped in similar sections and may not be representative of students as a whole. Additionally, survey instruments were not pilot-tested and therefore may have included items that were misinterpreted by students.

Individual student data such as overall grade point average (GPA), science GPA, age, previous degree, etc., was not available without the written signed consent of students. Obtaining individual written signed consent of each student would have made this evaluation unfeasible. Lastly, postexperience confidence assessments were administered after completion of the laboratory module rather than immediately following completion of the patient case and SOAP note. Therefore, despite completion of similar activities, it is possible that participation in laboratory activities influenced confidence in patient management.

Despite comparable student findings in overall confidence, use of online simulated patients led to favorable findings in engagement, relevance, and enjoyment of the simulated patient case. Additionally, students reported the simulation assisted in learning new content compared to paper patient cases. These findings support the transition of traditional paper cases to online simulated cases as the latter result not only in similar overall confidence and performance outcomes to paper cases, but also in increased student engagement in, and relevance and realism of, the learning experience. For these reasons, the authors plan to develop and implement additional simulated patient cases throughout the 4-semester pharmacotherapy sequence. Future studies could evaluate the use of virtual simulated patients on long-term retention of knowledge and confidence.

SUMMARY

Virtual simulated patients can be useful educational tools for pharmacy students as they give students an engaging, realistic, and safe environment in which they can assume the duties of a pharmacist caring for a simulated patient. Despite offering limited benefit over a traditional paper patient case in increasing overall student confidence in providing medication management, the virtual simulated patient case increased student engagement in the patient case and student-perceived relevance of laboratory activities to clinical practice. Additional research is needed to determine if online patient case simulations lead to increased student learning and performance.