A Virtual Patient Software Program to Improve Pharmacy Student Learning in a Comprehensive Disease Management Course

Abstract

Objective. To implement and assess the impact of a virtual patient pilot program on pharmacy students’ clinical competence skills.

Design. Pharmacy students completed interactive software-based patient case scenarios embedded with drug-therapy problems as part of a course requirement at the end of their third year.

Assessment. Assessments included drug-therapy problem competency achievement, performance on a pretest and posttest, and pilot evaluation survey instrument. Significant improvements in students’ posttest scores demonstrated advancement of clinical skills involving drug-therapy problem solving. Students agreed that completing the pilot program improved their chronic disease management skills and the program summarized the course series well.

Conclusion. Using virtual patient technology allowed for assessment of student competencies and improved learning outcomes.

INTRODUCTION

Within a traditional classroom setting, critical thinking, problem solving, and skills in identification and resolution of drug-therapy problems are typically integrated in lecture-based classes and reinforced through small group recitation sessions. Though this pedagogy has been the foundation of pharmacy curricula, and other healthcare curricula for years, students often find themselves disconnected and struggling to apply concepts in a real-world patient care setting.¹ As technology continues to revolutionize healthcare delivery systems, the principles of drug-therapy management must evolve as well. The American Council for Pharmacy Education (ACPE) recognizes simulation technologies as effective active-learning strategies and supports their integration into introductory pharmacy practice experiences (IPPEs).² The organization defines simulation as an “activity or event replicating pharmacy practice.” Advances in technology have made it possible to provide faculty members and students with unique opportunities to use human patient simulators, standardized patients, and virtual patients in an effort to bridge the gap between the classroom and clinical practice.³ Human patient simulators are examples of electronically controlled, high-fidelity mannequins that produce physiologic responses to medical interventions. Standardized patients are trained individuals portraying patients with specific conditions in a realistic, standardized, and repeatable way.⁴ Virtual patients are computer-based simulations designed to complement clinical training.⁵ Simulation learning complies with the Center for the Advancement of Pharmacy Educational Outcomes, which serve to guide curriculum development and assist students in making connections between learning experiences and pharmacy practice.⁶

Schools of medicine and nursing, and tertiary medical centers have been at the forefront of embracing this technology and several institutions have constructed sophisticated simulation centers that use advanced simulation technology to develop and strengthen hands-on clinical and communication skills.^7,8 More colleges and schools of pharmacy are now using human patient simulator technology and standardized patients scenarios to improve pharmacy students’ problem-solving and IPPE skills. However, developing and implementing physical simulation centers is a costly task and requires extensive resources.⁹ The cost of an 11,000-square-foot healthcare simulation center at the Medical University of South Carolina was estimated at $2.9 million, including renovations and equipment, with an additional $690,000 in yearly operational costs.¹⁰ While many schools use standardized patients in mock counseling sessions, students may perceive interactions with these actors as superficial and less emotionally engaging.¹¹ These live interviews, though less costly than full simulation centers, still require a classroom setting and are limited by the educators’ and participants’ availability.

Virtual patient software technology offers the distinct advantage of providing more versatility, mobility, and accessibility through a virtual environment that maximizes the realism of actual patient care at a low level of risk, avoiding the need for expanded physical space requirements and the associated overhead and staffing. These simulation scenarios can increase students’ exposure to patients living in a range of environments from rural communities to urban locations.¹² Demographic variables and comorbidities can easily be modified in a virtual case to encompass a more broad and diverse patient population compared to other forms of simulation.¹³ Unlike traditional methods of knowledge assessment, virtual patient simulation encourages students to move beyond rote memorization and toward clinical decision-making using evidence-based practice.¹⁴ While a short-answer or multiple-choice examination presents a patient at only 1 specific point in time, students can follow a virtual patient’s medical progress over time with the use of a simulated electronic medical record (EMR). Virtual patient platforms can also provide real time clinical guidance as students move through cases, as opposed to delayed feedback on written examination performance because of lengthy grading processes.¹⁵ Today’s Millennial generation pharmacy students have a natural affinity for technology. They embrace novel teaching and active- learning methodologies, such as virtual patient care applications, that bring the experience of patient care into the classroom-based setting.^16-18

There is a considerable volume of literature describing the use of simulation-based learning in healthcare education, but research has been heavily focused on human patient simulator technology and standardized patients.^19-23 There is a lack of published data exploring virtual patients in pharmacy education. Kane-Gill and colleagues conducted a MEDLINE search of simulation learning in pharmacy education; 8 of the 13 included studies involved standardized patients and high-fidelity mannequins, while only 2 articles used virtual patients.²⁴ A literature review conducted by Jabbur-Lopes and colleagues found only 7 of 263 potential studies related to the use of virtual patients in teaching pharmaceutical care, 3 of which were conducted outside of the United States.²⁵

Recognizing the potential value of virtual patient technology, a pilot program was developed and implemented within the school’s traditional comprehensive disease management (CDM) course. The objective was to determine if this novel interactive learning environment improved students’ clinical competence skills on subsequent examination assessments. In addition to documenting the achievement of these learning outcomes, we also assessed the program’s impact on students’ perceived value of this intervention.

DESIGN

The comprehensive disease management course at Northeastern University School of Pharmacy integrates self-care and nonprescription products, pathophysiology, therapeutics, and public health topics spanning 28 semester hours during the final 4 semesters of the doctor of pharmacy (PharmD) program. Foundational skills and uncomplicated medical problems are emphasized early in the faculty team-taught course followed by the application of these skills through complex cases involving multiple disease states in later semesters. The course series contains required seminars throughout all 4 semesters which are facilitated by faculty members or pharmacy residents. The seminars focused on the application of critical thinking, medication-therapy management, and intervention documentation skills using patient case scenarios. Traditional assessment measures, such as short answer classroom examinations, SOAP (subjective, objective, assessment, plan) note writing, and QuEST/SCHOLAR evaluations, are used to evaluate students’ clinical competence skills. Because the course series is comprehensive, the final semester includes patient cases with complex drug-therapy problems, and is particularly challenging for students.

In spring 2011, the school partnered with TheraSim (TheraSim, Durham, NC), a Web-based simulation software company for healthcare professionals. Through this partnership, 15 pharmacy practice faculty members who were curriculum content experts communicated with TheraSim’s medical and product development directors over an 8-month period while authoring 10 pharmacy-focused simulation cases. Each simulation case included multiple patient comorbidities and all 10 cases were designed to represent clinical content included in the 4-semester comprehensive disease management course. Clinical competencies were established for each case corresponding to faculty-authored medication drug-therapy problems. Students were required to identify and resolve these drug-therapy problems as interventions. The school places emphasis on a student’s ability to correctly identify patient drug-therapy problems and resolve them as standard-of-care clinical interventions that he or she would perform as a pharmacist. During the case-authoring process, faculty members provided detailed, patient-specific, evidence-based guidance that they furnished to the student during and after the case simulation.

The pilot software interface was designed to represent an EMR and each case included a picture and video clips of the patient answering a series of predetermined questions related to the encounter. Additional information provided to the student included the patient’s chief complaint, history of present illness, past medical history, medication-use history, physical examination findings, laboratory data, and diagnostic test results. The case simulation also incorporated a detailed history of prior clinic visits, associated relevant findings, and drug-therapy modifications.

Students received a Web address to their virtual patient’s “waiting room” and they were assigned a unique log-on identification code which allowed them access to the cases 24 hours a day, 7 days a week. Faculty members gave a brief software demonstration during a CDM class with an example patient case. Students were instructed to e-mail the TheraSim help desk to resolve technical issues as well as bring these to the attention of the course coordinator.

To progress through a simulation case, students had to log in to the system and evaluate a patient using the EMR as they would in a hospital or clinic setting. After performing a thorough review of the encounter and visit history, students were prompted to complete medication reconciliation for their patient and make drug-therapy modifications that they deemed were appropriate. Students were then prompted to identify pertinent patient drug-therapy problems from a dropdown box and document these changes as interventions. For example, if a student recognized that mediation therapy was indicated for a particular disease, he or she would select this drug-therapy problem and the corresponding “add prescription or nonprescription medication” intervention. Students were also able to identify and document preventative health, disease prevention, and patient education interventions. Upon case completion, students were provided with appropriate patient-specific disease state management decisions and a case summary with supporting clinical evidence. Users were provided with feedback for decisions they made and learned whether they achieved the competencies for that patient case (Table 1).

Table 1.

Key Features of a Virtual Patient Pilot Program Implemented in a Comprehensive Disease Management Course

Following development of the simulation cases, students were required to complete 2 cases per week during the 7-week comprehensive disease management spring seminar course block, and this assignment accounted for 10% of their seminar course grade. While completion of all 10 cases was initially required, we anticipated potential software implementation difficulties; therefore, students who completed at least 8 cases were given credit using an all-or-none approach. Users that made incorrect drug-therapy decisions were permitted an unlimited number of attempts to re-enter a decision and complete the case. The implementation of this pilot study was coordinated to overlap with CDM classroom examinations during this same semester. Because course content and application builds from one semester to the next, students were expected to demonstrate knowledge and skill retention on class examinations. Also, because the comprehensive disease management course represents the culmination of prior PharmD courses, examination questions transcend all levels of Bloom’s Taxonomy of Learning, and represent all domains of Fink’s Taxonomy of significant learning.^{26, 27}

During the spring semester of 2012, 10% of all course examinations contained embedded assessments of previously covered knowledge in the form of drug-therapy problems. The examination question objectives during the spring 2012 pilot implementation corresponded to the clinical competencies of the first 2 virtual patient cases because of course logistics. These examination questions were administered on classroom midterm examinations before and after the completion of the simulation cases. This pilot study evaluated whether these virtual patient case scenarios improved pharmacy students’ clinical-competency skills in solving drug-therapy problems on a subsequent examination assessment.

EVALUATION AND ASSESSMENT

During the development phase, faculty-authored drug-therapy problems were prioritized, weighted, and incorporated into the software interface by product development directors. Faculty members provided information regarding the significance of potential interventions made by the students using a tiered alert system. For example, an intervention that could result in patient harm was assigned the highest alert and incurred the greatest penalty, while less clinically significant decisions were assigned an informational alert that provided the student with guidance but carried no penalty. Clinical competency achievement and mastery of drug-therapy management skills for each simulation case were assessed through the TheraSim artificial intelligence rules-based system, which was based on the assigned alerts. To be consistent with a required passing grade for all other course assessments, students needed to demonstrate a threshold competency achievement rate of 73% on the simulation cases. In accordance with course practices for establishing the validity of questions during the examination item analysis, we predetermined that competencies with overall poor class performance would be removed from the data analysis. Poor performance on a particular competency was defined as achievement by less than 30% of the class.

It was important to determine whether students’ clinical competence, as measured by the pilot program intervention, would correspond to improved drug-therapy problem-management skills on a classroom examination. A pretest consisting of 4 short-answer questions was developed and added to the classroom midterm examination, which was administered prior to the pilot implementation in February 2012. A similar posttest was included on the subsequent midterm examination in March 2012 after students had completed the first 2 virtual patient cases. The examination questions were linked to the following course objectives and simulation case competencies: ordering appropriate antibiotic treatment considering a patient’s allergies (antibiotic-allergy), identifying drug-therapy problems and providing appropriate heart failure interventions (heart failure intervention), appropriate preventative health recommendations (preventative health), and incorporating diet and lifestyle changes to minimize disease progression and recommend adherence education for new medications (medication adherence). These competencies were introduced in previous modules of the comprehensive disease management course and not specifically covered during the intervention period.

The pilot program intervention was not designed to promote content retention but to improve clinical competence through drug-therapy problem-solving skills. Posttest questions were not developed verbatim from the pretest; however, drug-therapy problems and interventions were consistent with established simulation competencies and examination question objectives. Questions sets and grading keys were independently reviewed and validated by 3 faculty members to ensure that questions were appropriately linked to the stated objectives and corresponding case competencies.

A Wilcoxon signed-rank test was used to compare the results of the prepilot and postpilot implementation assessments. At the end of the semester, an anonymous survey instrument using a 4-point Likert scale was administered to evaluate students’ perceptions and reflection on the value of the patient cases and the impact of the novel software platform on student learning.

The evaluation of simulation case completion revealed that 119 of 135 (88%) students completed 8 or more cases and achieved 40 (73%) of 55 competencies. Additionally, 77 (57%) students achieved a score greater than 73% on all 55 competencies, which was adjusted to 74% after eliminating 6 case competencies with poor overall class performance. Students’ posttest performance was significantly improved on 3 of the 4 linked competencies (antibiotic-allergy, preventative health, and medication adherence) with nonsignificant improvements noted on heart failure intervention) (Table 2).

Table 2.

Summative Competency Assessment

One hundred seven of 135 (79%) students completed the postassessment survey instrument (Table 3). In response to whether the pilot study improved their chronic disease and medication therapy management skills, 84 (78%) students agreed or strongly agreed. When asked if the simulation cases were a good summary of the course series, 80 (76%) respondents agreed or strongly agreed. Seventy-six (71%) students agreed that the pilot program improved their understanding of how to use an EMR and 80 (75%) of respondents agreed it improved their ability to identify pertinent patient-specific information. Only 42 (39%) students agreed or strongly agreed that the pilot contributed new disease state management knowledge. Forty-five (43%) respondents agreed or strongly agreed that the program prepared them for upcoming advanced pharmacy practice experiences (APPEs). The most frequent responses to the open-ended questions about the pilot indicated that students valued the real time guidance function and felt that technology glitches were the biggest limitations of the software.

Table 3.

Doctor of Pharmacy Students’ Perceptions Regarding Virtual Patient Pilot Program Software

DISCUSSION

An interactive, virtual patient environment improved students’ clinical competence skills on traditional classroom examination assessments. The impact of this program on students’ perceptions of the value of this intervention was also positive.

The initial percentage of students who completed at least 40 (73%) competencies was unexpectedly low. Upon further investigation, 6 of the individual competencies involving the management and prevention of osteoporosis, uncontrolled hypertension, and chronic kidney disease had a successful completion rate of less than 30% and were removed from the overall analysis because of validity concerns. Competency performance data were valuable because they provided us with the opportunity to critically review cases and identify any case-based factors that may have limited student performance. Several additional factors might explain students’ relatively poor competency performance, such as software technical problems or knowledge and skills deficits. Suboptimal performance competency data were shared with the course coordinators and course content faculty members with recommendations to take this into account when revising and delivering these lectures in the future. After removing competencies with poor class achievement, we felt satisfied with the overall student performance on the remaining 49 competencies.

Our virtual patient pilot program had a positive impact on student pharmacists’ learning outcomes with improvement of all 4 competencies assessed on the examinations and significant improvement on 3 competencies. Students performed better on competencies focusing on skills, such as choosing appropriate therapy based on patient allergies, preventative health, and medication adherence, compared to questions specifically focusing on 1 disease state. The content related to the 4 competencies was not formally covered during the study; however, students may have acquired new skills independent of the virtual patient pilot program and the improvement of learning outcomes could have been the result of other factors.

Some literature related to the benefits of virtual patient software applications on student pharmacist learning outcomes exists and reveals results similar to our pilot program. Orr and colleagues studied pharmacy student learning and virtual patient simulation at the University of Rhode Island.²⁸ Eighty-one PharmD students in their third year corresponded with a virtual patient twice weekly via e-mail for an elective self-care course. Student assessment scores improved by 12% by the end of the program, reflecting an improvement in problem-solving, communication, and professional skills from baseline. Benedict and colleagues studied the use of patient simulation technology by pharmacy students at the University of Pittsburgh for evaluation of critically ill patients who had severe sepsis and septic shock.²⁹ Using a virtual patient software program similar to TheraSim, students’ drug-therapy decision-making and recommendation skills were assessed in several critical care patient cases. Students did not receive grades for the virtual simulation activities; their clinical decisions were evaluated through feedback provided at each decision-making point. Postsimulation learning outcomes were assessed through 12 questions on the course’s cumulative examination involving sepsis, and students’ average examination scores from this material were 87%. While there was no preassessment to gauge the students’ baseline scores prior to the simulation cases, the students scored much higher on the questions involving sepsis compared to an overall examination average of 76%. We attributed this improvement to the virtual experience. Most students (85%) enjoyed the simulation activities and 63% of respondents believed the sessions contributed to their learning.

Most pharmacy students in our study thought that the pilot program improved their understanding of how to use an EMR and manage chronic disease management skills more effectively. Pharmacy students thought that the pilot program improved their understanding of how to use an EMR and better manage chronic disease management skills, and was a good summary of the course series. Although most students reported that the virtual patient technology improved their drug-therapy management skills, less than 50% felt better prepared for upcoming APPEs. This may be explained by the uncertainty that students often feel immediately prior to APPE commencement and inability to objectively predict practice experience expectations. Only 42% of students agreed and strongly agreed the pilot program provided new knowledge to their existing disease state management database. This finding was not unexpected and the primary focus of the virtual patient simulation platform is to reinforce the foundational skills of the medication therapy management process taught throughout the PharmD curriculum. Student feedback also confirmed that the early decision to turn on the guidance feature within the software to provide formative feedback contributed to students’ perceptions of the value of this tool.

The improvements in student pharmacists’ drug-therapy, problem-solving, and critical-thinking skills, as well as the positive attitudes toward the pilot program, provide support for the full implementation of this virtual patient software into the course. Before this integration, the software requires further technical refinement. Students reported receiving clinically insignificant feedback (software system noise) as well as technical “glitches” that may have prevented them from successfully completing some of the cases. There was also a significant faculty time commitment in developing each patient case and time that was spent on answering student questions about technical aspects of the software.

SUMMARY

Implementation of a virtual patient pilot program at our institution improved pharmacy students’ clinical competence skills. Assessment data for competencies with poor performance are useful to further refine specific content delivery. Perceptions of the virtual software user experience were positive. Students reported increased confidence in chronic disease management skills and thought the cases provided a good review of the material covered throughout the comprehensive disease management course series. Despite the reported benefits of this active-learning tool, students encountered a number of technical issues during the pilot implementation that will need to be addressed before full integration into the PharmD curriculum.