Evaluation of the Use of a Virtual Patient on Student Competence and Confidence in Performing Simulated Clinic Visits

Abstract

Objective. To assess the effect of incorporating virtual patient activities in a pharmacy skills lab on student competence and confidence when conducting real-time comprehensive clinic visits with mock patients.

Methods. Students were randomly assigned to a control or intervention group. The control group

completed the clinic visit prior to completing virtual patient activities. The intervention group completed the virtual patient activities prior to the clinic visit. Student proficiency was evaluated in the mock lab. All students completed additional exercises with the virtual patient and were subsequently assessed. Student impressions were assessed via a pre- and post-experience survey.

Results. Student performance conducting clinic visits was higher in the intervention group compared to the control group. Overall student performance continued to improve in the subsequent module. There was no change in student confidence from pre- to post-experience. Student rating of the ease of use and realistic simulation of the virtual patient increased; however, student rating of the helpfulness of the virtual patient decreased. Despite student rating of the helpfulness of the virtual patient program, student performance improved.

Conclusion. Virtual patient activities enhanced student performance during mock clinic visits. Students felt the virtual patient realistically simulated a real patient. Virtual patients may provide additional learning opportunities for students.

INTRODUCTION

The perpetually evolving field of pharmacy demands a progressive curriculum that can effectively educate students on developing critical thinking and communication skills to improve patient care. Pharmacy students are expected to perform certain aspects of physical exams and communicate effectively with patients. The Accreditation Council for Pharmacy Education (ACPE) Standards 2016 Standard 2.1 requires that “the graduate is able to provide patient-centered care as the medication expert (collect and interpret evidence, prioritize, formulate assessments and recommendations, implement, monitor and adjust plans, and document activities)” and Standard 3.6 requires “the graduate is able to effectively communicate verbally and nonverbally when interacting with individuals, groups, and organizations.”¹ ACPE Standard 3.2 requires that graduates “Educate all audiences by determining the most effective and enduring ways to impart information and assess understanding.”¹ The Center for the Advancement of Pharmacy Education (CAPE) Educational Outcomes require the same outcomes.² Gaining proficiency in taking a health history, performing exams and communication skills requires multiple opportunities to practice and receive feedback.³ Common strategies for instruction include the use of standardized patients, actors, peers or faculty for students to practice techniques and receive feedback. The use of these modalities generally requires significant institutional resources, including faculty time and physical space.^4-6 During a curricula revision, Pharmacy Practice laboratory course coordinators decided to explore the use of Virtual Patient Simulation (VPS) as it has been reported to be an innovative and resource-low manner in which health care training can be accomplished. In general, utilization of VPS has been reported to be both dynamic and cost effective in training students on standardized clinical simulations.^6-9 This study will describe experiences with a virtual patient software and its’ effects on student performance in mock clinic visits.

Many medical, nursing and physical therapy programs have installed VPS in practice courses to measure student reception and academic impact.^4,10-13 VPS allows students to conduct patient exams in a stimulating, convenient, and realistic way. Shoemaker and colleagues found that VPS not only improved individual competency but also improved collaborative efforts.¹² This study involved a team with pharmacy, physician assistant, and physical therapy students. The experimental group put together a comprehensive treatment plan for a virtual patient with diabetes. The control group did not use VPS. The experimental group showed markedly higher results when it came to using the complementary roles of each member to optimize care. VPS may play an important role in interdisciplinary teamwork to improve health care.¹² This tool can be utilized during non-classroom time, at the discretion and convenience of students, therein aiming to enhance student acceptability. All these features can prove to be especially beneficial in a pharmacy curriculum.

Further use and study may allow for tailoring of VPS for pharmacy curricula. One study showed that pharmacy students found VPS to be enjoyable, helpful in learning, pertinent, and believed that more of these activities should be used in the curricula.¹⁴ The study also found that the VPS improved learning through test questions after the activity.¹⁴ Salter and colleagues found that, although there is limited evidence supporting the use of “e-learning,” it can still serve as a means to provide real-time feedback to both the students and instructors.¹⁵ Another study found that faculty-led problem-based learning in conjunction with VPS might serve as a more thorough means of teaching students if incorporated into pharmacy curricula.¹⁶ Smith and colleagues discussed various VPS platforms and outcomes, such as student performance on knowledge quiz scores and knowledge retention, which improved with VPS use. However, no reference was made regarding the impact on communication skills.⁶ Veronin and colleagues noted that a virtual environment improved students’ written communication skills. However, oral communication skills was not formally assessed.¹⁷ This study was designed to see if the use of a virtual patient would improve oral student communication in mock clinic visits.

Four VPS programs were considered for utilization in this study.¹⁸ Web-SP which was described by Zary and colleagues seemed to require significant faculty work to develop scenarios.^18,19 i-Human offered a comprehensive virtual patient interaction geared toward all health professions but was significantly more expensive than the two other investigated programs.¹⁸ Shadow Health and V-Sim seemed to be somewhat similar in features and price. Both are marketed for nursing students and faculty thought that although the practice and skills would be transferable, V-Sim is titled “V-Sim for Nursing” and pharmacy students may be unreceptive to a product geared for nursing. Shadow Health was interested in collaborating with a pharmacy school and offered a grant to supply the program at no charge to our pharmacy students for a pilot year. During the study year, faculty could utilize the program in a manner that integrated into the curricula and students would have lifetime access to the program. The Shadow Health virtual patient program Advanced Health Assessment Digital Clinical Experience was chosen for this study. (Shadow Health. (2016). Digital Clinical Experience (Version 4.0) [Software] Gainesville, FL. Available from http://www.shadowhealth.com.)

Shadow Health Digital Clinical Experience (DCE) is a web-based virtual patient simulation that affords students the opportunity to interview, examine, document and reflect on their experience with the Shadow Health Digital Standardized Patient.²⁰ The DCE is composed of different modules that correspond to different components of a history and physical exam.²⁰ The DCE has been predominantly used in schools of nursing, and to our knowledge, our study represents the first evaluation in the use of the simulation product in pharmacy education.^10,20,21 The DCE even allows for application of clinical knowledge. A Case Western University study of DCE in a nursing curriculum found that this type of VPS helped their students to communicate in a manner that is more appropriate when interacting with a patient.²¹ For instance, if a student asked a patient if they were having “dysuria,” an instructor could recommend that they ask if the patient is having “pain with urination.”²¹ Instructors can design specific case scenarios that encourage students to scrutinize all pertinent patient information. After a topic specific exam and history are conducted with the program, students are allowed to provide medical recommendations along with educational, informative, and empathetic remarks directly to the patient – thereby simulating a realistic interaction. Additionally, this program enables students to reflect on their own performance and apply better techniques and newfound knowledge to real life situations.²⁰ Students in this study are introduced to communication skills in two general education courses and then are taught the Indian Health Model for patient counseling in a didactic lecture in the first professional year. Students in the pharmacy practice labs in the first professional year engage in mock counseling using adjunct faculty members as standardized patients to practice their communication skills. Additionally, students complete modules on pulmonary, cardiovascular and endocrine devices in the first professional year practice labs which introduce the technical aspects of physical assessment. A core domain of the IPPE I experience that all students complete after the first professional year is patient communication.¹ Students complete a year of didactic lectures on pharmacotherapeutic topics, which include pulmonary, cardiovascular, and endocrine conditions during the second professional year. Throughout the second professional year students learn counseling techniques including motivational interviewing to improve patient engagement.

Prior to this curricular revision, physical assessment had been a separate and distinct laboratory component in the spring semester of a yearlong capstone course in the third professional year focusing on three distinct areas of physical assessment: pulmonary, cardiovascular and endocrine systems. Students participated in mock clinic visits as a team with each student assigned a portion of the clinic visit to conduct. Students were required to successfully complete individual objective structured clinical examination (OSCE) assessments of the technical aspects for each area. For example, in the pulmonary area, a student could conduct only the history portion of the clinic visit, yet would need to demonstrate a peak flow meter and two separate inhaler devices to successfully pass the OSCE portion. Students generally showed proficiency with the technical aspects prior to the curricular revision, with approximately 2% of students annually failing and needing to be remediated. However since each student was only evaluated as part of a team in conducting a clinic visit, it was hard to evaluate overall proficiency in the communication component of the clinic visit. A goal of the curricular revision was to assess each student on a complete mock clinic visit evaluating both communication and technical skill. A goal of implementing a computerized patient simulation or virtual patient was to increase student proficiency and confidence in the mock clinic visits by allowing students multiple attempts to practice prior to attending the mock lab.

During this curricular revision, physical assessment was incorporated into the community lab component of the capstone course with pulmonary occurring in the fall semester and cardiovascular and endocrine in the spring semester. Each student would conduct individual clinic visits and be assessed on both the technical component of the physical assessment skills and the communication component of the clinic visit. The length of each lab session was decreased from three hours to two hours with an expected addition of an online or pre-lab requirement to be completed prior to attending the laboratory session. The virtual patient exercises were intended to be a portion of the required pre-lab work.

METHODS

The learning objective for incorporating a virtual patient into the pharmacy practice lab is to improve student communication performance and confidence in the mock clinic visits in the practice lab. Use of a computerized virtual patient that could be completed asynchronously would accomplish the goals described in Bloom’s taxonomy of application, analysis, synthesis, and evaluation. Literature supports the use of technology could be an effective strategy to engage students in the learning process.^6,22 Documented benefits of VPS include the program providing feedback to students, allowing multiple student attempts, and students developing into self-directed learners.⁶ Faculty theorized that students who grew up surrounded by digital technology, would embrace the digital patient and its use would improve skills in the mock clinic visits conducted in the community practice laboratory.

Students enrolled in this required course participated in two 50-minute didactic lectures where faculty taught the pulmonary and cardiovascular assessments of the mock clinic visits. Students had access to instructional material through Blackboard, (Blackboard, Inc., Washington, DC) the University course management system. Links to approximately 60 minutes of video demonstration and individual patient instructional leaflets were provided to 17 different pulmonary devices for preparation for the pulmonary clinic visit. Links to approximately 8 minutes of videos on taking a radial pulse and blood pressure were provided for the cardiology module. Additionally the students attended a two-hour mandatory laboratory session prior to each clinic visit to practice interviewing and manipulating the various devices: inhalers, peak flow meters, sphygmomanometers, stethoscopes, etc.

In the practice laboratory session, peers acted as patients and faculty, adjunct faculty and/or Advanced Pharmacy Practice Experience (APPE) students provided feedback. The estimated time for students to complete the virtual patient (VP) assignments for the fall was 70 minutes, and in the spring was 85 minutes. Actual time varied considerably student to student. The inclusion of the use of the VP, which was a novel and different type of learning method, new to this institution, fostered independent and self-directed learning of adult learners. Students were allowed up to three weeks and multiple attempts to reach the required threshold for each assignment, allowing student control over their learning.

The study was designed as two distinct phases, delineated by semester (Table 1). The fall semester used a single blinded, randomized, controlled design. The spring semester design was open-label, nonrandomized and uncontrolled. Details and logistics of employing the Virtual Patient (VP) were completed after the publication of the fall course syllabus. Student participation in the fall was voluntary and students were awarded extra credit points for successfully completing the VP assignment. The awarding of extra credit points in the fall semester addresses the pedagogical principle where the importance of grades is high.

Table 1.

Timeline of Project

Each VP exercise allowed faculty to customize the subjective and objective findings that would be scored for each patient. Faculty then set a threshold to be reached for each assignment. The threshold could be simply completing the assignment, time based or content based. The threshold set for the fall was completion of the orientation, spend 30 minutes completing the health history and spend 30 minutes completing the comprehensive respiratory exam as well as achieving 25% of the clinical findings. The threshold for the spring semester was completion of the orientation, 40% of the clinical findings for each the cardiovascular and abdominal modules. The thresholds were determined by having fourth professional year rotation students test the exercise and then taking 50% of their score to account for some uncertainty on time required and utility.

As a required component of the class, in the spring semester, and based on the experience in the fall, the thresholds were increased. The time requirement was removed for two reasons: efficiency and reporting. Efficiency is developed as skills are mastered.³ Students develop mastery at different rates, so requiring students to spend a required amount of time was deemed counterproductive. Additionally, the way the VP grade center reported student time was cumbersome for a class of more than 300 students. Students were allowed multiple attempts completing the VP modules to build competence and reach the thresholds. After completing each exercise, model patient notes were revealed by the computer program for the virtual clinic visit so students could see what they had asked and documented compared to an ideal clinic visit. Attainment of the threshold constituted passing the specific VP exercise component and earning extra credit or a passing score.

The design of this study focused on communication performance in the mock clinic visit in the pharmacy practice lab, and student evaluation of the use of the virtual patient (Table 1). A survey was developed to evaluate student opinion on conducting physical exams and utilizing a virtual patient. Students were informed about the use of the virtual patient and the survey during a didactic lecture. All third professional year students enrolled in the required course were invited to participate in the anonymous survey. The research protocol, survey and grading rubric were reviewed and approved by the Institutional Review Board of the university. The 335 students enrolled in the course were randomly divided into groups to participate in the pharmacy practice labs. Each lab group was determined by taking the alphabetically arranged master roster and assigning a group letter (A-F) to each student name in order. Students in groups D, E, F became the control group, students in groups A, B, C became the intervention group (Table 1). In the fall semester, 318 students (95%) attempted the virtual patient assignment; 283 (85%) successfully completed the three modules and earned 4 points extra credit on an exam grade. In the spring semester, activities with the virtual patient were required graded modules of the course and included in the syllabus.

To assess communicative performance in the mock clinic visit, the physical assessment rubric was redesigned. Pharmacy practice faculty, who routinely see patients with asthma, hypertension and diabetes, worked with the lab coordinators to develop a system to assess the level of communication the students achieved in providing patient-centered care with each disease state. Four questions or assessments were identified for each disease state as demonstrating increasing levels of patient engagement and practitioner communication (Table 2). The questions were included on the grading rubric utilized in the pharmacy practice simulation lab, with the comment “This area is for student feedback only. It does not affect the overall grade” on the assessment area. Both groups of students, control and intervention, had access to the rubric in advance of the graded mock clinic visit. Adjunct faculty participating in the simulated clinic visit were instructed to check off each question the student asked or addressed during the session, as well as indicate null if an area wasn’t addressed. At the completion of each session the rubrics were collected and the data was evaluated. Rubrics that had no markings, neither a check nor a null symbol were excluded from analysis.

Table 2.

Mock Clinic Visit Rubric Questions or Assessments for Each Disease State as Demonstrating Increasing Levels of Patient Engagement and Practitioner Communication

The survey was administered via Qualtrics, Washington, DC. Students were asked four questions regarding their confidence in completing physical exams and six questions related to their opinion on utilizing a computerized virtual patient. The survey was administered prior to any student accessing the virtual patient computer program and completing any mock clinic visit (Table 1). The survey was administered a second time (interim survey) after completing three required VP exercises and completing the pulmonary mock clinic visit and a third time (final survey) after completing two additional required VP exercises and the cardiovascular mock clinic visit. All questions utilized a five-point Likert scale with responses ranging from strongly agree to strongly disagree. Three open text questions were included on the interim and final surveys to allow for student comments. Questions included were “list three positive aspects, three negative aspects of using the VP” and “do you have any additional thoughts or comments you would like to share about the DCE?” Statistical significance was evaluated at p< .05.

RESULTS

Student performance in the mock clinic visits is shown in Table 3. Performance was evaluated by the number of questions, indicated on the rubric, that the students asked during the mock clinic visit. More students in the intervention group, asked each question and attained a higher overall score, compared with the students in the control group, 45.18% vs. 53.19% respectively (p=<.001). In the control group, 60 (n=140) or 42.8% of students asked question number 2 and in the intervention group, 77 (n =141) or 54.6% of students addressed this item during their pulmonary clinic visit. This was the only question that showed a statistically significant improvement from pre- to post-virtual patient activity in the pulmonary lab (p=.049).

Table 3.

Student Performance. Number of Students Asking Each Question During Real Time Mock Clinic Visits in the Simulated Pharmacy Lab. (The score is the percent of questions asked in each group.)

In the cardiovascular mock clinic visit, after all students completed the VP assignments, more students asked more questions and the total score surpassed the combined score for students in the control group and intervention group, 49.20% to 62.15% (p<.001). When comparing the pulmonary control group with the total post-cardiovascular clinic visit group, it is evident that there was a statistically significant increase in the number of questions asked (p<.01 for all questions). The overall percentage of questions asked increased from 45.18% in the pulmonary control group to 62.15% for the total post-cardiovascular group. Question items 2-4 showed a statistically significant increase in the cardiovascular clinic visit group (p<.02 for those questions), which indicates an improvement in academic performance. Additionally asking more questions in general, addresses Bloom’s taxonomy learning objectives of application of improved communication skills. The specific inquiries of question #3 and #4 reach Bloom’s taxonomy of analysis, synthesis, and evaluation; as the questions address causes of disease exacerbations and then seek to work with the patient to solve the issues. The number of students who asked all four questions during each clinic visit increased from the control group to the intervention group to the cardiovascular clinic group, 23.4% to 29.8% to 33.8% respectively, although not statistically significant (p=.40). All these findings reinforce the idea that improved student performance may be linked to more exposure and continued practice with the virtual patient simulation experience.

Results from the student opinion survey are shown in Table 4. The table shows the percent of students who responded Strongly Agree or Agree to each question. Questions 1-4 on the survey assess the students’ confidence in taking an adequate history, performing physical exams, and recognizing abnormal laboratory values. It is interesting to note that student rating of confidence decreased after the pulmonary modules, (p<.001) compared to the interim survey, but increased to baseline after the cardiovascular modules, showing no significant change in aggregate student confidence pre to post (p=.209). The dip in confidence rating after the pulmonary module may be indicative of student realization of their overestimation of ability or overstatement of confidence prior to the VP and mock clinic visit. The reality of the complex VP and mock clinic visit may have caused the decrease in student confidence after the pulmonary module. Yet student confidence rebounded to baseline, after completing additional activities with the VP and additional mock clinic visits, perhaps indicating that practice and exposure to the VP and additional mock clinic visits increased confidence. These results may be attributed to increased familiarity and practice with the virtual patient client.

Table 4.

Student Survey Results. Percent of Students Who Responded Agree or Strongly Agree

Student agreement that VP was easy to use, question 5, increased significantly across the three surveys (p<.001). This may suggest that with consistent practice with the same program, students could subsequently carry out required tasks with more confidence in what they were doing. Survey questions 6-10 evaluate how useful the students felt the VP is to their learning. Fewer students agreed or strongly agreed that the VP was useful in helping them learn to obtain a history, perform advanced physical assessment skills, synthesize data or reflect on their performance after each module. However, student performance in the mock clinic visits improved after completing the modules with the VP.

Data were analyzed via SigmaPlot 13.0 software (Systat Software, Inc., San Jose, CA). With regards to both the performance data and the student perception data, the execution of a quantitative analysis required the assignment of numerical values to otherwise qualitative responses. In assessing comparisons between groups relative to the inclusion, or exclusion, of a clinic visit question, the inclusion of a question was assigned a numerical value of one, with exclusions coded as zero. A similar method of numerical assignment was applied to Likert scale items. For the present investigation, statistical analyses needed to be executed in accordance with two accommodations. Data were not normally distributed, as confirmed by the Shapiro-Wilk test, with such a characterization affecting Type I error risk. The second accommodation was implemented to address the unequal variance of the data, as confirmed by Brown-Forsythe tests. This latter point was very likely a consequence of differences in sample size between assessments. Such accommodations within data distribution and presentation favored the execution of Mann-Whitney Rank tests versus analyses of variance (ANOVA).

Qualitative analysis of the open text responses from the student survey was consistent with the Likert question responses. The students were asked to list positive and negative aspects of using the VP and the responses were analyzed by utilizing a direct content analysis.²³ One hundred seventy six positive and 145 negative comments were received. The most noted positive aspect identified by the students was that the program realistically simulated a real patient which was mentioned in 28% of the comments, followed by appreciating the ability to practice mentioned in 23% of the comments. Students also favored the availability of detailed model notes of ideal findings to use to compare their own findings (14% of comments) and the need to ask good questions to get helpful information from the virtual patient (9% of comments). The most noted negative aspect identified by the students was that they felt the program was not geared toward pharmacy, mentioned in 28% of the comments, followed by the patients and scenarios being preset and not realistic, 26%, experiencing technical issues, 17%, and the program being time consuming, 14%.

An additional 86 comments were provided by students when asked for additional comments or suggestions. These additional comments or suggestions were analyzed using a conventional content analysis;²³ 44% of the comments felt the program was geared more to other health professions and not pharmacy, while 15% of the comments addressed that it was good practice and 12% of the comments noted that purchasing the program would be too expensive or not worth it. Faculty had hypothesized that the use of a computerized VP would provide valid opportunities for student practice that students would embrace. Further evaluation of methods to gear the tool toward pharmacy is warranted but is outside the scope of this intervention.

DISCUSSION

The hypothesis that use of a virtual patient simulation would improve student performance and confidence is supported by these results. The data show that repeated exposure to a computerized virtual patient program increases student confidence and performance in a mock clinic visit in a pharmacy practice laboratory. The increasing attainment of the rubric items indicate that students were able to engage at increasingly meaningful levels, reaching Bloom’s taxonomy levels of analysis, synthesis, and evaluation, with the mock patients and potentially advance patient care. Ultimately, the goal is for students to effectively communicate and engage with patients during APPE rotations and as future practicing pharmacists. The long-term impact of student performance with and without the use of the VP is beyond the scope of this study.

The use of the virtual patient appears to have provided additional practice outside of the laboratory and improved student performance in a cost effective manner. We estimate that each student counseling or mock clinic visit session costs the university approximately $14.67 in adjunct faculty salary alone, not including indirect costs of laboratory space. The virtual patient program was supplied to the students for this study at no out-of-pocket cost via a grant from Shadow Health. The usual fee is $99 per student for lifetime access to the program. In this study, if the student only completed the six required exercises, with a single attempt, it would result in a $16.50 cost per exercise. Students in this study completed the respiratory assignment a mean of 1.54 times and the cardiovascular assignment a mean of 1.59 times with a range of 1-13 and 1-5 attempts respectively, netting costs as low as $4.50 per session for a hypothetical student who took maximum attempts on these two assignments and one attempt on each of the other assignments. The enrollment fee allows students unlimited access to 17 different assignments for life, so additional assignments and/or student practice sessions would drive the cost per session below the cost of a faculty involved session in a practice lab.

The survey results reveal promising findings in regards to this educational tool. Likert scale and open text student responses from the survey validated faculty hypothesis that a VP program would improve student confidence. An interesting theme emerged from the solicitation of positive comments on the student survey. Approximately 9% of the comments described that using the VP helped them realize that it took skill to elicit pertinent information from patients. Sample student comments included: “The VP responses were realistic and if you didn’t ask a good question, the VP’s answer reflected that.” “This program helped me understand the process of asking the patients questions. I learned that a pharmacist has to ask detailed questions to get specific answers from their patients.” “The DCE helped me formulate better questions.” Students’ enhanced appreciation of having additional practice and that actually interviewing a patient was more complex than they had perhaps assumed was seen by faculty as significant learning outcomes. Of note, continued use of the VP seemed to decrease student rating of usefulness as seen in the survey (Table 4) items 7-10, even though actual performance of skills in lab continued to improve throughout the study (Table 3).

This study has several limitations. The use of the VP was evaluated in a single course in the curriculum in a single school, so the results may not be generalizable to other pharmacy schools. Over the course of the study, participation varied both in performance as well as student perception. Concerns relating to the findings related to student performance include significantly fewer rubrics in the pulmonary modules which may have been due to faculty not correctly completing the rubric correctly, as rubrics not marked with a null symbol or checkmark were excluded. Such exclusions in data points further contributed to the lack of equal variance in the statistical analysis. Additionally, the decline in student participation on the surveys could have affected student perceptions. The changing thresholds for successfully completing the different exercises in the VP may have confused students and affected the study results. Also, the provision of extra credit for completing the pulmonary VP exercises compared to including the cardiovascular exercises being required graded components of the course could have affected student perception and survey results. Although we felt that the use of a virtual patient simulation program would be cost effective, some may feel that the $99 fee is cost prohibitive. A $99 fee would be less expensive than many textbooks and using the program in additional courses would increase utility and value. Additionally, the enrollment fee may be paid by the university, the student or shared by both allowing for a potential to shift the cost totally to the student, away from the university.

CONCLUSION

The major goal of this curricular innovation was to enhance student performance in mock clinic visits by utilizing a virtual patient computer program. Students who completed the activities with the VP performed better in the pulmonary mock clinic visit than students who completed the VP activities after the mock clinic visit. Overall student performance continued to improve during the cardiovascular mock clinic visit after all students completed the VP activities in advance of the mock clinic visit. The virtual patient simulation program may provide additional learning opportunities for students and improve performance.