Cost-Effectiveness of Using Standardized Patients to Assess Student-Pharmacist Communication Skills

Abstract

Objective. To explore the cost-effectiveness of including standardized patients (SP) in the didactic curriculum for application and assessment of students’ pharmacist-patient communication skills.

Methods. Five role play/case study (RP/CS) activities from a communication skills curriculum were replaced with five SP encounters. Communication was assessed using a rubric. This study developed an economic model to examine the costs and effectiveness of replacing RP/CS events with SP events in knowledge-application and communication assessment. Costs consisted of SP hourly wages for training and delivery of SP events. Outcomes examined were the incremental cost-effectiveness ratio (ICER) per student.

Results. The ICER comparing SP to RP/CS was $100.93 higher per student on first-attempt pass rates and $9.04 per one-point increase in the mean score.

Conclusion. SP was more effective and more costly than RP/CS. Further research into students’ willingness to pay needs to occur before determining if using SPs is cost-effective in teaching communication skills.

INTRODUCTION

Active learning is becoming increasingly prevalent in health professions education. In fact, the Accreditation Council for Pharmacy Education Standards 2016 requires all pharmacy schools to not only use active learning in the classroom, but emphasize active learning throughout the curriculum.¹ A widely used active learning technique in the education of health professionals is the utilization of standardized patients (SP).^2-5 Standardized patients have been incorporated into education through teaching and student assessment and have been used since at least the 1970s in the United States.^2-5 Standardized patients are individuals who are trained to portray a patient or caregiver of a patient in a manner consistent with actual patients or their families.^6,7 According to previous research, SPs are used at more than 77% of pharmacy schools, 94% of medical schools, 82% of physician assistant programs, and approximately 37% of nursing schools.^2-4 Simulation with the use of SPs facilitates student engagement through a sense of authenticity, allowing students to apply many different types of skills, including communication, patient assessment, medication-related problems, professionalism, psychomotor tasks, and other topics.^2-4,8,9 Simulation with SPs is also helpful because it allows students to make mistakes in the classroom/ laboratory before reaching real patients. Students across health care professions have been shown to view the inclusion of SPs favorably.^10-12 Incorporating SPs into the educational process may be associated with improved student outcomes, such as greater student self-confidence, higher scores on knowledge-based quizzes, and improved communication skills.^10,13,14

Smithburger and colleagues conducted a randomized crossover trial for student-pharmacists to compare whether high-fidelity simulation, SP interviews, or problem-based learning (PBL) was more effective with regard to knowledge and management of seizure disorders.¹⁰ High-fidelity simulation uses mannequins (which may have a pulse, heart and lung sounds, etc.) to simulate a real-life scenario that health care professionals may encounter. Both high-fidelity simulation and SP interviews were more effective than problem-based learning cases in all three scenarios, suggesting that more realistic portrayals in the classroom are associated with greater effectiveness. Medical students report that SP encounters are more applicable for future interactions with patients and view these activities as more “lifelike” and may take these activities more seriously than RP, which may explain the difference in effectiveness.¹²

Lupu and colleagues compared written cases (CS), peer role-play (RP), and mock-patient counseling to apply Motivational Interviewing skill development.¹³ This study found that those in the mock-patient counseling group performed significantly better than students in the written case and RP activities on knowledge-based quizzes. Gillette and colleagues compared RP/CS to SP activities to teach 220 second-year student-pharmacists communication skills and found that students in the SP group had significantly better first-attempt Objective Structured Clinical Examination (OSCE) pass rates.¹⁴ Additionally, students in the SP group scored approximately 11% higher than students in the RP/CS group, which reached statistical significance. The first Gillette and colleagues study was the basis for this cost-effectiveness analysis of SP use in pharmacy education.

Despite evidence showing that incorporating SP events into the educational process is effective, recent research has called into question the costs and effectiveness involved in employing SPs in didactic education in comparison with a much cheaper alternative, the students themselves (RP/CS).^15-18 Recent comparative research has shown that RP/CS may be just as effective at teaching communication as SP and at a lower cost to the institution. Further, simulation with SPs requires significant time on the part of faculty members who wish to incorporate SPs into teaching and assessment. Faculty members must build cases specifically for SPs and train them on the clinical nuances and how each case should be portrayed. After training and depending on the institution, faculty members may also need to help facilitate the SP activity. Simulation is also expensive for the institution, usually requiring multiple full-time employees (FTEs) dedicated to the use of simulation as well as the remuneration of SPs, usually as a salary or hourly wage.¹⁹ These faculty/staff maintain a roster of active SPs, recruit additional SPs, and engage in quality control/assurance, in addition to supporting faculty members who wish to use SPs during courses or examinations.

Even though simulation with SPs is associated with greater faculty time and institutional costs, employing SPs also has major advantages over RP/CS. In RP/CS, faculty members still must devote a significant amount of time to write realistic case scenarios, which may be similar to the amount of time as writing an SP case. Second, training SPs to portray clinical scenarios is an advantage because the actor/actress can be trained to portray nuances that are impossible to achieve in a peer role-play exercise because of time limits in the classroom. Other previously mentioned advantages of SP over RP/CS include student engagement with SP compared to RP/CS as well as students being able to make mistakes in a controlled environment before making those mistakes in a clinical setting that could result in patient harm.

Researchers have begun examining the costs and effectiveness of incorporating SPs into the educational environment because of the large expense to both faculty and institutions. However, existing research on the cost-effectiveness of SP versus other active learning methods such as in-class role-play is in its infancy. Most of the comparative research on SP and other active-learning methods on provider-patient communication has focused on medical students or medical residents.^15-18 To our knowledge, only one of these studies critically evaluated the cost-effectiveness of incorporating SPs compared to peer role-play (RP).¹⁷ Bosse and colleagues found that students in the RP group had better outcomes (ie, higher score) and lower costs than students in the SP group.

Pharmacoeconomics combines the scientific and economic fields to examine the value of pharmaceutical products in societies, with the goal being able to choose the optimal allocation of scarce, finite resources, such as financial capital.^19,20 These types of analyses allow decision-makers to choose between two courses of action and decide which action produces better outcomes. There are different types of pharmacoeconomic evaluations, such as cost-benefit analyses, cost-effectiveness analyses, and cost-utility analyses. Of these, cost-effectiveness analyses are one of the most common pharmacoeconomic evaluations and have been used to study a variety of clinical questions. The perspective of the analysis is one of the first and most important aspects to decide on prior to starting an economic evaluation because options from one viewpoint may not be attractive but would be attractive to other stakeholders.

To our knowledge, no study has ever used a formal cost-effectiveness analysis to examine the effectiveness of standardized patients on health care students’ acquisition of communication skills. The objective of this study is to compare the cost-effectiveness of employing SP compared to peer role-play/case study methods (RP/CS) to teach pharmacist-patient communication techniques to student-pharmacists from the perspective of an academic administrator.

METHODS

This study was approved by the Marshall University Institutional Review Board. This study used a quasi-experimental design to compare the effectiveness of two active learning methods in conjunction with the flipped classroom model for teaching patient communication to professional pharmacy students: RP/CS versus SP. A more thorough explanation of the methodology for the educational intervention can be found in a previous publication.¹⁴ The study was conducted over three administrations of a required second-year course. The first two administrations (2013, 2014) used case studies, discussion, and role play to teach the communication skills of interest. The third administration (2015) followed the same course structure and content, but included a series of five SP encounters in place of the case study, discussion, and role-play activities. Students in all three years were assessed using the same communication assessment. The sample size for the study was 220. Topics covered during the course included: theories applicable to pharmacist-patient communication (Health Belief Model, Shared Decision-making, Motivational Interviewing, Social Cognitive Theory, Fuzzy Trace Model, Andersen’s Behavioral Model of Health Services Use); pharmacist-patient discussion of risks/benefits associated with medications; communicating with patients who have low health literacy/cultural competency; Internet use and pharmacist-patient communication; and enabling patient self-management. Figure 1 presents the differences in the course administrations.

Figure 1.

Course Delivery Differences between Cohorts 1 & 2 (Role-play/case study) versus Cohort 3 (standardized patient event).

The final exam for the module in all three years was a high-stakes communication assessment that occurred at the end of the semester. Students who fail the communication assessment during the didactic curriculum are not allowed to progress to Advanced Pharmacy Practice Experiences (APPEs). The grading rubric has 30 possible points and is based on the Indian Health Service method of medication counseling and assessing self-efficacy for correct medication use is from Social Cognitive Theory. A modified Angoff method was used to establish minimum competency and a cut-score of 23 points out of 30 (76.67%) was established as the minimum score to pass the assessment.²¹ In all three instances, the case and the faculty grader were the same for all students.

We obtained the costs for the SPs for the first communication assessment for each cohort. The perspective for this analysis is an academic administrator for a pharmacy school. Only standardized patients’ hourly salaries ($12.50 per hour) were included in the analysis. The costs for years one and two included the training and delivery of both the practice assessment and the communication assessment. The costs for year three were the training and delivery of the five SP activities during the laboratory sessions and the communication assessment. All SP hourly salaries were constant over the three years. Based on stable SP salary across all administrations, costs were not discounted.

This analysis excluded faculty salary for case development and evaluation and time spent in training SPs for several reasons. First, we assumed that the amount of time for a faculty member to develop and evaluate a case for SPs is equal to an RP/CS case. Further, we excluded faculty salary for training SPs because we assumed that if not training SPs, the faculty member’s salary would be spent on a comparable task.

The probability of students passing the communication assessment was calculated from the first-time pass rates for the SP group (P(SP); treatment group) and the RP/CS group (P(RP/CS); control group.

The incremental cost-effectiveness ratio (ICER) was the outcome assessed in this study. An ICER is the difference in costs between two treatments divided by the difference in outcomes. The formula for calculating the ICER for this study was²⁰:

This study also calculated the ICER for the mean point difference between the three cohorts on the communication assessment²⁰:

RESULTS

From our recent publication, there were no significant differences among the student cohorts in undergraduate grade point average, or age (Table 1).¹⁴ The SP group had a significantly higher percentage of students who passed the communication assessment the first time (90.8% vs 61.1%, Chi-square=21.4, p<.001) than students who used RP/CS throughout the semester. Further, students in the SP cohort had significantly higher communication assessment scores than students in the cohorts who used RP/CS (26.5 vs 23.1, t=8.07, df=218, p<.001). Table 2 presents each cohort’s total cost (including training, communication assessment practice (if applicable), and communication assessment), first-attempt pass rate, and mean score.

Table 1.

Characteristics of Comparator Groups (N=220)

Table 2.

Total Cost, Pass Rate, and Mean Communication Assessment Score by Cohort (N=220)

In the two years prior to the implementation of SP throughout the semester (control group), SPs completed 1 hour of training for a practice communication assessment and worked an average of 3.75 hours during the practice assessment. SPs completed 1 hour of training and worked an average of 2.71 hours for the communication assessment. The total SP costs accrued for the two administrations with RP/CS during the semester, including the communication assessment were $1,143.75. The total SP cost for the first course administration was $756.25 ($9.82 per student) and the total SP cost for the second course administration was $387.50 ($5.78 per student).

In the year of SP implementation, students did not have a practice communication assessment. SPs completed 5 hours of training throughout the semester and worked 4 hours per week during the semester. During the communication assessment, SPs completed 1 hour of training and worked 6.5 hours during the assessment. The SP costs accrued for the most recent administration were $2,893.75 ($38.08 per student).

Due to the differences in the number of matriculated students for the first three course administrations (2016 cohort=77, 2017=67, 2018=76), we used the cost per student in the ICER calculation. Therefore, the ICER was $100.93 higher per student comparing SP to RP. The ICER comparing the mean differences in scores on the communication assessment was $9.04 per 1-point increase in the mean score per student (Table 2).

DISCUSSION

In an era of rising student indebtedness and state education budget cuts, it is important for health professional schools and colleges to critically evaluate educational processes that increase tuition and fees.^22-26 As shown in this study, incorporating SPs into student education can be expensive, as indicated by the positive ICER (more costly and more effective than RP/CS). Further, the effectiveness of simulation is being debated as research shows that SPs may not be any more effective at teaching communication than RP/CS. Even though the use of SPs is common in health professionals’ education, studies that critically examine the effectiveness as well as the costs of SPs are lacking but needed.

This study is among the first to use an economic evaluation to examine whether SPs are cost-effective in helping to teach communication skills to student-pharmacists. This study found that students were significantly more likely to pass the communication assessment the first time when standardized patients were used during the semester, even when the quality of the students and the course topics were equivalent. However, the true goal of the course is to develop improved student-pharmacist communication skills through repeated exposure to SPs and instructor and peer feedback on communication skills, not simply pass an assessment. Future research should examine other methods to measure communication skills (ie, open-ended question use during Introductory Pharmacy Practice Experiences) of student-pharmacists outside of the classroom. This would allow pharmacy educators to compare differences in long-term retention of concepts taught in the didactic communication module.

However, the question of whether SPs are truly cost-effective cannot be answered by this study because of the lack of research in the areas of willingness to pay (WTP) and student perception of the costs of stress or student perception of the costs of passing a high-stakes assessment. WTP is an analysis that attempts to examine the maximum price that a customer is willing to pay for a product/service.^27-30 To our knowledge, no study has ever examined the WTP of any health professional student in regards to how much students would be willing to pay to pass an assessment. It may very well be that students would be willing to pay more (or not at all) for an educational technique that will reduce the likelihood of failing an assessment. Future research should attempt to examine WTP in student-pharmacists. In cost-utility analyses, a common threshold is $50,000 per quality-adjusted life-year gained to decide if an intervention is cost-effective.^29,30 A threshold for education is crucial to help determine whether any educational strategy is cost-effective. In the case of SP vs RP/CS, students would be required to be willing to pay about $101 more during the second professional year to improve their chances of passing the communication assessment the first time.

This study contradicts recently published research in medical education, comparing SP to RP.^15-18 The differences between the measures used in this study versus previous research may help explain the differences. Further, the students in this study participated in five cases over five weeks. Previous research has used SPs over the course of 2 days to 3 weeks (even though the number of cases in each study may have been more than the current study’s). In contrast to the Bosse and colleagues study, which showed that RP led to better student outcomes than SP, our intervention consisted of one topic over the course of 5 weeks compared to nine cases covering multiple aspects of communication over a period of 3 weeks.¹⁷ Another difference between our study and the Bosse study is the use of the modified Angoff method in the current study to establish a minimum level of competency via a cut-score. The Bosse and colleagues study did not state whether an attempt was made to establish a minimum competency score, which also may explain the differences in the conclusions between the studies.

There are other limitations to our study that also need to be considered. SP salaries may not be the same throughout the country, which could decrease the generalizability of our results. Salaries could be higher or lower, depending on the institution, which could make our estimates either too conservative or too generous. Future research should be conducted with similar methodologies to determine if using SPs in the curriculum is cost-effective. Instructor experience may also explain the findings of this study. However, we believe the likelihood of this is low because the same instructor was used for all three course administrations as well as the limited amount of active lecturing based on the flipped classroom model. Further, the pre-work (lecture students accessed prior to coming to class) for each course administration was the same; the only difference between the three course administrations was the inclusion of repeated SP use during the most recent administration. Our analysis is also limited by not including any overhead of providing SP activities to students, such as faculty/staff time, construction of special facilities to house SP activities, or extra technological requirements that using SPs require over and above RP activities in the flipped classroom. Our analysis assumed that the technological requirements were the same because the SP activities used the same technology as the lecture-capture for the flipped classroom pedagogy. The flipped classroom approach may also limit the generalizability of our findings because other schools may teach communication using other methods, such as traditional lecture. A sensitivity analysis was not conducted because the overall decision concerning the costs will always stay the same (ie, that SP use will always be more expensive than RP/CS because students are not paid by the hour).

CONCLUSION

To our knowledge, this is the first study to critically examine the cost-effectiveness of SPs in the classroom to assess student-pharmacist communication skills. On one hand, the results of this study found that incorporating SPs into the classroom is effective when learning communication skills, but without reference to a cost threshold, we cannot determine if incorporating SPs is cost-effective. Future economic evaluations should attempt to establish a cost threshold so that stakeholders can assess whether a teaching technique is cost-effective. Future pharmacy education research should investigate the cost-effectiveness of employing SPs to aid instructor teaching of other skills, such as patient assessment, nonprescription medication counseling, etc.