Simulated Patients vs. Standardized Patients in Objective Structured Clinical Examinations

Abstract

Objectives

To describe the use of patient-actors as educators in a senior-level pharmacy practice course, and to contrast the value and application of “standardized patient” and “simulated patient” educational methodologies.

Design

The objective structured clinical examination (OSCE) of the licensing examination were utilized during and at the end of the course along with external assessment to determine the impact of this educational methodology. Interviews with a randomly selected cohort of 14 students were undertaken 3 years after graduation and licensure to evaluate long-term impact of this course.

Assessment

Overall, students responded positively to the shift from “standardized” patients to “simulated” patients, recognizing their value in teaching clinical and pharmaceutical care skills. Concerns were expressed regarding objectivity in assessment and individual grading. Over 98% of students successfully passed the OSCE component of the licensing examination. Long-term follow-up suggests students valued this approach to education and that it provided them with a foundation for better understanding of the psychosocial needs of patients in practice.

Conclusions

Simulated-patient educators can play an important role in the pharmacy curriculum, and can complement practitioner-educators in providing students with a real-world context for understanding complex patient care needs.

INTRODUCTION

An evolution towards performance-based teaching and assessment is occurring in the health professions at both the preclinical and trainee levels.¹ Initially pioneered in medicine in the 1970s (in response to lack of availability of “real patients” for ward-based teaching, and the recognition that students required more opportunities to practice in a controlled environment prior to actually being released in a clinical setting),^2,3 the use of actors to portray patients has become commonplace in many health professions.⁴ Performance-based teaching and assessment are terms used to describe methods that allow educators to focus on clinical skills rather than simply clinical knowledge. While traditional methods (ranging from lectures to case-based multiple-choice tests) may be effective for teaching and testing knowledge, teaching of clinical skills may not be as amenable to these methods.⁵ Performance-based methods provide an opportunity to teach and test the amalgam of knowledge, skills, and attitudes that are integral to health professionals’ work.^2,3 Particularly in the context of communicative and cultural competencies, performance-based methods possess intrinsic advantages over traditional methods since they require both declarative knowledge and procedural knowledge.⁶

The role of such patient-actors in performance-based teaching and testing may vary considerably. In many contexts, they are utilized for assessment. The objective structured clinical examination (or OSCE) was pioneered in medicine in the late 1970s as a tool for ensuring standardization and psychometric stability in high-stakes assessments of clinical skills.² The need for such examinations has been previously described, but are based on the recognition that simply “knowing” is insufficient within a health profession and that “knowing how” must be taught and assessed on its own. Within the OSCE context, there has been a considerable evolution in scientific understanding of how to construct an examination that is defensible, reliable, valid, and generalizable. While much of this work has been done in medicine, pharmacy has increasingly contributed to this growing body of literature. For example, the College of Pharmacists of British Columbia have utilized performance-based assessment as part of its entry-to-practice assessment procedures since the 1980s.⁶ The Ontario College of Pharmacists has utilized an OSCE component in its quality assurance peer-review process for maintenance of competency assessment of pharmacists already licensed and in practice since 1996.^7,8 Since 2000, the Pharmacy Examining Board of Canada has delivered a national, multi-site 16-station OSCE for all candidates for licensure across Canada.⁹ The psychometric stability of this examination has been previously described, and reflects the important, unique dimensions of pharmacy practice where a large component of professional work requires communicative/interpersonal competencies, over and above simple procedural or technical competencies.¹⁰ Unlike many other professionals, pharmacists must rely almost entirely upon speaking, listening, reading, writing, and observing to gather data. Most pharmacists (particularly those in community practice) do not routinely perform physical assessment as a general part of their practice. The design of a psychometrically stable OSCE for this context differs considerably from one in which technical-procedural skills (such as taking a blood pressure, or performing auscultation or palpitation) can be tested and measured.¹⁰

While much of the literature in performance-based methods has focused on assessment, there are increasing reports of the value of clinical simulations as important teaching tools.¹¹ Within this context, the level of psychometric stability is generally not as high, and there is considerably more latitude for formative, rather than summative assessment. While clinical simulations were initially pioneered in medical education, pharmacy education has embraced this methodology and adapted it to meet the unique needs of pharmacy practice and education.

The hallmark of clinical simulations is the need to develop authentic situations that are reflective of real-world practices. In this way, the design of a clinical simulation for a teaching purpose may be considerably different than that for a testing purpose. For testing, there is clearly a need for “an answer”: an objective, specific, observable, measureable endpoint to the simulation for which the candidate is rewarded, and which can be reproduced in a consistent manner. Of interest, this need for psychometric reliability may, in fact, introduce constraints on the validity of a case, since in many cases one answer is not the only available option, and in other cases, the “least bad” answer is all that can be selected.

When clinical simulations are utilized for teaching purposes, the need for reliability may be less significant than the need for ecological validity, depending upon the specific goal or objective of the educational intervention. For example, in medicine, clinical simulations are frequently utilized as a teaching tool to assist students to learn how to break bad news to patients. In such simulations, a case may be constructed in which a physician must explain to a patient's partner that he or she has died despite all attempts at resuscitation. In such a case, there is no specific “content” expectation; rather interpersonal skills (such as empathy) and communication skills (such as appropriate use of verbal and non-verbal cues) are the legitimate focus of teaching, learning, and formative assessment.

Curious by its omission is the lack of literature in pharmacy that describes these different types of clinical simulations and the different role expectations for patient-actors. Indeed, within the academic health sciences literature, roles and responsibilities of patient-actors are rarely described explicitly; instead there is an expectation that these individuals (most of whom are highly trained and highly skilled) are simply to be used as “vehicles” for teaching of students by the health care professional who happens to be instructing within a performance-based teaching session or evaluating within a performance-based testing session.

There is lack of consistency or reliability between patient-actors and health care professionals in high-stakes assessment processes. For example, in pharmacy, a study to determine reliability of global assessment of communicative and interpersonal competencies of candidates in a high-stakes licensing OSCE concluded that there was insufficient reliability to allow patient-actors to “replace” pharmacist-examiners as assessors.¹⁰ However, this study only aimed to quantify reliability, not to explain why differences in assessment between patient-actors and pharmacists might exist. One intriguing hypothesis that has not been evaluated suggests that as laypeople rather than trained health care professionals, patient-actors may in fact be more qualified to assess communicative and cultural competency skills of entry-to-practice candidates than those already “tainted” by their professional training, and the lack of reliability between patient-actors and professionals actually reflects an important and substantive difference in expectations that truly ought to be captured and measured as part of the evaluation of clinical skills competencies. As such, these reliability problems may be important sources of data regarding differing expectations of health care professionals’ clinical performance in simulations, and perhaps ought to be more explicitly incorporated as part of a triangulation process of data gathering and evaluation.

As patient-actors become increasingly ubiquitous in health professions training, certification, continuous professional development, and recertification, there is a need to more clearly characterize and describe the varying roles and responsibilities of these educators. Throughout North America there has been growing reliance upon use of clinical simulations as an important tool in pharmacy education. The unique dimensions of pharmacy practice and education, where communicative and cultural competency are of particular importance in both teaching and testing, requires new pedagological approaches to support the ongoing evolution of the profession.

DESIGN

Pharmacy Practice Seminar (PHM 429F) is a senior-level clinical skills course; the general design of this course has been previously described.¹² The major objective of this course is to model, teach, and assess pharmaceutical care within a controlled (simulated) environment. An emphasis of this course is the integration of clinical knowledge and communication skills within a context of caring and professionalism, as simulated within a community pharmacy practice context. A particular challenge in designing this course was the lack of available community practice sites and practitioners who truly provided pharmaceutical care as taught through the Faculty. In order to provide students with authentic simulations in practice, practitioners (rather than academics) were recruited and trained as teaching assistants to lead small-group tutorial sessions. While all of these individuals were experienced pharmacists, few had any specific expertise in the area of psychosocial needs of patients beyond that gained through experience as practitioners.

This course was initially conceptualized around a model of family and community health care, utilizing a determinants of health approach, recognizing the importance of psychosocial wellbeing as integral to health status. This 10-week course was initially designed to explore the evolution of one extended family over a period of time. An elaborate, multigenerational family tree was designed consisting of over 40 different family members. Over the 10 weeks of the course, students (working in groups of 8) had an opportunity to provide pharmaceutical care to members of this family, portrayed by trained patient-actors. A rotational system was established in order to ensure each student had exposure to at least 8 different patient-actors over the 10-week course. A unique feature of this course was the availability of follow-up and return visits; for example, a patient in week 2 who may be having difficulties with his inhaler and control of his asthma may return in week 7 because his mother has just been admitted to a nursing home and he is now the agent for her medications. Assessment in the course was mainly formative; however, summative evaluations were provided to students twice during the course itself and in a final multi-station OSCE conducted at the end of the course. Numerical or alpha-numerical grades for this course were eliminated, and replaced by a global assessment scale, resulting in 1 of 3 possible outcomes: “needs improvement” (or fail), “meets expectations” (or pass), and “exceeds expectation” (or honors).

Pharmacy Practice Seminar was unique insofar as the entire course was based on clinical simulations using patient-actors. Within pharmacy, there had been no previously described course that relied entirely upon performance-based teaching and assessment. While many courses involved patient actors to a greater or lesser degree, this course integrated patient actors into curriculum development and assessment from the outset.

Cases for this course were developed and piloted collaboratively with pharmacist-case writers and patient-actors. Importantly, the pharmacist case writers built cases around their actual experience in practice, but selectively highlighted important features or events as “teachable moments.” Patient-actors reviewed each case and provided history, context, and depth to the abbreviated patient histories that had been provided. Collaboratively, pharmacists and actors built contextually rich cases in which an individual's health-related needs were only a part (frequently a small part) of the overall needs of the patient.

This approach was consistent with the notion of determinants of health, a model of health behavior and care that suggests that “health” is not simply defined in terms of biochemistry and pathophysiology. Rather, in this approach, “health” includes employment, relationships, housing, and family networks as well. Recognizing the complementary strengths and expertise of pharmacists and patient-actors, a collaborative model of teaching and assessment was developed, one that leveraged the pharmacist-assessors expertise in clinical knowledge and skills, and the patient-actors expertise in providing contextually situated feedback as “the patient” who is receiving pharmaceutical care.

Importantly, this feedback was not simply based on formulaic prompts related to verbal and non verbal skills. Instead, actors were encouraged to develop unique personas for each case based upon their personal experiences and beliefs. While cases were developed that depicted a plot, a history, and an outcome, actors were encouraged to be creative in their interpretation of these cases in the classroom. As a result, there was little attempt at “standardization” of portrayals and instead great latitude to allow each actor to truly “simulate” (from his or her own unique personal perspective) this particular patient's life and situation.

Consequently, there was a conscious decision to downplay reliability (ie, consistency and standardization of interpretation) in an effort to enhance validity (ie, the authentic, nonstandard “messiness” of real-world practice). Since students were divided into small groups, all of which received the same case, the net result was divergent portrayals, based on each actor's own interpretation of the case, the situation, and the psyche of the patient.

Following each portrayal, students and pharmacist-assessors engaged in an open discussion focused on “teachable moments” inspired by the portrayal. Each small group evolved different teachable moments based on the unique dynamics of the group, the portrayal by the patient, and the experience of the pharmacist-facilitator.

Initially this design provoked anxiety on the part of both students and pharmacist-assessors, most of whom were accustomed to and familiar with standardization and reliability as cardinal virtues of teaching and assessment at the university level. For this course, cases and clinical simulations were designed to prompt reflection, analysis, and discussion regarding “care” in a pharmaceutical care context, with the understanding that care is patient-specific, nonstandardized, and highly variable.

In order to control some sources of variation and to expose students to a variety of teaching styles, actors and pharmacist-assessors were rotated on a periodic basis, allowing students to learn in a variety of different ways. Each week, at the conclusion of the session, each student and pharmacist-assessor completed an evaluation form that was utilized for course analysis and to provide quality control for the course.

To evaluate the success of this approach, a 3-part assessment system was developed. Written, anonymous, in-course and final course evaluations were completed by students and pharmacist-assessors. Performance on the entry-to-practice examination (licensing examinations), which include a 16-station OSCE; and a 3-year follow-up interview with a random sample of students to determine their perception of the value of the course on their professional practice and development. For this evaluation and assessment, the graduating class of 2001 was selected.

A total of 114 students were enrolled in the course during the school year; 112 of these students passed and subsequently graduated (the remaining students completed additional examinations/supplemental requirements and eventually passed). All graduating students completed the licensing examination in May of 2001. A random sample of 20 students was contacted in early 2004 (3 years following completion of the course) to participate in a follow-up interview. Of these 20, 14 agreed to participate in telephone-based interviews. Interviews were not audiotaped, but extensive field notes were compiled using a semi-structured interview guide. The focus of these interviews was to determine how students valued the course and in what ways it had (or had not) contributed to their professional development and practice. In an effort to control for social-response bias, the interviewer was a research associate not involved in the course in any way, and not familiar with or to the students.

ASSESSMENT

The graphs in Figures 1, 2, and 3 depict student satisfaction with the course over the 10-week period as defined by their response to 3 specific prompts: “The actor was effective in portraying a patient with [specific medical condition],” “The simulated patient provided me with feedback that will allow me to improve my pharmaceutical care skills,” and “As a result of this week's lab, I am better able to provide pharmaceutical care to patients with [specific medical condition].” Students were asked to respond to these statements along a 4-point scale ranging from “strongly agree” to “strongly disagree,” with no “neutral” option available. Variations in weeks 4 and 6 depicted in the graph in Figure 2 may be partially explained by the relatively more complex therapeutics content associated with those particular sessions (HIV in week 4 and ovarian cancer in week 6).

Figure 1.

Pharmacy students’ responses to the statement “The Simulated Patient was effective in portraying a patient with [specific medical condition].”

Figure 2.

Pharmacy students’ responses to the statement “The Simulated Patient provided with feedback that will allow me to improve my pharmaceutical care skills”

Figure 3.

Pharmacy students’ response to the statement, “As a result of this week's lab, I am better able to provide pharmaceutical care to patients with [specific medical condition].”

Table 1 depicts end-of-course assessment and students’ responses to key questions. While overall, student assessment of the course was positive, the majority favored a more traditional grading system and a more standardized approach to the course. In particular, students expressed concern that upon graduating they would be required to complete an OSCE utilizing “standardized” rather than “simulated” patients, and that this course did not provide sufficient experience in this model of teaching and assessment to assist them in meeting examination requirements.

Table 1.

End-of-Course Evaluations of Pharmacy Students, N = 99

*99 (86.8%) of 114 completed the survey instrument

All graduates of the program attempted the Pharmacy Examining Board of Canada's licensing examination, which included a 16-station objective structured clinical examination. Success rates were extremely high; over 98% (110/112) passed on the first attempt. The 2 remaining students passed on the second attempt.

Interviews with some students conducted after the course was completed indicated strong support for the teaching model utilized in this course, but some concern regarding the assessment system. Students interviewed recognized and supported the value of nonstandardized approaches to patient interviewing and care, acknowledging that this corresponded to the reality of professional practice. However, most students were concerned that a grade eventually had to be assigned and that this grade was based on performance in nonstandardized situations, which in turn introduced a potential issue of fairness. Despite these misgivings, all students interviewed acknowledged that their concerns over the assessment system were more than offset by the quality of the educational experience.

DISCUSSION

This nontraditional course introduced some major challenges with respect to assessment. Students and pharmacist-assessors generally recognized the value and importance of understanding “care,” and that “care” is by definition nonstandard. In a standardized educational program, where students are accustomed to receiving the same educational opportunities and experience and instructors are immersed in a culture that legitimately values reliability, consistency, and reproducibility, the notion of a nonstandardized course such as this initially provoked some discomfort. Based on data presented previously, students did not appear to experience any particular adverse effects from the experience of “simulated” vs. “standardized” patients.

All students in this course had prior experience with standardized patients utilized in an OSCE-type assessment in the third year, but no previous interaction with simulated patients as described in this course. As depicted previously, there was widespread agreement from students that they enjoyed and benefited from this learning environment. However, not captured by the postcourse survey was the frustration experienced by many students regarding a perceived lack of “fairness” in grading. While students recognized the value of this approach, there was considerable concern that there was so much variability between cases and patient-actors that it was simply not possible to compare the experiences of one group with those of another. While, from a learning perspective, this was acceptable, students believed it was problematic within an evaluation perspective. In short, they complained about “hard” patient-actors and “easy” patient-actors, and were concerned that this might adversely affect their academic standing.

The 3-year interviews revealed strong support for this course, rooted in the understanding that real-world practice is highly nonstandardized, and that the learning opportunities afforded by simulated (not standardized) patients are so much more significant and confer long-standing benefit for practitioners. Students commented on their initial skepticism of the “theatrical” nature of the course, and recalled feeling somewhat discomforted by the large role played by the patient-actors, and uncertain what that would mean for their pharmacy education. However, upon becoming practitioners, they recognized the value of this nonstandardized approach to teaching and learning.

Particularly resonant were students’ comments that the “types” of patients encountered in the course were precisely the “types” of patients they encountered in their early years of practice. Having had the opportunity to experiment with different approaches to pharmaceutical care and patient counseling, and to truly empathize with patients’ unique needs from a determinants of health perspective provided students with insights into practice and their own strengths and weaknesses that they could not have acquired otherwise.

The pedagogical model for this course – the pairing of pharmacist-assessors with patient-actors as co-teachers and co-evaluators of students – is nontraditional but effective. While patient-actors are traditionally limited in their scope and role in the classroom setting, even in formative feedback settings, the explicit acknowledgement of their expertise in this course was a distinct and well-received innovation.

While there is clearly a need for “standardized patients,” the experience of this course suggests further work is necessary to evaluate and characterize a specific, distinct role for “simulated” patients. This is particularly important in the pharmacy context where academically ideal models of pharmaceutical care are frequently not readily available in the community. Utilizing the expertise of patient-actors to contribute meaningfully to the teaching and assessing of students provides an alternative model for clinical skills development and evaluation.

There are considerable challenges associated with mounting a program such as this. First it is essential to have a well-developed pool of trained patient-actors and an administrative infrastructure to support performance-based teaching and assessment. Institutions affiliated with schools of medicine are likely to have such an infrastructure already in place and may be able to tap directly into this expertise; those without such an infrastructure must consider ways of developing one on their own, and this is admittedly an expensive and daunting proposition that may be beyond the range of some schools which may be located away from theatre arts programs or vibrant performing arts communities. Actors who can be trained to portray patients and provide feedback to students are an invaluable resource for this approach.

SUMMARY

This paper describes design, implementation, and assessment of a nontraditional skills-based course in pharmacy practice. Of particular importance is the way in which nonstandardized portrayals were used in the context of a university-based course. As described, students experience initial apprehension at the notion of being graded on the basis of nonstandardized events; however, based on course results and post-graduation licensing examination results, there appears to have been no adverse effect on students as a result of this approach. Importantly students themselves supported the nonstandardized learning environment in facilitating professional skills development and identified this experience as important in assisting them in becoming competent, caring practitioners. A key finding from this experience has been the value of using patient-actors as collaborators and educators, rather than simply vehicles for assessment. The feedback and assessment provided by simulated patient-educators complements the work of pharmacist-assessors and educators, and provides students with a rich context for developing their patient care skills.