Assessing the Believability of Standardized Patients Trained to Portray Communication Disorders

Abstract

Purpose

The purpose of this study was to evaluate the believability of standardized patients portraying individuals with communication disorders as part of a larger study in which standardized patients help train medical and allied health students about communication disorders.

Method

Two women portrayed persons with aphasia, and 2 men depicted persons with dysarthria associated with Parkinson's disease. Two stakeholder groups rated believability. Speech-language pathologists rated believability of videos online. Persons with aphasia rated aphasia videos during in-person sessions with the researchers.

Results

Targeted believability was 80 or higher (0–100 scale; 0 = not at all believable, 100 = very believable). For speech-language pathologist raters, average ratings met the target for the portrayals of the aphasia characteristics of word-finding problems, agrammaticism, nonverbal communication, and overall portrayal but not for auditory comprehension problems. Targets for the portrayals were met for the dysarthria characteristics of reduced speech movements, reduced loudness, reduced intonation, flat affect, and overall portrayal but not for speech rate. Ratings for different standardized patients portraying the same case were not significantly different from each other on most characteristics. Ratings from persons with aphasia were highly variable.

Conclusion

Standardized patients who do not have communication disorders can portray disorder characteristics in a believable manner.

Simulation is becoming increasingly prevalent in training students across many health care disciplines because it allows them to practice clinical communication, assessment, and intervention skills in safe and controlled settings (Bradley, 2006). Forms of simulation include standardized patients who are trained to portray a clinical case, mannequins that can be used to simulate discreet tasks or integrated body systems, and virtual patients in computerized simulations. Many authors have discussed advantages of simulation for student learning (Bradley, 2006; Hill, Davidson, & Theodoros, 2010; MacBean, Theodoros, Davidson, & Hill, 2013; Zraick, 2012; Zraick, Allen, & Johnson, 2003). Simulation provides a safe environment in which students can practice before using their new clinical skills with real patients. Thus, students can improve their skills, receive constructive feedback and guidance, and develop confidence before they apply their skills with actual patients. Use of simulation can improve the quality of clinical education by allowing instructors to exert more control over training. For example, instructors can control the type or severity of the clinical case provided to ensure that students are exposed to the range of clinical scenarios needed for comprehensive training. Instructors can also ensure consistent training and evaluation across students, as simulations allow for all students to be exposed to the same training scenarios. In addition, simulation may also improve safety and quality of care for patients by ensuring that students who work with them have had practice with clinical skills prior to the actual patient encounter. This can be important not only for the physical safety of clients but also for their psychological comfort in clinical encounters.

Consistent with the growth of simulation as a training method in other health care disciplines, examples of simulation are becoming increasingly prevalent in the field of speech-language pathology as well (Benadom & Potter, 2011; Hill et al., 2010; MacBean et al., 2013; Miles, Friary, Jackson, Sekula, & Braakhuis, 2016; Quail, Brundage, Spitalnick, Allen, & Beilby, 2016; Ward et al., 2014; Zraick, 2012). Recognition of the important role that simulation can have in training students is exemplified by the 2016 inclusion of simulation as an acceptable method to accrue a portion of clinical hours counting toward certification by the American Speech-Language-Hearing Association (ASHA; http://www.asha.org/Certification/2014-Speech-Language-Pathology-Certification-Standards/). With this growth in simulation, the challenges inherent with these processes must be addressed as well, including cost for time, equipment, and personnel, the need for strong pedagogy regarding when and how to include simulation for maximal learning benefit, and the design of authentic, realistic simulations (Hill et al., 2010; MacBean et al., 2013; Zraick, 2012). This is also an important time for exploring and comparing the advantages and challenges of different types of simulation (e.g., standardized patients versus virtual patients), such as the feasibility, realism, and impact on student learning across the different methods (Quail et al., 2016). The focus of this article is evaluating the potential for providing realistic, believable simulations with standardized patients who are portraying individuals with communication disorders.

Standardized patients are individuals who typically do not have the health care condition that they are portraying (Hill et al., 2010; Zraick, 2012). As part of their training, they have to learn the content of their script regarding what information to provide to students during their portrayals. In addition to the content of the script, they have to learn to portray any physical or affective features of the case that are relevant. Various terms have been used in the health care literature to refer to the quality of standardized patient portrayals, including realism , authenticity, accuracy, quality assurance, and fidelity (Bouter, van Weel-Baumgarten, & Bolhuis, 2013; Furman, 2008; Howley, 2013; Schlegel, Bonvin, Rethans, & van der Vleuten, 2015; Tamblyn, Klass, Schnabl, & Kopelow, 1990; Wind, van Dalen, Muijtjens, & Rethans, 2004). For the most part, these terms appear to be used interchangeably to reflect that the standardized patient adheres to the script as trained, provides information from the case accurately, does not inappropriately add information to or withhold information from the case, and portrays physical and/or affective characteristics as trained. Threats to the accuracy or fidelity of standardized patient portrayals can come from several sources (Tamblyn et al., 1990). One set of factors pertains to the selection of the standardized patients. Standardized patients should represent the age, gender, and other key demographic variables to be portrayed in the case. Their own past experiences either working as a standardized patient or having experience with the health care condition being portrayed may also influence their ability to realistically portray the case. Also included in this category is the number and nature of symptoms they are asked to portray. A second set of variables that might threaten the authenticity of case portrayal is related to the training procedures, such as the amount of training provided and who does the training. Also, logistical issues related to the actual encounters with students, such as the risk of standardized patient fatigue over long or repeated portrayals, might affect accuracy.

Although various methods have been used for training standardized patients, a common thread through these discussions is that standardized patients should be evaluated for accuracy and reliability of the portrayal before working with students (Adamo, 2003; Furman, 2008; Howley, 2013; Ker et al., 2005). In most examples drawn from the general health care literature in this article, trainers appear to develop case-specific checklists for evaluating standardized patients that detail the case content, as well as the specific physical and/or affective behaviors that the standardized patients are trained to portray. Accuracy of portrayals has generally been documented to be strong (85% accuracy or higher in studies reporting the percentage accuracy) for standardized patients portraying a range of cases with examples, including chest discomfort (Vu, Steward, & Marcy, 1987), diabetes with hypoglycemia (Schlegel et al., 2015), depression (Shirazi et al., 2011), genetic counseling needs (Erby, Roter, & Biesecker, 2011), physical therapy needs (Ladyshewsky, Baker, Jones, & Nelson, 2000), and additional various cases that might present to a general medical practice (Beaulieu et al., 2003; Errichetti & Boulet, 2006; Tamblyn et al., 1990).

Other researchers have developed more generic checklists and rating scales that could be used to evaluate the authenticity of standardized patient portrayals regardless of the symptoms simulated (Bouter et al., 2013; Howley, 2013; Wind et al., 2004). For example, the Maastricht Assessment of Simulated Patients (Wind et al., 2004, p. 42) asks evaluators to rate characteristics of the standardized patient, including that the standardized patient “appears authentic,” “might be a real patient,” and “answers questions in a natural manner.” The Nijmegen Evaluation of the Simulated Patient (Bouter et al., 2013) contains similar items, such as rating how well the standardized patient “resembled a real patient,” “played the role well,” and “reacted naturally during the consultation.” Use of these general scales would have the advantage of allowing comparison across standardized patients representing different cases, although raters would need additional information about the characteristics targeted for portrayal to assess what was realistic and natural for each case.

Although communication is a construct that is often rated in the standardized patient evaluations cited earlier, communication in the general health care literature typically refers to accuracy in stating the case history information, delivering the correct opening line or other aspects of the script, and not omitting case information (Errichetti & Boulet, 2006). Ratings of specific speech mannerisms are not as common, although one example was found for a case in which affect was to be conveyed and was evaluated through verbal mannerisms, such as low speech volume and clipped speech (Schlegel et al., 2015). None of the cases cited previously from the broader health care literature documenting accuracy of symptom portrayal by standardized patients involved the characterization of a patient with a communication disorder.

Evidence is growing for the use of standardized patients in the field of speech-language pathology, and this raises important questions regarding how standardized patients will portray communication disorders (Bressmann & Eriks-Brophy, 2012; Hill, Davidson, & Theodoros, 2013, 2015; Hill et al., 2010; Syder, 1996; Zraick, 2012; Zraick et al., 2003). Standardized patients have most often been used to assist with instruction and assessment of general clinical interviewing and communication skills, but also in the development of diagnostic skills specific to communication disorders. In some of these examples reported in the literature, the standardized patients are not actually portraying communication disorders but instead are depicting a family member of someone with a communication disorder or someone with an asymptomatic disorder, such as a very mild voice problem (Bressmann & Eriks-Brophy, 2012; Hill et al., 2013, 2015). In those cases, the standardized patients do not portray the symptoms of communication impairment. In other examples, individuals who do not have communication disorders have been trained to portray communication disorder symptoms, for example, aphasia (Zraick et al., 2003), as well as voice and fluency disorders (Syder, 1996). In these cases, the standardized patient must accurately portray the fluency, word finding, comprehension, or other impairments appropriate to the disorder.

A logical assumption might be that for students to buy in to the value and importance of the simulation experience, to engage with the experience authentically, and to benefit maximally from the experience, they need to believe that the standardized patient is providing a realistic exemplar of the communication disorder. To date, little systematic research has been published about evaluating the accuracy or authenticity of standardized patients in the field of communication disorders. Hill et al. (2013) conducted a study to evaluate the accuracy and quality of standardized patient portrayals in the field of communication disorders, but in that study, the standardized patients were depicting family members of children with communication disorders; thus, the standardized patients did not have to characterize any communication disorder symptoms themselves. There are gaps in our understanding of the extent and nature of training needed to prepare standardized patients to portray communication disorders. Furthermore, the discipline lacks systematic evidence as to whether standardized patients who do not have communication disorders can be trained to characterize the symptoms of communication disorders in a believable manner, meaning that stakeholders would regard the standardized patient portrayal as authentic.

The purpose of this study was to evaluate the believability of standardized patients trained to portray individuals with communication disorders, specifically aphasia due to stroke and dysarthria associated with Parkinson's disease (PD). The standardized patients in this project were trained as part of a larger program of research and education, focusing on training medical and allied health students how to communicate effectively with patients with communication disorders. Through that training program, medical and allied health students are provided with information about communication disorders, as well as practical strategies for communicating, so patients with communication disorders can take part in shared decision making in health care encounters. Preliminary information on that training program is available elsewhere (Burns, Baylor, Morris, McNalley, & Yorkston, 2012; Yorkston, Baylor, Burns, Morris, & McNalley, 2015), and outcomes of the training on student skills will be reported in future articles. Standardized patients were incorporated into the research program to assess the communication skills of medical and allied health students in simulated encounters before and after the training. This article will report on the component of the research program that involved training the standardized patients and evaluating their authenticity or believability for portraying individuals with communication disorders. The primary research question was whether standardized patients could realistically portray symptoms of the intended communication disorder from the perspective of speech-language pathologists (SLPs) who would be the professional experts in evaluating communication disorder characteristics. To seek perspective from additional stakeholders, a secondary study sought input on the believability of the standardized patients portraying a person with aphasia from the perspective of persons with aphasia (PWA). This secondary study was not conducted for the dysarthria standardized patient portrayal out of ethical concern for exposing persons with PD to examples of symptoms that might be much more severe than those they were currently experiencing but might experience in the future.

Study 1: Perspectives of SLPs

This section will report on the methods and results for the primary study to evaluate the quality of standardized patient portrayals of patients with communication disorders from the perspective of SLPs. A secondary study seeking feedback from PWA will be reported in the following section. All research methods were approved by the institutional review board at the University of Washington. All research participants were paid for their participation in the study.

Method

Standardized Patient Preparation

This section will describe the training of the standardized patients, as well as data collection with SLPs to evaluate the quality of their portrayals. The standardized patients were not considered research subjects but instead were regarded as members of the research team. They were paid employees of the University of Washington in the School of Medicine standardized patient program. Consistent with guidelines for reporting on inclusion of standardized patients in research, we will provide information regarding the physical characteristics, training, and responsibilities of the standardized patients (Howley et al., 2008).

Characteristics

Four standardized patients were hired for this program. The targeted age range of the standardized patients was 50–70 years of age to be consistent with the typical age range of people with the disorders they were portraying. To reduce possible influences of gender on communication styles, standardized patients of the same gender were selected to portray the same cases. Men were chosen to portray PD because PD is more common in men (they will be referred to as DYS1 and DYS2 in this article). Two women (referred to as APH1 and APH2) were recruited to portray aphasia after stroke. The standardized patients were required to have no self-reported neurologic disorders that would affect speech, language, or cognition, and no physical impairment that would be inconsistent with the characteristics of the disorders portrayed. No restrictions were placed on race or ethnicity for the standardized patients, but all were White.

Training

All of the standardized patients hired for this project had multiple years of prior experience working as standardized patients, although they had never portrayed communication disorders. Because of this prior experience, they did not require training in the basic elements of serving as standardized patients, but only required preparation for the specific communication disorder roles for this project. All standardized patients participated in approximately 6 hr of training to portray the cases for this study. The women representing the aphasia case were trained together, as were the men depicting the dysarthria case in an effort to improve consistency of the portrayals across the two standardized patients characterizing each case. The standardized patients were provided with detailed written documents consisting of a blueprint for the case. The blueprint provided information about the learning goals for the students and the targeted behaviors for the students to demonstrate so that the standardized patients were fully aware of the goals and purpose of the overall program. The blueprint contained information about the case being portrayed, including medical history, social history, and other relevant background information. In addition, the blueprint indicated detailed information about the communication disorder symptoms the standardized patients were to portray, as well as instructions as to how to respond to different communication strategies that the students might use with them. An overview of each of the cases’ targeted communication disorder symptoms and severity, as well as additional case history information, is provided in Appendix A.

In the training sessions, the standardized patients viewed videos of actual patients with their respective communication disorders. The SLPs on the research team provided instruction in the types of communication disorder symptoms that they should portray and pointed out examples of these in the videos. The standardized patients practiced with the SLP research team to portray the communication disorder symptoms and to practice how to respond to different strategies that students might use. For example, when a student used a strategy that should help with communication, the standardized patient should communicate more easily, whereas student use of a nonhelpful strategy would result in the standardized patient becoming confused or frustrated. These training sessions were video-recorded, with the videos viewed by the SLPs and standardized patients together, to provide feedback and further fine-tune the portrayals until the SLPs felt that the symptoms were realistic and the standardized patients expressed feeling comfortable in the roles. At that time, professionally recorded videos were created of each standardized patient. In these videos, a medical interview was portrayed with one of the research team members taking the role of the physician. The scenario portrayed the first few minutes of a typical outpatient medical interview in which the physician meets the patient for the first time, and the patient conveys his or her primary concerns or reason for the visit to the doctor. This was the general situation used in the medical student training program. The two standardized patients portraying each case used the same message so that the content of the interviews was the same within each diagnosis. However, although the standardized patients portraying the same cases were given the outline of a script for the message, they were not required to use the exact words. These videos were edited to the first 5 min of the interview for use in the data collection for this study.

SLP Raters

The perspectives of SLPs regarding the believability of the standardized patients were sought to represent the views of professional experts in the diagnosis and treatment of communication disorders. To be eligible for the study, SLPs needed to be certified by ASHA and to have at least 2 years of clinical experience working with adults with communication disorders. Because this study was conducted online, the SLPs also had to have access to a computer and the Internet. SLP participants were recruited by placing announcements on LISTSERVs affiliated with ASHA and the Washington State Speech-Language-Hearing Association. Interested SLPs contacted the researchers for enrollment.

Thirty-four SLPs completed the study. Mean age was 41.4 years (SD 10.2; range 27–66). Mean number of years of experience was 14.4 years (SD 8.5; range 3–32). The majority of SLPs were female (88.2%, with one SLP not reporting gender). The geographic regions of the United States as listed by the U.S. Census were used to examine the geographic distribution of the SLPs. Seventeen (50%) of the SLPs were from the West; five (14.7%) from the Midwest; five (14.7%) from the South; four (11.8%) from the Northeast; two (5.9%) were from outside of the United States; and one SLP did not report location.

Data Collection Procedures

An online questionnaire was created by using a website available through the University of Washington. The survey began with asking SLPs to provide demographic information, including age, gender, state of residence, number of years working as an SLP, and settings in which the SLP had worked. Then, the website presented videos of each of the four standardized patients. After each video, a set of questions was presented to assess the quality of the portrayals by the standardized patients. In the first question, the SLPs were presented with a list of 14 communication disorder characteristics (see Figure 1 for characteristics). Some characteristics were intended to be included in the portrayals, and others were not. The SLPs were asked to indicate those characteristics that they saw in the standardized patient's portrayal. The purpose of this question was to assess if SLPs identified presence of intended characteristics and absence of characteristics not intended for the portrayals.

Figure 1.

These graphs show the data from the speech-language pathologist (SLP; n = 34) ratings regarding the presence versus absence of communication disorder characteristics for the (a) aphasia standardized patients and (b) dysarthria standardized patients. Different communication disorder characteristics are presented along the y-axis, while the percentage of SLPs who indicated they detected that characteristic in each standardized patient is reported on the x-axis. APH1 = aphasia standardized patient 1; APH2 = aphasia standardized patient 2; DYS2 = dysarthria standardized patient 2; DYS1 = dysarthria standardized patient 1.

Following this task, the SLPs were presented with a shorter list of characteristics that were intended to be portrayed by the standardized patients. The SLPs were informed that the standardized patients were trained in these specific characteristics and were asked to rate how believable the standardized patient was in portraying each characteristic. The term believable was chosen for this study for several reasons. As mentioned earlier, different terms have been used in the literature to refer to the quality of standardized patient portrayals. In this study, terms such as accuracy and fidelity did not appear appropriate because these suggest that the raters know the details of the case, such as the intended message, which was not the intent in this study. The purpose of this study was to assess whether the portrayals of the communication disorder characteristics were realistic or authentic. Either of these terms would have been appropriate for use with the SLPs. However, one of the broader future purposes of this research is to assess and compare the quality of the standardized patient portrayals from the perspective of various stakeholders—not only the SLPs as professional experts but also students who work with the standardized patients to learn communication skills in the training programs, people who live with the communication disorders (and their families) who are, thus, represented by the standardized patients, and possibly others. The research team felt that these latter categories of stakeholders of students, patients, and families might not feel comfortable rating accuracy or realism of the characteristics because they would have less knowledge about and experience with the range and nature of disorder characteristics. The research team wanted to choose a different word that would allow these stakeholders to indicate if they found the standardized patients to be convincing or plausible, even if they did not feel that they could make an expert judgment about accuracy. Asking stakeholders how believable the standardized patients were was felt to convey the intended construct, while perhaps being more comfortable linguistically than the terms convincing, plausible, or authentic, particularly for PWA.

A summary of the questions presented to the SLPs to rate believability can be found in Appendix B. For example, for word-finding problems with aphasia, the website presented the question

Persons with non-fluent aphasia often experience word-finding impairments. They may not be able to think of a word they want to say, or they may substitute a word for the intended word. Expression is a struggle and is effortful. How believable is this standardized patient portraying word-finding impairments?

The SLPs were then asked to rate the believability on a 0–10 scale, with 0 = not at all believable and 10 = very believable (scores were multiplied by 10 to be on a 0–100 scale). For the aphasia standardized patients, SLPs were asked to rate the believability of problems with auditory comprehension, word finding, and grammatical formulation, as well as the believability of use of gestures. For the dysarthria case, the SLPs were asked to rate characteristics of reduced speech movements, reduced loudness, reduced intonation, short bursts of rushed speech, and flat affect. SLPs were also asked to rate the overall portrayal of each standardized patient and, as an additional option, to provide any written comments in text boxes provided on the website about the case portrayal or improvements that could be made in the characterization. SLPs could view the standardized patient videos multiple times if they wished, as they completed the ratings.

Data Analysis

Data were available for download in an Excel file (Microsoft Corporation, Redmond, WA). Descriptive data were calculated with SPSS (Version 18.0; Chicago, 2008). For the first rating task, which consisted of the SLPs endorsing presence or absence of each of 14 communication disorder characteristics, percentages were calculated for the number of SLPs who endorsed each characteristic. For the second rating task, which consisted of rating the believability of the smaller set of targeted characteristics for each disorder, descriptive data were generated, including the mean and standard deviation of the believability ratings for each communication disorder characteristic for each standardized patient. The data for each standardized patient were analyzed separately to allow for comparisons across standardized patients portraying the same case. To assess if there were significant differences in the portrayals across the two standardized patients depicting each disorder, we conducted paired t tests for each of the targeted communication behaviors and for the overall ratings. Despite the multiple t tests, significance level was kept at p = .05 because in this study, the goal was for nonsignificant results. Although nonsignificant differences are not the typically desired outcome of a t test, for the purposes of this study, the goal was to show that the two standardized patients did not differ significantly from each other. Adjusting the p value for multiple t tests would have made it easier to achieve the desired outcome for this study of showing nonsignificant differences; but keeping the significance level at p = .05 was a more conservative level for demonstrating nonsignificant differences and therefore more convincing to show that the standardized patients portrayed the symptoms in the same manner.

For this study, the benchmark of 80/100 or higher was chosen as the target to represent a high level of believability. Several studies from the general health care literature (Beaulieu et al., 2003; Erby et al., 2011; Errichetti & Boulet, 2006; Ladyshewsky et al., 2000; Schlegel et al., 2015; Shirazi et al., 2011; Tamblyn et al., 1990; Vu et al., 1987) suggest that accuracy of adherence to the script and portrayal of other physical and/or affective characteristics can be achieved at levels above 80%. SLPs had the option of providing written feedback about the standardized patients. Written comments were reviewed, and quotes representing the range of opinions expressed will be reported with a particular emphasis on changes recommended to improve believability.

Results

Checklist of Presence Versus Absence of 14 Characteristics

In their first task, the SLPs were presented with a list of 14 communication disorder characteristics, some intended to be in the portrayals and some not, and asked to indicate any characteristics they observed in each portrayal. Figure 1a presents the percentage of SLP raters who identified each feature for each standardized patient portraying a person with aphasia, and Figure 1b presents the same data for the characterizations of someone with dysarthria. Note that the profiles of features were different for the two communication disorders. As expected, problems with word finding, auditory comprehension, and agrammaticism were predominant in the aphasia portrayals, and problems with articulation, flat affect, and speech loudness were predominant in the dysarthria portrayals.

Believability of Targeted Characteristics

In their second task, the SLPs were asked to rate the believability of the characteristics targeted for portrayal in each case. Figure 2 presents the mean ratings and standard deviations across SLP raters for each targeted characteristic for the standardized patients portraying someone with aphasia (Figure 2a) and someone with dysarthria (Figure 2b). For the aphasia standardized patients, ratings on all characteristics met or exceeded the targeted believability rating of 80/100, except for the characteristic of showing problems with auditory comprehension. There were no significant differences between the two aphasia standardized patients on any of the disorder characteristics or on the overall rating. For the dysarthria standardized patients, all characteristics reached the targeted believability rating of 80/100, except for speech rate for showing short bursts of rushed speech. There were no significant differences between the two dysarthria standardized patients for any characteristic, except for the speech rate characteristic (showing short bursts of rushed speech) for which p = .002. There were no significant differences in the SLP ratings of overall believability for the portrayals of someone with dysarthria.

Figure 2.

These graphs present the mean and standard deviation of the SLP ratings for each communication characteristics for the (a) aphasia standardized and (b) dysarthria standardized patients. On the website, the speech-language pathologist raters saw a numeric scale from 0–10, with 10 = very believable. The data were converted to a 0–100 scale (by multiplying by 10) for analysis to allow comparisons with the data from the aphasia participants in this article and data from the students involved in the communication skills training (to be reported in a future article) who all did the ratings by using a 100-mm visual analog scale on paper. The rating scale is from 0–100, with 100 = very believable. Asterisks indicate characteristics for which the believability differed significantly between the two standardized patients (p < .05).

APH1 = aphasia standardized patient 1; APH2 = aphasia standardized patient 2; DYS2 = dysarthria standardized patient 2; DYS1 = dysarthria standardized patient 1.

SLP Comments

SLPs were invited to provide comments regarding the standardized patient portrayals if they wished. A number of the comments were positive. For example, with regard to the aphasia standardized patients, one SLP commented, “Overall she portrayed the frustration and hesitation of word-finding difficulty very well” (SLP 13 regarding APH1). Another commented, “The opposite head nod from the verbal response is a classic maneuver, great to include” (SLP 24 regarding APH2). The most consistent suggestions for improvement for the aphasia standardized patients were in the area of auditory comprehension. Auditory comprehension impairments were not highly evident in the aphasia interactions. One SLP wrote, “Comprehension impairments were not very apparent” (SLP 13 regarding APH1). However, some SLPs did indicate that some aspects of auditory comprehension problems were present, “Most evident when the doctor asks specific questions. I really liked that the ‘doctor’ asked 2 questions at once … she [standardized patient] is clearly not fully comprehending the complex question” (SLP 6 regarding APH2).

For the dysarthria portrayals, SLPs again had some positive comments such as, “Many strengths in his overall presentation, including right hand tremor, hypophonic and monotonic presentation, delayed initiation, and masked face” (SLP 19 regarding DYS1). Areas of improvement for the dysarthria standardized patients included improving portrayal of speech rate, “Short rushes of speech were not the most prominent characteristic” (SLP 32 regarding DYS1). SLPs also reported that some of the characteristics, such as intonation and rate, were too difficult to assess because the speech volume was so low, “His volume is too low to detect if he is monotone” (SLP 14 regarding DYS1).

In general, the feedback from the SLPs suggested that they found the standardized patient portrayals for both disorders to be believable: “Overall, I thought these patients were quite believable in their portrayal of the communication deficits targeted” (SLP 34). However, two issues seemed to detract most from believability. First, because they found some characteristics to be more believable than others, the package of symptoms for each standardized patient did not entirely meld together for an overall convincing portrayal. There were holes or gaps in the portrayals that affected believability, “I did not feel that any one patient exhibited the typical components for aphasia and hypokinetic dysphonia … it appears difficult for them to adopt the speech and gestural attitudes and divorce themselves from their own personal communication style” (SLP 1). The second issue was that the cases portrayed symptoms perhaps too far on the severe end of the continuum and therefore did not necessarily represent the typical patients that health care providers would see, “I would think that most patients that medical professionals encounter with PD and aphasia will not be as severe as these simulated patients” (SLP 13).

Study 2: PWA

The purpose of this secondary study was to seek the input from people who live with the communication disorders as to how well they felt the standardized patients portrayed symptoms of the disorders. Inclusion of people with the communication disorders portrayed in this study was regarded as a potential way for these stakeholders to have their voices heard in this project. However, there were concerns about showing the videos in this study to persons with PD because the dysarthria portrayed in the videos is severe, and the researchers did not want to distress potential participants with this degenerative condition. This was not as great of a concern for PWA because aphasia typically has a stable or improving course after stroke. For this reason, this secondary study was conducted only with PWA to assess the standardized patients portraying PWA.

Method

Participants

PWA were recruited from the Aphasia Registry and Repository at the University of Washington. The registry consists of individuals who have given permission to be contacted for research, and the repository contains background information on the PWA, including medical history, aphasia history and testing data, and demographic information. Note that individuals in the registry have been screened for cognitive impairments by using the Raven's Coloured Progressive Matrices (Raven, 1998) and have been screened for visual neglect and hearing loss. To be eligible for this study, PWA had to be age 18 years or over, with onset of aphasia due to stroke at least 9 months prior to participating in the study. Due to the linguistic demands of the study tasks and because this was our first effort in using these methods, we used purposive sampling to target PWA for recruitment who had relatively high levels of function and thus were likely able to participate comfortably and productively in the research sessions. This sampling was conducted through a combination of reviewing aphasia test data for potential participants and receiving recommendations from the personnel in the University of Washington Aphasia Lab who knew the individuals in the registry and could refer PWA who were most likely to be comfortable with the methods.

Twelve PWA were initially recruited for this study. Data from the first participant recruited to test the procedures as a pilot are not included due to changes in the protocol after this participant. Data from one other participant are not included because he had significant difficulty completing the task, and the researchers were not confident that his ratings reflected a clear understanding of the task. His Aphasia Quotient score on the Western Aphasia Battery—Revised (WAB) was 51.1, which is lower than the other participants who will be described in the following section (Kertesz, 2006). Data are presented here for the remaining 10 PWA. Mean age of PWA was 60.4 years (SD 8.7; range 46–71). Mean time since stroke was 8.2 years (SD 4.9; range 4–17). Six PWA (60%) were female. WAB Aphasia Quotient scores were available for eight of the PWA. The mean WAB Aphasia Quotient was 81.9 (SD 11.2; range 62.6–94.8).

Data Collection

Data collection for the PWA was conducted in one-on-one in-person sessions. The sessions were conducted by SLPs who could provide communication support to the PWA as needed. After being oriented to the nature of the study and the task, the PWA viewed both of the aphasia standardized patient videos. After each video, the PWA were asked to rate the standardized patient on the same characteristics as described previously for the believability ratings portion of the SLP study. The wording of the rating form was altered to be more layperson and aphasia-friendly (see Appendix B). The PWA completed the ratings on paper printed with a 100-mm visual analog scale, with the end points of 0 = not at all believable and 100 = very believable. Although the participants had been screened for visual neglect, the visual analog scale was presented vertically to allow it to be presented at midline. The PWA were invited to make any comments they wished regarding the portrayals. Any comments made by the PWA regarding the portrayal of the standardized patients were summarized in field notes by the researchers, and direct quotes were transcribed when possible. Two researchers attended most sessions with the PWA to assist with field notes.

Results

Believability Ratings

Due to the small sample size and highly variable nature of the data, results for each participant will be reported individually and summarized descriptively. Data from the PWA are presented in Figure 3. Because of the variability, it was difficult to discern many trends in the data. Ratings were higher (more believable) and more consistent for the behaviors of word-finding problems, agrammaticism, and overall portrayal than for the behaviors of auditory comprehension problems and use of gestures and other nonverbal language but only slightly so. Concerns arose regarding whether or not the PWA understood the task, but their comments, collected through field notes and summarized in the following paragraph, reflected that they generally did understand that the task was to evaluate if the standardized patients appeared to have aphasia. The PWA were able to provide feedback regarding what they liked and did not like about the portrayals. However, it often appeared that the PWA were comparing the communication behaviors of the standardized patients to themselves, despite instructions from the researchers to think about not only their own aphasia but also symptoms shown by other PWA they have met. It is possible that if their own symptoms were not highly similar to those of the standardized patient, that may have been a factor in their ratings.

Figure 3.

This figure presents the data for the participants with aphasia rating believability of the two aphasia standardized patients. Data points are for individual persons with aphasia. Data for aphasia standardized patient 1 (APH1) are presented with Xs, and data for aphasia standardized patient 2 (APH2) are presented with dots.

Comments from PWA

PWA followed the same trend as the SLPs in saying that both aphasia standardized patients were not sufficiently believable in showing problems with auditory comprehension. For verbal expression, the PWA suggested that a typical person with aphasia would be saying more words; thus, the case portrayed was more severe than would be expected. Similar to feedback from the SLPs, PWA felt that some words came out too easily for APH1, particularly the word “insurance.” The PWA also felt that to be believable, the standardized patients would be working harder to get their message across—perhaps using more words or gestures and showing more frustration with the difficulty in communicating. Feedback on the overall portrayals suggested that the PWA felt that the standardized patients were mostly believable but not entirely so. For example, comments about APH2 included, “Overall just something missing” (A6) and “She is very believable just not all the way” (A10).

Discussion

The purpose of this study was to assess the extent to which standardized patients who do not have communication disorders portrayed individuals with either aphasia or dysarthria in an authentic, believable manner. These standardized patients were being trained for inclusion in a larger study in which medical and allied health students would be evaluated by using communication strategies with them. Overall, the results suggest that the stakeholders who rated the standardized patients found them to be believable, but there were several areas in which improvements were recommended. For the standardized patients portraying someone with aphasia, raters typically indicated that believability would be improved by showing more severe problems with auditory comprehension, more verbal output so that problems with grammar would be more obvious, more frustration with difficulty communicating, and more efforts to try a wider variety of words, gestures, or other means of conveying their messages. Recommendations for improving the portrayal by the dysarthria standardized patients included not making the articulation restrictions so pronounced and allowing the standardized patients to be louder so that characteristics of lack of intonation and short bursts of speech could be more readily observed. Despite these areas of suggested improvement, the targeted believability goals were achieved for most characteristics, and stakeholders provided many positive comments regarding generally strong believability for the overall portrayals of the cases. For most characteristics, there were no significant differences between the two standardized patients trained to portray each role, suggesting that the standardized patients can be trained to portray these characteristics in a manner consistent with each other.

Both the aphasia and dysarthria cases were regarded by raters as representing the severe end of the continuum. This was intentional on the part of the researchers. These standardized patients were trained for the purposes of evaluating medical and allied health students' abilities to use appropriate conversational strategies with patients with communication disorders. For this purpose, the researchers wanted to portray severe enough communication disorders that the students would be compelled to use conversational strategies that were quite different from what they would use with patients without communication disorders, including multimodal and low-tech augmentative communication strategies. Therefore, although the comments from stakeholders are acknowledged that the standardized patients did not represent the most typical or common cases because of their severity, this severity level was designed to meet the needs of the larger project.

This study is consistent with findings from the broader health care literature suggesting that communication disorders are one of many health care concerns that standardized patients can be trained to portray with a relatively high level of realism (Beaulieu et al., 2003; Erby et al., 2011; Errichetti & Boulet, 2006; Ladyshewsky et al., 2000; Schlegel et al., 2015; Shirazi et al., 2011; Tamblyn et al., 1990; Vu et al., 1987). In all likelihood, portraying various medical conditions will pose different challenges to standardized patients, and future research may explore similar versus unique challenges and training needs for standardized patients portraying communication disorders versus other conditions. The findings of this study also extend the current literature regarding use of standardized patients in speech-language pathology (Bressmann & Eriks-Brophy, 2012; Hill et al., 2013, 2015; Syder, 1996; Zraick et al., 2003) and suggest that standardized patients can be trained to believably portray severe communication disorders. Several issues relevant to this study will be discussed in this section, including benchmarks for sufficient believability of standardized patients, qualified evaluators of believability, and suggestions for training and evaluating believability of standardized patients.

One of the first questions to consider in evaluating the believability of standardized patients is how good is good enough? In this study, the researchers targeted ratings of 80 or higher out of 100, with 100 = very believable. The target of 80/100 was chosen somewhat arbitrarily with the idea that ratings of 80/100 should indicate that the portrayals were mostly believable, although this level is fairly consistent with accuracy levels achieved in other health care areas per literature cited previously. The study was successful in that the mean ratings for most communication behaviors, as well as the overall ratings, reached 80/100 or higher, but the ratings did not extend much beyond 80. That would suggest that although the standardized patients met the goal of being mostly believable, there is room for improvement. Future research might explore different methods of rating believability or compare the believability of standardized patients to characteristics of actual patients, as a means of identifying an appropriate benchmark for authenticity.

One key issue to consider with regard to the question of how good is good enough is evaluating believability in the context of the broader purpose for including standardized patients in a program. In this program, the standardized patients were used in simulated interviews to evaluate the abilities of medical and allied health students to use a variety of different communication strategies to implement effective patient–provider communication. The goal was for the standardized patients to portray representative characteristics of the two communication disorders in a manner that would compel students to use different conversational techniques and communication materials to establish effective communication with the standardized patients. Given that the students were largely inexperienced with patients with communication disorders and that the overall goals of the program could be met with representative portrayals, there may have been more leeway for portraying some details of the disorders. There might be a different set of criteria needed to judge believability or a different level of believability needed for various purposes. For example, greater detail might be needed in differentiating some nuances of the portrayals if the standardized patients were being trained for working with graduate students in speech-language pathology who are learning detailed differential diagnosis procedures.

To develop methods for evaluating the believability of standardized patients to portray communication disorders, another question to consider is who is qualified to evaluate believability? SLPs were chosen as the primary stakeholders who would be the professional experts in communication disorders and thus the raters who would have the academic training, as well as the most depth and breadth of exposure to a range of different presentations of these communication disorders. It would seem logical that SLPs would play a central role in evaluating the believability of standardized patients for this reason. For this project, the perspectives of the medical and allied health students who participated in the training program might be valuable to further address the question of how good is good enough. The students would not be expected to be experts in the disorders, but for our particular training program, they are the key stakeholders, being the recipients of the communication skills training program. Their viewpoint would reflect if the students felt the standardized patients were believable enough to provide a high-quality and worthwhile training experience. If the students did not find the standardized patients to be believable, it might be harder for them to buy in to the training experience in terms of taking it seriously and trusting that they were learning useful skills. The perspective of the medical and allied health students participating in this training program, including their views regarding the standardized patients, will be presented in a future article.

The other stakeholder group that was included in the secondary study was PWA. The decision to include PWA was motivated by the premise that the viewpoint of people who live with the condition should be included in creating standardized patient cases to represent them. The researchers felt that the PWA should have a voice in how they were being portrayed. Several PWA asked why we did not just use people who actually have aphasia as our standardized patients. The authentic viewpoint of people with communication disorders is highly valued in teaching health care providers about these populations. However, there are many demands related to serving as a standardized patient that would be difficult for many people with certain types of communication disorders, including the ones in this study, to fulfill.

Completing the rating task for believability was somewhat challenging for most PWA in this study. All, except one (whose data were excluded), seemed to understand the nature of the study. Their comments revealed that they were genuinely attending to the communication behaviors of the standardized patients and providing feedback about what they liked and did not like about the portrayals. In that manner, their input was highly valuable. Two issues in particular may have affected the quality of the quantitative ratings data. First, the nature of the rating task was perhaps too abstract for some PWA. All rating questions asked the participants to rate the believability of the behavior. For the PWA, the task may have been easier if it had been more concrete with asking PWA to simply rate how severe the symptoms were. For example, instead of asking PWA to rate “How believable is this actor in showing problems with understanding people,” a better option might have been, “How much of a problem does this person have understanding what people say?” The first option was used in this study to keep the wording more consistent with that used for the SLPs, but a simpler task may have improved the quality of the data and the researchers' confidence in the data. A second possible complication with the PWA is that many of them compared the symptoms shown by the standardized patients with their own symptoms. The researchers instructed and reminded them that the symptoms did not have to match their own to be believable, but if the symptoms were similar to those that they had experienced themselves or had seen in other people, they could still be believable. However, this was a difficult point for some PWA.

Persons with PD were not included in this study out of concern that individuals with this degenerative condition might become upset at seeing symptoms more severe than their own and wondering if their speech would deteriorate in this manner. On the basis of our experiences with this project and feedback from the PWA who participated, there may be other powerful ways to include people who live with communication disorders in the standardized patient process. Several PWA suggested having someone who has the communication disorder work with the standardized patients during their training to provide a model for them. For this study, videos of actual people with the respective communication disorders were used for examples, but it is easy to imagine how being able to interact directly with a person with the communication disorder might improve the quality of the example that the standardized patients would have to follow.

There are limitations to this study that provide many opportunities for future research. First, the results of the believability analyses would benefit from larger samples of raters and more even distribution of gender across the raters. For improving standardized patient believability, it is important to explore variables, such as different training methods or the amount of training that would further improve the believability of the standardized patients. Given the resources required to train standardized patients, finding the most efficient methods to produce the best results would be highly beneficial. Another limitation of this study was that the videos shown to participants were restricted to 5 min in length, which was done primarily for feasibility by placing a manageable burden on the raters. However, questions arise as to whether judgments of believability would be different with a longer exposure to the standardized patient. An additional area to explore would be different ways to assess standardized patient believability to ensure that valid, reliable, and useful data are generated to guide standardized patient training. Evaluating the believability of standardized patients portraying communication disorders has to have at least two prongs: one being the portrayal of the actual communication disorder symptoms, as explored in this study, and the second being assessment of the accuracy and reliability in using the script content as explored in prior research (Hill et al., 2013). Another area for future growth would be expansion of portrayals of different types of communication disorders. Are there different aphasia or dysarthria characteristics that could be portrayed more believably than the ones targeted in this study? Would believability differ for other diagnoses, such as cognitive-communication impairments? In addition, it is important to consider that some people who actually do have communication disorders might enjoy serving as standardized patients if doing so would not place undue strain on them. For example, individuals with stuttering or laryngectomy would be two examples of people who should be able to serve as standardized patients, with maximum believability and manageable burden.

Acknowledging these many areas for growth and advancement of this research, the research team hopes that this study will provide a foundation for future studies to build upon for developing high-quality methods for including standardized patients in the many possible venues for educating different stakeholders about communication disorders. As we observed in this study, standardized patients can significantly enhance the experience and quality of education provided to students. The benefits gained are well worth the resources invested.

Appendix A

Case overview and communication disorder features

The following table shows the case overview and communication disorder features targeted for the standardized patient portrayals. These excerpts are from the instructions provided to the standardized patients.

Behaviors targeted for portrayal	Description and examples
Aphasia standardized patient
Case overview	A 60-year-old female status post left-brain stroke about 18 months ago, resulting in moderate aphasia and right upper extremity weakness. Patient is an elementary school teacher who has been on medical leave of absence since stroke. Patient is at the clinic today for a first-time visit to establish care after her prior primary care provider retired. She is married, but her husband is not able to be at the visit today.
Auditory comprehension	Moderate difficulty understanding what people say, particularly if they speak quickly, use long sentences, and present complex information. Patient is generally aware that she does not understand someone. If a student speaks fast or uses long or complex sentences, the patient shakes her head “no” and gives a confused look to indicate that she does not understand. Understands simple yes or no questions about 75% of the opportunities. Responses to all questions should be delayed 3–4 s, indicating that it takes a little longer to understand information.
Word finding	Significant problems coming up with words. Pause for at least 3–5 s before responding. The patient can think of the right word to say about 25% of the time. The rest of the time the patient cannot think of something to say or says a word that is not the intended one (e.g., says “dog” when means “cat”). Frequently perseverates saying the same word over and over. Verbally confuses yes and no about 50% of opportunities. Patient is aware of mistakes and gets frustrated but cannot always fix errors.
Agrammaticism	The patient can think of some of the content words in sentences but cannot put the words together in a grammatically complete sentence. For example, if she wanted to say, “My husband went to the store,” she might say “Brother—no–no—husband. Out.” She cannot say a complete sentence.
Gestures or body language	The patient readily uses gestures, with left hand only, to attempt to convey ideas. Gestures may be vague but will use pointing and simple charades to help convey message.
Reading comprehension a	Moderate difficulty understanding written information. If the interviewer writes down one or two key words, the patient can understand those, although it takes a few seconds to understand the words. If the interviewer writes something sentence-length or hands the patient anything to read that has sentences or paragraphs, that is too overwhelming and difficult, and the patient indicates that she cannot read that.
Written expression a	Has to use left, nondominant hand for any writing attempts. Patient can write first and last name slowly and with effort. Patient can spell husband's name. Beyond that, the patient can make an effort to write the first letter or two of a word (sometimes it is correct and sometimes not) but cannot write more than that. Cannot write well enough to convey ideas through writing.
General behavior and demeanor	Patient is alert, attentive, and eager to communicate. Patient has paralyzed right hand so unable to use it for any function. Patient is already sitting down when interviewer enters the room and remains sitting throughout so walking not part of presentation.
Parkinson's disease standardized patient
Case overview	A 62-year-old male diagnosed with Parkinson's disease about 10 years ago. He is a retired attorney who has just moved to Seattle to live closer to adult children, although he will be living in his own residence. Patient is at the clinic for a first-time visit today to establish care with a primary care provider after moving to Seattle.
Speech movements	Starting speech is difficult—it takes about 3 s to initiate speech each time the patient tries to say something. When speaking, the patient does not move his face or mouth much at all. Mumbled and slurred speech.
Loudness	Very quiet voice. The voice should be kept above a whisper but very quiet.
Intonation	Very little melody or intonation in the speech. Speech is kept monotone showing very little expression.
Rate	Words rush together in short rushes of speech—short phrases come out in a slurred rush with long pauses between.
Flat affect	The face stays largely expressionless. The patient maintains eye contact and follows activities with his eyes but otherwise little or no facial expression. The patient does not smile when meeting someone. Patient does not nod his head to acknowledge or show understanding. Face and head are largely motionless.
Written expression a	The patient's writing is very tiny (micrographia)—so small that it is hard to read—and it is effortful to do. The patient can write to a limited extent but really just a few words—writing is too effortful to do for long periods of time, such as writing out sentences or responses to all questions.
General behavior and demeanor	Demonstrates pill-rolling tremor in hand. Patient is already sitting down when interviewer enters the room and remains sitting throughout so walking not part of presentation. Posture somewhat stooped. Movements with upper extremities are slow and delayed—pauses at least 3 s before initiating movement. The patient understands what is said to him (unless the interviewer talks excessively fast or uses excessively technical jargon that most people would struggle to understand). There are no cognitive impairments.

^aThe standardized patients were trained in how to respond to opportunities for reading and writing. Due to inconsistent opportunities arising for this behavior in the interviews, data regarding believability of these behaviors were not included in this article.

Appendix B

Rating form wording

The following table summarizes the questions for speech-language pathologists and persons with aphasia to rate believability of standardized patient portrayals.

Aphasia standardized patient
Item	Speech-language pathologists	Persons with aphasia a
Auditory comprehension	Persons with nonfluent aphasia may have problems with auditory comprehension, although comprehension may be relatively spared compared with verbal expression. How believable is this standardized patient portraying auditory comprehension impairments?	Many persons with aphasia have problems understanding what other people say. Is this actor believable showing this?
Word finding	Persons with nonfluent aphasia often experience word-finding impairments. They may not be able to think of a word they want to say, or they may substitute a word for the intended word. Expression is a struggle and is effortful. How believable is this standardized patient portraying word-finding impairments?	Persons with aphasia often have lots of problems coming up with the words they want to say. They may not be able to think of a word. They may say a word they do not intend to say. Is this actor believable for this?
Grammar	Persons with nonfluent aphasia often demonstrate difficulties with grammar. Their speech may consist largely of content words with omission or errors of grammatical function words resulting in telegraphic speech. How believable is this standardized patient portraying agrammaticism?	Persons with aphasia often have problems with grammar. They may be able to say a few words but cannot always make a complete sentence. Is this actor believable for this?
Gestures	Persons with nonfluent aphasia may be able to use other nonverbal means of communication to help them, such as simple gestures, body language, and pointing to cues in the environment. They are alert and attentive to the environment around them for cues. However, they will not be able to easily use sign language or other complex systems. How believable is the nonverbal communication behavior (attentive to communication and appropriate gestures or body language) of this standardized patient?	Persons with aphasia often use other ways to communicate when speech is hard. They use gestures, body language, or facial expression. They may point to things around them. They pay close attention to their environment for clues. Is this actor believable for this?
Overall	Overall, how believable is this standardized patient in portraying someone with moderate nonfluent aphasia?	Overall, how believable is this actor in portraying someone with aphasia?
Parkinson's disease standardized patient
Item	Speech-language pathologists
Speech movements	Persons with dysarthria due to Parkinson's disease (PD) often present with significantly reduced speech movements. Speech may be slurred or imprecise. How believable is this standardized patient portraying reduced speech movements?
Loudness	Persons with dysarthria due to PD often have reduced speech loudness. Voice may be softer and weaker overall with reduced projection. How believable is this standardized patient in portraying reduced loudness?
Intonation	Persons with dysarthria due to PD often present with reduced intonation. Variation in both pitch and loudness is reduced. Speech may be characterized as monopitch and monoloudness with reduced stress variations. How believable is this standardized patient in portraying reduced intonation?
Speaking rate	Persons with dysarthria due to PD often have abnormal speaking rates. One pattern that is observed in some individuals is short bursts of rushed speech. How believable is this standardized patient in portraying short bursts of rushed speech?
Flat affect	Many persons with PD develop a flat affect or masklike facial presentation in that they do not use much facial expression when talking. This reduces the amount of nonverbal information available to the listener during a conversation because of the lack of cues from facial expression. How believable is this standardized patient in portraying flat or masklike affect?
Overall	Overall, how believable is this standardized patient in portraying someone with moderate to severe hypokinetic dysarthria due to PD?

^aOn the persons with aphasia forms, the term actor was used instead of standardized patient. The term actor is typically not preferred by standardized patients and therefore was not used otherwise in this study. However, for the purposes of working with persons with aphasia, the researchers felt that the term actor might be more aphasia-friendly.

Funding Statement

The authors gratefully acknowledge the National Institute on Deafness and Other Communication Disorders for funding this research (Grant R03DC012810; awarded to the principal investigator, Carolyn Baylor). Additional funding was provided to Helen Mach by the Stolov Research Fund in the Department of Rehabilitation Medicine at the University of Washington.