Videotaped Unannounced Standardized Patient Encounters to Evaluate Interpersonal and Communication Skills in Emergency Medicine Residents

Abstract

Background

Communication and interpersonal skills are important for effective patient care but are difficult to measure. Unannounced standardized patient encounters (USPEs) have the benefit of providing a standardized situation and provide a representation of usual care rather than best behavior, while also allowing for video recording without violating patient privacy. We conducted a feasibility pilot study to examine the use of videotaped USPEs in resident education of interpersonal and communication skills and specifically empathy.

Methods

This was a pilot study conducted at an urban community teaching hospital. About 16 first‐year emergency medicine engaged in four videotaped USPEs during their normal clinical shifts during 1 academic year. All visible recording equipment was concealed from the residents. The standardized patients completed two assessment forms after each encounter to measure empathy.

Results

All 16 residents engaged in four videotaped USPEs for a total of 64 encounters.

Conclusions

We were able to successfully demonstrate the feasibility of using USPEs for assessment of resident interpersonal and communication skills.

Effective communication is integral to the physician–patient relationship. Studies have shown that empathy in particular improves patient outcomes, results in a decreased rate of litigation, and increases patient and family satisfaction. ¹ , ² , ³ , ⁴ Unfortunately, communication and interpersonal skills are often overshadowed by medical knowledge and procedural skills during medical school and residency. One study found that 26% of interns failed to perform at entry‐level milestones related to professionalism and patient‐centered communication. ⁵ , ⁶ Furthermore, these skills have traditionally been notoriously difficult to measure. Attempts include multistation observed standardized clinical examination (OSCEs), direct observation, global evaluations, patient surveys, and multiple‐choice questionnaires. ⁵ , ⁷ , ⁸ However, each of these methods has its own limitations including, but not limited to, observer bias, recall bias, need for a large number of data points, lack of fidelity in the case of simulation or OSCEs, and the Hawthorne effect.

Unannounced standardized patient encounters (USPEs) may offer a novel method of addressing many of the barriers described above. USPEs provide a standardized situation across multiple encounters and a representation of usual care rather than best behavior, ⁹ while also allowing for video recording without violating patient privacy. Video review allows for the potential use of delayed objective feedback. We conducted a feasibility pilot study to examine the use of videotaped USPEs in resident assessment of interpersonal and communication skills, specifically empathy.

METHODS

Overall Study Design

This was a pilot study conducted at an urban community hospital. Sixteen first‐year emergency medicine (EM) residents in a single program were informed that they would encounter one or more unannounced standardized patients (SPs) during usual clinical care in the emergency department (ED) and that these encounters would be videotaped without their explicit knowledge at that time. All residents engaged in four videotaped USPEs. Following the conclusion of the study, all participants were allowed to view their video recordings if desired. This study was approved by the institutional review board at Maimonides Medical Center and supported by an institutional grant through the Maimonides Medical Center Research and Development Foundation.

Standardized Patient Recruitment and Training

Eight SPs were recruited from a directory of SPs who participated in activities at our simulation center. Each SP received an hourly pay rate consistent with local markets. Training was conducted by the investigators (initials blinded) and consisted of an introduction to the ED and familiarization with the room in which the encounters would occur. In particular, the SPs were oriented to the location of the video‐recording equipment to ensure that the view of the resident engaged in the case would not be blocked. The SPs were given the two measurement scales, detailed below, that they would be expected to use to assess each resident immediately following the USPEs. They received a written description of the case scenario and step‐by‐step instructions outlining each USPE (Data Supplement S1, Appendix S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10401/full). Two SPs were assigned to each of the four case scenarios. The anticipated time to run a scenario was 3 hours.

Case Scenario Development

Four case scenarios were developed by the investigators (ASC, CS). Each case was designed to present challenging situations that would require appropriate counseling, communication, and empathy skills to be demonstrated by the resident for successful resolution. Cases did not require orders for any medications, treatments, or diagnostic tests (for an example case, refer to Data Supplement S1, Appendix S2).

Video Recording

All USPEs took place in one designated room in the ED. Several rooms in the ED where the study took place had existing 24/7 video recording established for quality improvement purposes. Although residents may have been aware of the presence of the cameras, they were not informed that these specific cameras were being used for this study. Footage from the USPEs was extracted following each USPE and stored on a secure server for later review.

Electronic Medical Record

Two dummy personnel accounts (one registration clerk and one nurse) were created for this study. One investigator (ASC) received access to both accounts to register and triage the SPs. The following patient information was generated for the 64 unique USPEs: name, date of birth, medical record number, and account number. A new identity for the SP was assigned for each encounter. To facilitate later ease of identification, medical record and account numbers were one digit shorter than the standard length. Each SP received a patient identifier wrist band from a research assistant on the morning of arrival and was triaged according to the script for each case. The research assistant remained on the hospital campus during the USPE in case of any unexpected issues. All cases were triaged as an ESI Level 3 and assigned to the designated room of the ED. Residents documented in the chart and discharged the SP in an identical manner as other patients. USPEs were later removed from the EMR to prevent confounding with actual patient data.

Confederates

The day prior to a scheduled USPE, the attending physician and nursing leadership received an e‐mail notification. They were instructed to encourage the case progression and to conceal the true identity of the SP. The attending physician would be expected to treat the SP similar to any other patient while in the presence of the resident although a final signature on the medical chart was not required. Nursing leadership was involved to ensure that no medications or other treatments were actually administered to the SP.

Study Protocol

Unannounced standardized patient encounter occurred from 7:00 am to 10:00 am on Tuesdays, Thursdays, Fridays, and Sundays, which are characteristically low‐volume times in the ED. The SP was triaged into the EMR and placed in the designated room with video‐recording capabilities. The first‐year resident on duty was then instructed by his or her attending physician to assign themselves to the SP without being informed of the true nature of the encounter. Following discharge, the SP completed two assessment forms (Consultation and Relational Empathy [CARE], Jefferson Scale of Patient’s Perceived Physician Empathy [JSPPPE]). Residents remained unaware that the SP was not an actual patient until either they viewed their video or the conclusion of the study.

Outcome Measures

The CARE measure is a validated scoring system used to assess physician empathy in the clinical setting. ² , ¹⁰ This simple form consists of 10 questions designed for patients to rate a single interaction with a physician based on a 5‐point Likert scale. The JSPPPE has also been validated for use by patients to evaluate demonstration of empathy by physicians. Both forms can be easily adapted for use by a trained SP.

RESULTS

All 16 first‐year EM residents engaged in four USPEs as described under Methods for a total of 64 encounters. Simple descriptive statistics of the two outcome measures demonstrated a mean (±SD) CARE score of 37.16 (7.1) and JSPPPE score of 23.03 (5.23). Eight of the residents viewed their video recordings of the USPEs.

DISCUSSION

In business, “secret shoppers” allow companies to evaluate the usual behaviors and performance of their staff by circumventing the Hawthorne effect to improve metrics, especially those related to customer service. In a similar manner, our USPEs allowed assessment of resident interpersonal and communication skills, specifically empathy, during an episode of usual clinical care. We were able to successfully demonstrate the feasibility of designing and implementing this method of resident assessment, which has been widely described in other industries outside of medicine ¹¹ and within medicine, ¹² but has yet to be demonstrated in residency education until now.

Despite the preliminary nature of our study, we believe that these results are encouraging and may support the role for USPEs in residency education. We were able to gather four independent assessments of interpersonal and communication skills on each of our first‐year residents. Although the videos were not utilized in a significant way in this study, residents did have the opportunity to review them with a faculty member for additional insight into their skills. The ability to gather this type of feedback for residents performing in the actual ED environment using validated tools is a significant step forward compared to end‐of‐shift evaluations or other commonly used assessment methods in the clinical setting.

We acknowledge that the use of USPEs can be time‐ and cost‐intensive. ¹³ We propose that optimal use for USPEs would include: 1) a needs assessment of interpersonal and communication skills, 2) derivation of a less resource‐intensive measure that correlates well with scores obtained during USPEs, or 3) resident remediation of interpersonal and communication skills. In the future, we would like to see a comprehensive study to measure the impact of USPEs with or without delayed video review integrated into the feedback process on learner skill development. Additionally, this study could concurrently evaluate the impact on ED flow and other patients as part of the clinical learning environment.

CONCLUSIONS

In conclusion, we were able to successfully demonstrate the feasibility of using unannounced standardized patient encounters for assessment of emergency medicine resident interpersonal and communication skills.

Supporting information

Data Supplement S1 . Supplemental material.

AEM Education and Training 2020;4:419–422