Use of an Objective Structured Clinical Exam (OSCE) for Early Identification of Communication Skills Deficits in Interns

Abstract

Introduction

There is limited information about whether OSCE during GME orientation can identify trainee communication deficits before these become evident via clinical performance evaluations.

Methods

Ninety-seven interns matriculating to eight residency programs in six specialties at four hospitals participated in a nine-station communication skills OSCE. Ratings were based on the ‘Kalamazoo, adapted’ communication skills checklist. Possible association with intern performance evaluations was assessed by repeated-measures logistic regression and ROC curves were generated.

Results

The mean OSCE score was 4.08±.27 with a range of 3.3–4.6. Baseline OSCE scores were associated with subsequent communication concerns recorded by faculty, based on 1591 evaluations. A 0.1-unit decrease in the OSCE communication score was associated with an 18% higher odds of being identified with a communication concern by faculty evaluation (odds ratio 1.18, 95% CI 1.01–1.36, p=0.034). ROC curves did not demonstrate a “cut-off” score (AUC= 0.558). Non-faculty evaluators were 3 to 5 times more likely than faculty evaluators to identify communication deficits, based on 1,900 evaluations.

Conclusion

Lower OSCE performance was associated with faculty communication concerns on performance evaluations; however, a “cut-off” score was not demonstrated that could identify trainees for potential early intervention. Multi-source evaluation also identified trainees with communication skills deficits.

Introduction

Assessment of clinical trainees is widely recognized as an area needing greater attention (Epstein 2007). Early identification of poor performance is a logical goal to allow for prompt remediation efforts and increased supervision (Holmboe 2004). Resident performance deficits often relate to communication skills (Chang et al. 2009; Hauer et al. 2009), for which assessment can be particularly challenging (Richmond et al. 2011). The Objective Structured Clinical Examination (OSCE) at the time of matriculation into Graduate Medical Education (GME) (e.g. at the beginning of internship) has been utilized to broadly examine core skills and the results can inform curriculum enhancements in areas where trainees underperform (Lypson et al. 2004). While Objective Structured Clinical Exams are routinely used in medical school assessment and have been shown to predict students who will later perform poorly on subsequent clinical rotations in medical school (Martin & Jolly 2002), there is limited information about whether OSCE at the beginning of GME training can identify trainee deficits before these become evident via clinical evaluations.

The Partners Office of Graduate Medical Education developed an OSCE to provide incoming interns with an opportunity to practice communication skills and get feedback from trained standardized patients guided by a standardized checklist. This provided an opportunity to assess whether performance on a communication-focused OSCE at the beginning of internship was associated with subsequent performance evaluations. The purpose of this study was to determine if an OSCE at the time of matriculation into GME (e.g. beginning of internship) could identify trainees who would later be found to have communication deficits via faculty end-of-rotation evaluations during internship year. Further, the study sought to determine the frequency with which a communication concern was noted on routine end-of-rotation competency-based multi-source intern evaluations, and whether different evaluator groups were more or less likely to identify concerns about interns’ communication skills.

Methods

97 interns matriculating to eight residency programs in six specialties at four hospitals participated in a nine-station communication skills OSCE. Participating specialties included Internal Medicine, General Surgery, Orthopedics, Emergency Medicine, Psychiatry, and interns entering a transitional year of training. All interns from those programs were registered to participate as part of the programs’ orientation activities. Cases utilized a variety of clinical scenarios as shown in Table 1. Interns had 15 minutes to complete each case, followed by 5 minutes of immediate feedback from a trained Standardized Patient (SP). The interns were rated by SPs based on the ‘Kalamazoo, adapted’ communication skills checklist (Rider & Nawotniak 2010), with each item scored 1–5, (5 = highest level of skill). An overall OSCE score for each intern was determined using the mean score of all items on the checklist; summative OSCE scores were not provided to trainees as they were for research purposes only. Trained Standardized Patients (SPs) were utilized from an academic medical center that supplies SPs for various teaching programs around the region. SPs were specifically trained on the OSCE cases for this study. The Kalamazoo, adapted communications skills checklist is a reliable and valid measure of physician communication skills (Joyce et al. 2010).

Table 1.

OSCE Cases

Unexplained abdominal pain

Identifying symptoms due to medication interaction

Informed consent

Use of a medical interpreter

Breaking bad news and * HIPAA

Presentation of acute illness in an emergency ward patient

Hospital discharge and follow-up instructions

Starting a new medication

Telephone medicine in a chronic pain patient seeking opioids

^*Health Insurance Portability and Accountability Act

During internship year, written performance evaluations were collected for the participants after each rotation, as is done for all interns. The end-of-rotation evaluation forms included questions that assessed the six core competencies (at least one question per competency), as described and required by the Accreditation Council for Graduate Medical Education (ACGME). The six core competencies include patient care and procedural skills, medical knowledge, practice-based learning, systems-based practice, interpersonal and communication skills, and professionalism. Forty-five versions of competency-based evaluation forms were in use, completed by faculty, peers, patients, “self”, and other health care professionals. Three physician-educators (MEG, JC, DW) independently reviewed the evaluation forms to identify items relating to communication skills and designate ratings indicating concern (binary rating of ‘concern’ vs. ‘no concern’). The ACGME description of the core competency ‘interpersonal and communication skills’ (ACGME) was utilized to identify items on evaluations that related to communication skill; items pertaining to the remaining 5 core competencies of GME training were not scored, tallied, or included in the study analysis. Discrepancies were resolved by discussion and consensus was reached on all items. All study data was maintained in REDCap, a secure web-based application designed to support research (Harris et al. 2009).

Faculty, nurses/other healthcare professionals, peers, and patients who completed the end-of-rotation evaluations during the 12-month period of internship received the usual instruction provided on evaluation forms and by individual program directors, program staff, and nursing supervisors; additional rater training was not provided. Faculty evaluations were analyzed for the presence or absence of a communication concern in SAS 9.3 by repeated-measures logistic regression using generalized estimating equations to account for covariance among evaluations of the same intern. The model included terms for OSCE score and evaluator type (faculty, other healthcare professional/nurse, patient, peer) and used an exchangeable working covariance matrix. ROC curves were generated with logistic regression using each intern’s average OSCE score and communication concerns on faculty evaluations. Ethical approval was granted for studies involving human subjects by the Partners HealthCare Institutional Review Board, Protocol 2011P001079.

Results

OSCE scores among the 97 participating interns ranged from 3.3–4.6, with a mean ±SD of 4.08±.27. Routine multi-source performance evaluation forms included 4 to 6 communication questions. 1900 evaluations were collected for the 97 intern participants over the course of the academic year. There was variation across residency programs in terms of intern evaluation by different role groups: 100% of interns were evaluated by faculty, 43% by peers, 30% by other healthcare providers, and 10% by patients (Table 2). One of the eight residency programs included trainee self-assessment. Only faculty evaluations were included in the statistical model as all interns had faculty evaluations.

Table 2.

Intern evaluations by evaluator types

	Evaluations					Questions
	Evaluations that did not indicate concern		* Evaluation indicating concern		Evaluation total by evaluator type	Number of communication questions [answered] per evaluator type
Evaluation Type	Number of evaluations	Percent of evaluation without concern	Number of evaluations	Percent of evaluations indicating concern	N	Total number of questions per evaluator type	Mean number of questions per evaluator type
Faculty	1558	98%	33	2 %	1591	7575	4.8
* Nursing/Staff	60	90%	7	10%	67	405	6
* Patient	45	90%	5	10%	50	314	6.3
* Peer	147	94%	9	6%	156	571	3.7
Self	30	83%	6	17%	36	396	11
All evaluations	1840	97%	60	3%	1900	9261	4.9

^*These rates differed significantly after adjustment for baseline OSCE communication score (p = 0.020).

Interns with lower OSCE scores were more likely to have communication concerns on subsequent evaluations by faculty. A 0.1-unit decrease in the OSCE communication score was associated with an 18% higher odds of being identified with a communication concern by faculty evaluation (odds ratio 1.18, 95% CI 1.01–1.36, p=0.034). A logistic regression using each intern’s average OSCE score over all the communication items’ ratings from faculty evaluations did not yield an ROC “cutoff score” (AUC= 0.558, n= 19 concerns). Adjusting for a trainee’s OSCE score, non-faculty evaluators were more likely to flag communication concerns based on the results reported in Table 3. It is notable that other healthcare professionals (e.g. nurses) were more likely than physician faculty to identify intern communication deficits (OR 4.36, 95% CI 1.68–11.3), as were patients (OR 5.25, 95% CI 2.18–12.6) and peers (OR 3.01, 95% CI 1.54–5.89).

Table 3.

Multi-source evaluators other than physician faculty were more likely to identify communication deficits

Types of Evaluator	Odds Ratio
HC/Nurse vs. Faculty	4.36 (1.68,11.3), p = 0.002
Patient vs. Faculty	5.25 (2.18,12.6), p < 0.001
Peer vs. Faculty	3.01 (1.54,5.89), p = 0.001

Discussion

OSCEs are not commonly conducted at the time of matriculation into GME, though they have been reported as useful in broadly identifying strengths and knowledge gaps of incoming trainees to inform program curricula (Lypson et al. 2004; Wagner & Lypson 2009). Wallenstein (2010) conducted an OSCE that included 18 Emergency Medicine interns to determine if OSCE ratings on the six ACGME core competencies (ACGME) at the beginning of internship correlated to competency ratings on faculty Emergency Medicine evaluations during the first 18 months of training. This study was limited because only adult Emergency Medicine faculty evaluations were included (e.g. pediatric Emergency Medicine and all off-service rotations [which may have accounted for up to 50% of the intern year] were excluded). Correlations were found between the entry OSCE and adult Emergency Medicine faculty ratings in patient care, medical knowledge, and practice-based learning but not in systems-based practice, interpersonal and communication skills, and professionalism; however, the study did not include up to 50% of all faculty evaluations during internship and did not include multi-source evaluation.

The purpose of this study was to explore whether an OSCE at the beginning of internship could have prognostic value i.e. provide early identification of trainees who would later receive unfavorable faculty evaluation related to communication skills. To the best of our knowledge, this is the first study exploring the predictive potential of OSCEs among a diverse range of specialties and including all faculty and other multi-source evaluations collected during the internship year.

Our finding that lower scores on the OSCE were associated with subsequent communication concerns on faculty evaluations suggests that further research to identify an OSCE “cut off” score is essential for the OSCE to be utilized as a useful prognostic indicator of interns who might benefit from early intervention to improve communication skills. The finding that non-faculty evaluators were 3–5 times more likely than faculty to provide an unfavorable rating of an intern’s communication skills underscores the value of multi-source evaluation, which is required by the ACGME after each rotation (ACGME). Additional research is needed to confirm this finding and to see if it is generalizable across institutions. Several studies have examined trainee ratings by faculty, peers, patients, and nurses, but have been limited by a single-institution and/or single-specialty focus, and have yielded conflicting results (Brinkman et al. 2006, Chandler et al. 2010, Davis 2002, Joshi et al. 2004). For example, Brinkman (2006) found that nurses rated pediatric interns significantly lower than attending physicians on items relating to respect, accepting suggestions, teamwork, empathy, confidentiality, integrity, and accountability and rated them higher than faculty on anticipating post-discharge needs and effective care planning. Chandler (2010) found that faculty and nurses scored pediatric residents higher on professionalism and interpersonal skills than patients and families. In a qualitative study of internal medicine and surgical faculty, Dudek et al (2005) identified four broad themes that contributed to faculty reluctance to fail a trainee: Faculty were aware that the struggling trainee had no prior documentation of deficiencies, may not know how to specifically document weaknesses, may anticipate an appeals process, and were aware that there were limited options for remediation. Further research is needed to better understand the variation in trainee ratings among different evaluator groups.

A strength of this study was the inclusion of interns entering multiple specialties and correlation of an OSCE conducted during GME orientation to all faculty and multi-source evaluations completed during internship year. The study sought to compare the predictive potential of the GME orientation OSCE for early identification of interns with communication skills deficiencies to the routine end-of-rotation multi-source evaluation (ACGME) in each program. Thus, program specific evaluation forms were utilized and completed by both faculty and non-faculty evaluators (rather than one universal evaluation form for study purposes) who received the program’s ‘routine’ written and verbal instruction, without additional rater training. Utilization of multi-source evaluators who received ‘usual’ program instruction on evaluation completion (without additional rater training) is consistent with a number of studies in which faculty and other multi-source evaluators assessed trainees (Brinkman et al. 2006, Chandler et al. 2010, Davis 2002, Joshi et al. 2004, Wallenstein et al. 2010). It is possible that the formative feedback received from SPs as part of the orientation OSCE and the routine formative feedback provided (and required by the ACGME) during rotations throughout internship year impacted interns’ communication skills. Future studies could analyze evaluations by the time of year they were completed to determine if ongoing “on the job” feedback on communication skills (e.g. formative feedback during training and patient care experiences) influences subsequent performance on end-of-rotation evaluations. Results of this study could also have been influenced by the multi-source evaluator level of comfort and skill in rating intern communication skills.

Conclusion

By identifying a positive association between OSCE performance and subsequent unfavorable evaluation of communication skills, this study provides a preliminary suggestion that OSCE may be useful in early identification of interns with less effective communication skills. Given the importance of physician communication skills, and the recognized challenge in assessing these skills, further study of OSCE as a potential screening tool in GME may be indicated. Further research is also needed to better understand the differences in how various role groups evaluate residents so that the value of multi-source assessment can be maximized.

Practice Points.

1. In this study, trainee performance on a GME orientation communication-focused OSCE was associated with communication concerns subsequently noted on end of rotation faculty evaluations; however, ROC curves did not definitively establish an OSCE “cut-off” score for early identification of trainees who would benefit from remediation.

2. In this study, non-physician evaluators (nurses, peers, and patients) were 3–5 times more likely to identify a communication concern compared to faculty evaluators emphasizing the importance of multi-source evaluation of trainees.

3. Based on prior literature, OSCE provides the opportunity to practice skills essential for residency; however, further research is needed to develop an OSCE that can both provide skill practice and identify trainees who would benefit from early identification and remediation of communication skills.

Biographies

Dr. Goldhamer is Education Specialist, Partners Office of Graduate Medical Education, Partners HealthCare System and Instructor of Medicine, Division of General Internal Medicine at Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.

Ms. Cohen is Assistant Fellowship Director, Harvard Medical School Academy Fellowship in Medical Education Research and Director of Data Analytics and Instructor, Department of Health Policy and Management at the Harvard T.H. Chan School of Public Health, Boston, Massachusetts.

Ms. Brooks was Research Computing Core Administrative Manager, EDC Programmer and Clinical Consultant, Enterprise Research IS Group, Partners HealthCare, Boston, Massachusetts.

Dr. Macklin is Instructor in Medicine, Massachusetts General Hospital Biostatistics Center and Harvard Medical School, Boston, Massachusetts.

Dr. Co is Director for Graduate Medical Education at the Partners HealthCare System and Associate Professor of Pediatrics at Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.

Dr. Weinstein is Vice President for Graduate Medical Education at the Partners HealthCare System, and Associate Professor of Medicine at Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.