Piloting a Structured Practice Audit to Assess ACGME Milestones in Written Handoff Communication in Internal Medicine

Abstract

Background

Written communication skills are integral to patient care handoffs. Residency programs require feasible assessment tools that provide timely formative and summative feedback, ideally linked to the Accreditation Council for Graduate Medical Education Milestones.

Objective

We describe the use of 1 such tool—UPDATED—to assess written handoff communication skills in internal medicine interns.

Methods

During 2012–2013, the authors piloted a structured practice audit at 1 academic institution to audit written sign-outs completed by 45 interns, using the UPDATED tool, which scores 7 aspects of sign-out communication linked to milestones. Intern sign-outs were audited by trained faculty members throughout the year. Results were incorporated into intern performance reviews and Clinical Competency Committees.

Results

A total of 136 sign-outs were audited (averaging 3.1 audits per intern). In the first trimester, 14 interns (31%) had satisfactory audit results. Five interns (11%) had critical deficiencies and received immediate feedback, and the remaining 26 (58%) were assigned future audits due to missing audits or unsatisfactory scores. In the second trimester, 21 interns (68%) had satisfactory results, 1 had critical deficiencies, and 9 (29%) required future audits. Nine of the 10 remaining interns in the final trimester had satisfactory audits. Faculty time was estimated at 10 to 15 minutes per sign-out audited.

Conclusions

The UPDATED audit is a milestone-based tool that can be used to assess written sign-out communication skills in internal medicine residency programs. Future work is planned to adapt the tool for use by senior supervisory residents to appraise sign-outs in real time.

What was known and gap

Written communication skills are an important component of effective, accurate patient handoffs.

What is new

Structured audits using a specially designed tool (UPDATED) were used to assess intern sign-outs, and the results were incorporated into intern milestone reviews.

Limitations

Single site, single specialty study; lack of a comparison group.

Bottom line

The UPDATED structured audit tool is a readily implementable approach for assessing trainee competency in written handoff communication skills.

Introduction

In July 2011, the Accreditation Council for Graduate Medical Education (ACGME) issued new guidelines mandating that residency programs monitor and ensure resident competency in patient handoffs.¹ The enhanced focus on handoffs is also reflected in the Educational Milestones introduced by the ACGME's new accreditation system, with many specialties including milestones that address transitions of care.² Handoffs have increased in volume and complexity with duty hour restrictions, and they increasingly rely on the written sign-out document. Instruction on sign-out communication frequently focuses on verbal handoff skills due to the fact that curricular tools, as well as assessment methods for written sign-out communication, are often absent.³ Quality of residents' written sign-outs is often poor, with errors of omission, commission, and outdated information.⁴ Handoffs that are increasingly dependent on these means of communication have the potential for serious error.

Structured practice audits are a promising method of assessing resident communication skills that allow learner involvement in the assessment. Practice audits can stimulate reflection on performance as well as change behavior by encouraging the use of knowledge gained in future practice.⁵ These tools are most effective when the criteria used to judge clinical performance are explicit and reliable.⁵ Structured templates, in addition to being suitable for audit purposes, are also preferred methods of handoff documentation and recommended by many professional organizations.⁶ One such template, UPDATED, has been described as a tool to assess written sign-out communication in accordance with the ACGME Internal Medicine Milestones.⁷ We hypothesized that the UPDATED audit tool could be piloted to assess written communication skills of postgraduate year (PGY)-1 internal medicine trainees in a manner that would serve as an appropriate milestone-based evaluation for the residency program and would promote resident reflection on clinical performance.

Methods

Audit Instrument

UPDATED is a paper-based structured practice audit tool that was designed to evaluate the quality of written inpatient sign-out documents, using criteria linked to the ACGME Internal Medicine Educational Milestones (Figure 1). The UPDATED audit tool uses a structured template and evaluates several clinical skills corresponding to ACGME Milestones: effective communication during care transitions (interpersonal and communication skills [ICS]-C1); ensuring accurate documentation (ICS-F1, ICS-F2, professionalism [P]-A1); and appropriate synthesis and definition of clinical problems (patient care [PC]-C1).⁷ Seven sign-out elements corresponding to these milestones are scored individually; most are scored as 0 or 1 for presence or absence of information, though 2 elements (“Problem list prioritized and updated?” and “Directed tasks clear?”) are scored as 0, 1, or 2 to reflect additional levels of information. Scores are summed to tabulate a final audit score. A summed score of 0 to 3 is categorized as “poor,” 4 to 6 as “fair,” and 7 to 9 as “good.” The creation of the tool has been previously described.⁷

FIGURE 1 .

UPDATED Audit Tool Mapped to ACGME Internal Medicine Milestones

Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; ICS, interpersonal and communication skills; PC, patient care; P, professionalism.

Setting and Participants

This study was performed in the internal medicine residency program at the University of Chicago, a large, urban, university-based medical center with 45 PGY-1 interns. In 2012–2013, interns were introduced to UPDATED through a didactic presentation and given pocket cards with the mnemonic listed. Eight faculty auditors (4 chief residents and 4 hospitalists) were trained to use UPDATED by discussing the audit tool and scoring system with the authors who devised the tool (J.M.F., V.M.A.) and performing an example practice audit. The pilot project was designed to evaluate written sign-outs performed by all 45 interns on noncritical care internal medicine ward rotations.

Audit Process

The University of Chicago's electronic health record features a sign-out template that incorporates both automatically imported (eg, medication administration record) and free-text fields (eg, problem list, tasks, and anticipatory guidance). Interns on ward rotations update these free-text fields and refresh the auto-populated fields daily such that the information is updated, and an updated sign-out document is printed for a cross-covering team during a patient care handoff. The faculty auditors were granted access to the electronic patient lists for each ward team and printed sign-out documents completed by the interns on those teams. Each faculty member completed audits on 5 to 6 interns. Audits of 2 written sign-outs per intern were performed. Ratings among faculty members were compared for consensus, and final audit scores were compiled by 1 of the authors (S.K.M.). It was estimated that 75% (34 of 45) of interns would be scheduled for an inpatient ward rotation within the first trimester of the academic year (July through October). Thus the aim of the pilot project was to complete audits on 75% within these first 4 months.

The study was designated as exempt for educational purposes by the Institutional Review Board at the University of Chicago.

Evaluation

In discussion with the internal medicine residency program leadership, audit results were categorized into 3 possible designations. The intern's performance was considered satisfactory if both sign-outs were scored as “good” during an audit. If sign-outs received 1 “good” and 1 “fair” score, interns underwent future review with additional audits in subsequent months. If sign-outs received 2 “fair” scores, or any “poor” score, these sign-outs were considered to have critical deficiencies. These interns received immediate feedback from a chief resident and underwent additional audits in subsequent months.

Audit results were submitted to the Clinical Competency Committees and incorporated into semiannual performance reviews. Interns achieving a satisfactory audit were designated as “competent,” those awaiting additional audits as “undergoing further evaluation of competence,” and those with critical deficiencies as “received immediate feedback and undergoing further evaluation of competence.”

Results

Of the 45 interns audited in the first academic trimester, 32 (71%) were able to have data fully completed (Figure 2). Faculty auditors were unable to obtain sign-outs for the remaining 13 interns, commonly due to the senior resident on the team having completed the sign-out rather than the intern. Of the 32 interns who underwent a full audit in the first trimester, 43% (14) had satisfactory audit results. Five interns (15%) were identified as having critical deficiencies by receiving 2 “fair” scores on their sign-out audits and received immediate feedback. The remaining 13 interns (41%) received 1 “good” and 1 “fair” score and were assigned future review of additional sign-outs in the second trimester.

FIGURE 2 .

Results of UPDATED Structured Practice Audit Pilot (University of Chicago, 2012–2013)

^an=45 internal medicine interns.

Of the 31 interns audited in the second trimester (November through February), 25 (80%) fully completed sign-outs, and 7 (22%) were unable to be audited because of missing sign-outs. Of these 25 interns, 21 (84%) had satisfactory results, 2 (8%) required future review, and 1 had critical deficiencies and received immediate remediation.

Finally, in the third trimester (March through June), 9 of the remaining 10 interns had satisfactory audit results, and 1 audit was unable to be completed due to scheduling conflicts. In total, 44 of 45 internal medicine interns (98%) underwent a full sign-out audit in the yearlong pilot project, and 136 sign-outs were audited, for an average of 3.1 audits per intern.

Financial costs of piloting this project were minimal, as no additional materials or resources were required for implementation. The evaluation was incorporated into a previously existing resident assessment structure and required no additional staff resources or infrastructure. The most significant cost of the program was time incurred by faculty and chief resident auditors, which was estimated at 10 to 15 minutes per sign-out audited. Faculty were trained on how to use the tool at the beginning of the project in a 1-hour meeting, and met 3 additional times throughout the year for 1-hour meetings to compare results for consensus.

Discussion

Our experience suggests that a structured practice audit such as UPDATED can be feasibly and effectively piloted in an internal medicine residency training program to assess resident achievement of competence in ACGME Milestones related to written handoff communication. Audit results translated well into data submitted to the Clinical Competency Committee and were used to inform the residency program's comprehensive milestone-based assessment.

A particular success of our pilot was the ability to use the structured audit tool as a template for feedback regarding written sign-out communication skills, as the UPDATED mnemonic serves as a curricular element and an assessment tool for our residency program. When interns were identified as having serious deficiencies via the audit, they would meet with chief residents who first reviewed the audit results, and then provided the interns with a refresher on the UPDATED mnemonic and principles, with concrete suggestions for improvement. As structured practice audits strive to facilitate reflection and change future clinical practice, our pilot project supported that goal by creating a transparent and timely evaluation for interns.

Although we aimed to complete the majority of intern audits early in the year, the project fell slightly short of the 75% goal. The leading barrier impeding timely completion was that sign-outs were often completed by senior residents rather than interns. Completing a written sign-out has traditionally been considered a duty of intern physicians, but it is evident that all physicians, including attending physicians, participate in this important aspect of care transitions in the current era. While our pilot began as an intern-level assessment, this practice audit could easily be modified to assess these communication skills for all physicians participating in patient handoffs at multiple levels of training and practice.

This study has several limitations. It is a single program study with a short follow-up period. Our process used a limited number of assessments of each intern, which may reduce reliability. Apart from group consensus, no formal measure of interrater reliability was obtained. Our outcomes focused on feasibility. For the interns in our pilot, serious deficiencies in communication skills decreased throughout the academic year, and all interns with serious deficiencies received satisfactory scores in subsequent audits. However, intern written sign-out skills improved throughout the academic year by virtue of time and experience. It is unclear if the change we saw was attributed to the audit, as our study lacked a comparison group.

As this practice audit evolves, it will be necessary to evaluate additional outcomes, such as resident satisfaction with the audit process and intent to change behavior. Also, as skills deteriorate over time, it would be ideal to use this practice audit tool on a longitudinal basis to continually assess handoff communication skills. Additionally, it would be interesting to understand the effectiveness of the UPDATED audit process from the perspective of the handoff receivers, residents using the written sign-out document as a primary source of information for cross-cover patients. Finally, future research should address whether this process affects patient outcomes, such as the number of errors related to written handoff communication.

The next phase of our study of the UPDATED tool is to train senior supervising float residents on how to use it to audit sign-outs written by junior team members in real time. In our structure, float residents are responsible for receiving handoffs from interns and serve as supervising senior residents when the team resident is off-duty. We anticipate that the addition of this responsibility to float residents' role will strengthen the ability of the UPDATED tool to promote reflection on current practice and change behavior, along with adding another layer of peer assessment.

Conclusion

As residency training programs are now required to ensure competency in handoff communication, teaching and assessing residents' written sign-out skills has become essential for these programs. The UPDATED structured audit tool is an easily implemented approach to assess trainee competency in written handoff communication skills using a milestone-based approach.