A novel use of the discrete templated notes within an electronic health record software to monitor resident supervision

Abstract

Objective: Monitoring the supervision of residents can be a challenging task. We describe our experience with the implementation of a templated note system for documenting procedures with the aim of enabling automated, discrete, and standardized capture of documentation of supervision of residents performing floor-based procedures, with minimal extra effort from the residents.

Materials and methods: Procedural note templates were designed using the standard existing template within a commercial electronic health record software. Templates for common procedures were created such that residents could document every procedure performed outside of the formal procedural areas. Automated reports were generated and letters were sent to noncompliers.

Results: A total of 27 045 inpatient non–formal procedural area procedures were recorded from August 2012 to June 2014. Compliance with NoteWriter template usage averaged 86% in the first year and increased to 94.6% in the second year (P = .0055).

Initially, only 12.5% of residents documented supervision of any form. By the end of the first year, this was above 80%, with the gains maintained into the second year and beyond. Direct supervision was documented to have occurred where required in 62.8% in the first year and increased to 99.8% in the second year (P = .0001) after the addition of hard stops. Notification of attendings prior to procedures was documented 100% of the time by September 2013. Letters sent to errant residents decreased from 3.6 to 0.83 per 100 residents per week.

Conclusion: The templated procedure note system with hard stops and integrated reporting can successfully be used to improve monitoring of resident supervision. This has potential impact on resident education and patient safety.

BACKGROUND AND SIGNIFICANCE

The Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements mandate the provision of supervision for residents.¹ Program directors are required to “monitor resident supervision at all participating sites.” At the same time, residents “must be able to competently perform all medical, diagnostic, and surgical procedures considered essential for the area of practice.” Demonstrating these requirements for both parties may present challenges. Currently, one may try to meet these requirements by doing manual abstraction of medical records,^2,3 conducting interviews⁴ or surveys,^5,6 and keeping personal procedural logs.^7–9 However, these would require additional time and energy, and may be associated with less than precise results.¹⁰

OBJECTIVE

We set out to create and implement a semi-automated system of addressing these requirements in a discrete, standardized, and comprehensive manner that requires minimal extra effort on the residents’ part. Our vision was to have a system for discretely capturing and reporting on procedures performed across the institution. With this, we can ensure that appropriate levels of supervision are being documented, captured, and processed in real time to allow for institution-wide monitoring, reporting, and action based upon the data, where indicated. This potentially enhances resident education and affords the opportunity to impact patient safety by ensuring oversight.

MATERIALS AND METHODS

NoteWriter procedural note templates

Procedural note templates were designed using the standard existing electronic health record (EHR) template (known as NoteWriter) within the EHR software EpicCare Inpatient (Epic Systems Corp., WI, USA). This template allows users to document procedure notes by clicking buttons that cascade to other button choices as they move through the essential fields of procedural note documentation. A group of templates for common specific procedures was created along with a generic template, such that residents could document, in a standardized format, every procedure performed outside of formal procedural areas (FPAs), eg, operating rooms, pulmonary labs, interventional radiology suites (Figure 1). We felt that the area with the greatest need to ensure appropriate oversight would be inpatient procedures performed outside the FPAs, where the “Swiss cheese model” of accident causation is more likely to occur due to a reduction in the number of safety layers.^11,12 Indirect supervision¹³ is also more often utilized in the nonoperative setting, hence posing increased concerns regarding oversight. Likewise, the FPAs all have formalized and standardized processes for oversight of resident supervision. For this reason, as our initial endeavor we mandated compliance with use of the NoteWriter template when documenting procedures in the non-FPAs of the primary teaching hospital (Parkland Memorial Hospital) at our institution.

Figure 1.

Screenshot of NoteWriter procedural note template © 2015 Epic Systems Corporation. Used with permission.

In fulfilling our objectives, care was taken to minimize any adverse impact on the residents by implementing our intervention through a pre-existing system in use, and in the context of performing an activity they would already have to perform (documenting a procedure note). Each field within the NoteWriter template captured metadata that enabled discrete, automated reporting.

A total of 17 NoteWriter procedure note templates were launched in July 2012, with broad communication mandating their use to residents and all other providers at Parkland Memorial Hospital. Within every procedure template was a standard module containing a cascading question functionality to prompt for and mandate capture of documentation of oversight by learners, including whether they were privileged to perform without direct supervision, the level of supervision under which the procedure was performed, the supervising provider, and whether prior discussion of the procedure with an attending occurred (Figure 1). A procedural note would then be generated automatically, taking the place of a standard free-text chart entry, which might otherwise contain only limited relevant information that is typically not discrete, consistent, or auditable by system reports.

After an initial review of compliance with this new system at 3 months post implementation, in November 2012 a second memo by the chief medical information officer and the graduate medical education (GME) designated institutional official (DIO) was sent out to re-educate and encourage compliance with this process. From March 2013, the process was further enhanced whereby residents who did not abide by this standard process were sent letters directly from the DIO, directing their attention to the deficiency and reminding them to utilize the standardized templates. Finally, hard stops to encourage users to complete several mandatory fields, including the supervision portion of the template, were added in May 2013. A summary of the timeline with the introduction of interventions is shown in Figure 2.

Figure 2.

(A) Proportion of non–formal procedural area (FPA) procedures in the inpatient setting documented using the NoteWriter procedure note template since implementation in July 2012. (B) Proportion of non–FPA procedures documented using the NoteWriter procedure note template that included the level of supervision provided to residents. (C) Proportion of non–FPA procedures in the inpatient setting documenting direct supervision when it was required. (D) Number of letters sent per week to noncompliers in the first 6 months since the initiative started in March 2013. Data from July 2012 were omitted due to the introductory period and the changeover period of the academic year. For the months of August, September, and October 2012, no procedures reported the need for direct supervision.

Automated reports were generated from the EHR on a regular basis to monitor compliance. Our reporting allowed us to capture all procedure note note types. The use of nontemplated versions was detected by the absence of the structured elements that had metadata tags associated with them. Based on these reports, letters were sent out to noncompliers through the office of GME.

All procedure notes (whether using the template or not) were coded to require mandatory co-signature by the designated attending physician. This enabled verification of the accuracy of the designated supervising physician, the level of supervision documented and required, and other pertinent details of the procedure, minimizing the possibility of misattribution.

Parkland Memorial Hospital

Parkland Memorial Hospital is an 860-bed level 1 trauma center located in Dallas, Texas, serving as the primary teaching hospital for the University of Texas Southwestern Medical Center. In fiscal year 2014, Parkland saw 47 863 total discharges, 1 005 663 total outpatient visits, and 265 955 total emergency room visits. Eighty ACGME programs and all levels of residency, from interns to fellows, are represented. Documentation of medical care is performed electronically with EpicCare. Institutional Review Board exemption was granted for publication of these data.

Statistical analysis

The data from the generated reports were summarized and presented with descriptive statistics. Continuous data were subjected to the Student’s t test. The α level was set at 0.05. All P-values reported were based on 2-sided tests. Statistical analyses were performed in GraphPad QuickCalcs (GraphPad Software Inc., La Jolla, CA, USA).

RESULTS

A total of 27 045 inpatient procedures by 1075 residents were recorded from non–formal procedure areas between August 2012 and June 2014 (Tables 1–3). Compliance with use of this discrete documentation averaged 74.8% in the first 3 months.

Table 1.

Breakdown of the number of inpatient and outpatient procedure notes by level of residency

PGY Level	Inpatient	Outpatient	Unknown
1	6601	4032
2	10 885	5574	10
3	5423	4431	16
4	557	4641	18
5	50	77	1
6	38	3
7	4	1
8	1	4
Fellow	2395	1280	5
Total	25 954	20 043	50

PGY: Postgraduate year

A small number of procedure locations were not documented

Table 2.

Breakdown of the number of inpatient and outpatient procedure notes by specialty of provider

Specialty	IP/ED	OP	Unknown	Total
Emergency Medicine	8280	6		8286
Dermatology	259	4459	2	4720
Orthopedic Surgery	3913	563		4476
Ophthalmology	87	4023	4	4114
Internal Medicine	3338	423		3761
General Surgery	2498	1018		3516
Obstetrics and Gynecology	1553	1487		3040
Oral and Maxillofacial Surgery	597	2220		2817
Physical Medicine and Rehabilitation	23	1884		1907
Urology	164	1437	1	1602
Plastic Surgery	1189	138		1327
Family Medicine	250	841		1091
Otorhinolaryngology	338	471	3	812
Pediatrics	678	4		682
Hematology and Oncology	364	237		601
Nephrology	483	2		485
Pathology	76	340		416
Neurology	327	78		405
Cardiology	327	8	1	336
Neurological Surgery	318	2		320
Pulmonary Diseases	223	1		224
Anesthesiology	174	12	38	224
Rheumatology	51	90		141
Podiatry	105	16		121
Gastroenterology	74	14	1	89
Critical Care Surgery	56			56
Clinical Neurophysiology	35	15		50
Hematology	13	28		41
Neurocritical Care	31			31
Oncology Surgery	5	23		28
Vascular Surgery	1	10	11	22
Psychiatry	15	2		17
Endocrinology	11			11
Clinical Cardiac Electrophysiology	9			9
Allergy and Immunology	1	5		6
Hospice and Palliative Medicine		1		1
Totals	25 866	19 858	50	45 774

The total number of procedures on this table is less than the total number of procedures in Table 1 due to a tiny proportion of incorrectly documented specialties.

Table 3.

List of the top 20 most performed procedures in the non–formal procedural area inpatient setting from August 2012 to June 2014

Top 20 most performed procedures	IP/ED
Laceration repair	3868
Splint	3005
Vascular access	2896
Incision and drainage	2782
Lumbar puncture	1770
General procedure	1763
Paracentesis	1200
Arterial line	1091
Airway access	1029
Endometrial biopsy	738
Arthrocentesis	673
Thoracentesis	443
Chest tube	417
Reduction	304
Intubation	207
Bone marrow biopsy	148
Umbilical catheter	140
Nail removal	126
Echocardiogram	93
Intrauterine device removal	78

Further initiatives, including the second memo in November 2012, the individual letters from the DIO for noncompliers in March 2013, and the introduction of hard stops in May 2013, led to a steady improvement in compliance with NoteWriter usage (Figure 2A). This averaged 86% in the first year, increased to 94.6% in the second year (t = 3.10, 20 df, n = 25891, P = .0055), and stabilized in that range thereafter.

Level of supervision

By creating a templated note, we were also able to discretely capture other metrics of importance. When the project began, only 12.5% of residents documented supervision of any form (Figure 2B). By the end of the first year, over 80% of residents were documenting the level of supervision for procedures, with the gains maintained into the second year and beyond. Prior Notification of attendings for certain procedures (eg, thoracentesis or pericardiocentesis) was a requirement for residents, and these were also recorded and captured. This number began at 0%, but with feedback letters from the GME rose to be documented consistently at 100% by September 2013.

Direct supervision

As discussed in the Materials and Methods section, we felt that the highest risk of adverse events might occur in procedures performed in the non-FPA locations of the hospital. Likewise, residents performing procedures for which they had not met the criteria to perform without direct supervision were felt to be the most at-risk group. From the NoteWriter templated procedural notes, direct supervision was documented to have occurred in instances where it was required 62.8% in the first year (Figure 2C). This increased to an average of 99.8% in the second year (t = 4.96, 17 df, n = 3130, P = .0001) after the addition of hard stops in the template to ensure completion of the form. The specific name of the supervising physician was also discretely documented.

Letters to noncompliers

In March 2013, the GME office began to send letters to errant providers detailing their omissions and highlighting the new standard practice for documenting procedures and supervision. While the initial burden of work was high (averaging 52 letters per week over the first 5 weeks, or an average of 3.6 letters per 100 residents per week), the residents modified their behavior, and fewer letters were required each week until they plateaued at an average of 12 per week for this institution of 1452 residents (0.83 letters per 100 residents per week) (Figure 2D).

DISCUSSION

Over the past decade, the spotlight on resident education has been focused on the 80-hour-per-week duty restrictions and the 16-hour shift limit for interns.^14–25 Resident supervision has not received as much attention, and this has been reflected in the literature. To the best of our knowledge, our study represents the first attempt at addressing the ACGME’s requirement of provision of supervision to residents at an institutional level and in a semi-automated manner. Our results demonstrate the successful implementation of a simple and effective system that enables the monitoring of resident supervision to be objectively quantified.

At a large institution such as ours, ensuring compliance with a newly implemented policy requires the efforts of multiple parties. The initial memorandum informing the residents of the new policy was sent out by the chief medical informatics officer, along with e-learning modules to help the residents through the process. Individual residency program directors at the institution were also engaged in and informed of this new policy, and were encouraged to brief their residents on the subject. With this minimal effort, we were able to document capture of oversight 75% of the time. A second memorandum was sent out by the associate dean for GME, reminding residents to adhere to use of the templates. Along the way, the design of the templates was optimized to improve workflow based on feedback from the residents, and hard stops were introduced to maximize compliance. Individual letters sent out through the GME office to noncompliant providers also led to a positive response. As a result of all these efforts, compliance with the NoteWriter templated note system improved to the mid to upper 90% range and maintained a high level over the following 2 years. At this time, we are able to document oversight of our residents for non–formal procedural areas an average of 95% of the time via automated reports available to the GME and operational leadership.

The GME team is able to access this report and use it for a variety of purposes, including meeting the ACGME mandate, and also to report on procedures performed by residents, the types of procedures residents are performing, indications for procedures, complications, and specific names of supervising physicians.

The supervision of residents has often been left in the hands of the residency program director. Methods of ensuring adequate and appropriate supervision include surveying of residents and program directors,^26–29 direct observation of a select cohort of attending-resident interactions, retrospective audit of patient encounter documentation,³⁰ and use of personal procedural logs.^7–9

The discordant expectations of supervising attendings and residents on appropriate levels of supervision have previously been reported.⁶ Farnan et al. reported that attending physicians often thought they were not given the opportunity to be more involved in resident supervision, while residents felt that their attendings’ micromanagement reduced their autonomy in clinical decision-making. Parkland practices escalated levels of responsibility based on the achievement of competencies under supervision. It is expected that the level of supervision will often decrease as the resident achieves additional competencies and progresses through residency. An objective measure to chart a resident’s progress would facilitate discussions on the appropriate levels of supervision between a resident and a supervising attending. While this could be achieved by keeping a personal log of procedures performed, the NoteWriter templated note system enables generation of resident-specific reports without additional effort on top of standard patient care.

The use of a templated note system is not new. It has been shown previously to improve documentation of patient comorbidities and complications, before the era of EHRs.³¹ With EHRs, templated notes have been shown to increase completion of advance directive discussion notes.³²

With the previous system of documenting procedures in free text, consistency was lacking in the documentation of items such as the level of supervision, privilege to perform, etc. These procedural notes were also not easily auditable due to their free-text nature vs the coded system with templates and hard stops.

The feedback letters to the residents from the office of the GME were also useful, as they often led to discussions of opportunities for improvement in the process as well as feedback to the GME membership on standard practices. We experienced the greatest improvement in compliance after initiating this action. Residents seemed to appreciate the communication and took the opportunity to respond to the emailed letters frequently. It was this engagement that helped fine-tune the template. Not only was this benefit of great value, it had minimal impact on resources within the institution, averaging 0.83 letters per 100 residents weekly, using a templated letter. This was felt to be very tolerable by the GME staff.

Lastly, the use of templated notes along with our process of monitoring and providing feedback on compliance can allow for standardization of many other aspects of patient care that may be important to an organization, including regulatory items, graduate medical education, quality, or performance improvement. This system lends itself to all. The system we have described here can be easily implemented at any institution that utilizes an EHR platform with templated notes functionality. Keys to our success included extensive planning involving the various stakeholders, the availability of timely and comprehensive reports, and understanding the strengths and limitations of the EHR.

Limitations

This study was limited by the quality of data entered by resident physicians. While prompts and hard stops were put in place to ensure completion of the procedure note, there was nothing stopping residents from not documenting procedures at all. If a procedure was incorrectly documented as a note type other than “procedure” (eg, “progress note”), this would not be identified by the reporting system as an error. However, if the procedure was incorrectly documented in the traditional manner as a nontemplated, non–NoteWriter procedure note, the system would pick this up and flag the procedure as having been documented without using the template. This was the method used to calculate compliance with NoteWriter template usage (Figure 2A), which might be an underestimation of the true compliance.

Another limitation is that this system relies on the resident’s account of the procedure and the level of supervision required and received, thus reporting bias may occur. We feel that the residents are, by and large, honest in their documentation. Parkland uses a third-party product to capture resident procedure logs. Access to reports of residents’ proficiency in procedures is maintained outside the EMR via an intranet hyperlink from the hospital portal page. If a nurse or physician is unfamiliar with a particular resident’s qualifications, a quick online check will provide the necessary information. Likewise, the co-signing supervising physician would be able to identify any discrepancies between the level of supervision received and the level of supervision deemed necessary. There was also the possibility of errors in the assignment of the supervising physician by the resident. A countermeasure of these is that all procedure notes were sent to supervising attendings for co-signature. If misattributions were made, the co-signing attending would be made aware. While not foolproof, we believe that the additional layer of checks by forced co-signature would minimize errors and misattributions.

One may also find limit in that we implemented this templated standardized note in just one institution, and enforced it only in the inpatient aspect of patient care. While it was implemented in an enterprise fashion at our hospital, thus exposing a large variety of resident specialties to the experience, one cannot guarantee the same success when doing this at another institution. It should be mentioned, however, that we have begun to deploy this solution at our sister institution, the University of Texas Southwestern Medical Center, with similar initial success.

Future research directions

We have already begun to mandate compliance with this template for non–formal procedural area procedures in the ambulatory setting. This includes our standard process of working with the DIO to send letters to errant residents. The benefits of including this system in the formal procedural areas are being considered.

Finally, this templated procedural note system can also be implemented at other institutions with ease, especially if EpicCare is in use. Collaborating with other academic partners will provide external validity and may allow larger power for analysis of more subtle questions about supervision.

We hope to aim our future direction toward looking for hard outcomes in the use of this process.² Specifically, we may look to see if there are any differences in patient outcomes, safety, quality, and overall patient satisfaction based on documented supervision by residents. The impact of this system on patient safety will need to be evaluated in a prospective manner.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sector.

COMPETING INTERESTS

The authors have no competing interests to declare.

CONTRIBUTORS

Study design: all authors

Data acquisition, interpretation, and analysis: V.S.B., C.M., B.M.

Drafting manuscript: V.S.B., B.M.

Critically revising manuscript: all authors

Approval of final manuscript for submission: all authors

Study supervision: B.M.