An Educational Intervention to Improve Inpatient Documentation of High-Risk Diagnoses by Pediatric Residents

Abstract

Objective:

Diagnoses extracted from physician notes are used to calculate hospital quality metrics; failure to document high-risk diagnoses may lead to the appearance of worse-than-expected outcomes for complex patients. Academic hospitals often rely on documentation authored by trainees, yet residents receive very little training in this regard. This study evaluates inpatient pediatric resident notes to determine which high-risk diagnoses are commonly missed, and assesses the efficacy of a multi-tiered intervention to improve the documentation of these diagnoses.

Patients and Methods:

A baseline review of 220 charts identified 13 frequently missed high-risk diagnoses in 2013. Interventions began in 2014, including physician education and reference cards. The intervention also included note template prompts for four of the diagnoses. Using a standardized rubric, we reviewed charts for three years (2013, 2014, and 2015). The average within-disease probability of missed high-risk diagnoses was compared across time.

Results:

There was a decrease in the probability of undocumented target high-risk diagnoses after the intervention (52% versus 36% in 2014 [OR=0.51, p<0.001] and 37% in 2015 [OR=0.50, p<0.001]). Documentation of diagnoses prompted by the note template was not significantly better than those targeted by the other interventions alone (p=0.55).

Conclusions:

Pediatric resident notes were significantly less likely to omit a high-risk diagnosis after implementation of our documentation improvement program, suggesting that curriculum development is an effective method of improving documentation with the goal of improving the accuracy of health systems performance indices.

Introduction:

As pediatric hospitals work toward greater quality and patient safety, accurate outcome data is vital. Many children’s hospitals now submit this data to external sources as part of quality of care reporting and are moving toward increased transparency for patients and consumers, making outcome data for a given institution, department, or even individual provider increasingly available to the public.^1–4 Importantly, the information used to calculate quality metrics such as the mortality index is collected from physician documentation by certified medical coders as they capture diagnoses for billing. If the coded diagnoses fail to illustrate the severity of a patient’s illness due to incomplete physician documentation, aggregated quality data will not appropriately reflect the patient population’s complexity, leading to an appearance of inappropriately high mortality rates.^2,5 Based on established mathematical models, each International Classification of Diseases diagnosis is weighted according to the magnitude of its effect on hospital resource utilization and on an individual patient’s severity of illness and risk of mortality. ^1,2,6,7 If a patient has a greater number of “high-risk diagnoses” such as sepsis, pancytopenia, and thousands of others, that patient will be assigned a higher expected mortality risk. However, coders are not medically trained and cannot infer diagnoses that have not been explicitly documented by providers. For example, a physician may write that a patient is febrile, tachycardic, and has a positive blood culture, but if the term “sepsis” is not stated, it cannot be coded and will not be reflected in the patient’s estimated mortality risk.

Clinical documentation improvement (CDI) programs, whose aim is to ensure accurate documentation of these high-risk diagnoses, are now common in internal medicine and surgical fields, largely driven by clear financial incentives under Medicare payment structures. ^8–14 However, these financial incentives may not translate to pediatrics given the regional pay structure differences among children’s hospitals, so many pediatricians may not be supported by formal CDI initiatives.¹⁵ Additionally, residents often receive no formal training on CDI concepts despite being responsible for most documentation at academic medical centers.^16–18

To date there are no published studies on pediatric CDI efforts. The goal of this study was to establish a clinical documentation curriculum at our institution and to examine its efficacy in improving documentation of high-risk diagnoses, with a greater aim of improving the accuracy of our quality data.

Methods:

We performed this single-center time series study at a tertiary care academic children’s hospital within a hospital. The study was determined to be exempt from human subjects research review by the institutional review board at the University of California, Los Angeles.

In order to assess current documentation practices, we reviewed 200 charts that were randomly selected from all 1119 pediatric ward and intensive care unit (ICU) discharges between July 1 and December 31, 2013. We also reviewed all 20 mortalities from the calendar year, as these patients were presumed more likely to have high-risk diagnoses. This resulted in a total of 220 baseline charts reviewed. Eighty-four (38%) of these 220 charts were ICU patients as defined by spending time in the ICU at any point during that encounter. Four pediatric hospitalist attendings were trained in reviewing charts, specifically looking for missed high-risk diagnoses as defined by the Centers for Medicare and Medicaid Services.¹⁹ Only notes written by residents were reviewed. Of the baseline charts reviewed, 67% had one or more missing high-risk diagnosis. Diagnoses that were missed five or more times were targeted for intervention, resulting in the following list of 13 diagnoses: acidosis, alkalosis, acute kidney injury, chronic renal failure, epilepsy, heart failure, hypertension, malnutrition, neutropenia, anemia, pancytopenia, sepsis, and shock.

Interventions began in July 2014 at the start of the new academic year. Educational conferences on the importance of documenting specific diagnoses in terms that coders will recognize were presented to faculty and residents. These lectures were repeated over the course of several months in resident orientation, noon conferences, departmental quality meetings, and faculty meetings. A reference card (Figure 1) defining the target diagnoses was distributed electronically to residents and posted in their workrooms to educate them about when these diagnoses apply. The diagnoses were defined using established literature; when no clear guidelines were available, we relied upon expert consensus among subspecialists at our institution. We also revised the electronic medical record (EMR) note templates, which are used by all our pediatric residents, to include four of the target diagnoses that were easiest to incorporate without cluttering the note or overly encumbering note writers. Drop-down lists that were already present within the templates were modified to specifically mention the diagnoses of acute kidney injury, anemia, pancytopenia, and sepsis, to prompt residents to include them when applicable.

Figure 1:

Resident Reference Card for High-risk Diagnoses

A standardized rubric was created to specifically look for target diagnoses (Appendix A). Using this rubric, the four reviewers achieved over 90% inter-rater agreement on a test-sample of 10 charts prior to initiation of data collection. A total of 100 charts were then randomly selected from ward and ICU discharges between July 1 and December 31 of 2013 for our pre-intervention data, and 2014 and 2015 for post-intervention data. The 100 charts selected from 2013 were unique for this phase of the study, as we were now reviewing charts focusing only on the 13 diagnoses targeted after the baseline review. This sample size was selected based on an a priori power analysis (80% power, 5% alpha) to detect a 20% reduction in missed diagnoses. Additionally, all mortalities during those intervals were reviewed. Newborn nursery, neonatal ICU, and cardiothoracic ICU charts were excluded, as these populations have a unique set of diagnoses that are less generalizable to other patients. Patients admitted for two or fewer calendar days were also excluded, as they were less likely to have high-risk diagnoses. Charts that were found to meet exclusion criteria after chart review began were omitted without being replaced. This resulted in a total of 97 reviewed charts in 2013, 117 in 2014, and 114 in 2015 after applying inclusion and exclusion criteria. Each target diagnosis was coded as “present and documented,” “present but not documented,” or “not present” within each chart.

The primary outcome of this study was a reduction in undocumented target high-risk diagnoses after the intervention. Secondary outcomes were the efficacy of EMR templates, and whether or not improved documentation would lead to an increase in the expected mortality (i.e., the percent of patients expected to die based on illness severity) as measured by Vizient, formerly known as University Health-System Consortium.²⁰ The average within-disease probability of missed high-risk diagnoses was compared across time (pre- and post-intervention) using a mixed effects logistic regression model with disease random intercept. An interaction p-value for the differences between years was used to assess the utility of the note template. Differences in expected mortality were compared using an independent samples test of proportions.

Results:

Overall in 2013, there was a 52% probability of a target diagnosis being “present but not documented” (95% CI 0.47, 0.57), which decreased to 36% in 2014 (95% CI 0.32, 0.40) after implementation of the interventions (OR=0.51, p<0.001) (Table 1). This reduction was sustained in 2015 with a 37% probability (p<0.001, 95% CI 0.32, 0.42). As for the individual diagnoses, there was a reduction in missed documentation for three of the 13 target diagnoses (Figure 2). The results for the remainder of the diagnoses are available in Appendix B.

Table 1:

Analysis of Missed Target Diagnoses and Expected Mortality Over Time

	2013	2014	2015	p (2013 v2014)	p (2013 v2015)	p (2014v2015)
Number of missed diagnoses per chart*				0.18	0.03	0.46
Average ± 1 Standard Deviation	1.93 ± 1.13	1.72 ± 1.04	1.53 ± 0.84
Range	1,5	1,4	1,4
Probability of missed diagnosis+
Overall	0.52	0.36	0.37	<0.001	<0.001	0.94
ICU	0.49	0.43	0.40	0.35	0.13	0.59
Ward	0.59	0.38	0.39	<0.001	<0.001	0.81
Expected mortality ^	0.76%	0.78%	0.93%	0.93	0.49	0.54

^*p calculated using Wilcoxon rank-sum test

⁺p calculated using mixed effects logistic regression model with disease random intercept

^{^}p calculated using two sample test of proportions

Figure 2:

Reduction in Missed Target Diagnoses Over Time. Compared to 2013, statistically significant reduction in missed diagnoses was found for all target diagnoses combined (2014 [p<0.001] and 2015 [p<0.001]), acidosis (2015 only [p=0.04]), acute kidney injury (2014 [p=0.006] and 2015 [p=0.03]), and anemia (2014 only [p=0.04]).

The percent of the total charts reviewed per year that were mortalities remained consistent over the study interval (13%, 15%, and 16% in 2013, 2014, and 2015 respectively; p=0.89). The percent of patients with an ICU stay also remained the same (42%, 33%, and 39% in 2013, 2014, and 2015 respectively; p=0.32). When further analyzed by ICU status, high-risk diagnoses were more prevalent in ICU patients compared to ward patients (26% versus 15.7%, p<0.001). There was sustained improvement in missed diagnoses in ward charts, from 59% in 2013 to 38% in 2014 (p<0.001) and 39% in 2015 (p<0.001), but the improvement in ICU charts was not statistically significant.

For our secondary outcome measures, documentation of diagnoses included in the note template was not significantly better than those targeted by educational interventions alone (p=0.55). Expected mortality rates did increase by 22% over the study interval, but this was not statistically significant.

Discussion:

Our study showed that prior to our interventions, the probability of missing a target diagnosis was over 50%, suggesting that our patient population had a higher risk of mortality than our coded data would indicate. The educational intervention was effective at improving inpatient pediatric resident documentation of high-risk diagnoses, and sustaining improvement with ongoing education. This approach may be particularly useful for pediatric institutions without funding for formal CDI programs that may need to rely on physician-driven initiatives.

As expected, ICU patients had more high-risk diagnoses than ward patients, but the two groups were similarly susceptible to omissions in documentation; in fact, only the ward patient group showed statistically significant improvement. Thus based on our data, CDI efforts should include lower-acuity patients as well as ICU patients. We had anticipated that an increased ability to capture high-risk diagnoses would lead to an increased expected mortality, but this was not statistically significant.

Publications in adult literature have utilized similar measures such as lecture series and reference cards to achieve a 43% increase in expected mortality and increases in reimbursement up to 16%−24%.^21,22 However, many of these institutions also retain the services of a CDI team that audits notes in real time and queries physicians regarding missed diagnoses that may need to be added to their documentation.¹¹ In fact, at the time of this study, our institution’s CDI specialists were reviewing all adult inpatient documentation, but not pediatric charts due to funding limitations and the lack of perceived benefits. As a result of our work, our institution will be providing resources for a CDI team that will use our reference card to query documentation errors in real time, further bridging the gap between what physicians document and what coders interpret.

A notable strength of our study was that we evaluated undocumented and documented diagnoses as a percent of the total number of instances in which the diagnosis was clinically applicable. In this way, we were able to control for changes in patient complexity over time, in contrast to prior studies that only looked at absolute numbers of documented high-risk diagnoses, which may be confounded by increases in the prevalence of studied diagnoses over time.

A limitation of our study was its single-center design that focused solely on resident documentation, so our data may not be generalizable to non-teaching institutions. Additionally, because our educational interventions consisted of multiple elements, we cannot comment on the effectiveness of any single component. The retrospective nature of the review made it impossible to determine the “false positive” rate, i.e., the diagnoses that were documented but may not in fact have been applicable. The phenomenon of manipulating documentation to maximize reimbursement has been labeled “diagnosis-related group creep”, and is a potential adverse consequence of CDI efforts.²³ Finally, while we as physician reviewers had strong inter-rater agreement with each other, we were unable to correlate our findings with our coders as they review all physician notes, while our study only reviewed resident documentation.

Our educational interventions successfully decreased the probability of missing diagnoses by 16%; however there are still significant opportunities for further improvements to ensure that our quality metrics reflect the complexity of our patient population. Future work will expand on the current intervention to target additional diagnoses and to include attending and fellow documentation. There also may be more applications of EMR technology to assist in documentation based on existing data in other areas of the chart, such as importing physician-generated “problem lists” into daily notes to make it accessible to coders. Further studies may also assess the effect of these interventions on resident experience. Encouraging use of more precise language may help trainees recognize and treat high-risk diagnoses more promptly, but it is important to identify and prevent any adverse effects on resident workflow.

Supplementary Material

rubric A

Supplemental Appendix A: Standardized Rubric for Reviewing Charts

data B

Supplemental Appendix B: Differences in the Probability of a “Present but not Documented” Diagnosis Over Time

Abbreviations:

CDI

clinical documentation improvement

ICU

intensive care unit

EMR

electronic medical record