Mortality in the Neonatal Intensive Care Unit: Improving the Accuracy of Death Reporting

Abstract

Objective:

Death certificates commonly contain errors, which hinders understanding of infant mortality. We therefore undertook a quality improvement (QI) initiative to improve death reporting in our NICU.

Study design:

After our baseline assessment (January 1, 2015 to June 30, 2017), we implemented our QI initiatives using Plan, Do, Study, Act tests of change. We prospectively reviewed death certificates (July 1, 2017 to December 31, 2019) to evaluate the impact of our interventions.

Results:

The overall proportion of incorrect death certificates significantly decreased from 71% to 22% with special cause variation noted after the second PDSA cycle. The most common errors involved inaccurate or incomplete reporting of prematurity and errors in the sequence of events.

Conclusion:

Through a series of PDSA cycles focused on formal provider education and ongoing review, we significantly reduced inaccurate death reporting. These interventions are generalizable across NICUs and important to improve public health reporting accuracy.

Introduction

While nearly all providers in a neonatal intensive care unit (NICU) will complete a death certificate at some point, formal training on how to do so is rarely provided ^1–3. Though this deficit has been described previously in physicians caring for adults ^{4, 5}, fewer studies have evaluated the scope and nature of this problem in pediatrics, where deaths occur less frequently. When queried, pediatric residents have reported low rates of instruction in this process ². As a result, death certificates are often inaccurately completed both in the United States and internationally ^{4, 6–14}.

Because mortality statistics are generated using International Classification of Diseases, 10^th Revision (ICD-10 ¹⁵) codes assigned from these death certificates ^{16, 17}, failure to properly complete the death certificate leads to inaccuracy in mortality statistics ¹². This in turn may lead to incorrect estimates of healthcare disparities ¹⁸ and survival rates for certain conditions ^{5, 19} and by extension misdirected research funding and effort. We ²⁰ and others ^{14, 21} have shown that death certificates for infant deaths frequently contain non-specific or inaccurate information or omit important diagnoses ²². As accurate reporting is critically important in order to fully understand causes of neonatal mortality and ultimately improve neonatal outcomes ²³, we undertook an effort to reduce these inaccuracies in our neonatal intensive care unit.

Educational initiatives have previously been shown to improve the ability of medical providers to complete death certificates ^{4, 24} and have resulted in changes in mortality statistics ⁵. Reasons for inaccurate death certificate completion may include i) provider uncertainty regarding the cause of death, ii) lack of familiarity with the decedent, iii) failure to prioritize accurate death reporting as an important component of end-of-life care and iv) lack of education on how to properly complete the death certificate (Figure 1) ¹³. We therefore designed a series of interventions incorporating education, discussion, and “just in time” reminders aimed to improve death reporting. Prior to implementing our QI initiative, we found that a high proportion (72%) of death certificates were incorrect, with major inaccuracies including errors in the sequence of events leading to death, non-specific causes listed as the cause of death without further clarification, underlying cause of death not specified, and errors related to reporting of prematurity. The specific aim of this project was therefore to reduce the percentage of improperly completed death certificates in our NICU from our baseline 72% to 25% or less within two years, a goal which was thought to represent a substantial but achievable reduction in our error rate.

Figure 1.

Cause and effect diagram. The causes addressed by this QI project are highlighted by the shaded boxes.

Methods

This QI project was performed in a level IV NICU. Providers certifying deaths in our NICU include neonatal nurse practitioners (NNPs), neonatal-perinatal medicine or surgical critical care fellows, and attending neonatologists. To establish our baseline data, we reviewed all death certificates from our NICU for 10 quarters in 2015–2017 (January 1, 2015 to June 30, 2017). We prospectively reviewed death certificates by quarter over the next 10 quarters (July 1, 2017 to December 31, 2019) to evaluate the impact of our interventions. Our project followed the Institute for Healthcare Improvement Model for Improvement QI framework ²⁵.

Intervention

Our Plan-Do-Study-Act (PDSA) cycles sought to address the lack of education regarding death certificate completion as a primary contributing factor to inaccuracy in death reporting, in addition to raising awareness about the importance of this task (Figure 1). Our first intervention was an educational presentation given to providers who certify deaths in our unit, with content based on several publications reviewing optimal training modalities and publically-available protocols for death certification ^{6, 12, 26–28}. We reviewed how to complete a death certificate and common errors seen in our NICU directly drawn from our baseline data as well as prior studies (Figure 2) ^{4, 8, 11}. We also presented hypothetical cases and had participants practice composing mock death certificates. The presentation has been repeated yearly as new fellows enter our program to ensure that all NICU fellows have had this training at some point during their fellowship.

Figure 2.

Common errors on death certificates.

Our second intervention involved adding a “quick tips” sheet to the death certificate paperwork on our unit (Supplemental Figure 1). Providers could use this as a reference when completing death certificates if key points discussed during the educational presentations had been forgotten. A third intervention was to add discussion of the completed death certificates to our monthly NICU morbidity and mortality (M&M) conferences. After review of each mortality case, the death certificate is presented and discussed by all participants at this meeting. Any errors are reviewed and alternate ways to report the series of events leading to death are suggested. This allows for continued education that is interactive and provider-driven. Such repeated “audits” and feedback have been shown to improve death certificate quality ²⁶. A fourth intervention continued the mission of the first intervention by presenting to the NICU nurse practitioners (NPs) who also complete death certificates in our unit.

As errors related to reporting of prematurity continued to be high despite the above interventions during our first PDSA cycles, we subsequently added a note to each blank death certificate worksheet reminding providers to report a history of preterm birth and possible contributing maternal factors when these were thought to be relevant to the infant death.

Study of the Intervention

Our main outcome measure was the percentage of death certificates incorrectly completed, calculated per quarter. We systematically reviewed death certificates completed in our NICU and classified the errors into 5 main types, using error types similar to those in prior studies (Figure 2) ^{4, 8, 11}. As a review of the patient’s medical record was not undertaken as part of this project, we did not pass judgment on whether or not an alternate medical condition should have been chosen as the underlying cause of death. Three independent reviewers (MHW, KTL, AH), including one (AH) who was blinded to pre- or post-intervention status, assessed each cause-of-death statement and gave an overall designation of “incorrect” or “correct” by where “correct” cause-of-death statements had zero errors. All reviewers made independent assessments, which were then compared and differences highlighted. Disagreements between error types and the classification of death certificates were then discussed among the three reviewers until a consensus was reached. Of the 94 cases reviewed, only 13 (14%) had disagreements that needed to be resolved.

In addition to comparing the rates of overall correct or incorrect death certificates, we also compared the rates of each specific error before and after the intervention. Finally, in order to evaluate for factors that may influence our error rates, we recorded each infant’s age at death and time of death and analyzed differences in these variables.

As a process measure, we administered a questionnaire designed to evaluate the effectiveness of our educational presentation. Participants completed a brief set of questions both before and after the presentation, designed to assess both knowledge and satisfaction, and responses were compared.

Finally, we evaluated the impact of our intervention on ICD-10 codes assigned at the National Death Index (NDI) for each infant death in our cohort. These ICD-10 codes are used to generate national mortality statistics and thus this analysis allowed us to estimate the broader impact of our intervention. Data were obtained from the NDI following their policies ²⁹ and included both the underlying cause-of-death code for each infant as well as the entity and record-axis codes used to generate multiple cause statistics. We reviewed whether the ICD-10 code chosen as the underlying cause-of-death for each infant was accurate and whether reporting of prematurity on the entity axis increased after our intervention.

We created a control chart to evaluate for special cause variation based on standard control chart rules. Data were displayed over time in a statistical process control (SPC) p-chart and were presented as a rate with the numerator of the number of accurate death certificates and denominator of total number of death certificates per month. We reviewed these data quarterly to monitor the effects of our intervention on our outcome measure. We also compared the overall proportions of incorrect death certificates and specific error types pre- and post-intervention using a Fisher’s exact test. Descriptive and comparative statistics were performed using SPSS version 24 (IBM Corporation, Armonk, NY) and control chart analysis was performed using Chartrunner software version 3.6 (PQ Systems, Dayton, OH).

Ethical Considerations

This protocol was submitted our Institutional Review Board and was deemed Non-Human Subjects Research as the study subjects were deceased. This manuscript was prepared using the SQUIRE guidelines ³⁰.

Results

Our initial educational presentations were well-received, with all post-test takers responding “Yes” to the question “Did this workshop improve your ability to complete a cause-of-death statement”. These presentations also improved provider knowledge as reflected in an increase in the number of correct responses to two questions related to death certificate completion; an improvement in provider confidence was also seen (Supplemental Figure 2).

To evaluate the impact of our PDSA cycles, we reviewed a total of 94 deaths that occurred in our NICU during the baseline and post-intervention time. Demographic characteristics are presented in Table 1. Our control chart (Figure 3) illustrates the decline in the proportion of incorrectly-completed death certificates in relationship to our PDSA cycles. As our interim analyses revealed that the most common error in the incorrect death certificates was omission of details pertaining to preterm birth, we added our fourth intervention. With this, we successfully achieved our primary aim. Statistical process control p chart analysis shows special cause variation after October 1, 2017 with a significant decrease in the mean error rate from 71% to 22% (Figure 3). Greater than eight points were below the baseline mean indicating special cause variation and significance.

Table 1.

Demographics.

	Total N = 94	Pre-Intervention N = 46	Post-Intervention N = 48	p
Sex (male)	52, 55%	27, 59%	25, 52%	0.54
Age at death (days)	26.5, 7–72	25, 6.75–54.5	31, 9.25–88.75	0.58
Prematurity	63 (67%)	27 (59%)	36 (75%)	0.13
Gestational age at birth (weeks)	32, 26–37.5*	32, 26–38	31, 26.25–36	0.35

Data presented as percentage or median (IQR)

^*Missing data for 1 full-term infant (pre-intervention cohort).

Figure 3.

Control chart, with interventions noted in boxed text. The data are displayed over time in a statistical process control p-chart, where the data represent a rate with the numerator of the number of accurate death certificates and denominator of total number of death certificates per month

In comparing the 10 quarters prior to our first PDSA cycle (Q3, 2017) to the 10 quarters that followed, the demographic features of the cohort (Table 1) remained constant even though the proportion of death certificates completed incorrectly fell significantly. The most common errors types involved inaccurate or incomplete reporting of prematurity or errors in the sequence of events, and these were significantly reduced after our intervention as well (Table 2).

Table 2.

Errors in death reporting

	Pre-Intervention N = 46	Post-Intervention N = 48	p
Overall Incorrect
	33 (72%)	12 (25%)	<0.001
Type of Error (rate)
Mechanism as immediate cause of death	3 (7%)	1 (2%)	0.36
Error in the sequence of events or competing causes	14 (30%)	4 (8%)	0.008
Nonspecific process included without clarification	1 (2%)	0 (0%)	0.49
Underlying cause mistakenly entered as contributing cause	2 (4%)	1 (2%)	0.61
Prematurity reporting error*	25/27 (93%)	7/36 (19%)	<0.001
Prematurity omitted	7/27 (26%)	1/36 (3%)	0.01
Gestational age omitted**	17/18 (94%)	6/28 (21%)	<0.001
Reason preterm omitted**	17/18	4/28 (14%)	<0.001

^*only counting cases where the infant was born at <37 weeks gestational age

^**only counting cases where prematurity was reported

Across the entire cohort, we found no significant differences in the proportion of incorrect death certificates when comparing deaths occurring at night versus during the day (22/51 [43%] versus 23/43 [53%] incorrect, p = 0.41) or when comparing deaths occurring before or after 28 days of age (23/49 [47%] versus 22/45 [49%], p = 1.00), suggesting that time of death or age at death were not strong drivers of inaccurate death certificate completion.

ICD-10 data obtained from the NDI were available for 88/94 infants (six cases missing in the post-intervention group) and there was no significant difference between the proportion of ICD-10 codes that correctly identified the underlying diagnosis pre- or post-intervention (27/46 [59%] versus 28/42 [67%], p = 0.51) or in the proportion of cases where an ICD-10 code indicating prematurity was included in mortality statistics (18/27 [66%] versus 25/31 [80%], p = 0.25).

Discussion

After educating our providers on proper completion of death certificates and incorporating a discussion of death certificates into our monthly M&M conferences, we were able to achieve our goal of reducing our percentage of incorrect death certificates from 72% to 22%. Errors in the sequence of events were found to be a common source of error on our death certificates, as has also been seen in prior studies ^{4, 10, 11}. Interestingly, this has previously been attributed to the complexity of adult presentations, theorizing that the misreporting is related to the multitude of conditions that older adults acquire prior to death. We identified high rates of inaccuracy in the baseline data from our NICU, where infants generally die from a single underlying condition, and therefore focused our educational efforts on streamlining and simplifying descriptions of infant causes of death. Although the percentage of incorrect death certificates began to fall after our initial presentations, likely related to recognition of common errors in addition to increasing focus on death certificates, the sustained decrease is attributable to our monthly review of death certificates as part of our M&M conferences. Due to the rotating attending physician and fellow team in our NICU and the relative infrequency with which each individual provider completes a death certificate, constant reminders of common errors and how to avoid them are needed to sustain a positive change. Particularly, we identified in our pre-intervention data that errors in the sequence of events were most common in our NICU, and reading through the sequence at our M&M conferences helps to remind our providers how a logical series of events should be presented.

Although prior publications have also demonstrated a positive effect on death certificate accuracy following educational interventions, there are several notable differences both in the design of our intervention and assessment of its impact. The first is the population targeted by our quality improvement initiative which, to our knowledge, represents the first study in a United States NICU aimed at improving neonatal and infant death reporting. Though fewer deaths occur in pediatrics compared to adult medicine, many will take place in an intensive care unit setting. Prior research has found that of all components of end-of-life care, declaration of death and completion of the death certificate were the tasks that pediatric residents felt least equipped to perform ², demonstrating a particular need for improvement in this area. Neonatologists are more likely to certify deaths than most other pediatric care providers ³¹, further supporting the importance of improving death reporting in the NICU with the ultimate goal of improving the accuracy of mortality statistics for the country.

Additionally, unique aspects of our intervention contributed to its success and sustainability. Although prior groups have used an educational model to achieve a substantial reduction in death certificate inaccuracy ^{3–5, 24, 27}, the durability of these interventions remains challenging. We address durability by repeating our educational presentation yearly for each new group of NICU fellows and by our monthly review of death certificates in M&M conferences, which not only aid in continuing education but also in providing some degree of accountability for what is written on the death certificate.

Finally, although many prior studies have shown that formal instruction improves the ability of providers to properly complete a death certificate, they vary widely in the methodology used to assess the impact of these educational interventions, with some studies using a hypothetical case scenario that is then graded ^{1, 27} and others reviewing actual death certificates ⁴ or the resultant mortality statistics ^{3, 5}. We incorporated all of these approaches into our assessment, beginning with hypothetical cases presented before and after our educational presentation, then evaluating the death certificates completed during our PDSA cycles with comparison to our baseline data, and finally evaluating the effect on mortality statistics. This allowed us to fully examine the process of death certification at our institution and the public health impact of our intervention. Though it has been previously noted in hypothetical circumstances that incorrect cause-of-death statements would impact mortality statistics ³², and a population-based analysis found that errors in actual death certificates compared to ideal death certificates would alter the underlying cause of death reported in national statistics 60% of the time ¹², few prior studies have examined the impact of educational interventions on actual mortality statistics. An educational initiative launched in New York to specifically target hospitals that seemed to have higher-than-normal rates of death due to cardiac disease was successful in reducing deaths attributed to heart disease and increasing the number of reported conditions ³. A follow-up study replicated this result, demonstrating that education in death certification completion decreased discordance between medical record data and cause-of-death and increased the percentage of deaths attributed to sepsis, thought to be an under-reported cause of death ⁵. In our study, it appeared that our intervention did improve the accuracy of resultant ICD-10 codes, though this was not statistically significant likely owing to our small sample size and missing data for the post-intervention group.

One limitation to our study is that we did not judge the clinical accuracy of the diagnoses reported on the death certificate, similar to prior research relating specifically to the accuracy of death reporting rather than the accuracy of the clinical diagnosis ⁴. However, as the deaths reported in our study are generally relatively straightforward in terms of etiology due to the short life of the decedent, this is likely less of an issue in our study as it would be in studies in the adult population, where co-morbidities are more complex and numerous. An additional limitation is the subjectivity inherent in classifying death certificates as “correct” or “incorrect”; there may be more than one correct way to complete a death certificate. We addressed this by having our third death certificate reviewer blinded as to which certificates were pre- or post-intervention (certificates were also de-identified) in an attempt to limit bias in favor of our intervention.

In conclusion, though a call for quality improvement initiatives to improve death reporting was published nearly 30 years ago ⁷, inaccuracies continue to be pervasive. The downstream effects of our efforts to improve death reporting have important implications. As an example that we confirmed, prematurity has been frequently found to be underreported on infant death certificates ¹⁴ and in national mortality statistics ^{22, 33, 34}. This omission has historically led to these deaths instead being attributed to respiratory conditions and the suggestion that respiratory care needs to improve in order to reduce infant mortality – rather than preventing preterm birth or low birth weight ³⁴. Through our ongoing efforts, we plan to continue to improve the quality of death reporting from our NICU. The interventions that we have made would be feasible to implement more broadly in an effort to better understand the etiologies of infant deaths and how they might be prevented.

Abbreviations:

NICU

neonatal intensive care unit

QI

quality improvement

NDI

national death index