Relationship Between Nursing Documentation and Patients’ Mortality

Abstract

Background

Nurses alter their monitoring behavior as a patient’s clinical condition deteriorates, often detecting and documenting subtle changes before physiological trends are apparent. It was hypothesized that a nurse’s behavior of recording optional documentation (beyond what is required) reflects concern about a patient’s status and that mining data from patients’ electronic health records for the presence of these features could help predict patients’ mortality.

Methods

Data-mining methods were used to analyze electronic nursing documentation from a 15-month period at a large, urban academic medical center. Mortality rates and the frequency of vital sign measurements (beyond required) and optional nursing comment documentation were analyzed for a random set of patients and patients who experienced a cardiac arrest during their hospitalization. Patients were stratified by age-adjusted Charlson comorbidity index.

Results

A total of 15 000 acute care patients and 145 cardiac arrest patients were studied. Patients who died had a mean of 0.9 to 1.5 more optional comments and 6.1 to 10 more vital signs documented within 48 hours than did patients who survived. A higher frequency of comment and vital sign documentation was also associated with a higher likelihood of cardiac arrest. Of patients who had a cardiac arrest, those with more documented comments were more likely to die.

Conclusions

For the first time, nursing documentation patterns have been linked to patients’ mortality. Findings were consistent with the hypothesis that some features of nursing documentation within electronic health records can be used to predict mortality. With future work, these associations could be used in real time to establish a threshold of concern indicating a risk for deterioration in a patient’s condition.

Early recognition of patients’ deteriorating condition followed by effective communication and response by members of the interdisciplinary care team leads to decreased hospital mortality and is one of The Joint Commission’s National Patient Safety Goals.^1–3 Rapid response systems (RRS) aim to decrease barriers to such recognition, communication, and response. Despite the logical need for an RRS, their impact on patients’ outcomes has been inconsistent, possibly because of the imprecise methods and measurements used to recognize at-risk patients and communicate clinical concerns.^4–10 For example, electronic alerts based on vital sign measurements suffer from high sensitivity and poor specificity because of the natural variation in physiology and the difficulty in estimating accurate reference ranges.^1,11–15 Patients who benefit most from RRS often are not undergoing intensive monitoring of vital signs. However, nurses, more than any other health professional, have consistent direct contact with patients.¹⁶ This position enables nurses to detect subtle changes in patients’ condition and may be a factor in nurses’ use of the established and valid, yet poorly defined, criterion of “seriously worried about a patient” to initiate an RRS more than other health professionals.^16,17

In our prior work, we found that nurses use optional documentation in electronic health record (EHR) flowsheets to contextualize and highlight abnormal clinical data and to record their concerns and increased surveillance of patients.¹⁸ Optional documentation is defined as recording vital signs more often than the required minimal frequency and entering free-text comments associated with clinical measurements in EHR flowsheets. These optional documentation strategies were associated with patients’ status and outcomes.¹⁸ Based on our prior findings that the behavior of recording optional documentation is a feature of a nurse’s concern about a patient’s status, we hypothesized that these features of optional documentation within EHR data could be used to predict patients’ mortality.

Within the field of data mining, the term feature refers to the explanatory variables being analyzed. In other words, a nurse who enters vital signs for a patient every hour, when the required minimal frequency is every 4 hours, or documents comments in a patient’s flowsheet, most likely has a reason for doing this extra work. We think that this extra work reflects increased concern and surveillance of a patient perceived to be at risk of deteriorating condition or death. EHRs enable documentation patterns that have clinical significance to be detected and alerts to be sent to care providers. Therefore, the aim of this study was to identify the association between nurses’ optional EHR documentation and hospital mortality for a set of risk-stratified acute care patients.

Background

Clinicians document subjective assessments and concerns that inform complex decision making, and the analysis of these assessments and concerns may help identify and detect system weaknesses.^7,16–21 Physicians seek patient information from flowsheets more than any other clinical documentation.²² In our prior work, we found an association between optional documentation and mortality for cardiac arrest patients.¹⁸ In this study, we further investigate that finding by providing a more detailed analysis and controlling for acuity level in a broader population to demonstrate features of EHR nursing data that may be used to predict patients’ mortality. Most RRS calls are initiated because of a nurse’s concern,⁴ which is based on clinical knowledge, experience, and intuition and should supersede waiting to act until physiological criteria are met.^16,23 However, nurses’ articulation of their complex decision-making strategies and pattern recognition is often mismatched with physicians’ expectations for physiological data changes communicated by using only medical language, and this mismatch leads to delayed treatment.¹⁷

Our hypothesis that nurses’ behavior of recording optional documentation is associated with the outcome of patients’ mortality was informed by Donabedian’s framework of structure-process-outcome.^24,25 Donabedian’s framework posits that outcomes measurement must always take into account contextual factors, such as clinician behavior, as well as interventions. Specifically, Donabe-dian’s framework is consistent with motivations for mining EHR data. In this context, we analyzed the structure of EHR flowsheet documentation and requirements (vital signs and free-text comments), the process of flowsheet documentation (nurses’ behaviors in recording vital signs and free-text comments above and beyond documentation requirements), and the outcomes (cardiac arrest and mortality).

The context of human behavior is a major determinant of a behavioral intervention’s success (eg, rapid response teams, cardiac resuscitation) and cannot be controlled for within an experiment.²⁶ Clinicians’ behavior is central to the care process and an essential link between process improvement interventions and better outcomes.²⁶ In contrast, the fixed properties and functions of biological interventions interact in a consistent manner to affect outcomes. When an experiment is not feasible, such as assessing the impact of clinicians’ documentation behaviors on cardiac arrest outcomes, rigorous retrospective analysis of EHR data allows us to ferret out the relationship between behaviors and outcomes.

Methods

We collected all of the electronic nursing documentation within 48 hours of admission for 15 000 randomly selected acute care patients. The 15 000 patients were randomly selected from a set of 35 848 acute care patients who had not had a cardiac arrest and met the inclusion and exclusion criteria. We included all acute care adult patients (>18 years of age) hospitalized between September 2008 and December 2009. We excluded cardiac arrest patients and patients admitted directly to the intensive care unit, although patients may have had an unanticipated admission to the intensive care unit at some point during their hospitalization. We also analyzed documentation for acute care full-code cardiac arrest patients in the 48 hours preceding their cardiac arrest for admissions between September 2008 and December 2009 (n = 145).

We used data-mining methods to detect associations between the explanatory variables of optional documentation and the target (outcome) variable of patient survival. Specifically, we define optional documentation as 2 variables: (1) recording of vital signs more often than the required minimal frequency specified by clinical policy, (2) recording of the number of free-text comments associated with a clinical measurement in an EHR flowsheets. For our data mining analysis, the count (number of times) that the explanatory variable was recorded on the flowsheet was analyzed. We did not analyze the value of the vital sign or the content of information entered in the comment in this analysis, as this was completed in our prior work and was not the aim of this EHR data-mining study. IBM SPSS(c) statistical software (originally named Statistical Package for the Social Sciences) was used.

To control for different levels of acuity, we stratified patients by risk.²⁶ To calculate a risk score, we used the age-adjusted Charlson comorbidity (AACC) index, which has been well validated in a number of populations and uses ICD-9 (International Classification of Diseases, 9th revision) codes to predict the risk of death from 17 comorbid diseases in a 10-year period (Table 1).^27,28 The ICD-9 code for cardiac arrest, one of the measured outcomes in this study, is not included as a factor in the risk-score calculation. We used Deyo’s algorithm, the most widely cited,^29,30 to calculate the score and added 1 point for each decade of life beyond age 50 to account for the effect of increasing age.²⁸

Table 1.

Seventeen comorbid diseases used in calculating age-adjusted Charlson comorbidity (AACC) index^a

Acute myocardial infarction	Diabetes
Congestive heart failure	Diabetes complications
Peripheral vascular disease	Paraplegia
Cerebral vascular accident	Renal disease
Dementia	Cancer
Pulmonary disease	Metastatic cancer
Connective tissue disorder	Severe liver disease
Peptic ulcer	HIV infection
Liver disease

^aBased on information from Extermann,²⁷ Charlson et al,²⁸ and Dayo et al.²⁹

Comparisons were made on the basis of the mean difference in the frequency of vital sign measurements and number of comments recorded for acute care patients who died versus acute care patients who survived. We also analyzed the difference in the distribution of comment documentation for the population of 15 000 acute care patients who did not have a cardiac arrest versus the population of 145 cardiac arrest patients. A 95% confidence interval and an independent-samples 2-tailed t test with equal variances and sample size not assumed were used to assess for mean difference. Additionally, the Kolmogorov-Smirnov (KS) test was used to account for the variances among the graphs of the distributions of the populations. We calculated the Mann-Whitney U to strengthen confidence in our t-test results. Finally, because our data were stratified by AACC index, we performed a post-hoc power analysis for each t test by using G-Power software for each stratified sample to determine if the sample size was large enough to detect a statistically significant difference. Approval was obtained from Columbia University’s institutional review board (Protocol: IRBAAA-D8121).

Results

A total of 15 000 randomly selected acute care (non–cardiac arrest) patients and 145 cardiac arrest patients met the inclusion and exclusion criteria. Table 2 provides the baseline characteristics for the acute care and cardiac arrest populations and the stratification by AACC index. The populations differed significantly; therefore, we stratified the populations by AACC index to compare for differences in clinical status when comparing populations. For the acute care patients, the most frequent AACC index was 0, with 38.6% of patients falling into that category (Table 2). Most cardiac arrest patients (70%) had an AACC index of at least 6. The post-hoc power analysis indicated that some of the AACC stratified samples, specifically for the cardiac arrest patients, were not large enough to detect statistically significant differences. Tables 3 and 4 include statistically significant results for AACC stratified samples with a power greater than 40%.

Table 2.

Baseline characteristics and age-adjusted Charlson comorbidity (AACC) index

Baseline characteristics	Acute care patients (n = 15 000)	Cardiac arrest patients (n = 145)	P
Male sex, No. (%)	6099 (40)	83 (57)	<.001a
Length of stay, mean, d	5.7	24.4	<.001b
Age, mean, y	55	67	<.001b
AACC index, mean (range, 0–37)	2.25	8.7	<.001b
AACC index	No. (%) of patients
	Acute care patients (n = 15 000)	Cardiac arrest patients (n = 145)
0	5799 (38.6)	5 (3.4)
1	2093 (14.0)	4 (2.8)
2	2127 (14.1)	5 (3.4)
3	1886 (12.6)	5 (3.4)
4	1338 (9.0)	11 (7.6)
5	498 (3.3)	13 (9.0)
≥6	1259 (8.4)	102 (70.3)

^aZ test.

^bt test (2-tailed).

Table 3.

Comment documentation^a for acute care patients who died versus patients who survived

AACC index	No. of patients		No. of comments, mean (SE)		Mean difference (95% CI)	P		Power, % (effect size)c
	Died	Survived	Died	Survived		From t testb	From KS
0	16	5783	1.69 (0.82)	0.56 (0.02)	1.1 (−0.63 to 2.9)	.19	.25	42.0 (0.44)
1	27	2066	2.63 (0.91)	0.85 (0.04)	1.8 (−0.1 to 3.7)	.06	.33	73.0 (0.50)
2	52	2075	1.92 (0.39)	0.99 (0.04)	0.9 (0.15 to 1.7)	.02	.08	77.6 (0.38)
3	50	1836	2.10 (0.42)	0.96 (0.04)	1.14 (0.3 to 2.0)	.01	.02	88.7 (0.45)
4	47	1291	2.13 (0.51)	0.95 (0.05)	1.18 (0.14 to 2.2)	.03	.09	81.0 (0.43)
5	28	470	2.43 (0.73)	0.92 (0.09)	1.5 (0.003 to 3.0)	.05	.24	72.5 (0.50)

Abbreviations: AACC, age-adjusted Charlson comorbidity; KS, Kolmogorov-Smirnov test.

^aComment documentation count is 48 hours after hospital admission. The sample size was too small for cardiac arrest patients with AACC scores of 0 to 5, so only acute care patients are included.

^bIndependent samples t test (2-tailed, equal variances not assumed).

^cPost-hoc power analysis for t test.

Table 4.

Vital signs documentation^a for acute care patients who died versus patients who survived

AACC index	No. of patients		No. of comments, mean (SE)		Mean difference (95% CI)	P		Power, % (effect size)c
	Died	Survived	Died	Survived		From t testb	From KS
2	52	2075	23.23 (3.33)	15.41 (0.23)	7.8 (1.1–14.5)	.02	.06	85.0 (0.42)
3	50	1836	22.58 (2.70)	16.45 (0.26)	6.1 (0.7–11.5)	.03	.02	78.7 (0.40)
5	28	470	24.75 (4.00)	14.72 (0.48)	10.0 (1.8–18.3)	.02	.007	87.0 (0.60)

Abbreviations: AACC, age-adjusted Charlson comorbidity; KS, Kolmogorov-Smirnov test.

^aVital sign documentation count is 48 hours after hospital admission. Vital sign documentation frequency was above and beyond the minimum standard requirement. The sample size was too small for cardiac arrest patients with AACC scores of 0 to ≥6, so only acute care patients are included.

^bIndependent samples t test (2-tailed, equal variances not assumed).

^cPost-hoc power analysis for t test.

The mean differences in comment documentation for patients who died versus patients who survived are shown in Table 3. Patients in each low and moderate comorbidity risk category (AACC index, 0–5) who had increased comment documentation within the first 48 hours of admission were more likely to die. For example, the mean difference in comment documentation for patients who died versus patients who survived ranged from 0.9 more comments for patients with an AACC of 2 who died to 1.5 more comments for patients with an AACC of 5 who died. The results of the Mann-Whitney U test were consistent with our t-test results across each stratified sample.

The mean differences in vital sign documentation for patients who died versus survived are shown in Table 4. Patients with AACC indexes 2, 3, and 5 who had increased vital sign documentation were more likely to die by discharge. The t test and Mann Whitney U were significant for patients with AACC indexes of 3 and 5 for vital sign documentation, and there was a significant difference in the distribution of vital signs for patients who died and patients who survived (Table 4). All mean frequencies of vital sign documentation were beyond the minimum requirement for nurses to document vital signs (typically every 4–8 hours). Of note, patients with an AACC index of 3 had a significant difference in the distributions of both vital signs and comments for patients died and survived (P < .02, KS test; see Tables 3 and 4).

The distribution of comments for acute care non–cardiac arrest and acute care cardiac arrest populations differed, with a higher proportion of cardiac arrest patients having more comments (P < .001, KS test; see Figure). For example, about 95% of non–cardiac arrest patients had 5 or fewer comments documented, but only about 85% of cardiac arrest patients had 5 or fewer comments documented. A similar difference existed for the distribution of vital signs between non–cardiac arrest and cardiac arrest populations (P < .01, Kolmogorov-Smirnov test). Finally, cardiac arrest patients who died had a mean of 1.3 more comments documented in the 48 hours before a cardiac arrest than did patients who survived, when acuity (AACC index ≥ 6) was controlled for (P = .03, t₈₀ = 2.2).

Figure.

Kolmogorov-Smirnov test for equality of distributional functions, P < .001.

Discussion

To our knowledge, this study is the first to analyze nurses’ documentation patterns within a health record and to tie those patterns to patients’ mortality and outcomes. Our findings are consistent with our hypothesis that EHR data contain features of optional documentation recorded by nurses that may reflect a nurse’s concern about a patient and mining of these features could be done to predict mortality. We demonstrated that patients who died had a different mean frequency of vital signs and comments documented than patients with the same comorbidity risk who survived (t test, P < .05). Furthermore, cardiac arrest patients had a higher mean frequency of vital sign and comment documentation than non–cardiac arrest patients. Additionally, cardiac arrest patients who died had a higher mean number of comments documented than did cardiac arrest patients who survived (t test, P = .03).

We know clinicians do not document randomly, and our findings indicate that nurses’ documentation has a signal that gives information about the state of the patient and could be useful in predicting cardiac arrest and death. Patients’ mortality has been associated with nurses’ and physicians’ clinical judgment of illness severity upon admission,³¹ and we think that the signal of increased optional documentation is a marker of the nurses’ clinical judgment to increase surveillance and monitoring for patients deemed to be at risk of deterioration. The future applications of this research are broad; mining of EHR flowsheets to determine risk scores for patients is technologically feasible and natural language processing of comments may enhance knowledge development from flowsheet data.

Evidence of the clinical utility of predictive risk scores that use EHR data is lacking. Most risk calculation scores do not include nursing assessment data, beyond the raw physiological data. Perhaps current risk scores remain inadequate to predict patients’ outcomes because they do not account for the knowledge in a nurse’s assessment. The knowledge generated from this study sets the stage for future research with more targeted questions. For example, further work should be done to establish if the measurement of concerns documented by clinicians are more useful for predicting at-risk patients than existing predictive scores. This preliminary, descriptive study opens a new area of investigation that is becoming possible because of EHRs and raises the possibility of refining EHR systems to facilitate and connect comprehensive assessment of patients’ status and subtle changes to improve outcomes.

The weakness and shortcomings of nursing documentation stem from difficulties in finding important information because of a huge amount of routine notes.³² Physicians seek patient information from flow-sheets more than any other clinical documentation; yet, expert physicians have low rates of agreement on a common set of relevant flowsheet data that would support a particular diagnosis.^22,33 The design of intelligent digital displays of patient information to support clinical care and decision making assumes knowledge of the most important data elements.^34–36

We found that the existence of optional documentation is statistically correlated with the clinical state of the patient. This finding highlights the importance of considering documentation requirements when analyzing EHR data. We did not study required documentation versus optional documentation; however, we found that optional documentation has information correlated with the patient’s hospital mortality. We think that optional documentation increases as a nurse’s concern for the patient’s condition increases. Further research should be done to assess detection of signals from required documentation and when clinicians “document by exception” (ie, document only abnormal findings). Our prior work on physicians’ note quality highlighted the importance of salience as a characteristic of high-quality, useful notes.³⁷

In the inpatient setting, nurses are responsible for most EHR documentation.³⁸ Awareness of nurses’ documentation behaviors is essential to accurately interpreting EHR data-mining research and its implications in the clinical setting. Specifically, documentation policies in each clinical setting from which the EHR data are collected are most likely important biases to consider when analyzing EHR data. Our findings indicate a novel and promising area of EHR data-mining research—the documentation behaviors and processes that are associated with the clinical state of the patient.

Specifically, with future work, early detection of behaviors and processes associated with patients’ outcomes might allow us to decrease the time-to-response by an RRS. Many hospitals that have implemented an RRS have seen improved outcomes for patients, such as decreasing cardiac arrest rates outside of the ICU and ensuring that patients receive the appropriate level of care in a timely manner. This research begins the discussion of whether RRS teams are maximizing their resources to identify and prevent patients’ complications. Perhaps with future work, there may be added value to using a structured scoring system based on EHR data to increase clinicians’ awareness of who to watch closely and when to initiate an RRS call. Nursing documentation behaviors may be a useful variable to add to such a risk measurement score that informs the health care team of the clinical state of the patient.

Limitations

This study was a retrospective data-mining analysis conducted at a large urban teaching hospital in New York City that uses a commercial EHR. The power to detect differences between some cardiac arrest groups was limited because of risk stratification. The comparison of non–cardiac arrest and cardiac arrest populations is limited by the different time periods for EHR data collection.

Conclusion

This is the first study to link nursing documentation patterns to patients’ mortality outcomes. Our findings were consistent with our hypothesis that some features of nursing documentation behavior within EHR data may reflect a nurse’s concern about a patient and can be mined to predict mortality. We analyzed nurses’ optional flowsheet documentation for associations with mortality for a risk-stratified sample of acute care patients. Our findings indicate statistically significant associations between increased optional documentation of vital signs and comments and mortality for cardiac arrest patients with high comorbidity indexes. Patients who had increased optional documentation of vital signs and comments experienced significantly higher rates of mortality. Further analysis of such patterns may be useful for the measurement of patients’ risk for deteriorating condition and the design of interventions to recognize and mitigate such deterioration.