Validating administrative data for the detection of adverse events in older hospitalized patients

Abstract

Older hospitalized patients are at risk of experiencing adverse events including, but not limited to, hospital-acquired pressure ulcers, fall-related injuries, and adverse drug events. A significant challenge in monitoring and managing adverse events is lack of readily accessible information on their occurrence.

Purpose

The objective of this retrospective cross-sectional study was to validate diagnostic codes for pressure ulcers, fall-related injuries, and adverse drug events found in routinely collected administrative hospitalization data.

Methods

All patients 65 years of age or older discharged between April 1, 2009 and March 31, 2011 from a provincial academic health sciences center in Canada were eligible for inclusion in the validation study. For each of the three types of adverse events, a random sample of 50 patients whose records were positive and 50 patients whose records were not positive for an adverse event was sought for review in the validation study (n=300 records in total). A structured health record review was performed independently by two health care providers with experience in geriatrics, both of whom were unaware of the patient’s status with respect to adverse event coding. A physician reviewed 40 records (20 reviewed by each health care provider) to establish interrater agreement.

Results

A total of 39 pressure ulcers, 56 fall-related injuries, and 69 adverse drug events were identified through health record review. Of these, 34 pressure ulcers, 54 fall-related injuries, and 47 adverse drug events were also identified in administrative data. Overall, the diagnostic codes for adverse events had a sensitivity and specificity exceeding 0.67 (95% confidence interval [CI]: 0.56–0.99) and 0.89 (95% CI: 0.72–0.99), respectively.

Conclusion

It is feasible and valid to identify pressure ulcers, fall-related injuries, and adverse drug events in older hospitalized patients using routinely collected administrative hospitalization data. The information is relatively inexpensive and easy to access with no impact on clinical staff.

Video abstract

Download video file ^{(119.4MB, avi)}

Introduction

Every year, millions of older adults (≥65 years) receive safe and effective health care, yet there is clear evidence that some patients experience unintended harm from their care.1–3 Older hospitalized patients have a particularly high risk of experiencing adverse events. The elevated risk has been attributed to a number of factors, including the increased likelihood of more complicated illness and complex interventions, the presence of co-morbid conditions, and higher rates of hospitalization.1,2,4 An adverse event has been defined as an unintended injury or complication that is caused by health care management, rather than by the patient’s underlying disease.3 Adverse drug events, fall-related injuries, and pressure ulcers are examples of adverse events that older patients may experience while in hospital.1,2,4

Medications are the most common and cost-effective therapeutic intervention that older adults will obtain when they seek health care.5 It is estimated that more than 80% of older adults take at least one medication per day,6 and close to 40% take five or more per day, placing them at an increased risk of developing a drug-related problem.6,7 It has been reported that hospitalized older adults receive an average of eight or nine different medications during their hospital stay, including many “routine” medications, such as sedatives to promote sleep.8,9 Adverse drug events have been consistently identified in patient safety studies involving acute care.1–3,10,11 An adverse drug event is an injury that is caused by a drug, and can include injuries that result if an indicated drug is not taken as prescribed.12,13 Evidence from a report commissioned by the American Association of Retired Persons (AARP) to synthesize the literature on “medical injury” in older adults suggests that drug complications are two-and-a-half times more common in older hospitalized patients than in those less than 65 years of age.1 Certain drugs have also been associated with an increased risk of falls in this population.14

Unintentional falls comprise a significant proportion of injury-related morbidity and mortality in older adults.15,16 Approximately one-third of older adults will fall each year; of those, one-half will sustain a minor injury and up to 25% will sustain a more serious injury.14,16 Fall-related injuries can accelerate functional decline and nursing home admission.17,18 Hospitalization poses unique risks of falling in older patients. In addition to the consequences of immobilization, the unfamiliar environment, factors arising from the acute illness, and the effects of treatment may all place older hospitalized patients at higher risk of a fall.4,19 Rates cited in the literature for in-hospital falls for patients of all ages range from 2.2 to 7/1,000 bed days in acute care hospitals.19,20 However, it has been reported that older patients are nine times more likely to fall in hospital than younger patients.1

Pressure ulcers are another common problem in patients admitted to hospital. The incidence of pressure ulcers has been reported in the range of 0.4% to 38% in acute care, with up to 60% of new pressure ulcers developing while a person is in hospital.21,22 Older patients are a high-risk group for developing pressure ulcers, due in part to age-related changes in skin integrity and mobility limitations.4 The presence of comorbid conditions such as peripheral vascular disease or nutritional deficiencies exacerbates the effects of sustained pressure from immobility to increase the risk of developing pressure ulcers.23,24 Within hours of immobilization, some patients are at risk of developing pressure ulcers.4 In addition, their presence may adversely affect functional status, which, in turn, affects discharge disposition of the patient.21 Thus, on an organizational level, some adverse events may have an unintended impact on bed availability and patient flow.

Methods to detect adverse events

The first step in understanding the impact of adverse events and how to prevent them is the development and validation of methods to identify when they have occurred. However, one of the challenges in a plan to monitor and manage adverse events in an organization is the lack of readily accessible information on their occurrence. The presence of an adverse event may be documented on individual health records, but data abstraction from this source is resource intensive.25

Administrative data sources have been used to develop patient safety indicators and performance measures, as well as to enhance surveillance capability for adverse events.26–31 Using a public health approach to the problem, US investigators from Wisconsin developed and validated surveillance criteria for injuries related to medical and surgical care for use with administrative data.28 The Wisconsin surveillance criteria differ from other measures of adverse events in that they focus on harm to a patient as identified by an injury coded in the administrative data. Early studies of adverse events focused on medical error and negligence.10,11 Unlike some indicators and performance measures, the Wisconsin surveillance criteria are not limited to subsets of hospitalized patients. Their classification scheme uses a combination of diagnosis and external-cause-of-injury codes from the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) to identify adverse events or harm to patients. The diagnosis code describes the nature of the problem (eg, rash) and the external-cause codes identify the mechanism (eg, adverse effects of antibiotics in therapeutic use).28 A validation study examined the performance characteristics of the surveillance criteria when compared with the “gold standard” of the major epidemiologic studies in patient safety – the health record review. The surveillance criteria had an overall sensitivity of 59.9% (95% confidence interval [CI]: 42.8–75.0) and a specificity of 97.4% (95% CI: 94.1–98.8).28 The Wisconsin surveillance criteria have not been validated using the tenth revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10).32

Use of routinely collected diagnostic data has been underutilized as an ongoing decision-support resource despite its advantages over other methods for identifying adverse events: it is less costly; it captures new information on all patients admitted to hospital without having to rely on active reporting by health care professionals; the data are readily available in national and international jurisdictions; and the approach to event-finding can complement what is known from other sources and thus provide a more comprehensive view of the problem. The need for robust monitoring is increasingly important, as the relatively at-risk aging population seeking health care is rapidly growing. There is a research gap in using administrative diagnostic data specifically to inform patient safety improvements in the care of older patients.

Study objective

The objective of this retrospective cross-sectional study was to validate diagnostic codes for pressure ulcers, fall-related injuries, and adverse drug events found in routinely collected hospitalization data.

Methods

Setting

The validation study was conducted at the Queen Elizabeth II Health Sciences Centre (QEII HSC) in Halifax, Nova Scotia, Canada. The QEII HSC is the largest teaching hospital and academic health sciences center in Atlantic Canada, with approximately 1,100 beds and 31,000 admissions per year.

Subject selection and data source

Patients (≥65 years) discharged between April 1, 2009 and March 31, 2011 from the QEII HSC were eligible for inclusion in the validation study. Patients who spent any time in non-acute care during their hospital stay were excluded (eg, patients waiting for a nursing home bed) because it was anticipated that these patients would have multiple comorbidities and that there would be other factors contributing to the risk of adverse events. For patients with more than one hospitalization during the study period, only the first hospitalization was included. For each of the three types of adverse events, a random sample of 50 patients whose records were positive and 50 patients whose records were not positive for an adverse event was sought for review in the validation study (n=300 records in total). In Canada, the national standards for assigning diagnostic codes in administrative data are such that conditions are only coded if they have an impact on length of stay or management in hospital.33 Coding standards may vary in different countries.

Adverse events were identified by the diagnostic codes selected from ICD, Tenth Revision, Canada (ICD-10-CA) and found in the administrative hospitalization Discharge Abstract Database. The diagnostic codes for adverse drug events were derived from the Wisconsin surveillance criteria, as these have been previously used in our organization and they have strong performance characteristics compared with health record review.28,34 To determine diagnostic codes for pressure ulcers and fall-related injuries, the multidisciplinary research team systematically examined published sources of codes for these adverse events using explicit criteria established a priori (eg, yield, availability, coding rules, etc) to select candidate codes.26–28,35 Table S1 describes the list of diagnostic codes used in the validation study. The diagnosis type was also examined for those records in which adverse events were coded. The diagnosis type is used in the Discharge Abstract Database to identify preexisting conditions present at the time of admission (Type 1) and those that develop during the hospital admission (Type 2).36 Diagnosis types are critical for identifying harm to patients that occurs during hospitalization.

The adverse events identified by the administrative Discharge Abstract Database were compared with those identified by the gold standard of a structured health record review.3,10,11,28 Two health care providers (a nurse practitioner and a pharmacist) experienced in geriatrics independently reviewed each patient record for the occurrence of an adverse event without any knowledge of the coding in the administrative data. The reviewers used a standard data abstraction form and a structured review process.37 There was a brief education session about the review process and more detailed written instructions were provided to ensure that each reviewer went through a uniform process. The written instructions included items such as confirming the patient name, correct hospital admission, and the order in which different parts of the record were to be reviewed. Reviewers were asked to identify if there was documentation about the presence of a pressure ulcer, fall-related injury, and/or adverse drug event. Reviewers were asked to note the location in the health record where this information was documented. The form also included a section for free text where a reviewer could identify documentation that indicated a possible adverse event. The reviewers were not asked to assess potential preventability. Using the same structured review process, a physician reviewed 20 records from each sample reviewed by the health care providers (n=40) to establish interrater agreement. The study was reviewed and approved by the Capital District Health Authority Research Ethics Board.

Data analysis

The sensitivity, specificity, and 95% CIs were calculated for each type of adverse event. For pressure ulcers and fall-related injuries, diagnostic codes were compared with the results of the health record review by the nurse practitioner. For adverse drug events, diagnostic codes were compared with results of the review by the pharmacist. The interrater agreement was calculated for each type of adverse event based on the subset of records also reviewed by the physician.

Results

The nurse practitioner identified a total of 39 pressure ulcers and 56 fall-related injuries in the 185 complete records that were reviewed. Of these, 34 pressure ulcers and 54 fall-related injuries were identified in administrative data. The pharmacist reviewer identified a total of 69 adverse drug events in the 99 complete records that were reviewed. Of these, 47 were also identified in administrative data. Table 1 describes the sensitivity and specificity of the diagnostic coding for each type of adverse event as compared with the expert health record review. The physician reviewed 20 of the health records reviewed by each of the other reviewers. Interrater agreement was 100% for pressure ulcers and 87.5% for fall-related injuries, and ranged from 81.0%–95.2% for adverse drug events, depending on whether or not events coded as “possible” adverse drug events were included in the calculation. Although we randomly selected 300 records for review, 15 (5.0% of 300) records were missing one or more key documents needed to definitively confirm the presence or absence of a documented adverse event (eg, assessment note). One of the records reviewed by the nurse practitioner and the pharmacist was considered to be incomplete for the pharmacist review as it was missing key documents for determination of an adverse drug event.

Table 1.

Performance characteristics of diagnostic codes for adverse events compared with expert health record reviews

Type of adverse event	Sensitivity (95% CI)	Specificity (95% CI)
Pressure ulcera	0.87 (0.72–0.95)	0.96 (0.80–0.99)
Fall-related injurya	0.96 (0.87–0.99)	0.91 (0.74–0.98)
Adverse drug eventb	0.68 (0.56–0.79)	0.90 (0.72–0.97)

Notes:

^aNurse practitioner review

^bpharmacist review.

Abbreviation: CI, confidence interval.

Five pressure ulcers and two fall-related injuries were identified by the nurse practitioner, but were not coded in administrative data. Twenty-two adverse drug events were identified by the pharmacist, but were not coded in administrative data. The adverse events may not have been coded in the administrative data for reasons related to the coding standards (ie, coding restricted to conditions that have an impact on length of stay or management in the hospital). This limitation became apparent during the case selection phase of the validation study, as there were fewer than 50 records coded for in-hospital pressure ulcers and fall-related injuries during the first fiscal year. To address this limitation, the random selection process was applied to hospitalization data for an additional fiscal year (2010–2011) to identify sufficient records with diagnostic codes indicative of a pressure ulcer or fall-related injury.

Discussion

Valid epidemiologic data are central to the identification of risk factors and development of effective prevention strategies for adverse events. In this validation study, a total of 39 pressure ulcers, 56 fall-related injuries, and 69 adverse drug events were identified through health record review. Of these, 34 pressure ulcers, 54 fall-related injuries, and 47 adverse drug events were also identified in administrative data. Overall, the diagnostic codes for adverse events had a sensitivity and specificity exceeding 0.67 (95% CI: 0.56–0.99) and 0.89 (95% CI: 0.72–0.99), respectively. Different sources of information on adverse events provide varying degrees of depth and breadth with respect to an adverse event. Use of multiple sources of information will provide a more complete picture of the nature and extent of adverse events in an organization.38,39

One of the most common approaches to collecting information about adverse events is through spontaneous reporting using an institutional or other centralized incident-reporting system. Although reporting provides valuable detail about an event, one consistent finding is that adverse events are grossly underreported – by as much as 50%–96%.25,38,40 The biases associated with potential underreporting make it difficult to accurately determine the organization-wide impact of those adverse events. Accordingly, there are challenges with effectively measuring the results of interventions aimed at prevention and management of adverse events across an entire organization.

Use of administrative data to support system-level monitoring of adverse events has garnered increased attention.26–31,41–44 The reported sensitivity and specificity of diagnostic codes in administrative data used to identify adverse events is variable. The performance characteristics of the diagnostic codes in our validation study compare favorably with those of the Wisconsin surveillance criteria. The latter have a reported overall sensitivity of 59.9% (95% CI: 42.8–75.0) and specificity of 97.4% (95% CI: 94.1–98.8). The Wisconsin surveillance criteria include both the diagnosis code and external-cause-of-injury codes. The relative utility of the codes used in the Wisconsin validation study varied depending on the type of adverse event. The external cause of injury codes identified 98.5% of adverse drug events, which suggests that this is an important type of code to include.28 Other studies41,42 have reported substantially poorer performance characteristics. The variability is partially explained by differences in study methodology, case definitions, and care setting. A systematic review of ICD-10 codes used to identify adverse drug events in administrative data found 827 ICD-10 codes for adverse drug events that had been reported in the literature. Of the 41 papers included in the review, only two reported the sensitivity of their codes and both involved emergency department data.44 For instance, Hohl et al used a combination of prospectively collected data on adverse drug events linked with administrative data from two Canadian emergency departments. Their reported sensitivity ranged from 6.8% (95% CI: 4.0–11.2) to 28.1% (95% CI: 22.3–34.6), depending on the level of certainty that the event was attributable to a medication.42

There is scant literature on the performance characteristics of diagnostic codes in administrative data for identifying pressure ulcers and fall-related injuries. Verelst et al validated diagnostic codes to screen for five adverse events using the Belgian Hospital Discharge Dataset. The adverse events were drawn from the Agency for Healthcare Research and Quality Patient Safety Indicators and used ICD-9-CM diagnostic codes. They determined that the positive predictive value for diagnostic codes to identify pressure ulcers ranged from 54.3 to 74.5 for those that were hospital-acquired and present on admission, respectively.43 The positive predictive value is considerably higher in our study population (97.1 [95% CI: 83.3–1.0]). This may be related to the exclusive focus on older patients. Not only is this population at higher risk of developing a pressure ulcer, it may be that documentation is better because of the increased potential for complications in older patients. Although this study focused on three common types of adverse events in a high-risk population, the approach can be applied to patients from all age groups and in any acute care facility that collects routine administrative diagnostic data. The Wisconsin surveillance criteria have been used in pediatric and adult populations. Their criteria include validated diagnostic codes for other types of adverse events, such as those related to procedures or devices.28

Administrative data have important limitations with respect to adverse event surveillance. Administrative data are generally collected for purposes other than surveillance or research, so the quality and completeness of the data may be limited. There is often insufficient clinical detail to perform rigorous risk adjustment using administrative data.28,31,43 As described earlier, coding standards are such that conditions are only coded if they have contributed to an increased length of stay or change in management of care. In addition, coding standards typically require a documented physician diagnosis in order to be coded in administrative data. There is evidence that physician documentation about some types of adverse events (eg, pressure ulcers) is poor.45,46 This limitation became apparent during our validation study, as there were insufficient cases of pressure ulcers and fall-related injuries identified during 1 fiscal year. In a large validation study of patient safety indicators using administrative hospitalization data, similar challenges in reaching the target sample size for some indicators were reported.41 Fifteen (5.0%) of the 300 randomly selected records were missing one or more key documents. These limitations in documentation underscore the importance of using multiple sources of data on adverse events to understand the full scope of the problem.39,40 The use of diagnostic data from administrative sources will likely result in an underestimate of the number of adverse events, yet it is anticipated that it will yield important information using fewer resources than other methods of event-finding.

Use of administrative data for monitoring permits population-based analyses that can be used to identify risk factors, impact on hospital length of stay, disposition, and costs associated with adverse events. Administrative data also provide a feasible, relatively low-cost mechanism for evaluating the impact of prevention strategies on these same measures. These attributes offer important tools for organization-level monitoring that will augment data collected from other sources, such as incident reporting systems. Once high-risk patients and/or settings are identified using administrative data, more detailed methods such as quality reviews or case studies can be used to understand the contributing factors and system-, patient-, and/or provider-level issues so that prevention strategies can be appropriately tailored. As with all approaches to monitoring adverse events, an essential element includes one or more mechanisms by which to broadly share the information to maximize learning and the potential for making safety improvements to reduce harm to patients.

Conclusion

Patient safety has become a critical focus for health care organizations and accreditation bodies in the last decade. Prevention of adverse events is very relevant to a growing number of older patients seeking care, many of whom are especially vulnerable to the unintended harm that may occur in health care. Fall-related injuries, pressure ulcers, and adverse drug events are common and costly examples of potentially preventable adverse events in older hospitalized patients. The results of this validation study demonstrate that it is feasible and valid to identify adverse events in hospitalized patients using routinely collected administrative data. Although the approach likely underestimates the extent of the problem, the information is relatively inexpensive and easy to access, with no impact on clinical staff. Administrative data can provide a valuable tool for ongoing monitoring of the incidence and impact of adverse events as well as the identification of potentially modifiable risk factors upon which targeted prevention strategies can be developed and evaluated.

Supplementary material

Table S1.

Summary of ICD-10-CA codes for adverse events

Adverse drug events

G04.0, G21.0, G62.0, L25.1, L27.0, L27.1, O35.501, O35.503, O35.509, P04.4, T36.0–T37.5, T37.8–T39.4, T39.8–T40.0, T40.2–T40.6, T41.0–T41.5, T42.0–T42.8, T43.0–T43.6, T43.8–43.9, T44.0–T48.7, T49.0–T50.9, T78.8, T88.1–T88.3, T88.5, T88.7, T80.3–T80.6, T80.8–T80.9, T96.0, X40–X43, X60, X62–X64, X85, Y10–Y14, Y40.1–Y41.5, Y41.8–Y43.6, Y43.8–Y44.7, Y44.9, Y45.0–Y45.5, Y45.8–Y46.8, Y47.0–Y47.5, Y47.8–Y48.5, Y49.0–Y49.5, Y49.7–Y50.1, Y50.8–Y55.7, Y56.0–Y58.6, Y58.8–Y59.0, Y59.2, Y59.3, Y59.8, Y59.9, Y63.0, Y64.0, and Y65.0.1

Pressure ulcers

L89.0–L89.8.2

Falls

W01–W08, W10, and W17–W19 – cases selected on the basis of diagnosis type (post-admit; Type 2) and place of occurrence codes (school/other institutional and public area; U98.2).3,4