Navigating a ship with a broken compass: evaluating standard algorithms to measure patient safety

Abstract

Objective: Agency for Healthcare Research and Quality (AHRQ) software applies standardized algorithms to hospital administrative data to identify patient safety indicators (PSIs). The objective of this study was to assess the validity of PSI flags and report reasons for invalid flagging.

Material and Methods: At a 6-hospital academic medical center, a retrospective analysis was conducted of all PSIs flagged in fiscal year 2014. A multidisciplinary PSI Quality Team reviewed each flagged PSI based on quarterly reports. The positive predictive value (PPV, the percent of clinically validated cases) was calculated for 12 PSI categories. The documentation for each reversed case was reviewed to determine the reasons for PSI reversal.

Results: Of 657 PSI flags, 185 were reversed. Seven PSI categories had a PPV below 75%. Four broad categories of reasons for reversal were AHRQ algorithm limitations (38%), coding misinterpretations (45%), present upon admission (10%), and documentation insufficiency (7%). AHRQ algorithm limitations included 2 subcategories: an “incident” was inherent to the procedure, or highly likely (eg, vascular tumor bleed), or an “incident” was nonsignificant, easily controlled, and/or no intervention was needed.

Discussion: These findings support previous research highlighting administrative data problems. Additionally, AHRQ algorithm limitations was an emergent category not considered in previous research. Herein we present potential solutions to address these issues.

Conclusions: If, despite poor validity, US policy continues to rely on PSIs for incentive and penalty programs, improvements are needed in the quality of administrative data and the standardized PSI algorithms. These solutions require national motivation, research attention, and dissemination support.

BACKGROUND AND SIGNIFICANCE

While administrative data are commonly used in hospital report cards and physician profiling, and to identify and track adverse events,¹ studies have identified a variety of issues that can reduce data quality, including variations in coding and documentation practices across hospitals and failure to identify conditions present upon admission.^2,3 Collected during the process of health care delivery, administrative data (including demographic characteristics, patient diagnoses, and procedure codes) suffer from inherent ambiguity in the International Classification of Diseases, Ninth Revision, Clinical Module (ICD-9-CM) coding guidelines. There is clear documentation in the literature on significant variability in ICD-9-CM coded administrative data across institutions.^1,4–7 While US hospitals are currently transitioning to a new revision of ICD codes, ICD-10, researchers have found that the new schema does not significantly improve the quality of administrative data.^8,9

One important example is the collection and use of Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs). Released in 2003, 20 PSIs were established to facilitate identification of hospital complications and adverse events following surgeries, procedures, and childbirths. PSIs were developed to attempt to identify incidences where care could be perceived to be substandard as a result of a patient safety error, and are now being used in public reporting, performance assessment, and hospital reimbursement and penalty programs.^10,11 The Centers for Medicare and Medicaid Services (CMS) uses a weighted composite of 8 PSIs, the PSI-90, in its Hospital-Acquired Conditions and Hospital Value-Based Purchasing Programs. Additionally, US News & World Report uses PSIs in its hospital rankings, and the CMS Hospital Compare website includes PSIs in assessments of hospital performance. Given the new and expanding role for PSIs, it is imperative that the health care community understands these measures and the extent to which they are associated with clinical outcomes.

A significant concern raised about the use of PSIs has been the variability associated with coding practices; there is sufficient evidence to question the validity and reliability of quality comparisons between hospitals based on institution-level administrative data in general.^4,12 Hospitals with more robust data quality validation and control mechanisms that focus primarily on lowering false positives within reported data could receive higher quality rankings by virtue of reporting a lower number of inaccurately coded adverse events. Additionally, hospitals that lack robust data quality control procedures to filter out false positives could receive lower quality rankings despite clinical quality of care being similar to higher ranked hospitals. Despite these concerns, there is an ongoing focus on using administrative data for national quality and safety reporting programs, predominantly due to its ease of access and use.

The AHRQ indicators rely on the accuracy of administrative data; however, the low predictive validity reported by previous studies indicates that the administrative data used in the AHRQ indicator algorithms do not always correctly identify a patient safety incident. In recent studies, the validity of PSIs has been widely questioned. AHRQ provides a software program that applies standardized algorithms to hospital administrative data, specifically utilizing ICD codes and extensive exclusion criteria, described further in the Methods section.¹³ The most common factors contributing to incorrect PSI designation, and resultant low predictive validity, are presence of the condition upon admission, coding errors, and lack of documentation.^14,15 All 3 of these reasons can be attributed to problems with the accuracy of administrative datasets, and variability in administrative coding as described above could contribute to the low validity of AHRQ PSIs. In 3 recent studies, researchers sampled incidents that were flagged by the AHRQ software and utilized a single coder to validate each case. A more extensive multistep review process may validate even fewer incidents, as would evaluating each incident instead of just a random sample of incidents.

OBJECTIVE

To improve understanding of the reality of applying a standardized coding algorithm to administrative data, our clinical and research team conducted a complete PSI validation study of every incident identified by the AHRQ PSI software at our health system in fiscal year 2014. To expand upon previous studies, we convened a multidisciplinary PSI Quality Team that reviewed each PSI and received ongoing coding and documentation education. Due to the unique elements of our study, such as using a multistep process to validate the entirety of PSI flags in the study time frame, we hypothesized that we would find lower than typically reported predictive validity across the PSI indicators. When compared to medical record review, administrative or AHRQ software–identified PSIs have modest positive predictive values (PPVs) and wide variability in PPVs across specific indicators.^3,14–17 PPV is the percent of incidents identified by the AHRQ software that are verified by a retrospective chart review.

Secondarily, through these analyses, we aimed to test the hypothesis that low PPVs are due mainly to poor data quality in administrative databases. Two validation studies, one conducted at an academic medical center¹⁴ and another among a national sample of Veterans Administration hospitals,¹⁵ found that catheter-related bloodstream infections had the lowest PPVs, with other PSIs’ PPVs between 60% and 80%. While these studies were based on ICD-9 codes, a recent validation of PSIs using ICD-10 data from Canada found similarly low PPVs.¹⁸ There is also concern that the US’s transition from ICD-9 to ICD-10 will lead to even lower PPVs, due to the complexity of coding system translations.¹⁹ Both in the interest of transparency and to add to the ongoing discourse on this issue, we report the results of our hypothesis testing, including the reasons why an incident was not validated for each PSI flag.

METHODS

Setting

This study was conducted at a tertiary care academic medical center in a large metropolitan city. The medical center comprises a heart and vascular hospital, a cancer hospital, a rehabilitation hospital, an inpatient psychiatric facility, a community hospital, and a university hospital. Across these settings there are nearly 1400 inpatient beds and an annual average of 57 000 discharges, with ∼42% of patients either on Medicare/Medicaid and/or from underserved populations.

Study sample

We took a census approach that included all health system patients flagged for a patient safety incident by the AHRQ PSI software; the software used during the study period was version 4.5.²⁰ Our retrospective study was approved by the medical center’s Biomedical Institutional Review Board.

Patient safety indicators

This study was a retrospective analysis of data from patients that had incidents identified as 1 of 12 PSIs during fiscal year 2014. The PSIs identified were selected based on their importance to public reporting and reimbursement mechanisms. Table 1 lists each PSI number and name, and indicates which reporting mechanisms utilize that indicator. An SAS software module provided by AHRQ runs standard algorithms on the hospital’s discharge data, ICD-9-CM codes used for billing purposes, to calculate a rate for each PSI: outcome of interest (numerator) divided by population at risk (denominator).¹³ For example, for PSI 3, pressure ulcer, the denominator includes all medical and surgical discharges, excluding a principal diagnosis code of pressure ulcer, an ulcer present upon admission, or a diagnosis code of paralysis, such as quadriplegia. The numerator is discharges not excluded from the denominator, with any secondary diagnosis codes for pressure ulcer. The software can risk-adjust the PSI rates to reflect the age, sex, condition, and comorbidity distribution of the US population.

Table 1.

Groups using patient safety indicators (PSIs) for ranking and public reporting

PSI Number	PSI Name	US News & World Report	University Health System Consortium Award	CMS VBP bonuses and HAC penalties (PSI-90 composite score)
PSI 03	Pressure ulcer	X		X
PSI 04	Death among surgical inpatients	X
PSI 06	Iatrogenic pneumothorax	X	X	X
PSI 07	Central venous catheter–related bloodstream infections		X	X
PSI 08	Postoperative hip fracture rate	X		X
PSI 09	Postoperative hemorrhage/hematoma	X	X
PSI 10	Postoperative physiologic and metabolic derangementa			X
PSI 11	Postoperative respiratory failure	X	X
PSI 12	Postoperative pulmonary embolism or deep vein thrombosis			X
PSI 13	Postoperative sepsis rate			X
PSI 14	Postoperative wound dehiscence	X		X
PSI 15	Accidental puncture or laceration	X		X

^aPSI 10 is a proposed new Centers for Medicare and Medicaid Services measure.

The PSI quality team

In 2011, the health system established a PSI Quality Team, led by the chief quality and patient safety officer, comprising surgeons and physicians, a data manager, nurse quality reviewers (QRs) from the Quality Department, and a medical information management (MIM) documentation and coding quality expert. The data manager facilitated the relationship between the University Health System Consortium (UHC) and the clinical reviewers on the team. The MIM reviewer, a trained medical coder, served as the coding and documentation specialist and was a liaison between the 30 other coders employed by the medical center and the PSI Quality Team.

Clinical validation process

During the study time frame, data tables based on medical center billing data were automatically sent to UHC (a CMS vendor that calculates PSIs for CMS). UHC provides guidance to all members on the specific format of the data to optimize the performance of the AHRQ PSI software. UHC would run the data through the AHRQ PSI software and send quarterly reports of flagged PSIs back to the medical center. The data manager received the report and assigned all flagged incidents to no fewer than 2 nurse QRs and 2 physicians. Physicians and surgeons on the team were assigned PSIs pertinent to their clinical area of expertise. For each incident, the QRs identified and corrected issues in the documentation or coding of an encounter; including constructing a timeline.

After the QR review, a physician with expertise in the applicable specialty area then reviewed the QR’s documentation. Further review by the chief quality and patient safety officer provided an additional verification step. The MIM reviewer made the final PSI determination. The decision rules for PSI review were the AHRQ inclusion and exclusion criteria.¹³ If the electronic health record (EHR) documentation was not appropriate, consideration for an addendum was requested from the patient’s care team. If, based on the QR’s changes to documentation, the PSI code had been incorrectly assigned, the PSI flag was removed, thus reversing the PSI designation for that case. “Reversal” is defined as a lack of evidence in the record of a potentially preventable complication or iatrogenic event. If there was a disagreement among the PSI Quality Team, the original coding was left in place. This team decision process produced the data that were reviewed to determine the rate of PSI reversal, presented in the Results section.

Analysis

The PSI Quality Team maintained data records throughout the study time frame detailing the validation process and noting which PSIs were reversed and for what reasons. For each fiscal year, an initial count of PSIs and the number reversed were calculated. For each PSI category, the number reversed was divided by the initial count to determine the proportion of cases reversed. This proportion was subtracted from 1 and multiplied by 100 to calculate the PPV, the percent of clinically validated cases for each PSI category. The fourth and senior authors then independently reviewed the documentation for each case to determine the reasons for PSI reversal. This process was both inductive and deductive, meaning that an initial set of reasons was based on previously published literature, and emerging reasons were also recorded; when possible, reasons were collapsed into broader categories. Disagreements between coders were resolved through consensus-building discussion.

RESULTS

Table 2 presents the results of our calculations of PPVs. Eight PSIs had a PPV below 90%. The lowest PPVs were 29% for central venous catheter–related bloodstream infections, 44% for accidental puncture or laceration, and 53% for postoperative hemorrhage/hematoma. Pressure ulcer, iatrogenic pneumothorax, postoperative respiratory failure, and postoperative sepsis rates all had PPVs below 75%. Four PSIs – death among surgical inpatients, postoperative physiologic and metabolic derangement, postoperative pulmonary embolism or deep vein thrombosis, and postoperative wound dehiscence – had PPVs of nearly 100%.

Table 2.

The positive predictive value of PSIs flagged with AHRQ software in fiscal year 2014

PSI Number	PSI Name	Initial Count	Reversed	PPV (%)
PSI 03	Pressure ulcer	6	2	67
PSI 04	Death among surgical inpatients	88	0	100
PSI 06	Iatrogenic pneumothorax	28	8	71
PSI 07	CVC-related bloodstream infections	41	29	29
PSI 08	Postop hip fracture rate	2	0	100
PSI 09	Postop hemorrhage/ hematoma	145	68	53
PSI 10	Postop physiologic and metabolic derangement	12	0	100
PSI 11	Postop respiratory failure	88	25	72
PSI 12	Postop PE or DVT	126	0	100
PSI 13	Postop sepsis rate	35	9	74
PSI 14	Postop wound dehiscence	8	0	100
PSI 15	Accidental puncture or laceration	78	44	44
	Fiscal Year Total	657	185	72

Note: Reversal of a flagged PSI was a retrospective process of validation by a team of clinicians and documentation specialists.

Abbreviations: PSI: patient safety indicator; AHRQ: Agency for Healthcare Research and Quality; CVC: central venous catheter; PE: pulmonary embolism; DVT: deep vein thrombosis.

During fiscal year 2014, the PSI Quality Team reversed a total of 185 cases flagged as incidents by the AHRQ PSI software. A review of the reversed cases found 4 main factors that served as the basis for the changes: PSI algorithm limitations, coding misinterpretations, present upon admission, and documentation insufficiency (see Table 3 for a breakdown of numbers of cases by category and subcategory). PSI algorithm limitations accounted for 38% of PSI incident reversals. The most common subcategory of PSI algorithm limitations occurred when an “incident” was considered inherent to the procedure, or highly likely. For example, a vascular tumor will bleed if manipulated in any way, and that is not indicative of a safety issue. As a result, codes of hemorrhage were removed in cases of surgery on vascular tumors. Similarly, the other subcategory of algorithm limitations was classified as “nonsignificant PSI” and included cases where the PSI was easily controlled and/or no intervention was needed, eg, a mild hematoma resolved without clinical intervention. PSI software limitations was a highly prevalent reason for reversal among 3 PSIs: hemorrhage/hematoma, CVC-related bloodstream infections, and postoperative respiratory failure.

Table 3.

Reasons for reversal of a PSI flagged by the AHRQ PSI software during fiscal year 2014 (ie, the PSI Quality Team removed the flag upon case review)

Reasons	n (%)
PSI Algorithm Limitations
PSI inherent to the procedure or highly likely	44 (24)
Nonsignificant PSI
No intervention was needed	10 (5)
The PSI was controlled	16 (9)
Present on Admission	19 (10)
Documentation Insufficiency
Unclear documentation	8 (4)
Conflicting documentation	4 (2)
Missing documentation	2 (1)
Coding Misinterpretations
Incorrectly selected ICD-9 code	32 (17)
Information in the chart contradicted the ICD-9 code selected	41 (23)
ICD-9 exclusion criteria were not properly coded	9 (5)

Note: The dataset is 185 reversals from a review of all 657 PSIs at the academic medical center during the study year.

Abbreviations: PSI: patient safety indicator; AHRQ: Agency for Healthcare Research and Quality.

The 3 other reasons for reversal were all related to documentation within the EHR by the patient care team and trained coders. Present upon admission, 10% of reversals, was a documentation error that occurred during the patient admission process. Documentation insufficiency, 7% of reversals, included unclear, conflicting, or missing physician or nurse documentation that led to incorrect ICD coding by the billing coder. Coding misinterpretations occurred most frequently and refers to cases in which trained billing coders incorrectly assigned ICD-9 codes, resulting in the software incorrectly flagging a PSI. For example, in 25 cases across 4 types of incidents – CVC-related bloodstream infections, postoperative respiratory failure, sepsis, and hematoma/hemorrhage – the coders selected the ICD-9 code when the case details in the EHR indicated that the ICD inclusion criteria had not been met. Coding misinterpretations accounted for almost half of reversed PSI cases (45%) in fiscal year 2014.

DISCUSSION

Our internal process of retrospective clinical validation of incidents flagged by the AHRQ software found a PPV below 75% for 7 PSIs. Two recent validation studies, one at another academic medical center¹⁴ and one within the VA system,¹⁵ found qualitatively similar results in terms of which PSIs had low PPVs. CVC-related bloodstream infections had the lowest PPV across the 3 studies. However, the magnitude of these findings differed; while prior studies reported PPVs of 60–80%, our present study found PPVs as low as 30–40%. This means that our PSI Quality Team removed the PSI flag more frequently than occurred in other validation studies.

This difference in PPVs across studies may be explained in light of the new category of reasons for PSI reversal we found, algorithm limitations. Rosen and Ramanathan each reported reasons conceptually similar to the 3 other categories of reasons we present, all related to care team and coder documentation within the EHR. Algorithm limitations, however, is a new reason category that moves beyond issues of administrative data quality and encompasses problems with the specificity of the PSI coding algorithm. Our PSI Quality Team was considering issues of specificity not mentioned in previous studies, such as whether a flagged incident was a clinically significant event, or whether the incident was inherent to the procedure or highly likely given patient factors. The PSIs with the lowest PPVs in our study, hemorrhage/hematoma, CVC-related bloodstream infections, and postoperative respiratory failure, suffered disproportionality from issues related to the AHRQ PSI algorithm limitations.

Considering that the 2 most prevalent reasons for invalid classification of PSI cases were administrative data quality and the specificity of the ARHQ software, we propose a solution to address each issue: (1) national standardization of documentation and coding processes related to PSIs, and (2) risk adjustment, beyond standard case mix, of PSI rates at the institution level. We also propose that development of these solutions be sponsored and coordinated at the national level and disseminated through AHRQ as an augmentation of the PSI software. The following sections detail our proposed solutions.

National standardization of documentation and coding processes

While there is standardized coding training associated with ICD-9 and -10 coding, many studies have found persistent variability in ICD-coded administrative data across institutions.^3,4,9 The coders at our medical center are trained using American Health Information Management Association–certified coding courses, a process that we feel significantly improves the quality of coding at our institution. However, our results indicate that data going into the AHRQ PSI tool are subject to data quality issues. Previous studies have suggested ways to improve coding processes at the point of administrative data entry, including educating the patient care team about PSIs so that documentation can be as explicit and specific as possible, and educating coders about the AHRQ software, PSIs, and codes that correspond to PSIs.^3,14,15 A system of standardized coding education focused on PSIs would reduce variability in the quality of administrative databases both within an institution (ie, consistent coding among different providers and departments) and across hospitals in national-level comparisons.

The first step in developing this type of coding education at the point of data entry is to study current hospital-level PSI documentation education programs for residents, physicians, and care teams to identify best practices. The next step would be to use these best practices to guide development of standardized documentation and coding education programs to reduce the documentation problems frequently leading to incorrect PSI designation, thereby increasing the validity of the AHRQ PSI software. If developed by a nationally recognized body and disseminated across US health systems, such coding education programs could reduce the variability in PSI rates across hospitals that is due to differences in administrative documentation.

Risk adjustment of PSI rates at the institution level, beyond case-mix adjustment

The general consensus among academics who study the validity of PSIs is that until coding revisions are implemented to the AHRQ software, similar to those outlined at the end of our Results summary section above, use of PSIs in public reporting and pay-for-performance needs to be reconsidered.^{3,14,16–18,21,22} Rosen and colleagues¹⁵ have suggested that a “timing” code be added to the ICD codes, since certain PSIs have inclusion/exclusion criteria based on time, as well as adding a composite AHRQ PSI algorithm adjusting for PPV and present-on-admission rates.²¹ Our findings also suggest adding more extensive exclusion criteria to improve the sensitivity and specificity of the tool, such as an “inherent to the procedure” code or “bleeding greater than a certain threshold” code to allow flagging of only clinically significant events. This process would have to be specific to each PSI, would entail extensive and ongoing development work, and would result in significant additional coding complexity for both patient care documentation teams and post-discharge coding specialists.

Given the difficulty of the changes we have outlined above, as an alternative to adjusting the PSI algorithm at the level of individual patient identification, one option would be to integrate fault tolerance into the calculations at the institutional level that could be developed and applied to a hospital’s yearly PSI rates prior to making calculations for incentives and reimbursements. This risk scoring would provide for risk adjustment beyond what is already offered within the AHRQ PSI software, which currently adjusts rates for the age, sex, condition, and comorbidity distribution of the US population. This approach would require further research to determine cases where an identified incident has a high likelihood of post hoc determination that the incident was a non–patient-safety occurrence or inherent to the procedure for each PSI category, but could be an important step toward improving this process.

LIMITATIONS

The main limitation of this study is that it focuses on a single academic medical center. However, the problem being discussed is universal to every institution that provides patient care in the US health care system. All hospitals that accept CMS money have reimbursements and penalties tied to the PSI rates calculated from their administrative data. Our paper describes the work of a PSI Quality Team and presents a unique Band-Aid solution to a national problem. We propose broad solutions to increase the validity of PSI reporting, but these must be implemented on a national level with centralized coordination. Therefore, we feel this report of a single site’s experience could have national implications if such solutions are pursued.

A limitation of our analysis of reasons for PSI reversal is the difficulty we encountered distinguishing among the categories of documentation error and coding error. This analysis was done retrospectively using documentation from the PSI Quality Team, but without direct contact with the original care team or billing coders. It was occasionally difficult to ascertain whether the coder was not aware of certain exclusion criteria or if the front-line care team did not leave notes clear enough to identify the exclusion criteria. This limitation, however, can also be viewed as supporting evidence for this paper’s findings; there is a data quality problem inherent in administrative databases, and both trained coders and front-line documentation teams need additional education. Another limitation of this study is the inability to assess the sensitivity of the AHRQ PSI software. We could not determine the rate of “missed” PSIs. There may be a large number of patient safety incidents that go undetected for the same reasons identified in our analysis.

CONCLUSIONS

While our health system continues to individually validate all PSI incidents, starting in fiscal year 2015, this process became concurrent. Each PSI is now clinically validated upon the patient’s discharge, before the patient encounter is billed and the data are sent to external agencies. Concurrent validation of PSIs requires the daily resources of data managers, coding and quality specialists, and physicians. Clearly, this process is exceptionally labor-intensive and costly. Despite the low validity inherent in the use of administrative data for quality reporting, if national-level policy makers want to continue to rely on PSIs for incentive and penalty programs, potential improvements as proposed in this article are needed to increase the quality of administrative data through expanded documentation and coding education while simultaneously risk-adjusting hospital-level PSI rates to incorporate information on the number of incidents that are not clinically significant for each PSI. These solutions require national motivation, research attention, and dissemination support.

CONTRIBUTORS

All authors made substantial contributions to the conception of the work or analysis and interpretation of data. J.L.H., B.B., and T.R.H drafted the manuscript and conducted the data analysis. T.L. and A.S.M. contributed to results interpretations and manuscript preparation. S.M.B. oversaw and contributed to all aspects of the project. All authors read and approved the final manuscript.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sector.

COMPETING INTERESTS

The authors have no competing interests to declare.