Comparison of Automated Posttonsillectomy Bleed Capture With Self-report

This medical record review evaluates the feasibility and accuracy of an algorithm to capture posttonsillectomy bleeding in a single health care system compared with traditional self-reported bleeding by the surgical team.

Key Points

Question

Is automated capture of pediatric posttonsillectomy bleeding feasible and accurate compared with self-reported complications?

Meaning

Self-reported pediatric tonsillectomy bleeding complications were compared with an automated algorithm that searched the medical record for bleeding complications of 1017 tonsillectomies. The analysis revealed disagreement between the algorithm and self-reported database in 31 cases (3.05%).

Findings

Automated capture of complications from the medical record, although not perfect, is possible and may prove to be a useful adjunct to self-report.

Abstract

Importance

Tonsillectomy is one of the most common procedures performed by otolaryngologists and is associated with postoperative bleeding. Bleed rates are usually monitored by self-report.

Objective

To evaluate whether using automated capture and reporting of pediatric posttonsillectomy bleeding is feasible and accurate compared with traditional self-reporting by the surgical team.

Design, Setting, and Participants

An automated complication-reporting algorithm was designed to query the local health information exchange and then tested against self-reported tonsillectomy complication data collected from January 1, 2014, through December 31, 2015, at a tertiary pediatric hospital. The algorithm identified patients undergoing tonsillectomy and searched their postoperative encounters for a hand-selected set of diagnosis codes from the International Classification of Diseases, Ninth Revision and International Statistical Classification of Diseases and Related Health Problems, Tenth Revision and free-text words to identify complication events. Five months of the 2014-2015 data set were used to help design the algorithm. Data from the remaining 19 months were compared with self-reported complications.

Main Outcomes and Measures

Automated system findings compared with self-reported bleeding events.

Results

During the 19-month period, 1017 tonsillectomies were performed. We compared the algorithm’s effectiveness in finding tonsillectomy and adenotonsillectomy procedures for the evaluated surgeons with the hand-reviewed master tonsillectomy list. The algorithm reported 51 false-positive (5.01% missed) and 74 false-negative (7.28% misidentified) procedures. The algorithm agreed with self-report for 986 tonsillectomies and disagreed on 31 cases (3.05%) (κ = 0.69; 95% CI, 0.66-0.73). The algorithm was found to be sensitive to correctly identifying 60.53% (95% CI, 48.63%-71.34%) of tonsillectomies as having bleeding complications, with a specificity of 98.30% (95% CI, 97.19%-98.99%).

Conclusions and Relevance

Capture of posttonsillectomy bleeding is possible through an automatic search of the medical record, although the algorithm will require continued refinement. Leveraging health information exchange data increases the possibilities of capturing complications at hospitals outside the local health system. Use of these algorithms will allow repeatable automated feedback to be provided to surgeons on a cyclical basis.

Introduction

Tonsillectomy, one of the most common procedures performed by otolaryngologists, is associated with postoperative bleeding that can result in serious morbidity and mortality. The American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS) currently recommends that surgeons review their posttonsillectomy bleed rates at least annually. In many practices, including our health system, bleed rates are monitored by self-report. However, self-reporting is naturally subject to potential bias because of the various definitions of a complication, partial data capture, and human error, to name a few problems.

The recommendation by the AAO-HNS was not made in a vacuum; groups such as Washington State’s Surgical Care and Outcomes Assessment Program have shown a reduction in morbidity and mortality after surgery by critically reviewing data gathered at a regional level. However, the Surgical Care and Outcomes Assessment Program and other regional collaboratives currently depend on the use of trained human reviewers to collect patient-level data. Although the outcomes have been promising, the data capture is labor intensive and may not be practical for high-volume surgery such as tonsillectomy.

The use of electronic data capture could allow otolaryngologists to reduce the workload associated with tracking outcomes while increasing the accuracy of the data collected. Codes from the International Classification of Diseases, Ninth Revision (ICD-9) reported as the chief complaint for emergency department visits have been shown to be an effective mode for tracking medical diagnoses for epidemiology. The same might be possible for surgical outcomes.

To more fully capture surgical outcomes, an automated data capture system would benefit from the existence of a regional database. This database would contain information at the regional level so that if patients return for care at a facility other than where they received their index operation, the complication could be matched to the surgeon who performed the index case. Indiana has this type of database. The Indiana Network for Patient Care (INPC), a local health information exchange (HIE), is a resource that contains nearly 5 billion pieces of retrievable medical record data. With participation from 25 000 physicians, 94 hospitals, 110 clinics and surgery centers, and all major health care organizations in Indiana, the INPC covers a large portion of health care events in the state (http://www.ihie.org/).

The purpose of this study was to evaluate how automatic electronic identification of posttonsillectomy bleeding compares with self-report. We hypothesize that the use of an encounter’s chief complaint and diagnosis in coded and free-text format within the postoperative period would effectively capture a large portion of the clinically relevant posttonsillectomy bleeding complications. Not requiring hand curation of data should help improve accuracy and speed of outcomes analyses that can be repeated without increased effort. This process would ease local morbidity and mortality reporting and capture additional missed complications. If effective, it would be natural to extrapolate these efforts to help advance regional and national quality improvement efforts.

Methods

Permission of the institutional review board was sought and granted for retrospective review of the medical records for our department’s patients who underwent tonsillectomy. All private health data were maintained and stored in a manner compliant with the Health Insurance Portability and Accountability Act. Consent was not required for use of deidentified data.

We selected the years 2014 and 2015 to design and evaluate our algorithm. From February 1 through May 31, 2015, and December 1 through 31, 2015, self-reported posttonsillectomy hemorrhages (using REDCap [research electronic data capture; Vanderbilt University] and FileMaker Pro [Apple]) for otolaryngologists operating in our pediatric tertiary care center were used for validation and designing of the automated system. These self-reported databases included all oropharyngeal bleeding reported regardless of the reporter, even if not evaluated by a medical professional, because our group believed that even self-limited, noninjurious bleeding is an important event to be recorded owing to its effect on the patient’s and family’s perceptions. The month of December 2015 was chosen to sample the time when the health system transition to International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) (occurred in August 2015) was complete. The system was then evaluated using the 19 remaining months of 2014 and 2015. Tonsillectomies and complications for 6 of the 8 surgeons who performed tonsillectomies at the tertiary center during this period were included in the study because 2 of the surgeons had not yet been included in the HIE clinician database table.

The automated system was created using the Java programming language (Oracle) performing Structured Query Language (SQL) queries against the INPC health exchange’s database (Oracle Database 11g, Enterprise Edition) with the design and programming assistance from the Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, Indiana. This system queried patients who underwent a tonsillectomy or adenotonsillectomy in our tertiary pediatric care hospital by using a set of Current Procedural Terminology, 4th Edition (CPT-4) codes (Table 1). In addition, operative reports were included with tonsillectomy within the procedure list, with the caveat of rejecting reports with recent, prior, previous, and lingual describing tonsillectomy. Each patient’s medical record stored in the HIE was then queried across hospital systems to evaluate for encounters within a 21-day posttonsillectomy period (Figure). Encounters included emergency department visits, clinic visits, procedural notes, and notes written during inpatient stays. The INPC makes use of a global patient-matching algorithm that connects patients across hospital systems and connects orphaned medical records in the same hospital system. We compared the ICD-9 and ICD-10 codes used in the encounter’s chief complaint and diagnosis against a set of ICD-9 and ICD-10 code sets that we developed (Table 1). Free-text query analysis was also used to identify clinical notes with the term oropharyngeal hemorrhage. This phrase was chosen based on analysis of the training set of data. In addition, recognition of posttonsillectomy bleeding was augmented by a search for the CPT-4 codes used for oropharyngeal hemorrhage control (Table 1). These results were assembled and returned in unidentified format (an identified version was used for verification, evaluation, and troubleshooting) to be built into report form for individual surgeon evaluations of outcomes.

Table 1. Codes Used to Identify Tonsillectomy Procedures and Posttonsillectomy Hemorrhage Complications.

Code	Description
CPT-4 tonsillectomy procedure code
42820	Tonsillectomy and adenoidectomy; age <12 y
42821	Tonsillectomy and adenoidectomy; age ≥12 y
42825	Tonsillectomy, primary or secondary; age <12 y
42826	Tonsillectomy, primary or secondary; age ≥12 y
ICD-9 hemorrhage encounter code
998.11	Hemorrhage complicating a procedure
528.9	Other and unspecified diseases of the oral soft tissues
784.8	Hemorrhage from throat
ICD-10 code R041	Hemorrhage from throat
CPT-4 hemorrhage encounter code
42960	Control oropharyngeal hemorrhage, primary or secondary (eg, posttonsillectomy); simple
42961	Control oropharyngeal hemorrhage, primary or secondary (eg, posttonsillectomy); complicated requiring hospitalization
42962	Control oropharyngeal hemorrhage, primary or secondary (eg, posttonsillectomy); with secondary surgical intervention

Abbreviations: CPT-4, Current Procedural Terminology, 4th Edition; ICD-9, International Classification of Diseases, Ninth Revision; ICD-10, International Statistical Classification of Diseases and Related Health Problems, Tenth Revision.

Figure. System Query Design.

HIE indicates health information exchange.

A criterion standard database consisting of the 2014 and 2015 tonsillectomies was generated by reviewing financial data in combination with retrospective review of the tertiary pediatric hospital’s medical record. A manual review was performed of all self-reported posttonsillectomy bleeds in the medical record, noting whether a physical encounter was documented. This review prevented an evaluation of the system against self-reported bleeding complications that were documented by verbal communication or telephone but not recorded in the medical records available to the INPC and therefore not discoverable by the system.

We used Cohen κ analysis to compare the automated system’s findings with self-reported bleeding events. A κ statistic of 0.00 to 0.20 indicated poor agreement; 0.21 to 040, fair; 0.41 to 0.60, moderate; 0.61 to 0.80, substantial; and 0.81 to 1.00, almost perfect.

Results

We evaluated the automated system by comparing reported results with the self-reported complications from January 1, 2014, through January 31, 2015, and from June 1 through November 30, 2015, a period when 1017 tonsillectomies were performed. A Cohen κ analysis was initially performed to compare the agreement between the 2 self-reported databases at our disposal in Excel (Microsoft Corporation), and they were found to only disagree on 16 tonsillectomy cases (1.54%; κ = 0.82; 95% CI, 0.80-0.84).

We then compared the algorithm’s effectiveness in finding tonsillectomy and adenotonsillectomy procedures for the evaluated surgeons with the hand-reviewed master tonsillectomy list. The algorithm reported 51 false-positive (5.01% missed) and 74 false-negative (7.28% misidentified) procedures (Table 2).

Table 2. Tonsillectomies Reported by the Algorithm for the Evaluated Period vs Hand-Reviewed Master List of Tonsillectomies.

Automated Algorithm Finding	Hand-Reviewed List of Procedures		Total No.
	No. of Tonsillectomies	No. of Nontonsillectomies
Tonsillectomy	943	51a	994
Nontonsillectomy	74b	0	74
Total	1017	51	1068

^aFalse-positive procedures.

^bFalse-negative procedures.

We again chose κ analysis to evaluate the interrater agreement between the automated algorithm and our self-reported REDCap database. The algorithm agreed with self-report for 986 tonsillectomies and disagreed on 31 cases (3.05%) (κ = 0.69; 95% CI, 0.66-0.73) (Table 3).

Table 3. κ Analysis of REDCap Self-reported Posttonsillectomy Hemorrhage and Automated Algorithm^a.

Automated Algorithm Finding	Self-reported REDCap Database		Total No.
	Hemorrhage	No Hemorrhage
Hemorrhage	38	8	46
No hemorrhage	23	948	971
Total	61	956	1017
Total observed agreement	0.97	NA	NA
Agreement expected by chance	0.90	NA	NA
κ Statistic (95% CI)	0.69 (0.66-0.73)	NA	NA

Abbreviations: NA, not applicable; REDCap, research electronic data capture (Vanderbilt University).

^aData are presented as number of tonsillectomies, unless otherwise indicated.

The algorithm was further compared with the criterion standard of hand-reviewed records of tonsillectomies and associated bleeding complications and found to be sensitive to correctly identifying 60.53% (95% CI, 48.63%-71.34%) of tonsillectomies as having bleeding complications, with a specificity of 98.30% (95% CI, 97.19%-98.99%) (Table 4). When reviewing the codes that were used to identify bleeding events, ICD-10 code R041 was not found in the period tested after August 2015 during the ICD-10 transition at our institution, although we have found it used subsequently.

Table 4. Automated Algorithm vs Hand-Reviewed Criterion Standard List of Posttonsillectomy Hemorrhage Complications With Clinical Encounters in the Medical Record^a.

Automated Algorithm Finding	Hand-Reviewed Criterion Standard		Total No.
	Hemorrhage	No Hemorrhage
Hemorrhage	46	16	62
No hemorrhage	30	925	955
Total	76	941	1017
Sensitivity (95% CI), %	60.53 (48.63-71.34)	NA	NA
Specificity (95% CI), %	98.30 (97.19-98.99)	NA	NA

Abbreviation: NA, not applicable.

^aData are presented as number of tonsillectomies, unless otherwise indicated.

Discussion

Automation of complication capture in the medical record is important because it allows for better capture of complications for use during morbidity and mortality conferences. In time, automation may prove to be useful for local, regional, and national quality measures that have traditionally relied on personnel trained to hand review the medical record, such as the Surgical Care and Outcomes Assessment Program and the American College of Surgery’s National Surgical Quality Improvement Program. Taking advantage of new initiatives such as the AAO-HNS’s Regent data registry, automated complication capture may aid in the measurement of postoperative quality and for satisfying the new Medicare Access and CHIP Reauthorization Act of 2015 Merit-Based Incentive Payment System requirements. Automated reports could help surgeons recognize trends in their practices as they change over time, such as increased posttonsillectomy bleed rates, that might signify the need for closer analysis or change. In addition, the algorithm may improve the ability to perform quick population research on procedures and their outcomes.

The algorithm was successful in identifying tonsil bleeding that was not recorded in our self-reported databases. The algorithm identified at least 2 children who returned to the operating room for control of their hemorrhages at a hospital outside our health system but in the HIE’s network. The algorithm discovered bleeding complications in patients who returned to our pediatric hospital and went to the operating room that were not self-reported. It also identified minor bleeding that resolved in the emergency department without a report to our surgical team. The algorithm identified 2 patients who had health insurance data integrated into the INPC that suggested they were treated for bleeding approximately 1 week after their tonsillectomies. Altogether, the algorithm could identify approximately two-thirds of the bleeding complications that had encounters between a patient and a clinician in the medical record.

However, the algorithm was not perfect. It failed to identify approximately one-third of the bleeding complications. It did not identify all tonsillectomy cases in the record, missing 7.38%. Failures to correctly identify procedures and events are attributable to multiple reasons. First, identifying all true tonsillectomy procedures performed was difficult without discrete billing data for CPT-4 codes. A clear correspondence between a surgeon’s operative note and whether a tonsillectomy had been performed was not always available. Additional billing data would improve identifying when a tonsillectomy was performed and thereby improve some of the complications that were solely missed because the tonsillectomy was not identified. In addition, ICD-9 and ICD-10 codes are blunt instruments that do not fully capture the nuances of an event. One example includes a case in which bleeding was identified to be coming from the adenoidectomy site instead of the tonsillectomy that fell under the 998.11 code, hemorrhage complicating a procedure. This code identified a complication but not the specific one on which we were focused. The ICD-9 code 528.9, other unspecified diseases of the oral soft tissues, was found to be too broad in our experiment and created many false-positive findings because of other complications or events falling under the diagnosis of unspecified oral tissue disease. Use of this code will be discontinued. Certainly, refinement to the free-text searches of clinical notes during an encounter can be improved with care not to overfit the search algorithm. We believe that this change will improve the algorithm over time. The ICD-10 codes did not identify any bleeding events during the test period, although we have seen the codes used since. We suspect that ICD-10 was not fully integrated during the initial transition period.

Strengths and Limitations

This study demonstrates that existing systems with minimal augmentation can be retooled to create useful surgical analyses and to identify opportunities for improvement. We created a system that automates identification of a surgeon’s patients and procedures in the medical record and uses the power of HIEs to identify postoperative tonsillectomy complications regardless of whether the complication’s location is the same as where the original procedure was performed. Automatic tracking of postoperative tonsillectomy bleeding has been demonstrated as feasible but needs continued work to improve accuracy.

Use of HIEs to automatically gather perioperative complications can help inform the surgeon of the events surrounding an intervention. Surgeons and, indirectly, their patients and society can benefit from the growing health record system by identifying and analyzing perioperative events. These data can be used to provide continual feedback that informs their clinical practice. We plan to distribute the reports produced to surgeons as an adjunct to compare with their self-reported bleeding complications.

Health systems with similar HIEs can likely duplicate what has been achieved herein. We imagine that new research may build thoughtful interrogations of the medical record to develop new analyses and expand into other procedures, other complications, and even other events such as preoperative and intraoperative laboratory values. Examples could include hematoma events after thyroidectomies, chyle leaks after neck dissections, airway events after tracheostomy, preoperative nutrition status, and preoperative comorbidities. These automated analyses may also prove to be useful as health systems move toward removing waste and adding value through standardizing approaches to data collection and evaluation of outcomes.

Limitations to automation exist, and refinements need to be made. The algorithm by design did not capture bleeding that was reported over the telephone and documented in a telephone note. The system also missed tonsil complications that were self-reported in which the patient had subjective self-reported bleeding before physical evaluation in the emergency department and was not formally diagnosed with posttonsillectomy bleeding. We used the CPT-4 codes for control of oropharyngeal hemorrhage to improve the algorithm’s capture of complications. This system would not be available for many other perioperative events. Some operative notes were misidentified as tonsillectomies because the dictation would state phrases like “… prepped and draped in typical fashion as for adenotonsillectomy” but were describing a different procedure. Some emergency department notes, although typewritten, were reported to the HIE as portable document files (PDFs; Adobe Systems) and were not currently within our means to search.

Conclusions

The purpose of this study was to demonstrate the feasibility of automating the capture of posttonsillectomy hemorrhages. We believe that this algorithm demonstrates the promise of automated review of the medical record for complications. It also demonstrates the value of using HIEs for capture of cases not identified by self-report.