Derivation and Validation of Automated Electronic Search Strategies to Identify Pertinent Risk Factors for Postoperative Acute Lung Injury

Abstract

OBJECTIVE: To develop and validate time-efficient automated electronic search strategies for identifying preoperative risk factors for postoperative acute lung injury.

PATIENTS AND METHODS: This secondary analysis of a prospective cohort study included 249 patients undergoing high-risk surgery between November 1, 2005, and August 31, 2006. Two independent data-extraction strategies were compared. The first strategy used a manual review of medical records and the second a Web-based query-building tool. Web-based searches were derived and refined in a derivation cohort of 83 patients and subsequently validated in an independent cohort of 166 patients. Agreement between the 2 search strategies was assessed with percent agreement and Cohen κ statistics.

RESULTS: Cohen κ statistics ranged from 0.34 (95% confidence interval, 0.00-0.86) for amiodarone to 0.85 for cirrhosis (95% confidence interval, 0.57-1.00). Agreement between manual and automated electronic data extraction was almost complete for 3 variables (diabetes mellitus, cirrhosis, H₂-receptor antagonists), substantial for 3 (chronic obstructive pulmonary disease, proton pump inhibitors, statins), moderate for gastroesophageal reflux disease, and fair for 2 variables (restrictive lung disease and amiodarone). Automated electronic queries outperformed manual data collection in terms of sensitivities (median, 100% [range, 77%-100%] vs median, 87% [range, 0%-100%]). The specificities were uniformly high (≥96%) for both search strategies.

CONCLUSION: Automated electronic query building is an iterative process that ultimately results in accurate, highly efficient data extraction. These strategies may be useful for both clinicians and researchers when determining the risk of time-sensitive conditions such as postoperative acute lung injury.

Automated electronic query building is an iterative process that ultimately results in accurate, highly efficient data extraction, and these search strategies may be useful for both clinicians and researchers when determining risk of time-sensitive conditions such as postoperative acute lung injury.

ALI = acute lung injury; COPD = chronic obstructive pulmonary disease; DDQB = Data Discovery and Query Builder; DM = diabetes mellitus; EMR = electronic medical record; GERD = gastroesophageal reflux disease; MCLSS = Mayo Clinic Life Sciences System; PPI = proton pump inhibitor

Acute lung injury (ALI) is a devastating postoperative respiratory complication and a leading cause of postoperative respiratory failure,^1-3 with a mortality rate of up to 45% in certain surgical populations.^4,5 Moreover, treatment options are limited once the condition is fully established. Earlier identification of at-risk populations may allow the implementation of effective ALI prevention strategies. Recognizing that numerous baseline factors can modify a patient's response to illness or injury and the likelihood of developing ALI, we recently developed an ALI risk prediction model for mixed medical and surgical populations.^6,7 This score assigns points both for conditions that predispose patients to ALI (eg, shock, aspiration, sepsis, pancreatitis, pneumonia, high-risk surgery, high-risk trauma) and ALI-modifying factors (eg, sex, excess alcohol use, obesity, chemotherapy, diabetes mellitus [DM], smoking) at the time of hospital admission. We have shown the cumulative score to be a reliable predictor of the risk of developing ALI during hospitalization.

A key remaining limitation to the early identification of patients at high risk of postoperative ALI is the inability to identify risk factors in a timely manner. Currently, most investigators retrieve patient data manually from the medical record and reintroduce the data into research databases. This process, which is time-consuming and inefficient, risks inaccuracies due to errors in data entry.^8,9 Its scalability is also limited. Adoption of the electronic medical record (EMR) provides new opportunities for automated disease surveillance strategies. In particular, the availability of high-throughput information technology solutions for extracting clinical data provides an opportunity to improve the efficiency and conduct of clinical and translational research.^10-12 However, electronic data extraction is not without limitations.^13-15 Validation of data accuracy and completeness is necessary because data can be missing or incorrect as a result of the data-entry process or of corruption during data transmission, storage, or retrieval.¹¹

For editorial comment, see page 373

This investigation aimed to determine the accuracy of innovative, Web-based automated electronic data-extraction strategies for identifying risk factors for postoperative ALI. To achieve this objective, we compared these novel data-collection strategies with an independent manual data collection. We hypothesized that Web-based data-extraction strategies would identify pertinent risk factors for postoperative ALI with an accuracy comparable to or exceeding that achieved with manual data collection.

PATIENTS AND METHODS

After obtaining approval by the Mayo Clinic Institutional Review Board, we performed a secondary analysis of a prospective cohort study comparing 2 independent strategies for identifying pertinent preoperative risk factors for postoperative ALI.

Our study population was obtained from a previous prospective cohort evaluation investigating the association between intraoperative ventilatory parameters and the development of early (<5 days) postoperative ALI.³ Briefly, patients were included if mechanically ventilated for more than 3 hours during general anesthesia for intermediate- and high-risk surgical procedures. Patients were excluded if they denied permission to use their health information for research, they were younger than 18 years, they had prevalent major risk factors for lung injury or respiratory failure, or they had previously required mechanical ventilation. In a nested case-control design, 83 cases and 166 matched control patients (a total of 249 study patients) were identified from the full cohort from November 1, 2005, through August 31, 2006.

ALI Risk Factors

We analyzed data on preoperative clinical variables with evidence suggesting an association with postoperative ALI. Variables were categorized into 2 groups: preoperative comorbid conditions and preoperative medications. Preoperative comorbid conditions included DM,^3,16-18 chronic obstructive pulmonary disease (COPD),^3,19 restrictive lung disease,^20-22 cirrhosis, and gastroesophageal reflux disease (GERD). Classes of preoperative medications included proton pump inhibitors (PPIs), H₂-receptor antagonists, statins,²³ chemotherapeutic agents (within 6 months of the surgical procedure),^22,24 and immunosuppressants (within 6 months of the surgical procedure).^25-27 Amiodarone was also included as a medication of interest.^28,29 We interrogated the medical record for all medications contained within each class. Systemic corticosteroids were included in the list of medications considered for immunosuppressive therapy. Inhaled corticosteroids were not included.

It is important to note that a large portion of the included study population was referred from other health care facilities. As a result, documentation of the criteria leading to the diagnosis of a pertinent risk factor was often absent. Moreover, the retrospective nature of this investigation prevented prospective acquisition of this information. In light of these limitations, formal diagnostic criteria for the comorbid conditions of interest were not required. Rather, the comorbid conditions (eg, DM, COPD, restrictive lung disease, cirrhosis, GERD) were considered present if a physician documented the diagnosis in the EMR before the surgical procedure. Medications (eg, PPIs, H₂-receptor antagonists, statins, amiodarone) were considered present if they were being administered at the time of hospital admission. Because previous publications have suggested an increased risk of ALI with administration of chemotherapeutic and immunosuppressive medications up to 6 months before surgery, we increased the evaluation period for determining the presence of these medications to this interval (6 months before surgery).²⁴

Data-Extraction Strategies

Manual Data Extraction. The EMR of all study patients was interrogated by 1 of 2 trained study coordinators. Each was instructed to use standard operating procedures for the extraction of clinical data from the medical record. If a variable of interest was not identified in the EMR, it was assumed to be absent or negative. Similarly, if the variable was mentioned in the negative form (eg, “patient has no history of…” or “patient denies…”) or if it was listed in the patient's family history but was not specifically assigned to the study patient, it was again assumed to be absent or negative. The research coordinators responsible for manual data extraction were not involved in the automated electronic data-extraction process.

Automated Electronic Data-Extraction Strategies—Mayo Clinic Life Sciences System and Data Discovery and Query Builder. Mayo Clinic has established a partnership with IBM to collaboratively develop a sophisticated data warehouse (Mayo Clinic Life Sciences System [MCLSS]), which contains a near real-time normalized replicate of Mayo Clinic's EMR. This warehouse is developed from multiple original clinical data sources, including highly annotated, full-text clinical notes, laboratory tests, diagnostic findings, demographics, and related clinical data from the year 2000 onward. Mayo Clinic's EMR data are extracted, transformed, and loaded into MCLSS using IBM's WebSphere Commerce Analyzer, creating DB/2 Universal Database structures of Mayo Clinic's normalized clinical data. Clinical patient data are mapped to standard medical terminologies using LexGrid (Biomedical Statistics and Informatics, Mayo Clinic, Rochester, MN) natural language processing technology.

The MCLSS also provides approved users with a query-building tool called the Data Discovery and Query Builder (DDQB). The DDQB is a Web-based application configured for query building that is intended to help physicians and researchers interrogate data files contained in the MCLSS. The DDQB allows users to identify administrative, demographic, laboratory, and diagnostic data of interest within the EMR. At the center of the DDQB software is the data abstraction model, an XML-based component that maps end-user content expertise to physical representations. As the data within the MCLSS change, the data-abstraction model is modified while maintaining the validity of previously generated queries.

The DDQB allows users to build complex queries using Boolean logic, groupings, comparison operators, and relational joins without requiring programming knowledge. For new users, a 1-hour orientation is provided before use. The DDQB can then be used to query administrative data such as diagnostic and procedure-related codes. Complex query construction and free text searches of the EMR are also available to approved users. Because we wanted to develop near real-time data-collection strategies, we elected to pursue free text data searches rather than searching administrative codes such as International Classification of Diseases, Ninth Revision diagnostic codes; the latter are not applied in a time-efficient manner, and concerns have been raised about their accuracy in identifying comorbid conditions of interest.^14,30-32 Free text data queries were also used when interrogating the EMR for medications of interest. All MCLSS/DDQB searches were run by 1 of 2 physician investigators (A.A. or D.J.K.). Neither of these investigators participated in the manual extraction of clinical data.

For the automated extraction of medical comorbid conditions, MCLSS/DDQB was used to interrogate the EMR of each study patient during the 5-year period preceding surgery. Queries were restricted to the following sections of the clinical notes: history of the present illness, past medical/surgical history, and diagnosis. To optimize sensitivity, each query was designed to identify both the disease of interest and the common synonyms, acronyms, and abbreviations used to represent the disease. To improve specificity, we excluded negative forms of the condition of interest (eg, “patient does not have a history of…” or “there is no history of…”). We also excluded diagnoses referring to a family history of the condition. Negative forms such as denies, doesn't, and no were excluded as well. Query building was an iterative process in a derivation cohort of 83 study patients (ALI cases). Searches were performed and the results analyzed, and, on the basis of these results, more key terms, synonyms, acronyms, and abbreviations were added. Once finalized, the queries were validated in a second cohort of 166 patients (controls). A representative query is provided in the Figure. For the automated extraction of patient medications, free text searches were limited to the “current medication” section of the clinical notes. We limited the evaluation interval for medications of interest to the 3 months before surgery, extending it to 6 months for chemotherapy and immunosuppressive therapy. Clinical notes were interrogated for generic and trade names of all formulations within each specific pharmaceutical class. A representative medication search query is shown in Table 1.

FIGURE.

Electronic query for identifying a diagnosis of diabetes mellitus (DM) in the study population. HPI = history of present illness; MCLSS = Mayo Clinic Life Sciences System; PMH/PSH = past medical and surgical history.

TABLE 1.

Electronic Query for Identifying the Presence of Statin Therapy at the Time of Hospital Admission in the Study Population

Validation of the Manual and Electronic Data-Extraction Techniques

After all data had been extracted, concordance between the 2 data-extraction techniques was assessed. All discordant and concordant negative results were investigated with an exhaustive manual review of the medical record (A.A. and D.J.K.). The specific procedures for this secondary review were dependent on the initial data-extraction result. All positive findings resulting from an automated Web-based query provide a direct link to the EMR identifying the exact location of the variable of interest in the patient's medical record. For all discordant results, where the condition was identified as being present with a Web-based query, this link was used to evaluate the accuracy of the initial positive result. For discordant results with a positive finding during manual review of the EMR, a detailed secondary review was performed to ensure its validity. For concordant negative results, 2 physician investigators (A.A. and D.J.K.) extensively reevaluated the EMR to confirm the absence of the variable of interest. In this final exhaustive review, interrogation was not limited to specific sections of the clinical notes (eg, “past medical history” or “active medications”) but rather included all components. Concordant positive findings were considered true-positive results and were not further evaluated. Comorbid conditions (eg, DM, COPD, restrictive lung disease, cirrhosis, GERD) were considered present if documented in the EMR before the surgical procedure. Medications (eg, PPIs, H₂-receptor antagonists, statins, amiodarone) were considered present if they were being administered at the time of hospital admission. Medications identified as being present by either search strategy but determined during the final review of the EMR to have been discontinued before the surgical procedure were considered false-positive results. Chemotherapy and immunosuppressive therapy were considered present if administered at any time in the 6 months before the surgical procedure.

Statistical Analyses

Because a true criterion standard for determining the presence of comorbid conditions and medications of interest is lacking, we report percent agreement and Cohen κ statistics as our primary statistical analysis comparing manual and automated electronic data-extraction procedures. As a secondary analysis, we have determined the sensitivity and specificity of both manual data extraction and automated electronic data extraction for each variable of interest. The final exhaustive review of the medical record was used as the “criterion standard” when determining the sensitivities and specificities of the 2 search strategies.

RESULTS

The prevalence of medical comorbid conditions and administration of medications of interest (as determined by the final review of the EMR) ranged from 1% for immunosuppressive therapy to 50% for statin use. The percent agreement and associated Cohen κ statistics for each variable can be seen in Table 2. Due to a very low prevalence of preoperative chemotherapy and immunosuppressive therapy and the poor sensitivity for identifying these variables with manual review of the EMR, Cohen κ statistics are not reported for these variables. Of the comorbid conditions, DM and cirrhosis showed the greatest agreement and amiodarone therapy the least. Using the Cohen κ statistic magnitude guidelines described by Landis and Koch,³³ we determined that the agreement between manual and automated electronic data-extraction techniques was almost complete for 3 variables (DM, cirrhosis, H₂-receptor antagonists), substantial for 3 variables (COPD, PPIs, statins), moderate for a single variable (GERD), and fair for 2 variables (restrictive lung disease and amiodarone).

TABLE 2.

Percent Agreement and Cohen κ Statistics Comparing Manual vs Automated Electronic Data-Extraction Strategies in the Validation Cohort (N=166)

The rates of concordance and discordance are presented in Table 3. The sensitivity and specificity of each data-extraction strategy for all variables considered are presented in Table 4. Sensitivities for identifying variables of interest with manual data extraction ranged from a low of 0% for the infrequent variables chemotherapy and immunosuppressive therapy to a high of 100% for restrictive lung disease and cirrhosis (median sensitivity, 87%). Sensitivities for identifying variables of interest with MCLSS/ DDQB ranged from 77% for amiodarone therapy to 100% for COPD, restrictive lung disease, cirrhosis, H₂-receptor antagonists, PPIs, chemotherapy, and immunosuppressive therapy (median sensitivity, 100%). The sensitivities for identifying variables of interest with the Web-based electronic queries were higher than the corresponding sensitivities with manual review of the medical record for 7 of the 11 variables evaluated, identical for 3 variables, and lower for a single variable (statin therapy). The specificities were uniformly high (≥97%) for both manual and electronic data-extraction strategies.

TABLE 3.

Prevalence of Risk Factors for Postoperative Acute Lung Injury and the Concordance Between Manual and Automated Electronic Data-Extraction Strategies in the Validation Cohort (N=166)^a

TABLE 4.

Calculated Sensitivities and Specificities for Manual and Automated Electronic Search Strategies in the Validation Cohort (N=166)

DISCUSSION

This study compared 2 independent strategies for identifying pertinent risk factors for postoperative ALI. Our results suggest fair to excellent agreement between time-consuming manual review of the medical record and a highly efficient automated electronic search tool using data from a near real-time copy of the EMR. Comparison of the 2 strategies to an exhaustive review of the EMR revealed that the sensitivities of automated search strategies were consistently equal to or greater than that for the manual reviews of the medical record. The specificities of the searches were high and comparable for both data-collection strategies.

Early disease recognition with timely intervention has resulted in dramatic improvements in outcome for patients with acute coronary syndrome,^34,35 severe sepsis/septic shock,³⁶ and numerous other life-threatening conditions. Similarly effective strategies for the treatment of ALI have been suggested in preclinical studies.^37-41 Unfortunately, these promising preclinical results have not translated to the clinical setting, likely in part because strategies are applied too late in the progression of disease. Considering the limitations of clinical treatment trials and the disappointing results to date, prevention of ALI may prove a more effective approach than treating advanced disease. Working to this end, we have recently developed a lung injury prediction score for predicting a patient's risk of ALI at the time of hospital admission.^6,7 However, a remaining obstacle for the successful implementation of prediction models is the timely identification of variables known to portend risk. Historically, these variables have been identified with manual review of the medical record. This process is time-consuming, inefficient, and frequently prone to error.^8,9

The development and implementation of efficient automated electronic data-extraction strategies may provide an opportunity to identify pertinent risk factors and assess patient risk in near real-time. However, automated electronic search strategies have their own limitations. Pertinent data can be missing, errors can be made with data entry, and corruption can occur with data management, storage, and retrieval.¹¹ Valid results also depend on the accurate acquisition of patient information at the time of the initial patient encounter. Therefore, validation of new data-extraction techniques is mandatory.^8,9

It should also be emphasized that, although well-designed validated queries can be highly efficient, the query-building process is typically iterative and can often be quite laborious. This is particularly true of free text queries, in which abbreviations, synonyms, and acronyms are commonplace. Additionally, key terms are often presented in the negative form and in a non–patient-specific format such as family history. We noted the identification and handling of the negative forms of key terms to be particularly important when optimizing search specificity.

Alternatives exist to the free text search strategies used in this study. The Systematized Nomenclature of Medicine–Clinical Terms (SNOMED CT) is endorsed by health care standard organizations as the most comprehensive terminology for coding data in clinical information systems.⁴² Previous investigations have noted generally excellent content coverage with SNOMED CT.^43,44 However, limitations exist with this system as well. For time-sensitive searches, the coding process can introduce an undesirable delay in the time from data entry into the EMR to SNOMED CT code allocation. SNOMED CT also appears to map poorly to some important critical care conditions (eg, severe sepsis⁴⁵) as well as to numerous diagnoses included in common critical care severity of illness scoring systems, such as the APACHE (Acute Physiology and Chronic Health Evaluation) IV survey.⁴⁶ Similarly, SNOMED CT content coverage may not be as robust in the preoperative setting.⁴⁷ Finally, the reliability of SNOMED CT coding is imperfect and requires repeated training and testing.⁴⁸

More advanced natural language processing techniques could also be used for identifying the textual information of interest. However, these techniques require extensive training and specific expertise as well as large training datasets.^49,50 Instead, we aimed to develop and validate portable automated strategies that could be used by clinician scientists who lack this specific expertise. It is unclear that more advanced natural language processing techniques would improve the accuracy of data extraction.^51,52

Our study has several important limitations. First, we focused on 2 categories of data: patient comorbid conditions and medications. The completed queries were accurate, but considerable variability was noted in the early phases of the query-building process. The iterative nature of query building prevents us from generalizing our findings to other medical conditions or medications of interest because each would require independent validation. Similar validation would be required for other data elements such as demographics or laboratory results. A second limitation is our lack of strict definitions for the comorbid conditions of interest. The retrospective nature of this investigation and the substantial referral population prevented the use of strict diagnostic criteria. Rather, we chose to accept the documentation of a disease of interest in the EMR as sufficient for disease presence. It is possible that some of the patients in this investigation were inappropriately assigned comorbid conditions without meeting the strict diagnostic criteria. Although we attempted to identify active medication administration at the time of the surgical procedure, we cannot be certain that patients were taking the medications listed in their EMR. This limitation could only be effectively addressed with a prospective study design. Another limitation is the lack of a true criterion standard for determining the presence of our variables of interest. To allow a comparison of the 2 search strategies evaluated, we performed an exhaustive review of the medical record in an effort to create a criterion standard. This process has limitations and may have resulted in biased sensitivities and specificities for the queries performed. Moreover, the limited sample size results in rather imprecise estimates for the reported sensitivities and specificities, particularly among the lower frequency variables. A final important limitation is the single-center nature of this study and our study population. Although our specific results cannot be generalized to other institutions, the concepts presented have the potential for broad applicability.

CONCLUSION

Our findings validate the use of an automated electronic data-extraction tool for identifying pertinent preoperative risk factors for postoperative ALI. These strategies may be used to assist both clinicians and researchers when determining the risk of time-sensitive postoperative complications such as ALI.