Development and Validation of Electronic Health Record–based Triggers to Detect Delays in Follow-up of Abnormal Lung Imaging Findings

An electronic health record–based trigger algorithm was developed to identify patients who are at risk for delayed follow-up of abnormal imaging findings. The trigger achieved a positive predictive value of 57.3% and shows promise in detecting diagnostic delays in patients with imaging findings suspicious for lung cancer.

Abstract

Purpose

To develop an electronic health record (EHR)–based trigger algorithm to identify delays in follow-up of patients with imaging results that are suggestive of lung cancer and to validate this trigger on retrospective data.

Materials and Methods

The local institutional review board approved the study. A “trigger” algorithm was developed to automate the detection of delays in diagnostic evaluation of chest computed tomographic (CT) images and conventional radiographs that were electronically flagged by reviewing radiologists as being “suspicious for malignancy.” The trigger algorithm was developed through literature review and expert input. It included patients who were alive and 40–70 years old, and it excluded instances in which appropriate timely follow-up (defined as occurring within 30 days) was detected (eg, pulmonary visit) or when follow-up was unnecessary (eg, in patients with a terminal illness). The algorithm was iteratively applied to a retrospective test cohort in an EHR data warehouse at a large Veterans Affairs facility, and manual record reviews were used to validate each individual criterion. The final algorithm aimed at detecting an absence of timely follow-up was retrospectively applied to an independent validation cohort to determine the positive predictive value (PPV). Trigger performance, time to follow-up, reasons for lack of follow-up, and cancer outcomes were analyzed and reported by using descriptive statistics.

Results

The trigger algorithm was retrospectively applied to the records of 89 168 patients seen between January 1, 2009, and December 31, 2009. Of 538 records with an imaging report that was flagged as suspicious for malignancy, 131 were identified by the trigger as being high risk for delayed diagnostic evaluation. Manual chart reviews confirmed a true absence of follow-up in 75 cases (trigger PPV of 57.3% for detecting evaluation delays), of which four received a diagnosis of primary lung cancer within the subsequent 2 years.

Conclusion

EHR-based triggers can be used to identify patients with suspicious imaging findings in whom follow-up diagnostic evaluation was delayed.

^© RSNA, 2015

Introduction

Delays in the diagnosis of lung cancer from failure to follow up patients with abnormal imaging results remains a common problem that contributes to both malpractice litigation and poor patient outcomes (1–6). These delays in care, which typically involve well-intentioned clinicians, also occur when clinicians have good access to electronic diagnostic information. Multiple factors likely contribute to the delays, including time pressures, information overload, heavy workloads, and a lack of robust test-result tracking systems (7–15). Currently, identification of such delays in care requires a review of the records of all patients who undergo imaging. However, this nonselective review is too inefficient for practical use given the number of records that would require review and the vast amounts of information contained in electronic health records (EHRs). Likewise, sending providers information created with simple search queries (eg, all patients with abnormal chest radiography findings) for the purpose of identifying patients who need follow-up is not practical because most clinicians are already overloaded with information.

The increasing adoption of EHRs has led to the development of large repositories of patient data spanning the continuum of care (16). By taking advantage of this data, “trigger tools” could be developed and applied to automate the process of detecting records of patients who are experiencing delays in receiving diagnostic evaluation. Trigger tools use an algorithm based on certain clues to flag the records of patients who have a higher risk for harm so that their records can be reviewed for possible safety events (17–20). The use of trigger tools in radiology has been limited; a recent study proposed that the presence of lower extremity thrombosis at ultrasonography is a potential indicator of insufficient inpatient venous thromboembolism prophylaxis (21). Expanding on the capabilities of simple search queries, triggers can account for multiple inclusion and exclusion criteria to increase their positive predictive value and facilitate their prospective, potentially real-time, application. While triggers have traditionally been used to detect adverse drug events and nosocomial infections, we successfully developed them to detect delays in care related to follow-up of patients with positive fecal occult blood test results and abnormal prostate-specific antigen results (22–27).

Approximately 9% of patients with abnormal chest imaging findings suspicious for neoplasm fail to receive timely follow-up diagnostic evaluation (2,28). Development of a trigger to efficiently identify such delays in diagnostic evaluation could assist in the development of a back-up, or “safety net,” system to prospectively detect patients who are at risk for delays and alert designated personnel to take action (29). The EHRs at the Department of Veterans Affairs offer a unique opportunity to apply and evaluate the effectiveness of triggers on imaging results. When interpreting imaging results, radiologists at the Department of Veterans Affairs flag abnormal reports by using a structured computer code that alerts the referring provider (via their EHR inboxes) about the flagged abnormality. Most facilities use a “suspicious for malignancy” code to identify an imaging result that is suggestive of cancer. A similar system for rating mammograms (the Breast Imaging Reporting and Data System) is nearly universal throughout the United States, but the use of other structured ratings systems for imaging results is uncommon. In this study, we developed an electronic EHR-based trigger algorithm to identify delays in follow-up evaluation of patients with imaging findings suggestive of lung cancer, and we validated the trigger with retrospective data.

Materials and Methods

Setting

Data were obtained from the clinical data repository of a large tertiary care Veterans Affairs facility that provides both inpatient and outpatient care. The local institutional review board approved the study.

Trigger Development

We performed literature reviews and interviewed specialists and primary care providers to develop criteria for the trigger algorithm on the basis of existing processes for follow-up evaluation used by providers at the facility. We operationally defined each criterion to enable conversion into a set of computerized search criteria, and chart reviews were performed on a “test cohort” to determine the validity of our code for extracting data for each individual criterion. We previously evaluated the effectiveness and feasibility of the “age,” “terminal illness diagnoses,” and “hospice/palliative care enrollment” criteria during work on similar nonimaging triggers for patients with prostate and colorectal cancers (27). In these cases, we did not perform a repeat review of the identified records. The algorithm included criteria designed to identify patients with imaging findings consistent with a possible lung malignancy and exclude patients who do not need follow-up evaluations. The algorithm (Table 1) initially identified records of patients at risk for a delay in care on the basis of two sets of inclusion criteria: (a) “red flag” criteria to identify clinical clues, such as an abnormal imaging finding, and (b) demographic criteria, such as patients who were alive and within the age group in which follow-up action would be expected. The trigger subsequently removed records that did not require additional action on the basis of two sets of exclusion criteria: (a) clinical criteria to exclude patients who do not need follow-up evaluation, such as those with a terminal illness, and (b) expected follow-up criteria to exclude patients who already received appropriate follow-up care within a prespecified timeframe (Figure).

Table 1.

Criteria Used to Develop the Trigger Algorithm

Flow chart shows the trigger algorithm and chart review process.

Because there is no agreed upon definition of a delay in chest imaging follow-up, we chose 30 days as the maximum timeframe in which to complete an initial follow-up diagnostic evaluation. This timeframe was chosen on the basis of discussions with expert providers who considered the time needed to have patients see specialists and undergo testing. Furthermore, it is likely that, should lung cancer be definitively identified, the clinical impact of a delay of 30 days would be minimal. All criteria were based on structured data fields, including the International Classification of Diseases, Ninth Revision, and the Current Procedural Terminology codes, as well as a structured “suspicious for malignancy” code assigned to radiographs by radiologists at the time of their interpretation. The final criteria were reviewed by expert providers and are shown in Table 1. The criteria were coded into a structured query language computer algorithm and applied to the local data warehouse for the facility, a separate repository of clinical data extracted from the main EHR, twice per month.

Trigger Performance

We applied the final trigger algorithm to all patient records with a chest radiography or CT result during a 1-year period (the “validation cohort”) and calculated the positive predictive value (PPV), which is defined as the percentage of patients who were flagged by the trigger and were determined to need follow-up care but did not receive it. For practical use, triggers must achieve a reasonable PPV to avoid overloading recipients with false-positive results. We assumed that no recipient would want a false-positive rate of more than 50%, and our previous work suggests that triggers can achieve this level (27). Thus, we designed the study to detect a PPV of at least 50% with confidence intervals of plus or minus 10% and a two-sided α of 0.05, and we determined that a minimum of 97 records needed to be reviewed. By using a piloted chart review form, a trained physician assistant at the facility with over 30 years of clinical primary care experience performed manual chart reviews of all identified records to determine whether they truly contained a delay in diagnostic evaluation. Prior to proceeding, the initial 20 records were independently reviewed by a second physician reviewer (D.R.M.) with 9 years of internal medicine experience, and the findings were compared to ensure an interrater reliability of at least 80%. Records that were identified by the trigger but that were found to not have a delay in care were deemed false-positive results. The reasons for the false-positive results were recorded during review. The ratio of true-positive results to total records identified by the trigger was used to determine the PPV of the trigger algorithm. Finally, 2 years after the imaging test, we performed a follow-up review to determine whether patients subsequently received a diagnosis of lung cancer.

Data Analysis

Data were analyzed by using Excel software (Microsoft, Redmond, Wash) and SPSS version 21 (IBM, Armonk, NY). Trigger performance, time to follow-up evaluation, reasons for a lack of follow-up care, and cancer outcomes were analyzed and reported by using descriptive statistics.

Results

Trigger Development

We applied the trigger criteria to all patients at the facility who received chest imaging results between January 1, 2008, and December 31, 2008 (the test cohort). We performed manual record reviews on 36 randomly selected charts, making iterative refinements and reapplying the algorithm criteria after each review. Because Veterans Affairs facilities provide care to United States military veterans, the population was predominantly men (92%) aged 18 years and older (30).

Trigger Performance

The final algorithm was programmed as a computerized abnormal imaging trigger and was applied to the records of all patients with chest radiography or CT results between January 1, 2009, and December 31, 2009 (the validation cohort). Of 89 168 patients who were seen at the facility during the study period, 24 829 (27.8%) had at least one chest radiography or CT study performed, of which 538 records met both the red flag (an imaging result that was flagged as being suspicious for malignancy) and demographic criteria (Figure). Subsequently, 207 records were excluded on the basis of clinical exclusion criteria, and 200 records were excluded on the basis of expected follow-up criteria. The remaining 131 records (24.3% of those that met the inclusion criteria and 0.5% of all records for patients who underwent imaging) for patients at high risk for experiencing delays in diagnostic evaluation were manually reviewed. During manual review, 56 records were identified as false-positive results, while the remaining 75 were found to truly lack timely diagnostic evaluation (PPV, 57.3%; 95% confidence interval: 48.3, 65.8) (Table 2). Of the 131 records of patients at high risk for delayed diagnostic evaluation, 18 (13.7%) were ordered for screening indications (with “annual exam” or “pre-op” given as the reason for the examination), and the remaining 113 (86.3%) were ordered for diagnostic purposes.

Table 2.

Reasons for Delays in Follow-up of Patients with Abnormal Imaging Findings as Ascertained by Record Review

Note.—Data are in parentheses are percentages.

Among the 56 false-positive results (ie, records that were incorrectly identified by the trigger as positive), the most common reason was that the finding that was “suspicious for malignancy” was not in the lung, such as with a chest wall or mediastinal mass (in 17 [30.4%]). Follow-up actions for these findings would typically be handled differently than they would for a primary lung malignancy, and our trigger did not account for these actions. Other common reasons for false-positive results included patients who were receiving outside care (12 [21.4%]) or who declined to undergo further evaluation of the finding (eight [14.3%]). This information was only available within the free-text progress notes.

Of the 75 records without documented justification for the delay in follow-up (ie, true-positive results), in 35 records, follow-up studies were ordered in response to the abnormal findings but were not performed within 30 days. Within this subsample, the median time for completion of follow-up action was 49 days (interquartile range, 35–127 days). The remaining 40 records had no evidence of follow-up action. The reasons for delays in follow-up evaluation of abnormal imaging findings, as ascertained by a record review, are outlined in Table 2. Of the 75 patients with imaging results that were suspicious for lung cancer and who did not receive timely follow-up action, four (5.3%, with a median follow-up period of 237 days) subsequently received a diagnosis of primary lung cancer during the following 2 years, 47 (62.7%) had findings that were subsequently attributed to nonneoplastic causes, and 24 (32.0%) had no evidence of subsequent diagnostic evaluation or a documented reason for the absence of follow-up action. The latter findings were relayed to clinic leadership.

Discussion

An EHR-based trigger algorithm can help automate the process of identifying patients who did not receive timely follow-up of chest radiography and CT findings suggestive of lung malignancy. The trigger achieved a PPV of 57.3% (75 of 131) and reduced the number of records that required review to less than one-quarter (24.3%, 131 of 538) of the records that would require review had the process been nonselective. Triggers could harness the vast amounts of EHR data to help develop a safety net for detecting and mitigating delays in follow-up action for patients with abnormal imaging findings (21,26). Systems-based interventions to address this safety issue have been slow to implement, and EHR triggers could be a promising alternative (3,7,31–34).

Although our application included retrospective data, such triggers can be prospectively applied or used in near real time to identify records with potential delays in care. For example, trigger algorithms could be incorporated into panel management programs or registries, allowing the care team to identify and address delays before patient outcomes are affected. Some institutions use nurse navigator programs to track imaging findings that are suspicious for cancer until a definitive diagnosis is made (35–37). Prospective use of triggers by nurse navigators could enable the navigators to more efficiently identify and track suspicious findings. Building on prior work in tracking abnormal imaging findings, our triggers automate the process of both identifying high-risk patients and excluding those who do not require follow-up action. This would allow nurse navigators to focus their attention on patients who truly need action and direct less attention to reviewing false-positive results (38). Dedicated pulmonary nodule tracking programs that are operated by nurse navigators and use EHR-based triggers could be used to help patients navigate the health care system, improve consistency of follow-up actions, and reduce delays in care (39,40). The use of standardized codes for diagnostic criteria (eg, the International Classification of Diseases and the Current Procedural Terminology) enables portability among sites with only limited modification before implementation.

The ability of the trigger to identify imaging results that are suspicious for lung cancer depended on the radiologists’ use of structured “suspicious for malignancy” codes with which to tag reports during interpretation. Unfortunately, the use of such codes is uncommon outside the Department of Veterans Affairs, limiting widespread application of such triggers. While the use of sophisticated natural language processing and text mining algorithms may help, they are more difficult to implement and standardize across sites and have other accuracy and precision limitations. To address this problem, we recommend the development of a standardized set of radiology codes that could be more widely implemented across EHR systems. Similar to the Breast Imaging Reporting and Data System coding scheme, which greatly reduced ambiguity of mammogram interpretation and communication of results, a set of structured codes to indicate imaging findings that are suspicious for malignancy could be developed. These codes would allow EHRs to “understand” radiology report interpretations in clinical decision support tools (41). For example, regardless of the initial test result communication, an application could alert providers when action on an abnormal chest imaging finding has not been completed within a certain number of days and enable other types of institution-level measurements for an absence of follow-up (42). Future triggers could incorporate both structured data mining and natural language processing and leverage the benefits of each method while minimizing their limitations.

Several study limitations warrant discussion. We were unable to determine the false-positive rate and, thus, could not calculate the sensitivity and specificity of the triggers. This inability was because of the large number of negative record reviews that would be required to identify a single false-negative result and is a common limitation of data mining for outcomes with a low prevalence; however, the use of such triggers enables the detection of safety events in circumstances when manual reviews would otherwise be cost-prohibitive (43). This study was also not designed to evaluate the clinical impact of such triggers. Currently, there is no evidence that these triggers will improve outcomes such as morbidity, mortality, and disease stage at diagnosis. However, this study provides a basis for subsequent work on the prospective use of such triggers to test their impact on patient outcomes.

In conclusion, we developed an EHR-based trigger algorithm to identify patients with abnormal imaging findings who are at risk for delays in follow-up action. The trigger achieved a PPV of 57.3% and shows promise for detecting diagnostic delays in patients with imaging findings that are suspicious for lung cancer. Our study provides a starting point for future work on evaluating the clinical impact of the prospective use of such triggers, as well as the development of more advanced algorithms that extend beyond structured EHR data.

Advances in Knowledge

• ■ We developed and validated an electronic health record–based trigger algorithm to identify patients with imaging findings suspicious for lung cancer who lacked timely follow-up.

• ■ Over a 1-year period, the trigger identified 131 patients at a large health care facility, of whom 75 (57.3%) were found to have a delay in follow-up action, which was confirmed through a manual record review.

• ■ During the study period, 89 168 patients visited the facility, 24 829 had at least one chest radiograph or CT image obtained, and 538 had findings that radiologists flagged as being “suspicious for malignancy;” thus, the trigger reduced the number of records we needed to review to confirm follow-up delays to less than one-quarter of those with suspicious findings (24%; 131 of 538).

Implication for Patient Care

• ■ Health care organizations could leverage triggers to proactively detect potential delays in follow-up evaluation of patients with imaging findings suspicious for lung cancer and prevent delays in diagnosis.