Validating International Classification of Diseases Code (ICD) 10^th Revision Algorithms for Accurate Identification of Pulmonary Embolism

Abstract

Background:

Many research investigations for pulmonary embolism (PE) rely on the International Classification of Diseases 10^th modification (ICD-10) codes for analyses of electronic databases. The validity of ICD-10 codes in identifying PE remains uncertain.

Methods:

Using a pre-specified protocol, patients in the Mass General Brigham hospitals (2016–2021) with ICD-10 principal discharge codes for PE, those with secondary codes for PE, and those without PE codes were identified (N=578 from each group). Weighting was applied to represent each group proportionate to their true prevalence. The accuracy of ICD-10 codes for identifying PE was compared with adjudication by independent physicians. The F1 score, which incorporates sensitivity and positive predictive value (PPV), was assessed. Subset validation was performed at Yale-New Haven Health System.

Results:

1,712 patients were included (age: 60.6 years, 52.3% female). ICD-10 PE codes in the principal discharge position had sensitivity and PPV of 58.3% and 92.1%, respectively. Adding secondary discharge codes to the principal discharge codes improved the sensitivity to 83.2%, but the PPV was reduced to 79.1%. Using a combination of ICD-10 PE principal discharge codes, or secondary codes plus imaging codes for PE led to sensitivity and PPV of 81.6% and 84.7%, respectively, and the highest F1 score (83.1%, P<0.001 compared with other methods). Validation yielded largely similar results.

Conclusions:

Although the principal discharge codes for PE show excellent PPV, they miss 40% of acute PEs. A combination of principal discharge codes, and secondary codes plus PE imaging codes led to improved sensitivity without severe reduction in PPV.

INTRODUCTION

Clinical research and quality improvement initiatives for pulmonary embolism (PE) frequently rely on International Classification of Diseases (ICD) codes. These codes provide efficient ways for electronic phenotyping of PE in administrative claims databases and electronic health records (EHRs). Many research studies have used ICD, 10^th modification, (ICD-10) codes for PE to shed light on the epidemiology, safety, and comparative effectiveness of alternative therapeutics.[1–7] The American Heart Association (AHA) uses ICD-10 codes to identify patients for reporting annual Heart Disease and Stroke statistics.[8] The Centers for Disease Control and Prevention reports data based on ICD-10 codes to estimate the national incidence and mortality from PE.[9, 10]

However, the accuracy of the recent ICD-10 codes for identifying PE remains to be determined. The few existing validation studies did not fully ascertain whether the codes were in the principal (primary) or secondary positions, had a relatively small sample of patients, and did not represent a sample of patients with and without the ICD codes. Many were based on data from a single hospital.[1, 7, 11–17] This investigation aimed to validate ICD-10 PE claims codes using a multi-institutional cohort of patients with external validation in a separate, unique database.

METHODS

The rationale and general design features of this study (named PE-EHR+) were pre-specified and have been described previously.[1] Briefly, this multicenter study, funded by the AHA, took place between 2022 and 2024, and sought to validate efficient tools for identifying patients with PE in EHR databases. Patients from Mass General-Brigham (MGB) Health System, the largest health care system in Massachusetts, and a validation subset from Yale-New Haven Health System (YNHHS), the largest health system in Connecticut, were selected and reviewed according to pre-specified criteria.

Patients

Adult patients aged ≥18 years were selected in three equally sized groups from MGB between January 1, 2016, and December 31, 2021. Included patients required hospitalization with a principal discharge diagnosis of PE (group 1), a secondary discharge diagnosis of PE (group 2) (Table S1), or with no discharge diagnosis codes for PE (group 3). The choice for inclusion of the third group was intentional to ascertain whether false negatives existed (i.e., patients who had PE but were not assigned PE ICD-10 codes during the index admission).

Identification of PE via ICD-10 codes

The list of ICD-10 codes used for identifying PE has been described previously and is summarized in Table S1. Based on prior work with ICD-9 codes[1, 18, 19], we anticipated that adding ICD-10 secondary discharge diagnosis codes to principal discharge diagnosis codes would improve the sensitivity of detecting PE. However, there was concern about reduced positive predictive value (PPV) with secondary codes since the latter could indicate PE events in the past rather than acute PE or represent suspected and unconfirmed PE. Therefore, we pre-specified to assess the accuracy of combining PE secondary discharge codes with Current Procedure Terminology (CPT) imaging codes capable of diagnosing PE (Table S2) that occurred in close temporal proximity to the PE codes to determine whether false positives of secondary PE codes were reduced, and accuracy was improved (Table S2). In additional supplemental analyses, we explored the accuracy of results based on whether ICD-10 codes had a present on admission indicator.

Ascertaining the diagnosis of PE by independent review or records

The diagnosis of PE was ascertained in each case by two physicians (AB and CDK) who reviewed the charts independently using pre-specified criteria, including review of medical notes, vital signs, laboratory data, as well as imaging reports from computed tomography scans, ultrasound studies, and others as needed. Details about the criteria have been described previously, and a brief summary is provided in Table S3.[1] Discrepancies were resolved by consulting with a third physician (BB).

Outcomes

The main study outcomes were accuracy metrics for ICD-10 codes in the principal discharge position and in the principal or secondary discharge position, including in combination with imaging codes, for identification of patients with PE. Sensitivity, specificity, and predictive values were obtained using numerator and denominator values based on standard epidemiological definitions.[20] In addition to sensitivity and PPV, a holistic parameter, the F1 score, was considered to compare the diagnostic accuracy of the codes compared with manual review..[21]

Statistical considerations

Categorical variables were reported with frequency counts and percentages. Continuous data were reported as mean and standard deviation. With a confidence interval width of 10% and a two-sided α of 0.05, 550 individuals per each of the three groups provided an 80% power to detect a PPV of 80% for the PE-related principal ICD-10 codes compared with manual chart review. It was assumed that there may be exclusions from the analysis cohort because of duplicate records or incomplete information, including patients who were transferred from outside hospitals. Therefore, a randomly selected sample of 578 patients per group (a total of 1,734) was used for this study. For external validation at YNHHS, based on available resources, a convenience sample of 85 patients per group (a total of 255) was used.

At both MGB and YNHHS, a third group of patients (i.e., those without ICD-10 codes for PE) was intentionally chosen to be the same size as patients with ICD-10 codes. This relatively large size of included patients without PE codes allowed for the careful assessment of charts for potential false negatives, i.e., patients with PE who did not have ICD-10 PE codes during the index hospitalization.

Samples for patients with ICD-10 codes for PE in either the principal (group 1) or secondary (group 2) discharge position in the study markedly overrepresent their true prevalence in health systems, since most patients have neither acute PE nor a history of prior PE. Therefore, for the estimation of accuracy metrics, it was pre-specified to consider the appropriate weight of the three groups.[1] The parent denominators for the three patient groups were also ascertained at MGB and at YNHHS. From January 1, 2016, through December 31, 2021, there were 4,878 patients at MGB with principal discharge codes for PE, 3,224 patients with secondary discharge codes for PE, and 373,540 patients without codes for PE. These numbers were used to ascertain the weighted accuracy of ICD-10 codes in the principal discharge position or in the principal-or-secondary discharge position for the identification of PE. At YNHHS, there were 3,697 patients with principal discharge codes for PE, 19,988 patients with secondary discharge codes for PE, and 294,326 patients who had hospitalizations without any codes for PE (Figure S1). F1 scores were compared by Chi-squared tests.

Pre-specified subgroup analyses were planned to assess the accuracy of the codes in female versus male adults, older adults (aged ≥ 65 years) versus younger patients, across the ethnic-racial subgroups, and those with versus those without COVID-19, identified by positive polymerase chain reaction tests.[22–26]

RESULTS

Overall, 1,734 hospitalized patients were selected from MGB. After excluding the duplicates and ineligible cases with septic emboli, 1,712 entered the final analysis (mean age: 60.6 ±17.8 years, 52.3% female, Table 1). These included 568 patients with a principal discharge diagnosis of PE, 568 with a secondary discharge diagnosis for PE (without principal codes), and 576 patients without principal or secondary ICD-10 codes for PE.

Table 1.

Baseline Characteristics of Patients from MGB

	All Patients (n = 1,712)	PE* (n=863)	No PE* (n = 849)
Age (years ± SD)	60.6 ±17.8	62 ± 16.5	59 ±18.8
Female (%)	52	49	56
Race and Ethnicity (%)
Non-Hispanic White	80	80	80
Non-Hispanic Black	9.3	10	8.4
Non-Hispanic Asian	2.9	2.3	3.5
Non-Hispanic Other or Unknown	4.9	3.8	6.0
Hispanic or Latino	2.9	3.2	2.6
COVID-19 within 30 days (%)	3.0	4.1	2.0
Prior venous thromboembolism (%)	16	13.7	19
Diabetes (%)	3.3	3.2	3.4
Hypertension (%)	55	56	53
History of coronary disease (%)	19	19	18
History of peripheral artery disease (%)	8.6	8.9	8.4
History of cerebrovascular accident (%)	7.1	7.2	7.0
History of heart failure (%)	17	15.4	19
Hemodialysis (%)	1.5	1.4	1.6

MGB = Mass General-Brigham; PE = pulmonary embolism; SD = standard deviation

^*Confirmed via manual chart review

Accuracy of ICD-10 codes for identification of PE

Among 1,712 patients from MGB, 863 had PE, according to the review of records by two independent clinicians. Of these 863 patients with objectively confirmed PE according to independent medical review, 523 had an ICD-10 principal discharge diagnosis of PE, 338 had a secondary discharge diagnosis of PE, and two patients did not have PE codes listed at all. The sensitivity and specificity of ICD-10 principal discharge codes for PE were 60.6% and 94.7%, respectively, for detecting PE in the study sample. When adding secondary discharge codes (i.e., using principal or secondary discharge codes for PE), the sensitivity in the study sample improved to 99.8%, and the specificity was reduced to 67.6% (Table S3).

Weights were applied to assess the manually reviewed cases from groups 1–3 (i.e., those with principal discharge diagnosis codes for PE, secondary discharge codes for PE, and those without codes for PE) proportionate to their true prevalence at MGB. Using the weighted data, the sensitivity and specificity for the ICD-10 principal discharge diagnosis codes for PE were 58.3% and 99.9%, respectively. The PPV and the negative predictive value (NPV) were 92.1% and 99.1%, respectively. The weighted sensitivity and specificity for principal or secondary ICD-10 discharge codes were 83.2% and 99.5%, respectively, while the PPV and NPV were 79.1% and 99.7%, respectively (Figure 1, Table S4). Illustrative examples of false positives by either principal or secondary discharge codes for PE are presented in Table 2.

Figure 1. Accuracy Metrics for Various Combinations of Codes in the Mass General Brigham Health System (panel A) and the Yale-New Haven Health System (Panel B).

Data shown on the left-hand side derive from the study samples, while the data on the right side represent weighted health system-based estimates accounting for the true prevalence of the codes.

Table 2.

Illustrative Examples of False Positive ICD-10 Codes For PE

Principal discharge diagnosis of PE
Patient	Summary	Comment
Patient A	PE a month prior to index hospitalization	Nearly all cases had a prior history of PE. Yet, there was no acute PE in the index hospitalization or the few days prior to the hospitalization.
Patient B	PE 2 months prior to index hospitalization
Patient C	PE 4 months prior to index hospitalization
Patient D	History of recurrent PE (last in 2010)
Secondary discharge diagnosis of PE
Patient	Summary	Comment
Patient E	PE 2 months prior to index hospitalization	Most patients had a history of prior PE. Some had concerns about PE but no apparent PE during hospitalization.
Patient F	PE 5 years prior to index hospitalization
Patient G	History of PE >10 years prior
Patient H	New DVT during admission but no PE
Patient I	Initial concern for PE, but V/Q scan and lower extremity ultrasound were negative
Patient J	PE 10 days prior in the course of a major surgical procedure. Confirmatory imaging was performed in hospital J1. Subsequently transferred from hospital J1 to hospital J2 for reasons other than PE.	Since the patient was transferred, the original imaging that was billed took place at the first hospital and is not identified by procedure codes from the second hospital. Although some radiology review codes exist, they are not specific to PE tests.

PE: Pulmonary embolism

Accuracy of codes after incorporation of diagnostic imaging codes

In a pre-specified analysis, the accuracy of using either principal discharge diagnosis PE codes, or secondary discharge diagnosis codes plus imaging codes for PE was assessed. With this approach, the weighted sensitivity and specificity were 81.6% and 99.7%, and the PPV and NPV were 84.7% and 99.6% (Figure 1).

Considering a combination of sensitivity and PPV, using either principal discharge diagnosis PE codes, or secondary discharge diagnosis codes plus imaging codes for PE yielded an F1 score of 83.1%, which was higher than all other approaches to identify PE (P<0.001 compared with other algorithms, Table S4)

Subgroup analyses

Figure 2 and Tables S5 and S6 summarize the results of the subgroup analyses based on age, sex, COVID-19 status, race, and ethnicity for identification of PE based on principal codes, principal or secondary codes, and principal codes, or secondary codes plus diagnostic imaging codes. The results were largely similar among the subgroups and consistent with the main findings.

Figure 2. Subgroup Analyses Based on the Principal Codes (Panel A), Principal or Secondary Codes (Panel B), and Principal Codes, Or Secondary Codes Plus Diagnostic Imaging Codes (Panel C).

Data are from the unweighted MGB sample. ICD-10 = International Classification of Diseases-10; MGB = Mass General-Brigham; PE = pulmonary embolism. The Non-Hispanic Asian, Non-Hispanic Other or Unknown, and Hispanic or Latino subgroups were merged as “Other” subgroup due to the small number of patients in these subgroups. For detailed results regarding these subgroups, please refer to Table S5.

Chart review detected 360 patients with PE and cor pulmonale according to predefined clinical criteria.[1] Among these, the specific subset of PE codes that indicate cor pulmonale (Table S1) was used only in 36 patients, leading into very low sensitivity (≤10%) across all analyses. Additional test characteristics are summarized in Table S7.

Incorporation of present on admission indicator

Use of the present on admission indicator was associated with improved PPV, particularly those with secondary discharge diagnosis code of PE, but reduced sensitivity (Table S8).

Validation at YNHHS

During the study period, 318,011 hospitalized patients from YNHHS were identified, of whom a randomly selected sample of 255 patients was included (age: 63.3±18.9 years, 56.7% female), including 85 with principal, 85 with secondary discharge diagnosis codes for PE and 85 who did not have any ICD-10 codes for PE.

In the validation subset at YNHHS, among 254 studied patients who were eligible, a review of the records by two independent clinicians identified 90 cases with PE. The weighted sensitivity, specificity, PPV, and NPV for ICD-10 codes in the principal discharge position were 62.9%, 99.9%, 95.1%, and 99.2%, respectively. Respective numbers for using the codes in the principal or secondary positions and those for secondary codes paired with imaging codes are reported in Table S9 and were largely consistent with MGB analyses.

DISCUSSION

In this multicenter study, we demonstrate that the use of ICD-10 Principal discharge diagnosis codes for PE is associated with >90% specificity, PPV, and NPV, but limited sensitivity, missing nearly 40% of new PE events in hospitalized patients. The incorporation of secondary discharge diagnosis codes –as has happened in some prior studies and surveillance investigations–[8, 9, 27–30] improved the sensitivity. However, this came at the cost of diminished PPV (from 92.1% to 79.1%). Using a combination of secondary discharge codes plus diagnostic or procedure codes, in addition to principal discharge codes, improved sensitivity without a major compromise on PPV (84.7%), yielding the highest F1 score among all experimented approaches. Use of the present on admission indicator, instead, improved PPV but reduced sensitivity. Results were substantively similar in the external validation cohort of patients from YNHHS and consistent across age, sex, and ethno-racial subgroups, and with or without COVID-19.

Considering the main findings of this study, it is very likely that prior studies that used such approaches for identifying PE using ICD-10 codes, including those by the AHA and the CDC[8, 9], either missed many relevant cases with PE (if they exclusively used principal discharge codes), leading to underestimated incidence. It is also possible that those studies inaccurately included many non-PE patients in their cohort leading to the inclusion of heterogeneous and inaccurate populations, which may impact the estimates about the clinical course and outcomes that are being reported.

Unlike several prior studies,[31] the current investigation was based on a priori-specified goals, using data from multiple small and large hospitals, and included subset validation from an entirely different large health system – all of which improved the robustness of the results. In addition, a recent systematic review and meta-analysis of prior approaches for validation of ICD-10 codes revealed that most prior studies had not pre-specified a hybrid approach, had not looked into false negatives, and had not assessed the health system-level weighted estimates for the codes.[31] Lack of this latter step would lead to biased reporting of test characteristics, particularly PPV, which is strongly dependent on disease prevalence. Including patients with neither the principal nor secondary ICD-10 codes for PE in our study sample allowed us to detect patients who truly had PE during the index hospitalization but did not have ICD-10 PE codes listed in their index hospitalization. Although we only found two such patients among the 576 patients without either principal or secondary ICD-10 codes for PE, patients without ICD-10 codes for PE constitute most hospitalized patients. Therefore, the absolute relevance of the false negatives was higher in the weighted analyses.

The main implication of this investigation is providing robust estimates of accuracy metrics for various combinations of ICD-10 codes and other tools, such as CPT codes and present on admission indicators. These data can inform investigators of the appropriate selection of metrics for specific studies. For example, if PPV is most important, such as the case for identifying patients for a very targeted investigation, such as a clinical trial, principal discharge diagnosis codes may be most helpful. In turn, for surveillance investigation of the PE burden, a combination of either the principal discharge codes, or secondary codes plus CPT codes for relevant imaging studies may be most useful.

The study may also have implications outside of PE research. Although we have not specifically tested such algorithms in other thrombotic or non-thrombotic conditions, it is conceivable that a combination of ICD-10 codes in various positions, CPT codes, and other parameters such as present on admission indicator or multiple claims with the same codes in a given timeframe can similarly optimize patient or outcome identification for other conditions. In order to ascertain construct validity, it is strongly encouraged for such future studies to consider local and external validation.

Some other supplemental findings deserve attention. First, reassuringly, there was no major sex[32] or age or ethno-racial differences in the function of the code algorithms. Second, use of ICD-10 codes proved to be extremely insensitive for identification of patients with PE and cor pulmonale. Therefore, such an approach for patient identification should be avoided. Third, although use of the present on admission indicator in the subset that had the data element marked as present improved the PPV, it should be kept in mind that in many patients PE is a hospital acquired complication and enforcing the present on admission indicator will lead to missing of those patients. Fourth, A post-hoc observation was that a larger proportion of patients in the MGB health system had principal (rather than secondary) discharge diagnosis of PE, compared with patients from YNHHS. The reasons for this difference are uncertain but may include institutional variations in ways that ICD-10 codes are being utilized, or possibly differences in case mix such that MGB –in which two of the participating hospitals, Brigham and Women’s Hospital and Massachusetts General Hospital –have national and international reputation for PE care, receive a larger proportion of patients with principal discharge diagnosis of PE. Finally, the present on admission indicator in all health systems can also take unknown and missing values, which limit their practical utility.

This study also has some limitations that should be considered for appropriate interpretation. First, the PE-EHR+ study focused on data from two large academic health systems from the United States. We cannot exclude the possibility of some variations in other health systems in the United States and elsewhere. The team of investigators is exploring opportunities for additional validation studies, particularly outside the United States. Second, the current analysis did not focus on natural language processing (NLP), which may provide a potentially more efficient way for identification of patients with PE from electronic health records. However, many databases, such as data from Medicare beneficiaries, the National Inpatient Sample, the Center for Disease Control, World Health Organization Mortality Database and Prevention Wide-ranging Online Data for Epidemiologic Research database, and others, do not include electronic health records such as radiology reports or discharge summaries and are limited to ICD codes. Third, this study only focused on PE, and not PE-related outcomes. Future studies should similarly validate ICD-10 codes for identifying adverse outcomes, such as recurrent PE.[33]

In conclusion, in a pre-specified study including patients from two large US health systems, principal discharge diagnosis codes for PE showed >90% PPV. However, they missed nearly 40% of patients with PE. Integration of secondary discharge codes alone improved sensitivity but yielded a low PPV. Using a combination of principal discharge codes or secondary codes merged in conjunction with proximate PE imaging codes improved sensitivity while maintaining reasonable PPV, thereby yielding the highest F1 score. These results will enable the investigators to appropriately select the right combination of ICD-10 and CPT codes based on the preference for sensitivity, PPV, or a balance of both. Importantly, our findings may help provide more accurate future estimates of PE incidence by the AHA and CDC.