Validation of a Claims-Based Algorithm for Identifying Non-Infectious Pneumonitis in Patients Diagnosed with Lung Cancer

Abstract

Purpose:

Non-Infectious Pneumonitis (NIP) is a common complication of treatments for lung cancer. We know of no existing validated algorithm for identifying NIP in claims databases, limiting our ability to understand the morbidity and mortality of this toxicity in real-world data.

Methods:

Electronic health records (EHR), cancer registry, and administrative data from a National Cancer Institute-designated comprehensive cancer center were queried for patients diagnosed with lung cancer between 10/01/2015–12/31/2020. Health insurance claims were searched for ICD-10-CM codes that indicate an inpatient or outpatient diagnosis with possible NIP. A 20-code (Algorithm A) and 11-code (Algorithm B) algorithm were tested with and without requiring prescription with corticosteroids. Cases with a diagnosis of possible NIP in the 6 months before their first lung cancer diagnosis were excluded. The algorithms were validated by reviewing the EHR. The positive predictive value (PPV) for each algorithm was computed with 95% confidence intervals (CI).

Results:

Seventy patients with lung cancer had a diagnosis code compatible with NIP: 36 (51.4%) inpatients and 34 (48.6%) outpatients. The PPV of Algorithm A was 77.1% (95% CI: 65.6–86.3). The PPV of Algorithm B was 86.9% (95% CI: 75.8–94.2). Requiring a documented prescription for a systemic corticosteroid improved the PPV of both the least and most restrictive algorithms 92.5% (95% CI: 79.6–98.4) and 100.0% (95% CI: 90.0–100.0), respectively.

Conclusions:

This study validated ICD-10-CM and prescription-claims-based definitions of NIP in lung cancer patients. All algorithms have at least reasonable performance. Enriching the algorithm with corticosteroid prescription records results in excellent performance.

Introduction

Lung cancer is the leading cause of cancer death in both men and women in the United States¹. Development of novel medications and therapeutic strategies to treat lung cancer, including programmed cell death-1, programmed death-ligand 1, cytotoxic T-lymphocyte-associated protein 4 blockade, and use of targeted radiation to treat oligometastatic disease, have brought hope to patients who previously had few therapeutic options². These benefits, however, have not been without costs, as adverse events often impact quality of life and may result in the premature cessation of therapy, which can reduce the effectiveness of these approaches.

One toxicity that limits therapeutic delivery is pneumonitis, which is broadly defined as inflammation of the lung parenchyma. Although pneumonia (infectious pneumonitis) is also included in this category, lung cancer patients often experience a non-infectious pneumonitis (NIP). NIP in lung cancer patients can occur during treatment with chemotherapy or targeted therapy (chemo-pneumonitis), immunotherapy (immune-related pneumonitis), and radiation (radiation pneumonitis)³. Often identified and labeled by its presumed precipitant, NIP is a poorly understood yet common complication of therapy, affecting anywhere from 10–50% of patients^4–12.

Although these toxicities are well-described in clinical trials, more research is needed to understand the real-world incidence of NIP in patients with lung cancer and its impact on survivorship, and whether these treatments interact in causing pneumonitis. One potential reason for the paucity of real-world studies of non-infectious pneumonitis is the lack of formal validation studies using data easily available from large claims or medical records databases. Although two published studies have used ICD-9 and ICD-10-CM codes to identify cases of pneumonitis^4,13, the risk of misclassification bias and the lack of a validated algorithm has limited the interpretability of these studies.

There remain two major gaps in the literature: first, the development of an operationalizable definition for NIP within claims or medical record data that applies broadly to all causes of NIP – independent of the precipitant. One should not use a treatment-dependent, circular definition to conduct pharmacoepidemiology research of toxicity from particular treatments, although this is the current normal practice. The second gap is a formal validation of a definition of NIP. Otherwise, any research using real-world evidence to investigate pharmacoepidemiologic questions will face appropriate criticism and doubt. Any findings estimating the incidence of pneumonitis would be criticized for overestimation due to risk of misclassification by including pneumonia and other pulmonary complaints as NIP, and the typical definitions which include the exposure within the definition could be criticized for the introduction of bias in evaluating etiologic factors.

To our knowledge, this study is the first to provide a formally validated exposure-independent definition of NIP in lung cancer patients. This study will provide the foundation for future investigations of NIP using real-world data, which can further clarify the usefulness of novel therapies in patients who might be more ill, frail, and diverse, than those enrolling in clinical trials.

Methods

Study Population

The core study population was defined as patients diagnosed with any primary lung cancer (International Classification of Diseases for Oncology, third edition codes C34.0, C34.1, C34.2, C34.3, C34.8, and C34.9¹⁴) at the New Jersey’s only National Cancer Institute (NCI)-designated comprehensive cancer center (Rutgers Cancer Institute of New Jersey) and its on-campus, inpatient, tertiary care center (Robert Wood Johnson University Hospital) and outpatient counterparts (Robert Wood Johnson Medical School) during the study period (October 2015 to December 2020). These sites are a part of a major health system where patients often receive the spectrum of their cancer care including surgery, radiation therapy, chemo/immunotherapy, other specialist services, and emergency department visits. There were no requirements for service utilization or follow-up at the site.

Data Sources

Administrative claims data for the inpatient, emergency room, and outpatient settings originally collected for the purpose of reimbursement for medical services, were evaluated for ICD-10-CM codes that might indicate a diagnosis of NIP. The electronic health record (EHR) was reviewed for all patients who met the algorithm specifications in order to assess whether the identified cases were true cases of non-infectious pneumonitis. Additionally, prescription records were available from the EHR to simulate prescription dispensing records that would be available in a conventional claims database. Cancer registry data from the state’s NCI-funded Surveillance, Epidemiology, and End Results Program (SEER) cancer registry were used to confirm diagnoses of lung cancer during the study period.

Algorithm Definition

Cases of NIP were identified by searching for a priori defined diagnosis codes for NIP (Table 1) after the indexed cancer diagnosis in inpatient and outpatient claims databases. There were two algorithms tested, one with 20 ICD-10-CM codes (Algorithm A) and one with 11 ICD-10-CM codes (Algorithm B). Algorithm A was a subset of the codes used by Ryan et al. in an assessment of pneumonitis in lung cancer patients receiving chemoradiation¹³; two codes (J18.9 Pneumonia, unspecified organism and J98.4 Other disorders of lung were deliberately excluded, however, because of our focus on excluding cases of pneumonia and other pulmonary diseases. Algorithm B further narrowed the code list based on expert consensus. While Algorithm B included a subset of codes from Algorithm A, the algorithms were each implemented separately.

Table 1.

ICD-10-CM Codes for Non-Infectious Pneumonitis

Code	Description	Algorithm
		A	B
J67.9	Hypersensitivity pneumonitis due to unspecified organic dust	+	−
J70.0	Acute pulmonary manifestations due to radiation	+	+
J70.1	Chronic and other pulmonary manifestations due to radiation	+	+
J70.2	Acute drug-induced interstitial lung disorders	+	+
J70.3	Chronic drug-induced interstitial lung disorders	+	+
J70.4	Drug-induced interstitial lung disorders, unspecified	+	+
J70.8	Respiratory conditions due to other specified external agents	+	+
J70.9	Respiratory conditions due to unspecified external agent	+	+
J84.0	Alveolar and parieto-alveolar conditions	+	−
J84.1	Other interstitial pulmonary diseases with fibrosis	+	+
J84.111	Idiopathic interstitial pneumonia not otherwise specified	+	+
J84.112	Idiopathic pulmonary fibrosis	+	−
J84.113	Idiopathic non-specific interstitial pneumonitis	+	+
J84.114	Acute interstitial pneumonitis	+	+
J84.115	Respiratory bronchiolitis interstitial lung disease	+	−
J84.116	Cryptogenic organizing pneumonia	+	−
J84.117	Desquamative interstitial pneumonia	+	−
J84.2	Lymphoid interstitial pneumonia	+	−
J84.89	Other specified interstitial pulmonary diseases	+	−
J84.9	Interstitial pulmonary disease, unspecified	+	−

All patients with at least one code for NIP as a primary or secondary diagnosis after the lung cancer diagnosis were included as a potential case. Patients with a code of NIP within the 180 days prior (baseline period) to the cancer diagnosis were excluded. Additional algorithms were tested requiring the ICD-10-CM code definition and also prescription of systemic oral corticosteroids (National Drug Code list adapted from the National Committee for Quality Assurance¹⁵ and included in supplementary material) within 1 day after the first NIP diagnosis (either 1 day after discharge for inpatient cases, and 1 day after date of visit for outpatient cases) or discharged deceased (so they were not eligible to receive discharge medications). Including corticosteroids in our algorithm was hypothesized to improve the algorithm performance, because NIP is routinely treated with corticosteroids. Additionally, if a NIP code occurred prior to 180 days before the lung cancer diagnosis, the patient was excluded in a sensitivity analysis.

Case Adjudication

In order to determine whether the cases identified by the algorithm were true cases of NIP, the EHR was reviewed by two trained investigators (the reviewers). All the identified cases were adjudicated. First, the reviewers (two medical student investigators SSN and DEP) underwent training by a faculty radiation oncologist and faculty medical oncologist (JM, SKJ) to ensure that a definition of NIP was understood. Then, they reviewed all potential cases identified by the algorithms, assessing outpatient and inpatient medical records for the indexed NIP diagnosis and 2-weeks after. They reviewed the admission notes, progress notes, and discharge/visit summaries: chief complaint listed in history of present illness, preliminary diagnoses (the differential diagnosis and any tentative working diagnoses outlined in the assessment and plan), and treatment strategies (the assessment and plan for orders placed for corticosteroids and antibiotics).

Cases were adjudicated by assessing if objective criteria (Table 2) were met for a definitive case of NIP (true positive), possible cases of NIP, or definitive false positive. These criteria were determined by consulting with the faculty co-investigators (JM, SKJ) who routinely manage patients with NIP and have published prospective and retrospective studies where NIP was a key outcome^16–20. For example, a case was a definitive false positive if there was evidence of infection and the chief complaint was pneumonia. The first 10 cases adjudicated by the reviewers were assessed also by these co-investigators to ensure that the reviewers and the experts had 100% agreement. If there were any discrepancies, the reviewers would have been retrained per expert feedback in an iterative process until 100% agreement was reached. Any case that was unclear (possible case) or had disagreement between the reviewers after applying the criteria were to be sent to review by the expert medical oncologist and radiation oncologist (JM, SKJ). The experts would have reviewed the clinical information and made a subjective judgement; if they both reached consensus about the case, it would be reclassified as true positive or false positive, otherwise it would be classified as a possible case. In fact, after initial training, agreement between the reviewers was 100% and there were no cases classified as “possible case”; thus no expert review was required.

Table 2.

Adjudication Strategy for Identifying Cases of Non-Infectious Pneumonitis

Case Type	Criteria
Definite Case (True Positive) Any one of the following.	Clinical notes in history of present illness (HPI) or assessment and plan (A&P) indicating the patient was definitively diagnosed with any “pneumonitis” or that the most likely diagnosis was “pneumonitis” or “non-infectious interstitial inflammation” (e.g., “shortness of breath likely secondary to pneumonitis, pneumonia unlikely”) Radiology reports indicating “pneumonitis” or “inflammation of lung parenchyma”. Bronchioalveolar lavage or biopsy indicating elevated leukocytes and NO organisms – must include rule out of viral, bacterial, and fungal etiologies. Patients treated with steroids with clinical improvement without treatment with antibiotics.
Not Case (False Positive) Any one of the following.	Clinical notes in HPI or A&P indicating that the patient was diagnosed with “pneumonia”. Pneumonia might be listed on the differential diagnosis of true positives, but if the clinical notes indicate that pneumonia is the most likely diagnosis, this observation will be deemed not-case. Confirmed evidence of infection. Patient treated with antibiotics WITHOUT steroids that clinically improved. No mention of ‘pneumonitis’ in any clinical notes for the admission (inpatient) or visit (outpatient).
Possible Case (SEND TO EXPERT REVIEW)	Does not meet any of above criteria or meets both criteria. No evidence of infection.

Statistical Analysis

The PPV was calculated by taking the ratio of true positives to all potential cases. Ninety-five percent confidence intervals (CI) were calculated using the Clopper-Pearson method²¹. A pre-specified cut-off of the lower bound of the 95% CI of 70% was considered acceptable performance for application in future studies. For analyzing the feasibility of our study, we estimated that 22–32 patients would be needed to identify a PPV of 83–85% with a lower-bound of the 95% confidence interval of 70%. Inpatient and outpatient PPVs were also calculated separately: the inpatient PPV was calculated by analyzing the baseline information from the inpatient and outpatient record, and only inpatient records after the indexed lung cancer diagnosis; the outpatient PPV was calculating using the same baseline period for excluding cases, but only counted cases of NIP in patients who had no admissions for NIP. We chose to focus on measuring the PPVs of our algorithms in order to ensure that the cases identified by the claims were true cases of NIP.

Ethics Approval

This study was approved by the Rutgers Biomedical and Health Sciences Institutional Review Board.

Results

There were 70 patients diagnosed with lung cancer who had at least one diagnosis code from Algorithm A (Table 3). Thirty-six (51.4%) patients were identified in the inpatient database, and 34 (48.6%) outpatients were identified in the outpatient database. For Algorithm B, there were 61 total patients; 31 (50.8%) patients were identified in the inpatient database vs. 30 (49.2%) in the outpatient database. After full adjudication there were no patients classified as possible cases. The PPV of Algorithm A was 77.1% (95% CI: 65.6–86.3). Algorithm B, excluding non-specific codes for NIP, had a PPV of 86.9% (95% CI: 75.8–94.2). Requiring patients to have a documented prescription of corticosteroid improved the PPV of Algorithms A and B to 92.5% (95% CI: 79.6–98.4) and 100.0% (95% CI: 90.0–100.0), respectively.

Table 3.

Positive Predictive Value of Proposed Algorithms

Algorithm	(TP/N) Total PPV [95% CI]	(TP/N) Inpatient PPV [95% C]	(TP/N) Outpatient PPV [95% CI]
Primary Analysis: ICD-10-CM Codes Only
A	(54/70) 77.1% [65.6 86.3]	(29/36) 80.6% [64.0 91.8]	(25/34) 73.5% [55.6 87.1]
B	(53/61) 86.9% [75.8 94.2]	(28/31) 90.3% [74.2 98.0]	(25/30) 83.3% [65.3 94.4]
Secondary Analysis: ICD-10-CM Codes + Corticosteroids upon discharge
A	(37/40) 92.5% [79.6 98.4]	(26/27) 96.3% [81.0 99.9]	(11/13) 84.6% [54.6 98.1]
B	(35/35) 100.0% [90.0 100.0]	(25/25) 100.0% [86.3 100.0]	(10/10) 100.0% [69.2 100.0]

Sensitivity analyses seeking to exclude NIP at any time prior to the cancer diagnosis did not lead to any additional exclusions; hence the PPVs were identical (data not shown).

Discussion

This study formally validated a claims-based definition of NIP. The algorithms tested, chosen a priori, all had reasonable performance. The first algorithm (Algorithm A) included a group of 20 ICD-10-CM codes. After excluding 9 codes, the next algorithm (Algorithm B) had even better performance; the lower bound of the 95% CI exceeded our a priori threshold of 70%, indicating acceptable performance. After then requiring corticosteroid treatment within 1 day after NIP diagnosis, the algorithms performance improved further.

Not surprisingly, the algorithms were qualitatively superior in the inpatient setting compared to the outpatient setting. Less specific codes for NIP might be used in the outpatient setting because diagnosis codes are not linked directly to reimbursement. Using medication data to enrich claims-based algorithms is another potential method to improve the PPV. The performance of our algorithms after requiring a corticosteroid prescription within 1 day of diagnosis greatly improved the performance of Algorithms A and B. However, requiring corticosteroid treatment within 1 day of diagnosis as part of the algorithm for identifying pneumonitis halved the identified cases. The cost of this exclusion is primarily a reduction in the sensitivity, although this could not be quantified in this study because we did not seek in our methods to identify all true cases.

A prior study by Ryan and colleagues applied one of the first algorithms to identify cases of pneumonitis in patients with lung cancer receiving chemoradiotherapy¹³. There were no formally validated definitions of pneumonitis at that time, and the investigators did not make a distinction between pneumonitis generally and NIP. One important difference between our algorithms and the Ryan algorithms was their inclusion of the ICD-10-CM codes J18.9 (Pneumonia, unspecified organism) and J98.4 (Other disorders of lung). Because our algorithms were specifically aiming to evaluate non-infectious causes of pneumonitis, we intentionally excluded the J18.9 code. However, we may have missed cases of NIP that had been misclassified with J18.9 – a code primarily used for cases of pneumonia. The deliberate exclusion of the J98.4 code was also important because this code may be used nonspecifically for lung diseases that are commonly comorbid with lung cancer.

The primary challenge of identifying NIP in claims databases is the misclassification of cases of pneumonia as cases of NIP. One could argue that using dispensing of antibiotics as an exclusion criterion could improve the performance as well. However, many patients with NIP are treated empirically with antibiotics, in both outpatient and inpatient settings²². While we would anticipate excluding patients who are treated with antibiotics would reduce misclassification of cases of pneumonia, it may reduce the algorithm performance by increasing the relative amount of other lung diseases (not NIP and pneumonia) identified. We would also expect the sensitivity of this algorithm to be reduced.

The strengths of this study were the access to all medical records for identified cases and the robust adjudication strategy. A common weakness of validation studies of claims data is the low response rate when requesting medical records to adjudicate cases identified by claims-based algorithms, especially of outpatient records. Because we had full access to both the administrative databases and electronic health record, reporting bias does not affect our estimates of the PPV of our algorithms. Further, our adjudication strategy improves the confidence of our estimates. By having two independent investigators review the identified cases of NIP using objective criteria, our results have internal validity and can also be reproduced by other investigators.

The major limitations of this study are three-fold. First, the algorithm could lack external validity because claims data are being collected from only one NCI-designated comprehensive cancer center and its on-campus inpatient and outpatient counterparts. The evidence presented only truly validates the implementation of the algorithm at the one institution, because coding practices will vary across hospital systems and different settings and these algorithms, while not data-derived, might not perform as well other settings. Also, because claims data were not collected directly from an insurer (private health plan or Medicare/Medicaid), we do not have access to claims from all patient contacts with the US health system. The absence of claims from other local institutions could have overestimated or underestimated our algorithm performance. Second, the adjudicated outcome of NIP is a clinical diagnosis, without a clear gold standard. While objective case adjudication criteria were applied by the reviewers, the validity of our proposed algorithms are limited by manual review of physician documentation. Third, we were unable to calculate the algorithms’ negative predictive values, sensitivities, or specificities because we lacked a well-characterized population of true cases of NIP. While this study does effectively characterize the PPVs of our algorithms to ensure true cases of NIP were identified, we lack important information about the false negatives and true negatives of the proposed algorithms. Our algorithms could have acceptable PPVs but have low sensitivity, which might limit the usefulness for some investigations.

This study formally validated a claims-based definition of NIP enabling future research of NIP in administrative databases. The algorithms’ performances were all at least reasonable. All algorithms, except the 20-code algorithm (Algorithm A) without corticosteroid prescription data, met the pre-specified threshold for adequacy. Implementation of these algorithms for studying drug effects in a particular database should be preceded by further validation within that database.

Key Points:

• Using 20 ICD-10-CM codes, the algorithm had a PPV of 77.1% (80.6% and 73.5% using inpatient and outpatient data, respectively)

• Using 11 ICD-10-CM codes, the algorithm had a PPV of 86.9% (90.3% and 83.3% using inpatient and outpatient data, respectively)

• Only including cases with corticosteroid prescriptions, improved the 20-code algorithm to 92.5% (96.3% and 84.6% using inpatient and outpatient data, respectively) and the 11-code algorithm to 100.0%.

• We propose using an 11-code algorithm with or without enrichment with corticosteroids or a 20-code algorithm with enrichment with corticosteroids to identify cases of NIP in pharmacoepidemiology studies of lung cancer patients.