Natural Language Processing to Identify Pneumonia from Radiology Reports

Abstract

Purpose

To develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports.

Methods

We trained one NLP system, ONYX, using radiograph reports from children and adults that were previously manually reviewed. We then assessed its validity on a test set of 5,000 reports. We aimed to substantially decrease manual review, not replace it entirely, and so we classified reports as 1) consistent or 2) not consistent with pneumonia or 3) requiring manual review due to complex features. We developed processes tailored either to optimize accuracy or to minimize manual review. Using logistic regression, we jointly modeled sensitivity and specificity of ONYX in relation to patient age, comorbidity, and care setting. We estimated positive and negative predictive value (PPV and NPV) assuming pneumonia prevalence in the source data.

Results

Tailored for accuracy, ONYX identified 25% of reports as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). For the remainder, ONYX’s sensitivity was 92% (95% confidence interval [CI] 90–93%), specificity 87% (86–88%), PPV 74% (72– 76%), and NPV 96% (96–97%). Tailored to minimize manual review, ONYX classified 12% as needing manual review and for the remainder had sensitivity 75% (72-77%), specificity 95% (94-96%), PPV 86% (83-88%), and NPV 91% (90-91%).

Conclusions

For pneumonia validation, ONYX can replace almost 90% of manual review while maintaining low to moderate misclassification rates. It can be tailored for different outcomes and study needs and thus warrants exploration in other settings.

INTRODUCTION

Pneumonia is common and can have severe consequences in older adults. A growing literature suggests that some medications increase pneumonia risk.^1-5 Pharmacoepidemiologic studies often identify pneumonia cases within large databases using International Classification of Diseases, version 9 (ICD-9) codes or the equivalent. However, ICD-9 codes lack accuracy for pneumonia: in validation studies, their sensitivity has ranged from 48 to 80% and positive predictive value (PPV) from 73 to 81%.^6,7 Misclassification of outcomes can limit statistical power and bias study results. Some studies have reviewed medical records to validate cases,^1,3,8 but this approach is costly and time-consuming. Automated methods for outcome validation would be very helpful and could also be used for clinical decision support or public health surveillance.

With the growing use of electronic medical records, automated outcome validation may be possible using Natural Language Processing (NLP), in which a computer processes free text to create structured variables. Pneumonia is suited to this approach because the diagnosis requires a positive chest radiograph,^9,10 and chest radiograph reports have fairly standard format and language. Several studies have used NLP to identify pneumonia from clinical texts,^11-14 with sensitivity from 64 to 100% and specificity from 85 to 99%. Most prior studies examined relatively few reports and included few pneumonia cases. Little is known about accuracy in the outpatient setting, where pneumonia is often diagnosed. No prior study evaluated NLP as a filter for chart review.

Our aim was to develop NLP approaches to validate pneumonia cases from electronic radiology reports. Because we knew that certain reports are challenging to classify, we aimed to replace a large portion of manual review with NLP, but not all. We used reports that were previously manually reviewed to train one NLP tool, ONYX,¹⁵ and assess its accuracy. We tailored our approach for different scenarios to explore trade-offs between efficiency and accuracy.

METHODS

Setting

Group Health (GH) is an integrated healthcare delivery system in the Northwest United States with extensive electronic health data. GH members have coverage through employer-based plans, individual plans, Medicare, and Medicaid. The racial and ethnic composition is similar to the surrounding region, including 79% Caucasian, 3% African-American, 8% Asian/Pacific Islander, 1% Native American, 5% Hispanic and 3% other race.

This study was approved by the GH Human Subjects Review Committee with a waiver of consent.

Data Sources

The gold standard measure of pneumonia came from medical record reviews performed for the Pneumonia Surveillance Study (PSS).¹⁶ Presumptive cases were identified from ICD-9 codes (480-487.0 or 507.0) for GH members of all ages between 1998 and 2004.¹⁶ Trained abstractors reviewed about 93,000 electronic chest radiograph reports (Table 1) to determine if an infiltrate was present or the radiologist interpreted the report as showing pneumonia. To improve consistency and ensure that abnormal findings were likely to represent pneumonia, abstractors were given detailed instructions (manual available on request). For instance, infiltrates described as streaky, nodular, mass-like or consistent with atelectasis did not qualify. Inter-rater agreement was 94% (kappa 0.84) and intra-rater agreement 95% (kappa 0.87). For the current analyses, the gold standard for pneumonia was that a report described an infiltrate meeting study criteria or contained a radiologist’s interpretation that pneumonia was present.

Table 1.

Characteristics of the source population of radiology reports and the test set of 5,000 reports used for validation

	Source dataset*			Test set
Characteristic	All reports (N=93,110)	Positive for pneumonia† (N=26,345)	Negative for pneumonia† (N=66,765)	(N=5,000)‡
	n (%)	n (%)	n (%)	n (%)
Patient age (years)
0-4	7,130 (8)	3,084 (12)	4,046 (6)	423 (8)
5-19	7,839 (8)	3,426 (13)	4,413 (7)	439 (9)
20-44	12,248 (13)	3,311 (13)	8,937 (13)	609 (12)
45-64	24,984 (27)	5,840 (22)	19,144 (29)	1,256 (25)
65-74	14,466 (16)	3,538 (13)	10,928 (16)	840 (17)
75-84	17,940 (19)	4,752 (18)	13,188 (20)	987 (20)
85+	8,503 (9)	2,394 (9)	6,109 (9)	446 (9)
Pneumonia case†	26,345 (28)	26,345 (100)	0 (0)	2200 (44)
Comorbidities
Congestive heart failure	14,424 (15)	3,712 (14)	10,712 (16)	877 (18)
Chronic lung disease	29,079 (31)	8,240 (31)	20,839 (31)	1767 (35)
Cancer	12,285 (13)	3,255 (12)	9,030 (14)	747 (15)
Setting of care
Outpatient	86,028 (92)	23,926 (91)	62,102 (93)	4598 (92)
Inpatient	6,366 (7)	2,191 (8)	4,175 (6)	362 (7)
Missing	716 (1)	228 (1)	488 (1)	40 (1)

^*A dataset of 93,110 chest radiograph reports that were previously manually reviewed for a study of pneumococcal conjugate vaccine.¹⁶

^†According to manual review.

^‡Oversampled for true pneumonia cases (according to manual review) and comorbidities.

In addition to using PSS data, we also trained ONYX using reports from a second study which used similar methods.

Information about subjects’ age, care setting (outpatient vs. inpatient), and comorbidities (congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), or cancer) came from GH automated data. Comorbidities were defined based on ICD-9 codes from the prior 12 months. We used automated data to identify the individual radiologist reading the report.

NLP Tools: ONYX and ConText

ONYX is an open-source NLP system (available at http://code.google.com/p/onyx-mplus/) that integrates knowledge about syntax (the structure of sentences) and semantics (the meaning of words) to interpret free text and produce structured output.¹⁵ ONYX can be trained on documents from a particular domain (e.g., the pulmonary domain), and it helps the user create training cases. Its output is a set of concepts identified from individual sentences. For instance, from the phrase “ill-defined density in the right lower lobe,” ONYX generates a concept of “localized infiltrate” with a location of “right lower lobe.”

Decision rules are applied to ONYX’s output to classify the report as a whole. We designed rules to classify reports as 1) consistent or 2) inconsistent with pneumonia or 3) needing manual review (when a report had certain prespecified complex features). Because studies’ needs differ, we developed two classifiers. Both are based on the PSS abstraction manual, and both treat ONYX as a filter for chart review, but the extent of filtering differs. The first classifier is designed to prioritize accuracy –to minimize false positives and false negatives – at the cost of more reports needing manual review. This approach might be useful for a study with greater resources to support manual review. The second classifier is designed to minimize manual review, as might be preferred when resources are limited, although we thought this efficiency might come at the cost of higher misclassification.

The first classifier identifies reports as needing manual review if any of the following are present: 1) seemingly inconsistent statements about pneumonia (e.g. infiltrates present and also absent); 2) atelectasis and pneumonia identified in a single report; or 3) state change language (e.g. infiltrate “improving” or “resolved”). Online Appendix Table 1 provides more detail. For the second classifier we made the following changes: 1) reports containing inconsistent statements were classified based on the most frequently occurring concept (pneumonia vs. not pneumonia); 2) reports with both atelectasis and pneumonia were classified as not pneumonia; and 3) ONYX’s output was processed by an algorithm called ConText to more accurately detect negated and resolved findings and to distinguish clinical history from current findings.¹⁷

ONYX is based on a family of NLP systems^12,18 developed largely for chest radiograph interpretation. Our initial training data came from these systems; it consisted of phrases extracted from chest radiograph reports and their relationship to relevant concepts. Next, ONYX was applied to 30 GH reports and the results were manually reviewed and corrected. This process allows ONYX to “learn” from mistakes, creating new associations that are applied to future texts. Next, we applied ONYX to 1000 PSS reports and targeted training based on ONYX’s mistakes. In addition to editing errors in ONYX’s output, we made changes to ONYX’s processing engine, such as adding a new semantic or syntactic grammar rule. Finally, ONYX’s performance was assessed in an independent test set of 5,000 PSS reports oversampled for pneumonia (based on manual review) and comorbidities to improve the precision of estimates of the test statistics for patient subgroups.

Statistical Analyses

For each classifier, we calculated the proportion of reports that ONYX classified as requiring manual review and then estimated sensitivity and specificity for the remaining reports. Sensitivity is the proportion of true pneumonia reports that ONYX correctly identified as showing pneumonia. Specificity is the proportion of non-pneumonia reports that ONYX correctly identified as not showing pneumonia. Because we wanted to examine how these measures varied by patient characteristics, we did not calculate them directly but instead modeled them using multivariable logistic regression. Specifically, we jointly modeled ONYX’s true positive rate (sensitivity) and false positive rate (1 minus specificity) as a function of patient age, comorbidities, and care setting.¹⁹ Interaction terms between each characteristic and pneumonia status were used to facilitate estimation of both the true and false positive rates in a single model. We used weights to account for oversampling and generalized estimating equations to provide standard errors that account for potential correlation between multiple reports from the same patient.²⁰ Using the coefficients from this model, we estimated ONYX’s overall sensitivity and specificity in a population with the same characteristics as the source population (using the predictive margins method).^21,22 Finally, from the estimated sensitivity and specificity we calculated PPV and NPV assuming pneumonia prevalence as in the source population. Specifying prevalence is important because PPV increases as prevalence goes up, because the proportion of true positives increases relative to false positives. For similar reasons, NPV goes down as prevalence goes up.

We also estimated ONYX’s performance for individual radiologists who had read at least 100 reports from the test set.

Results showed that Classifier 2 had markedly higher specificity and PPV than Classifier 1. This was in part expected because of the new decision rule that arbitrarily classified certain ambiguous reports as not showing pneumonia. As a post hoc analysis, we explored the factors contributing to Classifier 2’s higher specificity by comparing the classifiers’ performance on the subset of reports that both could classify and examining Classifier 2’s accuracy for the reports that only it was able to classify.

We examined the implications of our findings for a hypothetical study seeking to validate 10,000 potential pneumonia cases, assuming the same prevalence as the source population. We calculated the number of reports that would need manual review and the number of false negatives and false positives under three scenarios: manual review without ONYX’s assistance; ONYX with Classifier 1 (tailored for accuracy); and ONYX with Classifier 2 (tailored to decrease manual review.)

Analyses were performed using SAS software, version 9.2 (SAS Institute, Inc., Cary, NC).

RESULTS

Table 1 shows characteristics of chest radiograph reports in the source dataset of 93,110 reports and the test set of 5,000 reports. Patients’ mean age in the test set was 55 years, and 92% of reports came from the outpatient setting. Outpatient reports predominate because GH patients requiring hospitalization are cared for in outside hospitals, so their inpatient radiograph reports are not available.

When ONYX and the pneumonia classifier were tailored to maximize accuracy (Classifier 1), 25% of reports were identified as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). This proportion varied by age, care setting, and presence of chronic lung disease (see Online Appendix Figure 1.) For the remaining reports, ONYX had an estimated sensitivity of 92% (95% CI, 90 – 93%), specificity 87% (86 – 88%), PPV 74% (72 – 76%), and NPV 96% (96 – 97%); see Figures 1A-1C. Sensitivity differed slightly by age but not comorbidity or care setting. Specificity was lowest in those aged 65 or above and was lower for people with COPD than those without COPD.

Figure 1.

ONYX’s performance under conditions aimed at improving accuracy (allows more reports to be designated as needing manual review)

A. Specificity*

B. Sensitivity*

C. Positive predictive value*†

*Compared to gold standard from manual review of reports.

†PPV estimated assuming prevalence of pneumonia in the source dataset (28%).

‡Comorbid conditions ascertained from health plan automated diagnosis data (International Classification of Diseases, version 9, codes.)

Tailored to minimize manual review (Classifier 2), ONYX classified 12% of reports as needing manual review, including 18% of true pneumonias and 7% of non-pneumonias. For the remaining reports, sensitivity was 75% (72 – 77%) and specificity 95% (94 – 96%). Sensitivity varied considerably by age but specificity varied little, and neither varied substantially by comorbidity or care setting (Figures 2A-2C). The estimated PPV was 86% (83 – 88%) and NPV, 91% (90 – 91%).

Figure 2.

ONYX’s performance under conditions aimed at decreasing the proportion of reports needing manual review

A. Specificity*

B. Sensitivity*

C. Positive predictive value*†

*Compared to gold standard from manual review of reports.

†PPV estimated assuming prevalence of pneumonia in the source dataset (28%).

‡Comorbid conditions ascertained from health plan automated diagnosis data (International Classification of Diseases, version 9, codes.)

Online Appendix Table 2 shows results of analyses exploring reasons for Classifier 2’s higher specificity. Classifier 2 had high specificity for the reports that were able to be classified by both classifiers and also for the reports that only it could classify.

Figure 3 shows results for the 15 radiologists who had read at least 100 reports, graphed separately for each classifier. A few had particularly high (or low) values.

Figure 3.

ONYX’s accuracy for individual radiologists*

A. ONYX with Classifier 1 (prioritizes accuracy)

B. ONYX with Classifier 2 (prioritizes decreasing the amount of manual review). Thus, additional reports are included in the analyses for Figure 3B that were not included in Figure 3A.

*Limited to the 15 radiologists who read 100 or more reports in the test set. Each number represents one radiologist, and each radiologist is assigned the same number in both graphs.

Table 2 shows the estimated outcomes if a hypothetical study used each of 3 approaches to validate 10,000 potential cases.

Table 2.

Trade-offs between efficiency and accuracy in a hypothetical study with 10,000 potential cases and true pneumonia prevalence the same as in the source population^*

	Manual review of all records	ONYX with Classifier 1 (prioritizes accuracy)	ONYX with Classifier 2 (prioritizes fewer manual reviews)
Number of charts requiring manual review	10,000	2248	1008
With true pneumonia	2800	952	504
Without pneumonia	7200	1296	504
ONYX: total incorrectly classified	0	916	909
False positives†	0	768	335
False negatives†	0	148	574
ONYX: total correctly classified (no manual review needed)	0	6836	8083
True positives†	0	1700	1722
True negatives†	0	5136	6361

^*Assumes that reports classified by NLP as either consistent or inconsistent with pneumonia do not undergo further manual review.

^†Compared to the gold standard of manual review and thus, by definition, there are no false positives or negatives when all reports undergo manual review.

DISCUSSION

We found that one NLP tool, ONYX, accurately identified a large proportion of pneumonia cases from electronic chest radiograph reports and could dramatically reduce manual review for outcome validation. Because research projects have varying needs, we created two versions of our tools. The first (aiming to maximize accuracy) had sensitivity 92%, specificity 87% and PPV 74% but designated 25% of reports as “needing manual review”. The second (aiming to decrease manual review) had sensitivity 75%, specificity 95%, and PPV 86%, with 12% of reports needing manual review. With this second classifier, ONYX could eliminate almost 90% of manual medical record review while maintaining a low false positive rate (5%) and moderate false negative rate (25%).

The amount of outcome misclassification that is acceptable will vary according to a project’s goals and resources. In some contexts, a false negative rate of 25% may be too high. It could lead to underestimation of pneumonia incidence or selection of cases that are not representative. In these settings, our first classifier may be preferable, with its false negative rate of only 8%. The trade-off is that more resources will be needed for manual review. Still, the false negative rates for both classifiers should be placed in context: most pharmacoepidemiologic studies define pneumonia from ICD-9 codes or the equivalent, which have false negative rates of 20-52%.^6,7 With NLP research proceeding at a rapid pace, the accuracy of tools like ONYX will probably continue to improve.

Differences in accuracy between Classifier 1 and 2 likely stem from several causes, including the addition of ConText, an NLP algorithm designed to improve handling of negation and historical information. Adding ConText should improve both sensitivity and specificity. Another factor was that we changed a decision rule so that certain ambiguous reports were automatically classified as not pneumonia. This decision, which was somewhat arbitrary, favors specificity over sensitivity, because shifting reports into the “not pneumonia” category will inevitably result in more false negatives (reducing sensitivity) as well as true negatives (improving specificity).

ONYX’s accuracy varied modestly by patient age and comorbidity. For Classifier 1, sensitivity was consistently high for all groups. Specificity was more variable, with lower values for older people and those with chronic lung disease. Classifier 2 had high specificity across all groups but lower sensitivity, especially in the oldest patients. Older patients and those with chronic lung disease may be more likely to have chronic lung abnormalities, and NLP may have difficulty distinguishing chronic, stable abnormal findings from acute infiltrates.

Compared with prior NLP studies, our results indicate similar or somewhat better accuracy, although it is difficult to compare studies because of differences in the gold standards and patient populations, including pneumonia prevalence (which influences PPV and NPV). Also, prior studies used NLP to classify all reports—even those with complex features that our study diverted for manual review. This difference may make our tools’ accuracy appear higher. Mendonca et al. applied an NLP system, MedLEE, to chest radiograph reports from the neonatal intensive care unit.¹⁴ Sensitivity was 71%, specificity 99%, and positive predictive value only 7.5% compared to medical record review. The low PPV arose because pneumonia prevalence was very low (2%) and also the gold standard required signs and symptoms not captured on chest radiographs (and thus not available to MedLee.) Fiszman et al. applied SymText to 292 reports (pneumonia prevalence, 38%) and reported accuracy similar to ours: sensitivity 95%, specificity 85%, and PPV 78%.¹² Their gold standard was physicians’ interpretation of the reports. Elkin et al. applied the Multithreaded Clinical Vocabulary Server to 400 reports (pneumonia prevalence, 3.5%).¹³ Compared to physician review of the reports, NLP had sensitivity of 100%, specificity 90% and PPV 70%. Prior studies included relatively few reports. Most did not describe the care setting (inpatient vs. outpatient), and none reported accuracy for subgroups.

Strengths of our study include the large test set and the availability of clinical information including age, comorbidity, and care setting. These resources allowed more precise estimates of accuracy as well as subgroup-specific estimates. The preponderance of outpatient exams makes our findings more relevant to the general population. To our knowledge, ours is the first study to explore tailoring NLP tools to different study needs.

Thus far, we have applied NLP only to chest radiograph reports and not other clinical data relevant to pneumonia. Radiologists’ interpretation of chest radiographs can be inconsistent, with studies reporting kappa of 0.37 for infiltrate in adults²³ and 0.58 in children.²⁴ Established case definitions, e.g. the CDC/National Healthcare Safety Network definition for hospital-acquired pneumonia,¹⁰ typically require a positive radiograph in addition to clinical symptoms and findings. One could argue that NLP should draw on symptoms, vital signs, and laboratory findings in addition to radiology reports. Murff et al. took such an approach.¹¹ Compared to medical record review, their NLP algorithms had sensitivity of 64% (95% CI, 58-70%) and specificity of 95% (94-96%) for post-operative pneumonia. While a multi-faceted approach may be desirable, challenges to implementation exist, including that many of the data required are rarely measured in outpatients. Still, a crucial building block is an accurate method for classifying chest radiographs, which are central to most pneumonia definitions.

Our study has limitations. We measured comorbid illnesses from administrative data, which do not have perfect accuracy. Our data included relatively few inpatient cases, resulting in less precise estimates for this subgroup. Reports came from a single health care system, which may limit generalizability. Radiologists from different institutions or geographic regions may use different language, and so an important next step will be to assess transferability. It would also be useful to investigate transferability of our tools to other conditions. As a next step, we plan to assess our NLP tools’ accuracy for reports from other institutions and for other conditions. In the current study, considerable effort was needed to construct a classifier to create a binary pneumonia variable from ONYX’s output. We developed a rule-based classifier based on expert opinion, but machine learning is an alternative approach that could improve efficiency. It would also be valuable to explore the use of machine learning alone, without an NLP tool such as ONYX, to detect pneumonia from radiograph reports. Schuemie et al. recently described such efforts; he found that the best approaches achieved sensitivity of 82-93% with PPV 81-90%.²⁵ We currently use only features of the radiology report to determine which reports need manual review. Future work could explore whether incorporating patient characteristics (e.g. age or comorbidity) can improve this determination.

In conclusion, vast amounts of clinical data are becoming available in free text within EMRs. They could be valuable for many purposes but currently are expensive and time consuming to access. New technologies such as NLP offer tremendous opportunities. Studies such as ours provide insight into the potential of NLP to improve research and clinical care.

Supplementary Material

01

Online Appendix Table 1. Pneumonia classifier: further explanation of criteria used to identify reports needing manual review

Online Appendix Table 2. Factors Contributing to Changes in Performance Measures Between Classifier 1 and Classifier 2

Key points.

• When disease outcomes are identified from diagnosis codes in automated databases, misclassification can limit statistical power and bias results.

• Natural Language Processing (NLP) can extract information from free-text electronic medical records in an automated fashion, which could improve validity and efficiency.

• We found that one NLP system, ONYX, works well to identify pneumonia from free-text radiology reports: after training, ONYX could replace nearly 90% of manual medical record review, with sensitivity 75%, specificity 95%, and positive predictive value 86%.

• ONYX is available open-source and can be adapted for different outcomes and study needs.