Using Natural Language Processing to Extract Abnormal Results from Cancer Screening Reports

Abstract

OBJECTIVES

Numerous studies show that follow-up of abnormal cancer screening results, such as mammography and Papanicolaou (Pap) smears, are frequently not performed in a timely manner. A contributing factor is that abnormal results may go unrecognized because they are buried in free-text documents in electronic medical records (EMRs) and, as a result, patients are lost to follow-up. By identifying abnormal results from free-text reports in EMRs and generating alerts to clinicians, natural language processing (NLP) technology has the potential for improving patient care. The goal of the current study is to evaluate the performance of NLP software for extracting abnormal results from free-text mammography and Pap smear reports stored in an EMR.

METHODS

A sample of 421 and 500 free-text mammography and Pap reports; respectively, were manually reviewed by a physician and the results categorized for each report. We tested the performance of NLP to extract results from the reports. The two assessments (‘gold’ standard vs. NLP) were compared to determine the precision, recall and accuracy of NLP.

RESULTS

When NLP was compared to manual review for mammography reports, the results were as follows: precision = 98% (96–99%), recall = 100% (98–100%) and accuracy = 98% (96–99%). For Pap smear reports, the precision, recall and accuracy of NLP were all 100%.

CONCLUSION

Our study developed NLP models that accurately extract abnormal results from mammography and Pap smear reports. Future plans include using NLP technology to generate real-time alerts and reminders for providers to facilitate timely follow-up of abnormal results.

INTRODUCTION

Inadequate follow-up of abnormal test results represents a significant patient safety and malpractice concern.^1–3 In fact, the fastest growing area of malpractice litigation involves failures or delays in diagnosis. Of these, 25% are attributable to avoidable failures in the test follow-up system.⁴ Accordingly, follow-up of outpatient test results has become a major priority for organizations and policy makers concerned with healthcare quality and patient safety; so much so, that the National Committee for Quality Assurance has included implementation of reliable systems to effectively track test results as a criteria for primary care practices to attain Patient-Centered Medical Home designation.⁵ These safety concerns are echoed by physicians who perceive that follow-up of abnormal test results, as well as the systems used for follow-up, are sub-optimal.^6–9

Test result follow-up is especially problematic for cancer screening, and inadequate test result follow-up has been implicated in missed and delayed cancer diagnoses.^3,10 For example, 28% of women don’t receive timely follow-up of abnormal cervical cancer screening results and minority women are at especially high risk.^11,12 Major factors contributing to the lack of timely follow-up are sub-optimally designed work processes and the dearth of clinical decision support systems in the healthcare system.^13–15 In this environment, abnormal test results often fall through the cracks⁸ and in the case of cancer screening test results, may lead to delayed diagnoses and increased morbidity and mortality.

Clinical Decision Support Systems (CDSSs) that generate alerts and reminders to clinicians improve guideline adherence to recommended care.¹⁶ CDSSs use structured data elements in electronic medical records (EMRs) and conditional (if …, then …) logic to trigger alerts when pre-defined conditions are satisfied. Currently, unstructured clinical data, such as that found in free-text cancer screening reports (Pap smear and mammography), cannot be used in CDSSs.

Natural language processing (NLP) has the ability to extract results from free-text clinical reports stored in EMRs and convert the results into a structured format suitable for use by CDSSs.^17,18 The overall goal of our study is to develop and validate NLP that will accurately identify results in free-text mammography and Pap smear reports and convert them into a structured format suitable for CDSS.

METHODS

Study Setting and Population

The UNC Health Care System includes NC Memorial Hospital, a Neuroscience Hospital, the UNC Women’s Hospital, the UNC Children’s Hospital and an Ambulatory Care Center in which 154,224 unique patients were seen at UNC Ambulatory Centers for 973,122 visits. Overall, 10% of patients are Latino, 58% female, and 97,851 were between 19 and 59 years of age, while 34,862 were aged 60 or older.

Natural Language Processing (NLP) of Free-Text Clinical Documents

The NLP software used for the study employs dictionary-based named entity recognition¹⁸ to identify the standard terminologies used to document mammography and Pap smear results in the free-text reports. All NLP models were developed using the IBM Content Analytics software package residing on a computer server at the Information Services Division of the University of North Carolina Hospital. The study was conducted at the University of North Carolina Healthcare System and was approved by the institutional review board (IRB number 11-1197). The project was funded by a National Institutes of Health grant 1UL1TR001111 through the North Carolina Translational and Clinical Sciences Institute.

Our goal is to attain NLP performance characteristics (precision, recall and accuracy) greater than 95%, with 95% confidence intervals of +/− 1.5% (range of 3%). For these performance specifications, we calculated that a minimum sample size of 400 reports, each for mammograms and Pap smears, was required.

Mammography Reports

Mammography results are reported as Breast Imaging Reporting and Data System (BIRADS) assessment categories ranging from 0 to 6, each with a specific recommendation for follow-up (table 1).¹⁹ Figure 1 shows an example of a mammography report with the BIRADS result and follow-up recommendations highlighted in yellow. In the semi-structured mammography reports, results are always located after the word “IMPRESSION:” and the follow-up recommendations appear after a detailed description of the mammography findings. NLP algorithms were developed to detect the term “IMRESSION:” in the free-text mammography documents and to extract the BIRADS results that appear afterwards. In the corpus of mammography reports, there was some variability in how results were documented; for example, BIRADS 1 might appear as “BI-RADS 1”, “BIRADS I”, “BIRADS-1” or “BIRADS #1”. Therefore algorithms were developed to correctly map all possible iterations of BIRADS 1 using conditional logic (if …, then … rules) in the NLP soft software.

TABLE 1.

Breast Imaging Reporting and Data System (BIRADS)

BIRADS Category	Follow-up Recommendation	% of Sample (N=421)
0: Incomplete	Need for further evaluation	3.6% (15)
1: Normal	Normal interval follow-up	13.5% (57)
2: Benign	Normal interval follow-up	57.9% (244)
3: Probably benign	A short-interval follow-up	12.1% (51)
4: Suspicious abnormality	A biopsy should be considered	8.6% (36)
5: Highly suggestive for malignancy	Biopsy or surgery should be performed	1.2% (5)
6: Biopsy-proven carcinoma	Appropriate action should be taken	3.1% (13)

FIGURE 1.

Sample Mammography Report.

We selected a random sample of 421 free-text mammography reports completed between January 2003 and January 2012 from the Carolina Data Warehouse for Health (CDW-H). The reports were manually reviewed by a general internist (CM) and the BIRADS result determined for each. We developed and tested the performance of an NLP model to extract the BIRADS results from the same set of reports. The two assessments (‘gold’ standard vs. NLP) were compared to determine the precision, recall and accuracy of NLP for extracting mammography results. Precision, recall and accuracy were calculated as follows:

Precision = TP/(TP+FP), Recall = TP/(TP+FN), and Accuracy = (TP+TN)/Total; where TP=true-positive, FP=false-positive, TN=true-negative, and FN=false-negative.

Papanicolaou (Pap) Smear Reports

Figure 2 shows a portion of a Pap smear report with results highlighted in yellow. Pap smear results are always located in free-text after the expression “RESULTS/INTERPRETATION” and are reported using the Bethesda Classification²⁰ of cervical cytology (table 2). For example, results of the Pap report in figure 2 would map to “ASC-H” in table 2. NLP algorithms were developed that map the results identified in the free-text documents with the corresponding Bethesda Classification shown in table 2.

FIGURE 2.

Sample Pap Smear Report.

Table 2.

Pap Report Results (501 results in 500 reports)^†

Pap Results	% (n)
Negative	89.4% (447)
Unsatisfactory for evaluation	2.8% (14)
Abnormal Results (40)
Atypical Squamous Cells of Undetermined Significance (ASCUS)	3.0% (15)
Atypical Squamous Cells, Cannot Exclude High-Grade Squamous Cell Intraepithelial Lesion (ASC-H)	0.0% (0)
Low-grade Squamous Intraepithelial Lesion (LSIL)	3.8% (19)
High-grade Squamous Intraepithelial Lesion (HSIL)	0.4% (2)
Atypical Glandular Cells (AGC)	0.4% (2)
Squamous Cell Carcinoma (SCC)	0.2% (1)
Endocervical Adenocarcinoma in Situ (AIS)	0.2% (1)

^†One Pap report had 2 results.

We used a random sample of 500 free-text Pap smear reports completed between January 2003 and January 2012 and stored in the CDW-H. Pap smear reports were manually reviewed by a general internist (CM) and the results (Bethesda Classification²¹) were determined for each report; this was used as the “gold” standard to compare with NLP results. The performance of NLP was assessed using Precision, Recall and Accuracy as described previously.

Finally, we determined inter-rater reliability for manual review of mammogram and Pap smear results by having a second reviewer (EA) independently abstract results from a 20% random sample of mammogram and Pap smear reports. We compared the results to those obtained by the 1^st reviewer (CM). Inter-rater agreement was calculated using the kappa coefficient.

Follow-up of Abnormal Pap Spear and Mammogram Results

For each abnormal Pap smear and mammogram (BIRADS 4, 5, and 6) result identified using NLP, we reviewed the electronic medical record (EMR) to assess the amount of time to repeat testing or follow-up procedure (example; colposcopy or core biopsy). If none was found, we reviewed clinical notes to determine if the abnormal result was acknowledged and that medical management other than repeat testing was chosen or that follow-up was to be completed at an outside institution. All calculations were performed using Stata for Windows (Stata Corporation, College Station, Tx).

RESULTS

Mammography Reports

Manual review (‘gold’ standard) of the 421 mammography reports determined that 3.6%, 13.5%, 57.9%, 12.1%, 8.6%, 1.2%, and 3.1% of results were BIRADS 0 thru 6; respectively (table 1). Inter-rater agreement (kappa coefficient) between the two reviewers was 1.0. When we compared NLP against the ‘gold’ standard manual review, the results were as follows:

Precision = 98% (96–99%), Recall = 100% (98–100%) and Accuracy = 98% (96–99%). The only inaccuracies in the NLP model occurred with mammography reports that had initial BIRADS results of zero (0) with later addendums documenting result changes (example; changing BIRADS 0 to BIRADS 2) based on radiologists’ reviews of previous mammography results. In these situations, the NLP model identified the BIRADS 0 as the final result and did not detect results in the addenda. This occurred in a total of 10 (2.4%) of the 421 reports and we have since modified the NLP model so that it now identifies results in report addenda.

Follow-up of Abnormal Mammograms

For abnormal mammogram results (BIRADS 4, 5 and 6) identified using NLP, we reviewed the medical record to determine time between incident abnormal results and any repeat testing or procedures (table 3). For BIRADS 4 mammogram results, 34 of 36 patients had follow-up (usually biopsy) within 2 months. Of the two patients not followed up within 2 months, one had fine needle aspiration of a left axillary mass (identified on the mammogram) approximately 8 months (251 days) after the incident BIRADS 4 result. Results of the fine needle aspiration showed metastatic adenocarcinoma consistent with breast primary. The second patient with BIRADS 4 and no follow-up within 2-months had biopsies of bilateral breast masses 79 days later with results showing benign pathology.

TABLE 3.

Follow-up of Abnormal Mammograms (N=54)

BIRADS Category	Follow-up within 1 Month	Follow-up within 2 months	Description of Results not Followed up within 2 Months
4	86.5% (31 of 36)	94.6% (34 of 36)	Mammography report recommended fine needle aspiration of left axillary mass. Fine needle aspiration was performed 251 days later and the pathology results showed metastatic adenocarcinoma consistent with breast primary. Mammography report recommended biopsies of bilateral hypoechoic masses in each breast. The biopsies were performed 79 days later and the pathology results were benign.
5	80.0% (4 of 5)	80.0% (4 of 5)	Mammography report recommended biopsy of two breast masses suspicious for malignancy. Biopsy was performed 239 days later and pathology showed invasive ductal carcinoma.
6	100.0% (13 of 13)	100.0% (13 of 13)	All BIRADS 6 results were followed up within 1 month
TOTAL	89.1% (48 of 54)	94.6% (51 of 54)

Four of 5 patients with BIRADS 5 had follow-up at 2 months (table 3). The one remaining patient had biopsies of 2 breast masses approximately 8 months (239 days) after the incident BIRADS 5 result and the pathology was consistent with invasive ductal carcinoma. All patients with BIRADS 6 results were followed up within 1 month.

Papanicolaou (Pap) Smear Reports

Results of the manual review (‘gold’ standard) of 500 Pap reports are shown in table 4. Inter-rater agreement between the two reviewers was 1.0. When we compared Pap results determined by NLP against the ‘gold’ standard, we obtained 100% each for Precision, Recall, and Accuracy for the NLP model.

TABLE 4.

Follow-up Testing for Abnormal Pap Smear Results (N=40)

Abnormal Pap Smear Category	No Documented Follow-up Testing	If Follow-up, Average Months to Follow-up
Atypical Squamous Cells of Undetermined Significance (ASCUS)	47% (7 of 15)	8.1 months
Low-grade Squamous Intraepithelial Lesion (LSIL)	37% (7 of 19)	8.5 months
High-grade Squamous Intraepithelial Lesion (HSIL)	0% (0 of 2)	12.5 months
Atypical Glandular Cells (AGC)	0% (0 of 2)	0.6 months
Squamous Cell Carcinoma (SCC)	0% (0 of 1)	0.1 months
Endocervical Adenocarcinoma in Situ (AIS)	0% (0 of 1)	1.6 months

Follow-up of Abnormal Pap Smear Results

For the abnormal Pap smear results identified using NLP, we reviewed the medical record in order to determine the time between the incident abnormal Pap result and any repeat testing or procedures (example; repeat Pap smear or colposcopy). For women with Pap smear results showing low-grade squamous intraepithelial lesion (LSIL), 37% had no documented follow-up testing at 18-months after their incident abnormal Pap (table 4). For the atypical squamous cells of undetermined significance (ASCUS) Pap abnormalities, 47% of women had no documented follow-up testing at 18-months.

DISCUSSION

Although mammography and Pap smears have well-established lexicons for results reporting, results of these diagnostic procedures are not routinely stored in clinical information systems as discrete query-able data elements, but are “buried” in free-text narrative reports. As a result, the full benefit of using standard lexicons to report results are not realized.^22,23 A major barrier to structured reporting can be found in current adoption patterns in which breast imaging, cardiology, and gastroenterology are the most common use cases.²⁴ Diagnostic reports in these disciplines describe well-circumscribed anatomical sites that are subject to fairly narrow pathologies, thus making results reporting conducive to manageable structured templates. However, the spectrum of possible anatomical sites and potential pathologies increases significantly for other studies such as magnetic resonance imaging (MRI) scans, thus making structured results reporting potentially subject to inaccuracies and lack of completeness.²⁵ For example, a study by Johnson²⁵ found that completeness and accuracy scores for MRI results declined significantly when physicians switched from a narrative reporting format to a structured format. However, other investigators have found improvements in satisfaction with diagnostic reports after adoption of structured reporting.²⁶ Therefore, for some diagnostic studies, there may be trade-offs between result quality (completeness and accuracy) and structured formatting of results. Finally, some radiologists are concerned that structured data entry of results may adversely affect their ability to effectively interpret radiographic images because of sub-optimally designed human-computer interfaces.²⁷ Future research is needed to investigate potential trade-offs of narrative versus structured results reporting on outcomes such as physician workflow, report quality, and timely follow-up of abnormal results.

In summary, much of the data generated by our healthcare system, including radiology reports, pathology reports, clinical notes and discharge summaries, will continue to be entered into EMRs as narrative text. Given the increasing need for data analytics to help drive evidence-based patient care, regulatory policies and NLP-enabled technologies will be needed to effectively leverage this unstructured data. Along these lines, one of the Stage-3 meaningful use regulations proposed by the Office of the National Coordinator for Health Information Technology is that EMRs have the ability to identify abnormal test results and notify ordering providers.²⁸

Presumably, this should include abnormal results stored as both structured and unstructured data in EMRs. Our study developed NLP models that accurately identify abnormal results from mammography and Pap smear reports and convert the results into structured data that can be used to generate computerized alerts and reminders for clinicians. Future studies should focus on integrating accurate NLP models into CDSSs that will automatically alert providers of abnormal results in free-text reports. Wagholikar and colleagues¹⁷ conducted a pilot study of a CDSS that used NLP to identify and extract results from free-text electronic Pap smear reports and make follow-up recommendations. The authors found that the CDSS recommendations were correct in 73 out of 74 test patients when compared with physician judgment. Also, two cases were correctly identified by the NLP-based CDSS for gynecological referral for abnormal results that were not initially identified by the physician. The one incorrect CDSS recommendation in the Wagholikar¹⁷ study and the excellent, but imperfect, NLP model performance in our study highlight a major issue going forward with NLP-enabled CDSSs; what level of accuracy is required before these systems can be safely used in patient care? For example, does the NLP model need to perform perfectly to be used for patients care or alternatively, is 100% sensitivity with less than perfect positive predictive value sufficient? Also, what type of governance structures should institutions develop to monitor and ensure the satisfactory performance of NLP-based CDSSs. To illustrate further, the performance characteristics of a previously high-performing NLP model may become unacceptable because the format of a free-text report in the EMR has changed. If this goes undetected, then patients’ can be harmed when alerts do not fire appropriately. Therefore, as NLP-based CDSSs are implemented, institutions need to establish governance structures that are responsible for monitoring performance and upgrading NLP-based CDSSs as test report formats and recommendation guidelines change. There is growing literature showing that poorly designed or poorly implemented health information technology can result in patients harm²⁹; this is certainly the case with NLP-based CDSSs. For example, electronic health records that generate excessive numbers of alerts for relatively minor result abnormalities may increase the frequency of missed test results due to information overload and alert fatigue.^30,31 To prevent alert fatigue, systems should permit a degree of customization by physicians that prioritize alerts based on urgency level.^32,33

Our study found that significant numbers of abnormal mammogram and Pap smear results identified using NLP were not followed up in a timely manner. One woman with a BIRADS 4 result on her mammogram with a recommendation that several suspicious axillary masses be followed up via fine needle aspiration did not have the recommended procedure for another 8 months (251 days). Results of fine needle aspiration of the axillary masses were consistent with metastatic adenocarcinoma (breast primary). A second woman in our study with a BIRADS 5 result did not have recommended follow-up (biopsy) for approximately 8 months (239 days) and the results showed invasive ductal carcinoma. We also found instances of abnormal Pap smears with inadequate follow-up. Guidelines indicate that patients having Pap smear results with LSIL should be referred for colposcopy (if HPV testing positive or not performed) or for repeat Pap smear testing in one year (if HPV negative). However, 33% of patients with LSIL results in our study had no evidence of follow-up at 18 months from the incident abnormal Pap smear result. Our results are similar to that reported in the literature showing follow-up of abnormal Pap smear results is frequently inadequate.^11,34,35 For example, up to 28% of women don’t receive timely follow-up of abnormal results; especially minority women.^11,12 Although multiple factors affect timeliness of follow-up for abnormal cancer screening results¹³, the recognition of abnormal results by clinicians is the first step in the process - a process found to be sub-optimal and susceptible to error^7,8,13. Poon et al⁸ found that 83% of physicians reported at least one delay in reviewing patients’ test results during the previous 2-months; automated alerts and reminders may significantly reduce these delays. Presumably a well-designed NLP-based CDSS can help avoid or ameliorate some of the diagnostic delays we found in our study. For example, using structured and unstructured data from an EMR, a rules-based CDSS can generate physician alerts if patients with BIRADS 4–6 results have no evidence of subsequent follow-up (breast surgery clinic, biopsy, fine needle aspiration, etc.) within a prescribed time interval. Physicians and/or care managers can turn off alerts if they know patients have had follow-ups at outside institutions or if patients had subsequent follow-ups not included in the CDSS algorithm (example, palliative care or hospice).

There are several limitations in our study. The first limitation is that the NLP models we developed may not be readily portable to other institutions that have different formats and word pattern in their mammography and Pap smear reports. However, institution-to-institution variation in the report formats is likely minimized because of the ubiquity of result reporting standards for these tests. A second limitation is that we chose use cases (mammography and Pap smears) in which result reporting is standardized (BIRADS¹⁹ and Bethesda²⁰) and the number of possible result categories fairly small, with seven for mammography (table 1) and nine for Pap smears (table 2). Other types of reports, such as radiographs (other than mammograms), present much more of a challenge for NLP in that result reporting is less standardized and the number of possible results numerous. Consequently, NLP performance is highly dependent on the diagnostic radiology procedure being used, and recall (sensitivity) can range from 98% for detecting pulmonary embolus on chest CT reports to 68% for intracranial hemorrhage on head CT reports.³⁶ The last limitation is that we relied on documentation in the electronic medical record to determine if abnormal mammogram and Pap smear results were followed up. It is possible that patients had appropriate and timely follow-up at outside institutions and this was not documented in our institution’s EMR.

Finally, patients play an important role in the result follow-up process. For example, direct notification of test results to patients via letter, email, telephone, or patient portal might serve as a safety net to help facilitate timely follow-up of abnormal results and help empower patients to be active participants in their care.^3,37–39 In fact, proposed Stage-3 meaningful use objectives for electronic health records stipulate that patients must be able to view their test results on line within 4 business days of the information becoming available to their physicians.²⁸ However, as the trend towards direct patient notification continues, systems need to be designed that facilitate patients’ understanding of their test results.^38,40

CONCLUSION

In summary, this study outlines the development of NLP models that convert free-text results of mammography and Pap smear reports into a structured format that can be used to generate alerts and reminders for clinicians. Future directions include linking the NLP-derived Pap smear and mammography results to explicit evidence-based guidelines that can be provided to clinicians via an EMR-based CDSS and studying the impact of the CDSS on timely follow-up of abnormal cancer screening results.