Applying a Natural Language Processing Tool to Electronic Health Records to Assess Performance on Colonoscopy Quality Measures

Abstract

Background

Gastroenterology specialty societies have advocated that providers routinely assess their performance on colonoscopy quality measures. Such routine measurement has been hampered by the costs and time required to manually review colonoscopy and pathology reports. Natural Language Processing (NLP) is a field of computer science in which programs are trained to extract relevant information from text reports in an automated fashion.

Objective

To demonstrate the efficiency and potential of NLP-based colonoscopy quality measurement

Design

In a cross-sectional study design, we used a previously validated NLP program to analyze colonoscopy reports and associated pathology notes. The resulting data were used to generate provider performance on colonoscopy quality measures.

Setting

Nine hospitals in the UPMC health care system.

Patients

Study sample consisted of the 24,157 colonoscopy reports and associated pathology reports from 2008-9

Main Outcome Measurements

Provider performance on seven quality measures

Results

Performance on the colonoscopy quality measures was generally poor and there was a wide range of performance. For example, across hospitals, adequacy of preparation was noted overall in only 45.7% of procedures (range 14.6% to 86.1% across nine hospitals), documentation of cecal landmarks was noted in 62.7% of procedures (range 11.6% to 90.0%), and the adenoma detection rate was 25.2% (range 14.9% to 33.9%).

Limitations

Our quality assessment was limited to a single health care system in Western Pennsylvania

Conclusions

Our study illustrates how NLP can mine free-text data in electronic records to measure and report on the quality of care. Even within a single academic hospital system there is considerable variation in the performance on colonoscopy quality measures, demonstrating the need for better methods to regularly and efficiently assess quality.

Introduction

Colorectal cancer is the 2nd leading cause of cancer death in the US.¹ Colorectal cancer is preventable because removing adenomas, the precursors to colorectal cancer, can reduce colorectal cancer incidence. Colonoscopy is a cost-effective and common method of screening for colorectal cancer. ^2,3 However, colonoscopy has shortfalls in screening, because, among other reasons, physician miss adenomas during a colonoscopy.^4-8 There is great variation across physicians in the fraction of colonoscopies in which an adenoma is found (in one study rates ranged from 9.4% to 32.7%).⁹ Further, a patient who has a colonoscopy performed by a physician with a low adenoma detection rate is at increased risk of developing subsequent colorectal cancer.¹⁰

There is also variation in other aspects of colonoscopy quality.^8,11 For example, physician's recommendations for follow-up colonoscopies are often shorter than guideline recommendations.^12-14 In a large fraction of colonoscopies, there is no indication provided on the colonoscopy report or the indication listed is not on the list of recommended indications. This overuse of colonoscopy decreases access to care, drives up health care costs, and exposes patients to unnecessary risk.¹⁵ Access is important because there of a shortage of colonoscopy capacity.¹⁶ These data demonstrate, as one editorialist believes the “effectiveness of colonoscopy as a screening tool is a lot better (and worse) in some endoscopists' hands than others.”⁵

This evidence has led gastroenterology specialty societies to call for physicians to regularly monitor their performance on colonoscopy quality measures so care can be improved.^17,18 Unfortunately, such routine measurement is not taking place, primarily due to the inconvenience and expense.¹⁵ Measuring adenoma detection rates and other quality measures typically requires manual review of colonoscopy and pathology reports. To address the difficulty in measuring physician quality, we developed the first natural language processing (NLP)-based computer software application (C-QUAL) for measuring performance on colonoscopy quality indicators.¹⁹

Background

NLP is a field of computer science in which the computer is trained to “read” text to identify relevant data.²⁰ C-QUAL automatically analyzes both colonoscopy and pathology reports in the electronic health record (EHR) and abstracts the necessary information (e.g. indication, polyp detection, cecal intubation). It can thereby assess all the important aspects of colonoscopy quality. We tested C-QUAL by comparing it to the gold standard of manual abstraction by a physician and found that the C-QUAL tool had excellent accuracy on nine different quality measures advocated by specialty societies.¹⁹

In this study, our objective was to demonstrate the efficiency and potential of NLP-based colonoscopy quality measurement. To do so, we used C-QUAL to analyze almost 25 thousand reports in a large hospital system to to highlight performance variation across hospitals and physicians.

Methods

We conducted a cross-sectional analysis of reports from relevant colonoscopy procedures over a two-year period in a single hospital system.

Development of Quality Measures

The American Society for Gastrointestinal Endoscopy/American College of Gastroenterology¹⁷ and the U.S. Multi-Society Task Force on Colorectal Cancer¹⁸ have published quality measures for colonoscopy. From these publications, we identified 20 potential measures. For each measure, we created detailed specifications on which colonoscopy reports were eligible and which reports met criteria for passing [Appendix describes process of identifying 20 measures and the specifications]. For example, one published quality measure is notation and photo documentation of cecal landmarks. We judged visualization of the cecal landmarks to be met if the report stated any of the following, (1) appendiceal orifice or ileo-cecal valve (2) entering ileum, (3) any of the following phrases “cecal landmarks”, “typical landmarks”, “landmarks of cecum”, “characteristic anatomy”, (4) or noted that patient has a prior resection (e.g. “ileocolonic anastomosies”). If the report included negation of relevant phrases (e.g. “cecal landmarks not visualized”, we did not count this as visualization of cecal landmarks. Based on published quality measures for this measure,¹⁷ we excluded from the denominator reports where the preparation was inadequate and where the procedure was aborted prematurely.

Development and Validation of NLP Tool

NLP is intended to take the place of a manual abstractor, automatically extracting key information from colonoscopy reports and categorizing the information as passing or not passing a given quality measure based on the specifications. Our NLP-based colonoscopy measurement tool has three basic processing steps (1) parsing the text of the reports into sentences, words, and sections, (2) identifying key concepts (e.g., gastrointestinal bleeding) by matching the text against clinical vocabularies, and (3) understanding the meaning of key concepts (e.g., is the gastrointestinal bleeding the indication for procedure or a complication?) by examining their context in the report.

Details on the development and validation of our NLP-based colonoscopy measurement tool are published elsewhere.¹⁹ Briefly, we used an iterative process to develop the tool. Clinicians identified relevant information from randomly selected colonoscopy reports, and these samples of manually abstracted data were used to formulate the rules for the NLP tool. After this process was complete and the NLP tool was finalized, we tested its performance by applying it to 453 new colonoscopy and associated pathology reports. We compared the NLP tool's output against a manual evaluation of the same reports by a physician. We tested both the accuracy of abstraction of specific data elements as well as quality measures. As detailed in the Appendix, nine of the 20 measures showed high agreement (kappa > 0.7) between the NLP tool and clinician abstraction. Of these nine measures, we report seven in the manuscript. These quality measures are listed in Table 2 and each measure's specifications are included in the Appendix. As detailed in the Appendix, we dropped two measures (withdrawal time >6 minutes, fraction of colonoscopies where preparation adequate) as they were redundant and highly correlated with two measures that were retained (withdrawal time documented, fraction of colonoscopies where preparation documented).

Table 2. Performance on Quality Indicators for Individual Hospitals.

Hospital	Total number of colonoscopy reports	ASA Classification of Physical Status is Indicated#		Informed Consent from Patient Documented		Quality of Bowel Preparation Described		Cecal Landmarks Noted		Adenoma Detected		Withdrawal Time for Colonoscopy Noted		If Indication for Colonoscopy is Chronic Diarrhea, Biopsy Taken
		No. of relevant reports*	Rate (%)	No. of relevant reports*	Rate (%)	No. of relevant reports*	Rate (%)	No. of relevant reports*	Rate (%)	No. of relevant reports*	Rate (%)	No. of relevant reports*	Rate (%)	No. of relevant reports*	Rate (%)
A	1216	1216	15.3	1216	25.4	1216	14.6	1191	46.6	1137	33.9	1216	0.0	54	74.1
B	1248	1248	0.0	1248	98.3	1248	86.1	1227	86.6	1142	19.5	1248	2.2	46	95.7
C	2539	2539	0.5	2539	2.1	2539	15.4	2496	11.6	2335	18.8	2539	0.0	59	84.7
D	1004	974	0.0	974	43.6	1004	44.9	957	79.8	911	24.3	1004	3.4	67	82.1
E	6337	5446	0.0	5446	38.9	6337	36.7	6153	82.1	5938	26.0	6337	0.2	312	73.4
F	2323	2323	0.9	2323	90.2	2323	84.8	2243	90.0	2100	25.8	2323	1.8	149	87.2
G	3643	3643	39.8	3643	49.5	3643	51.8	3444	46.9	3205	25.8	3644	3.7	177	82.5
H	835	779	0.1	779	64.1	835	61.3	784	76.7	753	14.9	835	5.6	62	77.4
I	5012	4941	0.0	4941	61.3	5012	44.8	4830	55.2	4597	27.7	5012	1.5	271	78.2

^*As detailed in quality measure specifications in Appendix, reports not relevant to quality indicator have been dropped. For example, procedures with inadequate preparation were not included in the denominator for adenoma detection rates.

^#American Society of Anesthesiologists (ASA) classification of physical status and fitness for surgery

Dataset

Our data came from the UPMC Medical Archival System data repository,²¹ which includes clinical data that comes directly from the electronic medical record of patients seen at UPMC's hospitals and physician offices. We limited our analyses to reports from 2008-9 at the nine UPMC hospitals where colonoscopies are performed, which are not children's hospitals, and where pathology reports are available. Some UPMC hospitals are still in a transition stage of interfacing transcription data into the database and all reports were not available for analysis. The data repository permits longitudinal profiles of patient events to be created with direct links to all the elements of the clinical electronic record. All records were de-identified using the De-ID program.²²

Applying NLP Tool to Sample of Colonoscopy Reports

After identifying all colonoscopy and associated pathology reports, we excluded reports where (1) there was more than one endoscopic procedure (e.g. upper endoscopy and colonoscopy) and/or pathology report on the same day; (2) the same report described a combined upper endoscopy and colonoscopy procedure; (3) the patient identifier was missing and (4) the report was generated via ProVation software. ProVation software allows a clinician to generate a colonoscopy report with pre-populated drop-down menus. Because the reports are already generated in a structured format, we did not apply the NLP tool to these reports. We did include reports generated by Pentax software (14% of all colonoscopy reports in our sample), in which the physician is prompted to input free text to fill in template sentences and to record additional information. These Pentax reports represent a combination of free text and structured sentences. The remaining colonoscopy reports in our study sample were dictated and transcribed (86% of reports).

After applying the NLP tool to the reports, we linked the reports to information on the patient, the physician who performed the procedure, and the hospital where it was performed. For some physicians we had access to only the subset of the colonoscopies they performed at UPMC hospitals. These physicians are in private practice and performed procedures at UPMC hospitals, non-UPMC hospitals, and unaffiliated private endoscopy centers.

Statistical Analysis

We report the overall performance for each quality measure by hospital and by physician. We restricted our analyses on variation in physician performance to physicians with at least 30 reports in our dataset; 30 is a cut off used in prior work.¹⁰ To assess the face validity of the data reported by the NLP tool, we assessed the adenoma detection rate by selected patient characteristics known to be associated with higher risk of adenomas.^14,23-25 All statistical analyses were conducted using SAS (Version 9.2, SAS Institute, Cary, NC, 2010). Approval for this study was obtained from both the University of Pittsburgh and RAND Human Subjects Protection Committees.

Results

A total of 24,157 reports were analyzed using the NLP tool. Of all colonoscopies, 54.1% were performed on females and the majority of patients (59.0%) were between 50-69 years of age.[Table 1] All 9 hospitals were in urban areas and four were members of the Council of Teaching Hospitals. The number of admissions per year at the hospitals varied from 5000 to <10,000 (n=2, 22.2%), 10,000 to <20,000 (n=5, 55.6%), and ≥20,000 (n=2, 22.2%) (Appendix Table 5). The specialty of the 77 physicians with 30 or more cases was gastroenterology/internal medicine (n=60, 77.9%) or general/colorectal surgery (n=17, 22.1%). Twenty-four (31.2%) had primary teaching appointments at the University of Pittsburgh School of Medicine (Appendix Table 6).

Table 1. Patient and Provider Characteristics of 24,157 Colonoscopy Reports^#.

Characteristic	Value
Sex - no. (%)*
Female	13,038	(54.1)
Male	11,063	(45.9)
Age group - no. (%)
≤18 yr	14	(0.1)
19-29 yr	669	(2.8)
30-39 yr	1,067	(4.4)
40-49 yr	2,717	(11.2)
50-59 yr	8,375	(34.7)
60-69 yr	5,884	(24.4)
70-79 yr	3,690	(15.3)
≥80 yr	1,741	(7.2)
Race - no. (%)
White	20,532	(85.2)
Black	2,195	(9.1)
Other	1,430	(5.9)
Insurance - no. (%)
Medicare	6,262	(26.0)
Private	16,290	(67.6)
Medicaid	1,002	(4.2)
Other	603	(2.5)
Faculty status of physician performing procedure - no. (%)*
Primary teaching appointment	4,434	(18.4)
Other	19,519	(81.0)
Specialty of physician performing procedure - no. (%)*
Gastroenterology or internal medicine	22,104	(91.7)
General surgery or colorectal surgery	1,849	(7.7)

^*For 56 reports sex not recorded. For 204 reports, physician name not recorded or faculty status or specialty not available. Typically faculty status and specialty is not recorded for trainees.

^#Totals may not equal 100% because of rounding

Analysis of the data generated by our NLP tool revealed that performance on most quality measures was poor, with a wide range of performance across hospitals and physicians (Table 2, Figure 1, Appendix Table 6). American Society of Anesthesiologists (ASA) classification for fitness for surgery was reported in 7.2% of reports (range 0% to 39.8% across 9 hospitals), informed consent for procedure was noted in 50.0% (range 2.1% to 98.3%), preparation of colon was recorded in 45.7% (range 14.6% to 86.1%), cecal landmarks in 62.7% (range 11.6% to 90.0%), withdrawal time was recorded in 1.5% (range 0% to 5.6%), and a biopsy was obtained for patients with chronic diarrhea in 79.7% (range 73.3% to 95.7%). One or more adenomas were detected in 25.2% of colonoscopies (range across hospitals 14.9% to 33.9%).

Figure 1. Performance on Quality Indicators for 9 Individual Hospitals and 77 Physicians with ≥30 Reports^*.

^*Heavy vertical line indicates median performance and the surrounding box represents the boundaries of the 25^th and 75^th percentile of performance. The extended horizontal line (“whisker”) extends one and a half times the interquartile range beyond the box. The individual circles mark the performance of providers whose performance is outside the boundaries of the box (i.e. the 25^th and 75^th percentile).

On six of the seven quality measures, colonoscopy reports using the structured report system had a higher quality score than dictated reports (Figure 2). For example, quality of preparation was noted in 90.3% of reports generated by the stuctured report system and 36.4% of dictated reports (p<0.001 for test of comparison). Only with adenoma detection rate was there lower performance among reports generated by the stuctured reports systems vs. dictation (21.9% vs. 25.9% respectively, p<0.001).

Figure 2. Performance on Quality Indicators Based on How Colonoscopy Report Was Created^*.

^*Difference between two types of methods of generating reports statistically significant at p<0.001 level for 6 of the 7 quality measures. For difference on withdrawal time, p-value 0.02

Several patient and physician factors were associated with differential adenoma detection rates (Appendix Table 8). For example, adenoma detection rates were higher among male vs. female patients (31.0% vs. 20.3%, p<0.001) and older patients (e.g. >80 yo patients (31.3%) vs. 40-49 yo (16.7%)). Colonoscopies performed by gastroenterologists had higher adenoma detection rates than those performed by general surgeons (25.6% vs. 20.5%, p<0.001).

Discussion

Our results highlight the potential of NLP to measure performance on colonoscopy quality measures. Our NLP tool efficiently analyzed a large sample of colonoscopy reports. Our findings demonstrate there is clear variation in performance, even within a highly-regarded academic healthcare system. Across the nine hospitals, there was almost three-fold variation in the adenoma detection rate. The variation in performance on the quality measures across physicians was even larger.

Prior work has identified this variation in quality as one of the limitations of colonoscopy as a screening tool for colorectal cancer screening.⁵ Specialty societies and experts have called for providers to routinely report on their colonoscopy performance,⁵ but few physicians are reporting their quality.^11,26 NLP could be one means of making routine reporting more common.

Our results highlight several advantages and disadvantages of using NLP for routine quality measurement using data in EHRs. The key advantage of NLP is that it is economically feasible. It would be prohibitively expensive and time consuming to manually review tens of thousands of colonoscopy reports. Another advantage is that NLP allows providers to continue to use natural narrative when describing patient care. There has been criticism that structured note systems in current EHRs force providers to create unnatural and overly structured notes²⁷ which take extra time to create^28,29 and impede communication because these notes are difficult to read.²⁹ The NLP tool also has the advantage is that it can be easily adapted over time to incorporate changes to the quality measures.

There are also disadvantages of NLP-based quality measurement. As we document in the Appendix, the accuracy of the tool compared to physician abstraction decreased as the complexity of the data and language increased (e.g. indication for procedure). We focused on colonoscopy reports which are relatively limited in scope and length. NLP-based tools will have more difficulty with documents such as outpatient notes or discharge summaries. For high-stakes applications such as public reporting of quality or pay-for-performance tools, providers may not accept the level of accuracy currently achieved by our NLP tool. However, we note that we are working to refine the NLP program to improve its performance and that the alternatives to NLP also have limitations. Data manually abstracted from medical records frequently contain errors^28,30 and claims data, another common source of data for quality measures, may not always accurately reflect clinical care.³¹ Another issue with our current tool is that we only focus on the text within the colonoscopy and pathology notes. Important information could be included elsewhere. For example, the indication for the procedure might be on a progress note and the withdrawal time could be calculated by comparing the time stamps on the photographs of the appendiceal orifice and retroflexion in the rectum (if such photos were included).

Instead of dictations, some physicians are using software (e.g., CORI, gMed, ProVation) with pull-down menus to create colonoscopy reports. These report tools facilitate quality measurement and some might be concerned that these software programs preclude the need for a NLP tool. We believe our NLP tool still has advantages. To generate adenoma detection rates using such software, one must still manually abstract pathology reports. Also, within and across endoscopy centers, physicians might use different software tools and/or dictation. C-QUAL can analyze any type of report.

In our analyses, reports generated by software tools had higher performance on documentation-specific measures (e.g., indicating informed consent), but there was no difference in performance for adenoma detection rates. There was also no clear correlation across hospitals between documentation-specific measures and adenoma detection rates. For example, the hospital with the highest adenoma detection rate had the lowest performance on reporting bowel preparation and withdrawal time. These findings highlight a larger debate regarding the focus on quality measurement. From a clinical perspective, colonoscopy quality measures that reflect documentation may be seen as less important than measures of outcomes such as adenoma detection rate.³² In other words, devoting resources to software that improves the completeness of documentation may not necessarily lead to better clinical outcomes.

Consistent with prior studies,^14,23-25 we find that the adenoma detection rate is higher among males and older patients. This consistency with the prior literature increases the face validity of the NLP tool. Though there is great variation in the adenoma detection rate across providers in our sample, the overall adenoma detection rate observed in our study population exceeds the recommended benchmarks for the adenoma detection rate (≥25% for men over 50, ≥15% for women over 50) published in quality guidelines.¹⁷

Our study has several limitations. Because it is limited to a single hospital system and there is variation in the manner physicians record colonoscopy reports, it is likely that our NLP tool would need to be adapted to the reporting style and language used by other physicians to achieve comparable performance in another setting. Consistent with prior studies, we did not adjust provider scores for differences in patient population. This type of risk adjustment will have to be considered, especially if one begins profiling physicians who sub-specialize and who treat patients with clearly different adenoma detection rates (e.g. patients with inflammatory bowel disease or younger patients).

In summary, our analyses highlight the potential for NLP to evaluate performance on colonoscopy quality measures in an inexpensive and automated fashion. This type of routine quality meaurement can be the foundation for efforts to improve colonoscopy quality.