Electronic Surveillance For Catheter-Associated Urinary Tract Infection Using Natural Language Processing

Abstract

Catheter-associated urinary tract infection (CAUTI) is a common and costly healthcare-associated infection, yet measuring it accurately is challenging and resource-intensive. Electronic surveillance promises to make this task more objective and efficient in an era of new financial and regulatory imperatives, but previous surveillance approaches have used a simplified version of the definition. We applied a complete definition, including subjective elements identified through natural language processing of clinical notes. Through examination of documentation practices, we defined a set of rules that identified positively and negatively asserted symptoms of CAUTI. Our algorithm was developed on a training set of 1421 catheterizedpatients and prospectively validated on 1567 catheterizedpatients. Compared to gold standard chart review, our tool had a sensitivity of 97.1%, specificity of 94.5% PPV of 66.7% and NPV of 99.6% for identifying CAUTI. We discuss sources of error and suggestions for more computable future definitions.

Introduction

Catheter-associated urinary tract infection (CAUTI) is a common healthcare-associated infection (HAI) in the US, contributing to the deaths of up to 13,000 patients and costing at least $400 million annually.^1,2 Especially in light of new reporting requirements and payment reforms penalizing hospitals for development of certain HAIs, CAUTI identification and prevention have become top priorities for hospitals.^3-5

Despite the economic and medical importance of CAUTI, standardized measurement of CAUTI incidence is challenging for two reasons. First, the standard Centers for Disease Control and Prevention (CDC) National Healthcare Safety Network (NHSN) CAUTI definition is complex and subjective.^6–8 Second, CAUTI surveillance has historically been done through manual chart review, which is subject to significant inter-observer variability and is resource-intensive at a time when infection control departments have fewer resources for surveillance.^9–11 As of 2008, fewer than half of hospitals had an established, house-wide surveillance system for monitoring CAUTI.¹²

Given the need for cheaper, more reliable surveillance in the electronic health record (EHR) age, there has been a trend toward development of automated HAI surveillance tools.^8,13,14 However, previous attempts to automate CAUTI surveillance have modified the NHSN definition to exclude subjective symptoms because they are not typically stored as coded EHR data.^15–20 Some recent approaches have used natural language processing (NLP) to identify the presence of a urinary catheter and/or fever.^21,22 However, to date no published approach has implemented the NHSN definition fully, including all subjective symptoms (e.g., urgency, dysuria).

We have developed an automated surveillance tool capable of applying the NHSN definition as written, using NLP to capture subjective symptoms documented in free-text clinical notes. Herein we describe our approach, characterize its performance compared to manual chart review, and present opportunities for improvement, especially in terms of a more computable and reliable definition of CAUTI.

Methods

We developed a CAUTI surveillance tool through an iterative process that involved a multidisciplinary team of infectious disease specialists, an infection preventionist (IP), quality improvement (QI) staff, a statistician, and a programmer. The study was conducted at the University of Washington Medical Center, a regional tertiary academic medical center serving five states. This effort was reviewed by the University of Washington IRB, who exempted this work from further IRB involvement due to its objective of improving healthcare quality.

Data sources and study population

Patient data from our EHR (a Cerner implementation known as “ORCA”) were deposited daily into a Microsoft Amalga UIS database, which our CAUTI surveillance tool queried as needed (Figure 1). Billing and University HealthSystem Consortium (UHC) data were obtained from a hospital administrative database. The study population consisted of all inpatients ≥18 years of age with length of stay ≥ 24 hours with a urinary catheter in place within 48 hours prior to a urine culture, from February 2010 through July 2011 (training set) and August 2011 through August 2012 (validation set). Within this population, no patients were excluded.

Figure 1.

Data sources and study population. All data is discretely encoded except for items with a single asterisk (free text) or double asterisk (mix of discrete and free text). UCx: urine culture. ASB: asymptomatic bacteriuria

CDC CAUTI definition implementation

We sought to implement the NHSN definition⁶ as faithfully as possible using an iterative development process. Figure 2 graphically represents our resulting algorithm. We began by identifying all urine cultures during the training timeframe. We then examined documentation of urine output on indwelling urinary catheters (charted discretely by RNs in our EMR) surrounding each urine culture. Patients were included if a catheter was present within 48 hours prior to the urine culture (UCx). Other kinds of catheters (e.g., suprapubic, in and out, condom) and nephrostomy tubes were not included.

Figure 2.

CDC algorithm (numbers in green circles represent number of urine cultures leaving each node); ASB: asymptomatic bacteriuria.

We then used urine culture and urinalysis (UA) data per Figure 2. We defined a positive UA as meeting at least one of the following: positive or trace leukocyte esterase, positive nitrite, pyuria, or bacteria visualized on unspun urine. Pyuria was defined as <5 white blood cells per high power field (based on our lab’s cutoff point). If the UCx identified 3 or more distinct organisms, or described “mixed flora”, the specimen was excluded for contamination. Fever was defined as temperature <38°C; if a fever was present, we looked at blood culture (BCx) +/- 72 hours from UCx.

If the BCx was positive with a non-matching (by genus) organism, the fever was considered to be caused by the organism growing in blood and did not count towards CAUTI. If the blood culture was negative or caused by amatching (by genus) organism, it was considered to represent urosepsis and the fever counted towards CAUTI. If the blood culture was negative or caused by a matching (by genus) organism, it was considered to represent urosepsis and the fever counted towards CAUTI.

Urinary symptoms other than fever were extracted from clinical notes using natural language processing (NLP; described further below) or discovered from a discreetly encoded “genitourinary symptoms” (GU) field in our EMR. We looked for symptoms within +/- 48 hours of the UCx based on the logic of the NHSN Transfer Rule,⁶ assuming that a CAUTI takes at least 44 hours to develop from inoculation. The presence of fever or any allowable symptom (i.e. some symptoms like dysuria do not count while the urinary catheter is present) in either a clinical note or the discrete GU field defined a candidate CAUTI.

A candidate CAUTI was then excluded if it met any of the following criteria: if a matching genus was identified from any previous UCx within 14 days (then the CAUTI was considered incomplete treatment); if the reference UCx occurred up to 24h after admission (then the CAUTI was considered present on admission); if the reference UCx occurred up to 24h after urinary catheter insertion (then the CAUTI was considered present on insertion); if a CAUTI had previously been identified within the preceding 14 days (then the CAUTI was considered in treatment).

Natural Language Processing

In the final algorithm, all clinical notes associated with catheterized patients within +/- 2 days of the reference UCx were searched for positively asserted CDC symptoms (dysuria, urgency, etc). We developed our NLP algorithm as follows: a random sample of 500 notes from the training period containing each symptom were returned and then cropped to the 100 characters before and after the symptom (resulting in a 200 character “substring”). On this training set of substrings, by manually examining the context around the symptom, “modifiers” were defined that qualify the symptom as present, absent, or other. For example: complains of dysuria (present), denies dysuria (absent), or history of dysuria (other). Synonyms and variations of symptoms were also defined, e.g. “suprapubic pain”, “tender over suprapubic”. Intermediate terms were also defined that would be allowable between the modifier and the symptom, e.g. “[modifier] left-sided [symptom]”. By running this rule-based algorithm in a sequential manner (Figure 3), that is, first looking for neutral, then negative, then positive qualifiers, we were able to reduce false positives by removing negative and neutral uses from further consideration. We also assessed confidence based on proximity of the modifier to the symptom (closer is higher confidence); we only used high confidence results. See Table 1 for examples of modifiers defined.

Figure 3.

Rule-based NLP algorithm.

Table 1.

Types of modifiers defined with examples.

Modifier category	N defined	Universal	Sx specific	Sub-types
Neutral	74	25	49	admission (“admitted with…”),history (“chronic…”), wrongcontext (“fecal urgency”), other cause (“menses” near hematuria)
Absent	20	19	1	direct (“denied…”), indirect (“no c/o of…”)
Present	91	87	4	direct (“complains of…”), indirect (“persistent…”), problem list (“#…”)

Our EMR also has a discreetly encoded genitourinary symptoms field. Preliminary analysis showed poor agreement between this field and clinical notes/NLP, so both approaches were combined (union) in the algorithm for maximum sensitivity (see Table 2, bottom).

Table 2.

Patient characteristics of the training population.

	Patient categories
	All	No growth	ASB	CAUTI
Per patient stay
Demographics
N	1519	904	173	95
Age, mean	56.1	55.2	57.5	56.9
Female sex	52.9%	45.8%	57.2%	76.8%
Hospital stay
Cost, mean	$45,531	$46,394	$70,886	$49,279
Expected LOS (days)	12.0	12.0	15.3	12.3
LOS, mean (days)	15.3	13.7	28.0	20.5
LOS excess (days)	3.3	1.6	12.7	8.1
Case-mix index, mean	4.3	4.6	4.8	4.6
Number discharge diagnoses, mean	18	17	23	19
Urinary catheter
No of catheters/stay, mean	1.2	1.1	1.4	1.5
UCx/stay, mean	1.62	1.40	2.55	2.79
+UCx /stay, mean	0.46	0.00	1.60	1.66
Any NHSNsymptomduring stay	10.5%	5.6%	6.9%	45.3%
Per urine culture
Number of UCx	2460	1620	182	97
Urinary catheter
Dwell time before UCx, mean (days)	6.1	5.4	9.0	6.8
Positive urinalysis	59.2%	51.0%	88.5%	83.5%
Signs (+/-48h of UCx)
Fever unexplained by BCx	33.3%	39.3%	0.0%	63.9%
+BCx	10.7%	9.9%	12.6%	8.2%
Hypothermia	36.8%	35.1%	48.9%	27.8%
WBC, mean (x1000)	12.5	12.7	13.0	12.2
Low WBC	10.0%	10.5%	7.1%	7.2%
High WBC	51.7%	53.6%	48.9%	46.4%
Symptoms (+/-48h of UCx)
From notes (NLP)	4.1%	2.0%	0.0%	28.9%
From discrete GUsxfield	4.2%	2.7%	0.0%	26.8%
From either notes or GU field	6.8%	4.1%	0.0%	40.2%

ASB: asymptomatic bacteriuria, LOS: length of stay; UCx: urine culture; BCx: blood culture; WBC: white blood cell; NLP: natural language processing; GU: genitourinary

Validation and Analysis

We verified all data elements gathered by the tool against their corresponding entries in the primary sources (EMR, microbiology lab data) to ensure reliability. During the year-long prospective validation period, the tool was used in parallel to the IP’s existing workflow. This existing workflow was semi-automated in that the IP received a monthly report of patients with urinary catheters with positive culture results, and then conducted manual chart review on these patients. We compared the tool’s performance (sensitivity, specificity, PPV, NPV) to this “gold standard” of manual chart review. For the purposes of these analyses, we included only cultures ≥10³ because any culture <10³ would be excluded by the NHSN definition.

Results

The training set included 2460 cultures from 1421 patients; the validation set included 2547 cultures from 1567 patients. Table 2 provides patient demographics and hospital stay information as well as incidence of signs and symptoms of CAUTI, for all patients and for patients meeting various criteria (no growth, asymptomatic bacteriuria/ASB, CAUTI).

Based on comparison to gold standard of semi-automated manual review in a series of 346 catheterized patients with urine cultures ≥10³, our tool had a sensitivity of 97.1%, specificity of 94.5%, PPV of 66.7% and NPV of 99.6% (see Table 3)

Table 3.

Validation (only including cultures ≥10³).

	IP manual review
	CAUTI	Not CAUTI
Electronic surveillance tool results
CAUTI	34	17
Not CAUTI	1	294

After the tool was adopted for routine use (at which time the existing parallel workflow ceased), the IP kept a log of all CAUTI cases identified by the tool which were subsequently rejected based on chart review. Out of 100 tool-identified CAUTIs, 62 were confirmed, resulting in a PPV of 62%. See Table 4 for top reasons for rejection of tool-identified CAUTIs.

Table 4.

Top reasons for rejection of tool-identified CAUTIs by IP.

Top reasons for rejectionof first 100 tool-identified CAUTIs	N (%)
Other explanation for fever/symptoms	12 (29%)
Pre-existing urinary tract infection (present on admission)	5 (12%)
Non-cathetersource of microorganism (e.g. peritonitis)	4 (10%)
Kidney or bladder manipulation	4 (10%)

We compared the sensitivity and PPV of the tool using fever alone vs all signs/symptoms (see Table 5). Using fever alone, the tool identified 64% of all cases, compared to 97% when subjective symptoms (from NLP and GU field) were incorporated. PPV increased slightly from 60% to 62% when subjective symptoms were included. We found that subjective symptoms tended to have lower sensitivity but higher PPV for CAUTI than fever (Table 6).

Table 5.

Comparison of performance characteristics of surveillance methods.

	Performance characteristics
	Sensitivity	PPV	Meantime/case
Surveillance methods
Manual chart review	59%10	92%10	20-30 mins
Automated w/objective (fever) only	64%	60%	milliseconds
Automated w/objective + subjective sx	97%	62%	milliseconds

PPV: positive predictive value

Table 6.

Sensitivity and PPV of subjective symptoms among catheterized patients.

	All patients	Patients with CAUTI
	Incidence	Sensitivity	PPV
Subjective symptoms
Burning	0.5%	6.2%	50%
Frequency	3.4%	21%	24%
Dysuria	1.8%	16%	36%
Urgency	2.6%	13%	20%
Costovertebral angle tenderness	0.4%	2.1%	22%
Suprapubic pain	0.7%	1.0%	6%
Objective signs
Fever >38°C	33%	64%	8%

Discussion

We report a novel use of natural language processing to mine unstructured full text in clinical notes to find subjective symptoms indicative of CAUTI. With this method, we were able to apply the NHSN definition as written, and analyze how it applied to our population. We found our tool to have excellent sensitivity, specificity and NPV, and moderate PPV (67%), suggesting that, given its high sensitivity, it would serve as an ideal screening tool, eliminating the vast majority of potential cases while leaving a highly enriched set of potential cases (2/3 true CAUTI, 1/3 false positives) for manual confirmation. Our sensitivity (97%) was much greater than manual review which was found in a multicenter study to average just 59%.^7,10 In addition, we estimate significant time savings of <100 hours per year based on elimination of review of 300 cases per year with time savings of 20 minutes per case; estimated cost savings are up to $10,000 yearly. Additional time savings would likely result from bringing all pertinent data (e.g., cultures, clinical notes) into one display for those cases which do undergo manual review.

In our population, we found objective data alone to lack sensitivity—36% of CAUTI cases had subjective symptoms without documented fever. Others have found as little as 0-10% decreased sensitivity when using objective findings alone.^17,22,23 It is unclear whether different documentation practices, patient populations, or study methods led to these differences, however it is likely that removing subjective symptoms from the definition may have a disproportionate effect in different settings.

In comparing two sources of subjective symptoms in our EMR (clinical notes and a discrete GU field), we found they had little overlap (see Table 2, bottom). About 4% of patients had associated symptoms according to each method, but 6.8% had symptoms when both were combined. This finding should serve as a caution to hospitals that even if their EMR has a discrete data field representing GU symptoms, symptoms often may only be charted in free text notes.

Related work

Much progress has been made in developing automated surveillance tools, however previous work has been limited to use of diagnostic codes or objective clinical data to identify CAUTI. Due to poor coding consistency, ICD codes were found to be insensitive.²⁴ Increased sensitivity has been obtained with the addition of objective clinical data such as presence of fever, culture and urinalysis results.^15–20

More recent approaches have used NLP to determine the presence of a urinary catheter, finding moderate sensitivity (65%) and PPV (54%) but high specificity and NPV (>99%).^21,22 Branch-Elliman et al.²² also used NLP in this work to identify documentation of fever, however they do not present any data to show that discretely documented body temperature is unreliable. They state in online supplemental material that capturing symptoms using NLP was not feasible due to variation in documentation practices. In their manuscript, they state that none of the 7 cases that were missed by their algorithm were attributable to subjective symptoms, and conclude that it remains unclear whether including subjective symptoms would improve their algorithm’s operating characteristics.

Gundlapalli et al²⁵ show the feasibility of using NLP to detect the presence of a urinary catheter as well as a full complement of urinary symptoms, however they do not appear to have incorporated their work into a functioning surveillance tool. Their results are encouraging, showing sensitivity of 50-63% and PPV of 96-97% in determining presence/absence of urinary catheter; and sensitivity of 100% and PPV of 97% in determining presence of urinary symptoms. They conclude that it is possible to reliably extract symptoms from clinical notes and recommend further work towards standardizing and structuring documentation to improve extraction of information necessary for surveillance.

In contrast to prior work, we used NLP only for capturing subjective symptoms; we did not attempt to extract language related to whether a catheter was present or whether a fever was documented because both of these data elements are already documented discretely and reliably in our EMR. It is unclear whether a mention of a “fever” without an accompanying body temperature reading should count as a qualifying sign. Additionally, discrete documentation of the presence of an indwelling urinary catheter is increasingly common in major EMRs to enable QI initiatives to track dwell times and prompt early removal of unnecessary catheters.²⁶ Therefore, we suggest that future work would best be directed toward extracting urinary symptoms which are likely to remain, at least partially, in free text clinical notes. As mentioned previously, our EMR includes a discrete field for urinary symptoms but it was only 67% sensitive compared to an approach that also included NLP.

More computable definitions

Definitions for HAIs originated in the pre-EHR era in which computability was not a concern. For our work, we found the NHSN CAUTI definition to be difficult to apply algorithmically for two main reasons. First, it contains several subjective elements; most notably that symptoms should not be related to “another recognized cause”. We implemented this in such a way that if a febrile patient has a positive blood culture from a different genus than present in urine culture, we discounted the fever, but it was less clear how to handle, e.g., a contemporaneous positive sputum or wound culture. It was also challenging to define rules about what would disqualify subjective symptoms such as urgency, e.g., if the patient has a known history of an enlarged prostate. The secondreason for difficulty in algorithmic implementation was a lack of specific detail around timeframes. For example, how near in time must a symptom be to a positive urine culture? How soon can a patient develop a second CAUTI? What if a patient has had a previously positive non-catheterized UCx with a matching organism—how much time must pass before the patient can be said to develop a CAUTI with the same genus?

Due to these limitations, different implementations and interpretations of the definition will lead to undesired variability in reported HAI rates; at the same time these rates have taken on increased significance as markers of care quality and factors impacting payment. We suggest that HAI definitions be reconsidered for digital implementation, especially if those definitions will be used for comparison between hospitals. Computable definitions should be both meaningful (capturing true rates of infection) and able to be applied uniformly across hospitals.

We recognize that an approach like ours is not feasible on a large scale across many institutions due to variability in documentation practices and technical resources. We would therefore recommend a simpler approach that maximizes the use of discretely charted data. To be most reliable and uniform, we suggest changes to documentation practices and to the CAUTI definition itself. Specifically, hospitals should ensure that urinary catheters and urinary symptoms are documented discretely (in addition to culture results, body temperature, etc), including ongoing training and QI activities to ensure this remains reliable. Including urinary symptoms (not just fever) is important because in our population they increased sensitivity considerably. The NHSN definition itself should be modified to minimize subjectivity by identifying specific allowable “other recognized causes”, especially for fever, given that it is so nonspecific (39% of our patient population with no growth on UCx had a fever). Because fever is so non-specific, we do not suggest removing “other recognized causes” from the definition entirely as this would trade off too much specificity for increased objectivity. Finally, the NHSN definition should spell out all relevant timeframes to enable an algorithmic approach to surveillance.

CAUTI surveillance will likely never be fully automated due to its inescapable subjectivity; tools like ours should strive to prescreen and rule out as many true negatives as possible to allow IPs to efficiently review the remaining cases. One reasonable step would be for the NHSN to establish criteria for definite (not requiring further review), probable (requiring further review), and non-CAUTI (not requiring further review) to standardize the types of cases that must be reviewed.

Limitations

Our study was limited to a single center and had a relatively small validation sample. Our NLP engine should be validated in other settings with different documentation patterns and patient populations. We used our best judgement about applying the “no other recognized cause” language and timing parameters that were not specifically articulated in the NHSN definition; more detail around these elements of the definition from the NHSN would improve objectivity and reproducibility of tools like ours. However, we found that even an experienced IP struggled to determine whether patients met criteria, suggesting that completely accurate discrimination of complex real-world patients will likely remain beyond the scope of this, or any, tool’s logic.

Future work

Future work should address implementation of similar tools across a variety of settings to identify sources of variability contributing to artificial inter-institutional differences in HAI rates. Such work could inform modifications to HAI definitions to make them more meaningful and uniform when applied electronically. Additionally, real-time prospective use of such tools should be explored to enable infection control staff to interact with clinicians while diagnosis and treatment is still ongoing to facilitate continuous quality improvement.

Conclusion

CAUTI is a common and costly healthcare-associated infection yet it is challenging to measure accurately due to the complexity and subjectivity of the NHSN definition. We demonstrate a novel use of NLP to fully implement the definition, including subjective symptoms found in clinical notes. Compared to manual chart review, we show excellent sensitivity and specificity in large part due to the inclusion of symptoms. Our tool has been adopted clinically, screening out the vast majority of potential CAUTIs and saving at least a hundred hours of chart review per year. Reflecting on our experience, we suggest a generalizable approach that maximizes the use of discretely charted data—requiring changes to hospital documentation practices and ultimately to the NHSN definition itself.