Automated algorithm in catheter-associated urinary tract infection identification using digital health records: validity and barriers in implementation into clinical practice

Abstract

Objective

This study aims to evaluate the accuracy of an automated algorithm designed to identify catheter-associated urinary tract infections (CAUTIs) using electronic health records from a hospital. We assess the algorithm’s effectiveness as a clinical decision support tool by analyzing its ability to accurately identify CAUTIs based on predefined clinical parameters.

Materials and methods

We conducted a retrospective analysis of all patients hospitalized in the acute internal medicine wards of the Cantonal Hospital of Baden between November 2022 and October 2024. Automated algorithm identifies potential CAUTI cases based on standard centers for disease control (CDC) criteria. The automated algorithm identified patients meeting these criteria as potential cases and the number of CAUTIs is visualized in a dash board. All records were manually reviewed and inconsistencies validated by an infection specialist. Sensitivity, specificity, area under the receiver operating characteristic (ROC) curve (AUC), likelihood ratios (LR) and 95% confidence intervals were calculated to assess performance.

Results

Between November 2022 and October 2024, a cohort of 1424 patients with indwelling catheters was assessed, resulting in the identification of 11 manually diagnosed CAUTIs. The automated algorithm identified 6 of these cases, yielding 6 true positives (TP), 1,385 true negatives (TN), 28 false positives (FP), and 5 false negatives (FN). The algorithm demonstrated a sensitivity of 55% (95% CI: 27.3% − 81.8%) and a specificity of 98% (95% CI: 97.3% – 98.7%), positive likelihood ratio (LR+) of 27.3 and a negative likelihood ratio (LR−) of 0.5. AUC of 0.76 (95% CI: 0.73–0.80) with P-value of 0.0001.

Conclusion

In this validation study, the automated algorithm demonstrated high specificity (98%) but limited sensitivity (55%) for the detection of CAUTIs. While a positive algorithm result was strongly associated with true infection, nearly half of confirmed CAUTI cases were not identified. Therefore, the algorithm may support surveillance by prioritizing cases for review; however, due to its limited sensitivity, it cannot reliably exclude non-cases and should not replace manual case adjudication. The primary reasons for missed cases were gaps in electronic health record (EHR) documentation. In hospitals with comprehensive EHR use across all departments, improved sensitivity is expected.

Introduction

Business intelligence (BI) tools are increasingly integrated into healthcare environments to assist in clinical decision-making, quality improvement measures, and operational efficiency [1]. Diverse data analytics platforms, have found application in generating automated reports for various healthcare outcomes [2], including infection monitoring. In the current study, automated algorithm has been used to analyze clinical data and generate reports to aid in identifying catheter-associated urinary tract infections (CAUTIs), which are a common and serious hospital-acquired infection.

However, these automated reports must be validated against a gold standard before they can be fully trusted in clinical settings [3, 4]. While automated algorithm is not designed to detect infections independently, it visualizes and reports on defined clinical parameters that might suggest infection, thereby supporting healthcare professionals in identifying potential CAUTI cases.

A study by van der Werff et al. similarly explored the use of automated surveillance for healthcare-associated urinary tract infections [5], though their approach incorporated natural language processing (NLP) to analyze free-text clinical notes. Their model achieved a sensitivity of 66.7% and specificity of 99.7%, demonstrating that supplementing structured EHR data with unstructured text improved detection performance. The accuracy of such tools in generating clinically useful reports is paramount [6], particularly in high-risk areas like infection detection. This study aims to validate the accuracy of automated CAUTI reports by comparing them to manual diagnoses performed by an infection specialist.

Methods

Study design and setting

This retrospective study was conducted in patients hospitalized in the acute internal medicine wards of the Cantonal Hospital of Baden (KSB). KSB is one of the largest Cantonal Hospitals in Switzerland which provides a 24-hour service for a population of 350’000 living in the eastern part of the Canton Aargau and the western part of the Canton Zürich. Highly specialized services cover most areas except for brain and heart surgery. The study utilized routinely collected electronic health records (EHRs) stored in the hospital’s database between November 2022 and October 2024. Data included demographics, hospital administrative data, microbiological results, laboratory findings, vital signs, medications, and physician notes.

The study population consisted of all patients hospitalized in the acute internal medicine wards during the study period who had indwelling urinary catheters. In acute internal medicine settings, common indications typically include urinary retention, accurate urine output monitoring in hemodynamically unstable patients, perioperative management, and neurological impairment affecting bladder function. To establish the reference standard, all catheterized patients were reviewed by an infectious disease specialist (BW). Case adjudication was performed independently of the automated algorithm output. The infectious disease specialist conducting the review was blinded to the algorithm classification. The patient inclusion and exclusion process is illustrated in Fig. 1.

Fig. 1.

Flow chart of study

The STARD guidelines were used for reporting of the study [7]. Description of checklists according the STARD guideline can be found in Table 1.

Table 1.

Description of checklists according the STARD guideline

STARD Item	Description in Study
1. Identification as a diagnostic accuracy study,	Automated Algorithm in catheter-associated urinary tract infection Identification using digital health records: Validity and barriers in the implantation into clinical practice
2. Structured summary,	The abstract provides a summary of study design, methodology, diagnostic measures, and conclusions based on the performance of the Automated algorithm
3. Scientific background,	The study discusses the clinical relevance of CAUTI detection and Automated algorithm potential as a supplementary diagnostic tool, enhancing infection control.
4. Study objectives and hypotheses,	This study aims to evaluate the diagnostic accuracy of Automated algorithm for CAUTI detection and compare it with specialist-reviewed diagnoses.
5. Study design (prospective or retrospective),	This is a retrospective study using EHR data, analyzing historical CAUTI cases flagged by Automated algorithm.
6. Eligibility criteria,	Patients with > 24-hour catheterization and symptoms matching CAUTI criteria were eligible, ensuring a clinically relevant sample.
7. Basis for identifying eligible participants	CAUTI symptoms and catheterization details from hospital EHR data flagged by Automated algorithm formed the basis of participant inclusion.
8. Setting, location, and dates	This study was conducted at Cantonal Hospital Baden, including cases from August 2022 to September 2024.
9. Sampling series	There is a consecutive sampling of all cases flagged by Automated algorithm within the study timeframe, increasing study representativeness.
10a. Description of the index test (Automated algorithm)	Automated algorithm identifies potential CAUTI cases based on predefined symptoms and catheterization data; used for automated CAUTI screening.
10b. Description of the reference standard	Manual review and validation by infection specialists, based on clinical CAUTI definitions, serves as the diagnostic reference test
11. Rationale for reference standard choice	, Infection specialists' manual diagnoses offer high reliability, making it an appropriate gold standard for CAUTI detection validation.
12a. Test positivity criteria for index test,	Automated algorithm flags cases based on catheterization duration (> 48 hours) and symptoms indicative of CAUTI, following predefined criteria.
12b. Test positivity criteria for reference standard	Infection specialist diagnos based on clinical signs, symptoms, and lab results consistent with CAUTI guidelines, ensuring diagnostic accuracy.
13a. Blinding of index test readers to reference standard	Infection specialist conducting manual diagnosis were blinded to Automated algorithm's outputs to avoid bias.
13b. Blinding of reference standard readers to index test	Manual review data were not accessible to the Automated algorithm, ensuring an independent evaluation.
14. Statistical methods	Used sensitivity, specificity, and ROC analysis to assess diagnostic accuracy, with 95% confidence intervals.
15. Handling indeterminate results	Cases with missing or incomplete EHR data were flagged and discussed as a limitation impacting sensitivity.
16. Handling missing data	Missing emergency department records are noted as a limitation affecting the Automated algorithm accuracy, and the limitation section discusses this further.
17. Analysis of accuracy variability	Bootstrapping and confidence interval was calculated
18. Sample size determination	Consecutive case inclusion allowed for a representative sample within the timeframe, although a formal sample size calculation was not detailed.
19. Flow diagram of participants	A flowchart is suggested to clarify patient eligibility, exclusions, and the final sample, visualizing study inclusion processes.
20. Baseline demographics	Demographic data on age, catheter duration, and underlying conditions are included, detailing the clinical background of participants.
21a. Disease severity distribution,	None
21b. Distribution of alternative diagnoses	None
22. Interval between index and reference tests	As a retrospective study, the time interval was not directly relevant, but the blinding of reviewers prevented bias in diagnosis timing.
23. Cross-tabulation of test results	Cross-tabulation shows the diagnostic performance of Automated algorithm, highlighting true positives, false positives, true negatives, and false negatives.
24. Accuracy estimates and confidence intervals	Sensitivity, specificity, and AUC are reported with confidence intervals, demonstrating diagnostic reliability.
25. Adverse events	No adverse events occurred.
26. Study limitations,	Limitations such as missing emergency records and low sensitivity are discussed, impacting Automated algorithm’s diagnostic reliability.
27. Clinical implications	Automated algorithm’s low sensitivity suggests it is best used as a supplementary tool with manual oversight, highlighting its role in clinical practice.
28. Registration	NA
29. Protocol availability	Detailed methodology and procedures are included within the manuscript, though no separate protocol is cited.
30. Funding sources	Funding information is included, specifying that no conflicts of interest affected study design or reporting.

CAUTI definition

The CAUTI definition was based on CDC criteria [8]. CAUTI was defined as the presence of a urinary catheter for over 48 h, with the day of device placement counted as day 1, and the occurrence of infection either while the catheter was in place or on the day of removal or the following day. In addition, a positive urine culture (≥ 10⁵ CFU/mL with no more than two microorganisms) and at least one compatible clinical sign or symptom were required. Clinical criteria included fever (> 38 °C), suprapubic tenderness, costovertebral angle pain or tenderness, or if the catheter had been removed: urinary urgency, frequency, or dysuria.

Automated algorithm

The algorithm identifies CAUTI cases based on the following criteria (Fig. 2):

Fig. 2.

CAUTI surveillance dashboard

1. Duration of Catheter Use: The urinary catheter must be in place for at least two days: Catheter Days = Removal Date − Insertion Date, where Catheter Days ≥ 2.

2. Timing of Urine Culture Orders: Order Date−Insertion Date ≥ 2 and Order Date−Removal Date ≤ 1.

3. Fever Criteria: Confirmed CAUTI cases require a documented body temperature Temp > 38^∘C.

Outputs from this logic were visualized through interactive dashboards that integrate real time data.

Reference test

A trained physician and research fellow reviewed all medical health records in patients hospitalized at the acute internal medicine wards that were catheterized or were potentially meeting CAUTI definitions. Based on the medical records, all CAUTIs were confirmed or in case of unconfirmed CAUTI definition, all cases were reviewed by an infection specialist (BW).

Validation metrics and statistical analysis

We calculated the true positive (TP), the true negative (TN), the false positive (FP), and the false negative (FN) rates. We calculated the sensitivity and specificity. The diagnostic performance of the automated algorithm for detecting catheter-associated urinary tract infections (CAUTI) was assessed using the receiver operating characteristic (ROC) curve, from which the area under the curve (AUC) was calculated. To determine the statistical significance of the AUC, a p-value was computed using the DeLong test, which compares the observed AUC against a null hypothesis of AUC = 0.5 (no discrimination). A p-value < 0.05 was considered statistically significant. The optimal threshold for classification was determined using the Youden Index. Sensitivity and specificity were computed along with their 95% confidence intervals (CI). To further characterize the diagnostic strength of the algorithm, we calculated positive likelihood ratio (LR+) and negative likelihood ratio (LR−). Analyses were conducted using the R Statistical language (version 4.4.1; R Core # Team, 2024) with the pROC [9] package for ROC curve analysis.

Results

Description of study population

The study population included 1,424 patients (Table 2) with indwelling urinary catheters who were hospitalized in the acute internal medicine wards of the Cantonal Hospital of Baden between August 2022 and September 2024 accounting to a total of 8193 catheter days. Among these patients, 11 CAUTIs were confirmed through manual validation. The automated algorithm correctly identified 6 of these cases as true positives, while 28 were classified as false positives. Among non-infected patients, the model correctly identified 1,385 true negatives, whereas 5 cases were misclassified as false negatives.

Table 2.

Description of the study population

Study Period	November 2022-October 2024
Total number of patients screened	10,019
Total number of patients with urinary catheter (N)	1424
Age (Mean ± SD)	78.9 (1.9)
Gender Distribution	Female(46.4%), Male (53.7)
Total catheter days	8193
No. of Patients with > 38 °C (N)	310
Hospitalization Duration (days) (Mean ± SD)	7.8 (0.6)
Manually diagnosed CAUTI Cases	11
CAUTI rate per 100 patient days	0.11
CAUTI rate per 100 patient days§	0.70
CAUTI rate per 1000 catheter days	1.3
Algorithm`s true positive	6
Algorithm`s true negative	1385
Algorithm`s false positive	28
Algorithm`s false negative	5

§ -> Number of patients with urinary catheter

Sensitivity and specificity

The automated reports for CAUTI demonstrated a sensitivity of 55% (95% (CI) of 27.27% to 81.82%), indicating the proportion of true infection cases correctly identified by the algorithm. The specificity was 98% (95% CI: 97.25% – 98.73%), reflecting the proportion of non-infected patients correctly classified.

ROC curve and AUC

The area under the ROC curve (AUC) was calculated as 0.7630 (Fig. 3), with a 95% confidence interval ranging from (0.7254–0.8012), indicating a good discriminatory ability, demonstrating that the tool performs significantly better than random chance (AUC = 0.5) in distinguishing between true infection and non-infection cases.

Fig. 3.

Summary of receiver operating characteristic (ROC) curve between manual detection and automated algorithm reports of CAUTI

Likelihood ratio

To better interpret the clinical significance of the likelihood ratio (LR) values observed in this study, we refer to established evidence-based diagnostic guidelines [10]. Our analysis showed that the automated algorithm had a positive likelihood ratio (LR+) of 27.25 and a negative likelihood ratio (LR−) of 0.464. While the high LR+ suggests that a positive result strongly correlates with infection, the tool’s limited sensitivity highlights its potential for false negatives.

Discussion

This study assessed the effectiveness of an automated algorithm for detecting CAUTIs using data from EHRs. Our findings highlight both the potential and limitations of automated surveillance tools, particularly in the context of diseases with inherently low incidence rates. The algorithm demonstrated a high specificity of 98%, effectively identifying non-CAUTI cases. However, its moderate sensitivity of 55% indicates that a significant number of true CAUTI cases were not detected. Several factors contributed to the missed CAUTI cases in our study (Table 3). Gaps in EHR documentation, including missing catheterization records likely impeded accurate detection. In our institution, catheter insertion is primarily documented by nursing staff at the time of placement, whereas catheter removal is recorded by nursing or medical staff depending on the clinical setting. During emergency admissions or high-acuity situations, documentation may be delayed, leading to incomplete structured data capture. Improving documentation consistency may require embedding mandatory structured fields for catheter insertion and removal, implementing automated daily prompts for catheter necessity reassessment, and integrating catheter status into standardized nursing workflow templates within the EHR. Such system-level integration may not only improve algorithm sensitivity but also support catheter stewardship and infection prevention efforts. Limited documentation coverage across different hospital departments, particularly for patients with prolonged stays in the emergency department (ED), may have resulted in an underestimation of catheter days, which could have affected CAUTI detection rates. The low prevalence of CAUTI in our dataset (11 out of 1,424 patients, approximately 0.8%) also contributed to class imbalance, potentially biasing the algorithm toward predicting non-CAUTI cases. In contrast, automated algorithms may demonstrate more stable and generalizable performance in higher-prevalence infectious conditions, where larger case numbers reduce class imbalance and statistical uncertainty. Examples include community-acquired pneumonia, sepsis syndromes, influenza, COVID-19, or non–catheter-associated urinary tract infections. In such settings, sensitivity estimates are less vulnerable to fluctuation from small changes in case classification. An additional limitation relates to the reference. Furthermore, catheter indications were not analyzed, which limits interpretation of whether specific clinical subgroups may have been at higher risk for CAUTI or whether variation in underlying indications may have influenced algorithm performance. Although manual chart review was used as the gold standard, this approach is not immune to misclassification. Prior studies have shown that manual surveillance may both under-detect infections due to incomplete documentation and overestimate CAUTI when distinguishing true infection from asymptomatic bacteriuria or nonspecific clinical symptoms is challenging [14, 15]. Variability in documentation quality and subjective interpretation of symptoms may therefore have influenced case adjudication. Another important observation is that statistical uncertainty influenced the sensitivity estimate [16, 17]. Given the small number of confirmed CAUTI cases in the dataset, the sensitivity estimate had a wide confidence interval, meaning that small changes in case classification could have led to substantial shifts in performance metrics. This variability makes it difficult to generalize the model’s performance across different hospital settings. The small number of confirmed CAUTI cases in our dataset al.so highlights a fundamental challenge in developing automated infection detection algorithms for diseases with inherently low prevalence. Algorithms trained on small datasets may struggle to distinguish between true infections and non-infectious urinary catheter colonization, leading to a higher risk of false negatives. Expanding the training dataset to include multi-institutional data or implementing synthetic data generation techniques could improve the model’s robustness and performance across different clinical settings [18]. Despite these challenges, the algorithm’s high specificity (98%) suggests that it was optimized to minimize false positives, which inadvertently increased the number of false negatives. This trade-off underscores the importance of carefully selecting detection thresholds to balance sensitivity and specificity according to clinical priorities [19]. While business intelligence tools and automated systems offer potential advantages in improving workflow efficiency and infection monitoring, their success is heavily dependent on the completeness and accuracy of digital health records. Additionally, the reliance on structured data alone may miss important clinical information recorded in free-text notes, further reducing detection sensitivity. The integration of NLP-based approaches, as demonstrated in prior studies [5, 14, 20], could potentially improve CAUTI detection. Beyond CAUTI, automated surveillance approaches may be applicable to other healthcare-associated infections (HAIs) where structured microbiological, procedural, and device-related data are available. These include central line–associated bloodstream infections (CLABSI), ventilator-associated pneumonia (VAP), surgical site infections (SSI), and Clostridioides difficile infections. In such contexts, algorithm-based detection may support infection control teams by enabling real-time monitoring, standardized reporting, and benchmarking across institutions. In conclusion, while automated surveillance systems hold promise as a supplementary infection monitoring tool, our findings indicate that manual review remains essential to ensuring accurate diagnosis. Future improvements should focus on refining detection thresholds, integrating unstructured clinical data through NLP, and improving the timing and completeness of EHR documentation. Additionally, training new algorithm on larger, datasets is crucial to improving its reliability, particularly in diseases where small sample sizes can lead to unstable performance estimates. Beyond retrospective detection, future developments may shift toward prospective risk stratification. Automated identification of patients at increased risk for CAUTI based on catheter duration, comorbidities, and other clinical risk factors could enable earlier preventive interventions, such as timely catheter removal or intensified monitoring [21]. Such predictive approaches may ultimately provide greater clinical value than post hoc case identification alone. Until these refinements are achieved, automated CAUTI detection should serve as a decision-support tool rather than a replacement for clinical judgment, with healthcare professionals playing a central role in infection diagnosis and treatment decisions.

Table 3.

Suggested reasons for missed CAUTIs

Gaps in timely Electronic Health Records (EHR) Coverage	Some CAUTI cases were missed due to gaps in EHR documentation, such as missing catheterization records. These gaps may have occurred in emergency situations where the timing of indwelling catheter placement was not entered promptly. As a result, the algorithm’s ability to accurately detect infections was limited
Class Imbalance and Algorithm Bias	The dataset contained far more non-CAUTI cases than CAUTI cases, leading the algorithm to favor predicting ‘no CAUTI.’ Machine learning models trained on imbalanced datasets tend to optimize for specificity [11, 12], reducing false positives but missing true cases.
Statistical Uncertainty	When there are very few CAUTI cases in a dataset, the sensitivity estimate becomes less reliable and more variable. Sensitivity is typically reported with a confidence interval, which provides a range within which the true sensitivity value is likely to fall. However, with a low number of cases, this confidence interval becomes wider, meaning that the estimate is more uncertain. For example, if a study had 100 CAUTI cases, the sensitivity might be estimated with a narrow confidence interval of 65%–75%, making it a precise measure. In contrast, with only 11 CAUTI cases, the sensitivity could be estimated at 54.5% but with a much wider confidence interval, such as 27.27% − 81.82% This wider range indicates greater uncertainty in the sensitivity measurement. A consequence of this variability is that the algorithm’s sensitivity may not generalize well to different datasets [13]. If tested in another hospital or a larger population, the reported sensitivity could fluctuate significantly. This instability makes it difficult to determine whether the algorithm will perform consistently in real-world clinical settings. Additionally, even a small change in the number of detected CAUTI cases can cause a large shift in sensitivity. If only two additional cases were detected correctly, the sensitivity would increase from 54.5% to 72.7%, which is a substantial increase. This shows that the sensitivity estimate is highly unstable when based on a small number of cases. The limited number of CAUTI cases also impacts the overall assessment of the algorithm’s effectiveness. If the sensitivity estimate is uncertain, it is difficult to determine whether the algorithm reliably detects infections. A more stable and precise sensitivity measurement would require a larger sample size with more CAUTI cases. Until then, the model’s performance remains variable, which could lead to inconsistent results across different clinical settings.

Abbreviations

AUC

Area Under the Receiver Operating Characteristic Curve

BI

Business Intelligence

CAUTI

Catheter-Associated Urinary Tract Infection

CDC

Centers for Disease Control and Prevention

CI

Confidence Interval

EHR

Electronic Health Record

ED

Emergency Department

FN

False Negative

FP

False Positive

KSB

Kantonsspital Baden

LR

Likelihood Ratio

LR+

Positive Likelihood Ratio

LR−

Negative Likelihood Ratio

NLP

Natural Language Processing

ROC

Receiver Operating Characteristic

SD

Standard Deviation

STARD

Standards for Reporting of Diagnostic Accuracy Studies

TP

True Positive

TN

True Negative

Author contributions

L.A and B.W contributed to the conception and data collection. L.P and N.S assisted with data extraction, J.A, BW and M.M.W contributed to manuscript drafting, methodological supervision, critical review, and validation of the results. All authors reviewed and approved the final manuscript.

Funding

This study received no external funding.

Data availability

All data is provided within the manuscript.

Declarations

Consent for publication

All authors have reviewed and approved the final manuscript and consent to its publication.

Ethics approval

All methods were carried out in accordance with relevant guidelines and regulations. The protocol underwent the formal evaluation and approved by the Ethics Committee Northwest and Central Switzerland (ethical approval number EKNZ, Red-2024-00819). The Ethical Committee Northwest and Central Switzerland classified the study as quality improvement study and thus, based on the Swiss law for human research, no formal ethical approval was necessary.

Consent to participate

As the study was classified as a quality improvement study by the responsible IRB (Ethics Committee Northwest and Central Switzerland, Number EKNZ, Red-2024-00819). In accordance with national law for human subject research (https://www.edoeb.admin.ch/en/human-research, Art. 33 HRA), the requirement for individual patient consent was deemed unnecessary. The study was conducted in accordance to good clinical practice (GCP) procedures and in accordance to all international and national laws. No personal identifying information was used in the study data.

Competing interests

The authors declare no competing interests.