Targeted workup after initial febrile Urinary Tract Infection: Using a novel machine learning model to identify children most likely to benefit from VCUG

Abstract

BACKGROUND:

Significant debate persists regarding appropriate work-up of children with initial UTI. Greatly preferable to “all-or-none” approaches in current guideline would be a model that can identify children at highest risk for both recurrent UTI and VUR to allow targeted VCUG, while children at low risk could be observed. We sought to develop a model to predict probability of both recurrent UTI and VUR (“rUTI-associated VUR”) among children after initial UTI.

METHODS:

We included the RIVUR and CUTIE subjects, excluding the prophylaxis treatment arm from RIVUR. The main outcome was defined as rUTI-associated VUR. Missing data was imputed using optimal tree imputation. Data were split into training/validation/testing sets. Machine learning algorithm hyperparameters were tuned by the validation set with 5-fold cross validation.

RESULTS:

500 subjects (305 RIVUR and 195 CUTIE) were included in the study (90% female). Mean age was 21±19 months. 72 patients developed rUTI, of whom 53 also had VUR (10.6% of total). The final model included age, gender, race, weight, systolic blood pressure percentile, dysuria, urine albumin/creatinine ratio, prior antibiotics exposure, and current medication. The model predicted rUTI-associated VUR with AUC at 0.761 (95CI: 0.714– 0.808) on the testing set.

CONCLUSIONS:

Our predictive model using a novel machine learning algorithm provides promising performance to facilitate individualized management of children with initial UTI, and identify those most likely to benefit from VCUG after the initial UTI. This would allow more selective use of this test, increasing the yield while also minimizing overutilization.

INTRODUCTION

Urinary tract infection (UTI) affects up to 1.8% of boys and 8.4% of girls within the first 6 years of life.¹ Children with a febrile UTI have increased risk for anatomic anomalies, including 25–40% with vesicoureteral reflux (VUR),^{2, 3} which is associated with recurrent pyelonephritis and renal scarring. Historically, this elevated risk has prompted genitourinary imaging to identify VUR and other anomalies, leading to interventions.⁴ However, reports of high resolution rates in lower grade VUR, successful conservative management in selected patients, and the association between bladder and bowel dysfunction (BBD) and febrile UTI (fUTI) suggest that VUR is heterogeneous, and individualized management is warranted but difficult to achieve^{5, 6}

Voiding cystourethrogram (VCUG) is at the center of uncertainty in the management of children after initial UTI. Given its invasive nature, infection risks, and radiation, there is a desire for its judicious use.⁷ In 2011, the American Academy of Pediatrics (AAP) modified its guidelines for fUTI management in children ages two to 24 months.⁸ Unlike the previous version, which recommended VCUG following the first fUTI, the 2011 guidelines recommended deferring VCUG until after the second UTI unless abnormalites noted on renal ultrasound. However, the correlation between VUR and renal ultrasound abnormalities is poor.⁹ Therefore, the guidelines are controversial, and groups have expressed concern regarding the potential consequences of a delayed diagnosis of VUR, even while acknowledging concern for over-diagnosis with universal testing.¹⁰ This controvery suggests that our current clinical tools are inadequate to guide management in targeted fashion.

In such circumstances, modern machine learning techniques may be useful to improve clinical decision-making.¹¹ Unlike traditional methods that rely heavily on statitical assumptions, machine learning algorithms can identify highly complex patterns from data, allowing robust predictions. If the risk of recurrent fUTI and VUR in children presenting with their first fUTI could be predicted accurately, informed decisions regarding diagnostic imaging and management would be possible. Such model requires high-performing algorithms and robust data source. Randomized Intervention for Children with Vesico-Ureteral Reflux (RIVUR) and the ancillary Careful Urinary Tract Infection Evaluation (CUTIE) Trials provide an unique opportunity. They represent one of the largest groups of children who presented with their first UTI followed meticulously for 2 years.

We sought to use machine-learning techniques to predict risk of both recurrent UTI and presence of VUR (“rUTI-associated VUR”) among children presenting with initial UTI. By identifiying such children, we can determine those most likely to benefit from early VCUG, while also identifying children in whom the VCUG can be reasonably deferred (unlikely to have either recurrent UTI or VUR). We hypothesized that by using this model we could characterize the probability of rUTI-associated VUR given a limited set of patient baseline characteristics.

MATERIALS and METHODS

Data source and Cohort Selection

RIVUR was a multicenter (19 across the US), randomized, double-blinded, placebo-controlled trial designed to determine whether daily antimicrobial prophylaxis effective in preventing recurrence of UTI in children with VUR. The trial cohort and rationale were published previously.¹² Briefly, the eligibility included (1) age at randomization between 2 months and 6 years, (2) a diagnosed first or second febrile or symptomatic index UTI within 112 days to randomization, and (3) presence of VUR based on VCUG. Patients were followed for 2 years with primary outcome of recurrent UTI.

In addition to VUR patients, 3 centers collaborated to develop a parallel cohort of children (CUTIE) who were otherwise eligible for RIVUR without VUR. Participants were also followed for 2 years with the same protocol.

Together, the RIVUR placebo group and CUTIE represented a unique cohort of children whose UTI recurrence and VUR status were known and followed over a 2-year period without treatment (Figure 1-1).

Figure 1.

A and B, cohort flow diagrams.

Covariates Selection and Outcome Definition

The goal for the model development was to identify patients with rUTI-associated VUR using data available after the index UTI but prior to VCUG. These included basic demographic factors, physical exam, history, BBD, urine specimen, and ultrasound. Additionally, interaction terms were made a priori. A complete list of predictor candidates were listed in Appendix 1.

RIVUR and CUTIE both used stringent criteria for UTI.¹³ For both index or recurrent UTI, the event must have met all of the following criteria: pyuria on urinalysis, culture-proven infection with single organism (≥50,000 CFU/mm3 for catheterized or suprapubic aspirated specimen; ≥100,000 CFU/mm3 for clean voided specimen), fever (≥38°C) or UTI symptoms within 24 hours of urine collection (suprapubic, abdominal, flank pain/tenderness, urgency, frequency, hisitancy, dysuria, foul-smelling urine, or failure-to-thrive, dehydration, hypothermia in infants ≤ 4 months old).

The primary outcome in this study, rUTI-associated VUR, was defined as any VUR diagnosed on VCUG, and recurrent febrile UTI during the study period. Recurrent fUTI was defined as above with fever (≥38°C). Subjects with VUR but no recurrent UTI, with recurrent UTI but no VUR, and those with neither were defined as negative for the primary outcome.

Statistical Analysis and Model Development

Bivariate analyses were performed to compare potential predictors between those with rUTI-associated VUR and those without. We used the Chi-Square test, Fisher’s exact test, t-test, or Wilcoxon rank-sum test as appropriate based on data characteristics and distribution.

The details of the machine learning model development are explained in Appendix 2. Briefly, missing data were imputed using a novel technique, namely optimal tree imputation.¹⁴ The flow of oversampling, train/test set splitting, model training, and performance evaluation (AUC) are demonstrated in Figure 1-2. Optimal classification trees (OCTs) were chosen to be the technique due to high performance and interpretability.¹⁵

An alpha of 0.05 and 95% confidence intervals (CI) were used as criteria for statistical significance. All analyses were performed using Julia v 0.6.0 (package OptimalTrees, OptImpute), Python 2.7.13 (package Scikit-learn), and SAS 9.4 (SAS Institute, Cary, NC).

RESULTS

Demographics

We identified 500 participants without the exposure of prophylactic antibiotics including 305 from RIVUR and 195 participants from CUTIE. Mean age was 21±19 months. Female patients constituted 90% of the cohort.

Compared with children without rUTI-associated VUR, patients with rUTI-associated VUR were significantly more likely to be white (91% vs 72%, p<0.01 for chi-square test vs non-whites), taking over-the-counter or prescription medication (74% vs 49%, p<0.01), have a higher index UTI temperature (mean 39.8 vs 39.4°C, p=0.03). Additionally, children with rUTI-associated VUR were found to have lower weight percentile (median 57^th percentile vs 72^th percentile, p=0.05), prior antibiotics treatment history (median 1 month vs 0, p=0.07), and a dilated right ureter (6% vs 1%, p=0.05). The full demographics by the outcome were listed in Table 1.

Table 1-.

Patient characteristics by outcome

Characteristics	Total n=500	rUTI-associated VUR n=53(%)	No rUTI-associated VUR n=447(%)	p value
Age at enrollment				0.41
2–11 months	238	26(49.1)	212(47.4)
12–23 months	91	9(17.0)	82(18.3)
24–35 months	48	2(3.8)	46(10.3)
36–71 months	123	16(30.2)	107(23.9)
Missing	0
Gender				0.46
Female	452	50(94.3)	402(89.9)
Male	48	3(5.7)	45(10.1)
Missing	0	3(5.7)	45(10.1)
Race				<0.01*
White	368	48(90.6)	320(71.6)
Black	52	0(0)	52(11.6)
Mixed	37	2(3.8)	35(7.8)
Others	35	3(5.7)	32(7.2)
Missing	8
index UTI temperature (missing 54)				0.03
Mean±SD		103.6±1.4	102.9±1.8
Index UTI organism				0.61
E coli	451	47(88.7)	404(90.4)
Others	46	6(11.3)	40(8.9)
Missing	3
Index UTI symptoms				0.047
Febrile	160	21(39.6)	139(31.1)
Symptomatic	83	3(5.7)	80(17.9)
Both	257	29(54.7)	228(51.0)
Missing	0
Index UTI				0.47
Susceptible	248	29(54.7)	219(49.0)
Resistant	252	24(45.3)	228(51.0)
Missing	0
Abx tx in past 6 months (missing 1)				0.07
Median(IQR)		1(0–2)	0(0–1)
Taking any medication				<0.01
Yes	260	39(73.6)	221(49.4)
no	239	14(26.4)	225(50.3)
Missing	1
Family history
sibling VUR	14	3(5.7)	11(2.5)	0.32
sibling recurrent UTI	28	4(7.5)	24(5.4)	0.70
Missing	1
History of constipation				0.37
Yes	139	12(22.6)	127(28.4)
No	359	41(77.4)	318(71.1)
Missing	2
Ever treated for constipation				0.23
Yes	79	5(9.4)	74(16.6)
No	419	48(90.6)	371(83.0)
Missing	2
Bladder bowel dysfunction				0.30
Yes	63	10(18.9)	53(11.9)
No	57	4(7.5)	53(11.9)
Not toilet-trained	373	39(73.6)	334(74.7)
Missing	7
Weight percentile (missing 0)
Median(IQR)		57(38–80)	72(45–88)	0.05
SBP ≥90th percentile				0.72
Yes	115	14(26.4)	101(22.6)
No	283	30(56.6)	253(56.6)
Missing	102
Ultrasound findings
Dilated right ureter	8	3(5.7)	5(1.1)	0.05
Dilated left ureter	9	2(3.8)	7(1.6)	0.26
Thickened bladder wall	6	0(0)	6(1.3)	0.50
Missing	40

^*chi-square test significant for white vs non-white, and fisher’s exact test white vs black vs others vs mixed

Prediction Model

The prediction model was illustrated in Figure 2-1. Each patient was categorized into a terminal “leaf”, comprised of patients who shared similar probability of rUTI-associated VUR. Baseline variables included BBD (defined as positive (=1) if the Dysfunctional Voiding Questionnaire score (Appendix 3) was ≥ 6 in female or ≥ 9 in male patients, all non-toilet trained children were treated as negative (=0)),^{16, 17} age (months), gender, race, weight percentile, number of antibiotic resistance in urine culture, urine albumin to creatinine ratio (urine ACR), dysuria, taking prescription or over-the-counter medication (Y/N), antibiotics within past 6 months (non-UTI), ≥90th percentile systolic blood pressure.

Figure 2.

A, full prediction OCT model with high-low risk cutoff at 0.5. SBP, systolic blood pressure. B, example 1. C, example 2. UCx, urine culture.

The tree structure was intuitive and interpretable for clinical use. For example, patient A presented as a 20-month black male with no BBD as shown in Figure 2-2. Having no BBD (No=0) and age at 20-month would give initial BBD*Age=0 which was less than the cutoff at 42. At the next node (Race*Sex), he would go down further left as a black male and end up over the low-risk group. Take another example (Figure 2-3): a 5 y/o girl with BBD, 1st UTI urine culture showed bacteria with multiple Abx resistance. She had BBD with age as 60-month and therefore the first value was >42 that leads to the right. Multiple resistance in UCx further leads to a high-risk group. VCUG should be strongly considered in this case. All leaves and associated characteristics are listed in Table 2 (extracted from Figure 2-1).

Table 2-.

Probability of rUTI-associated VUR by risk groups

Groups at high risk of rUTI-associated VUR	Probability(%)(95CI)
BBD*age≥42, Abx resistance≥3	82.8(69.1–96.5)
BBD*age≥42, Abx resistance≤2, weight percentile<44.12	98.2(94.7–100)
Whites/Other females/Mixed males, Urine ACR≥44.75(mg/g), dysuria(+), ≥4 mo Abx tx for infections other than UTI	100(60.2–100)
Whites/Other females/Mixed males, Urine ACR≥44.75(mg/g), dysuria(−), on other medications	85.7(79.6–91.8)
Whites/Other females/Mixed males, Urine ACR<44.75(mg/g), ≥1 mo Abx tx for infections other than UTI, weight percentile≥60.06, SBP>90th percentile	76.4(65.1–87.6)
Whites/Other females/Mixed males, Urine ACR<44.75(mg/g), ≥1 mo Abx tx for infections other than UTI, weight percentile<60.06, Urine ACR≥23.98	84.5(79.7–92.2)
Whites/Other females/Mixed males, Urine ACR<44.75(mg/g), no Abx tx, dysuria(−), age*weight≥549.9	97.3(92.3–100)
Groups at low risk of rUTI-associated VUR	Probability(%)
BBD*age≥42, Abx resistance≤2, weight percentile≥44.12	0(0–10.1)
Blacks/Other males/Mixed females	0(0–4.1)
Whites/Other females/Mixed males, Urine ACR≥44.75(mg/g), dysuria(+), ≤3 mo Abx tx for infections other than UTI	0(0–17.1)
Whites/Other females/Mixed males, Urine ACR≥44.75(mg/g), dysuria(−), not on other medications	0(0–12.3)
Whites/Other females/Mixed males, Urine ACR<44.75(mg/g), ≥1 mo Abx tx for infections other than UTI, weight percentile≥60.06, SBP≤90th percentile	19.4(9.7–29.0)
Whites/Other females/Mixed males, Urine ACR<44.75(mg/g), ≥1 mo Abx tx for infections other than UTI, weight percentile<60.06, Urine ACR<23.98	0(0–13.3)
Whites/Other females/Mixed males, Urine ACR<44.75(mg/g), no Abx tx, dysuria(+)	0(0–7.8)
Whites/Other females/Mixed males, Urine ACR<44.75(mg/g), no Abx tx, dysuria(−), age*weight<549.9	23.3(15.1–31.2)

Abx=antibiotics

ACR=microalbumin to creatinine ratio

SBP=systolic blood pressure

The receiver operating characteristics (ROC) curve for the proposed OCT model on the testing set was shown in Appendix 4. Area under curve (AUC) for OCT model was 0.761 (95CI: 0.714– 0.808). This demonstrated good performance of the predictive model (a perfect predictive model would have AUC of 1.0, while a model that is no better than a coin toss would have an AUC of 0.5. Sensitivity analysis was performed comparing algorithms (logistic regression, random foreste, gradient boosted trees). OCT was chosen because good performance, tighter AUC confidence interval, and intepretability (Appendix 2).

DISCUSSION

To our knowledge, this is the first machine learning model created to predict clinical outcomes in children with a UTI. Our model effectively sorted children into probabilistic groups for rUTI-associated VUR. By quantifying the risk of both recurrent UTI and VUR in a given child, our model allowed data-driven management. For those children predicted to have a very high risk for rUTI-associated VUR, providers may order VCUG after the initial fUTI, despite the 2011 AAP guidelines’ recommendation. Conversely, for patients predicted to have a very low risk of rUTI-associated VUR, deferral of the VCUG would seem reasonable, since VCUG in such children is likely to be low-yield, even among those who experience recurrent UTI. The model also offered the capability to quantify the probability of the outcome, to allow for individual interpretation.

Unlike many machine learning techniques with “black-box” reputation, OCT models enable physicians to look “under the hood” while maintaining high performance.¹⁵ This allows incorporation of medical knowledge into the conceptual model framework, ensuring a clinically sound model with real-world applicability. For example, during the model building phase, serum parameters were initially included. With the goal of reducing unnecessary invasive testing, analysis was performed with and without the serum data. Model performance was minimally affected, and therefore serum-related variables were removed. Additionally, we were able to reduce the number of variables from more than 50 to fewer than 8 needed per patient while maintaining performance. The variables ultimately included are easy to obtain in a typical clinical setting. The only specimen needed to fully utilize the model predition power was urine ACR- a common test which does not require sterile urine collection.

Encouragingly, most of the splits in the OCT model correlated well with clinical intuition. For example, BBD has been found to be highly associated with VUR and UTI.⁶ While BBD has the limitation of being applicable only to toilet-trained children, our model overcame this issue by combining BBD and age, which resulted in the identification of older children with BBD as higher risk. Our model also showed a higher risk of rUTI-associated VUR for whites and female, which were consistent with reported data.^{18, 19} Similarly, previous antibiotics exposures and resistance were highly related with rUTI in VUR patients.^{20, 21} Unfortunately circumcision was too rare in our cohort to be incorporated into the model meaningfully. Hypertension was also a known sequela from reflux nephropathy and pyelonephritis, and was selected by the algorithm.^{22, 23}

During model development, we also noted several interesting findings. Urine ACR stood out in the final model and subsequent sensitivity analyses. Despite relatively scarce urology literature, this test is commonly done in nephrology to screen albuminuria. Multiple groups have reported significant association between high-grade VUR, UTI, and urine ACR.^24–26 Although the exact mechanism is unknown, it is hypothesized that retrograde urine flow, glomerulosclerosis, and subsequent hyperfiltration are significantly associated with this finding. Similar to urine ACR, weight consistently was chosen as important tree-splitting predictor. The association between weight and VUR or UTI are poorly defined in the literature, and conflicting data have been reported.^27–29 Our hypothesis is that weight may be associated with VUR and UTI in a non-linear fashion, and its influence may be dependent on complex interactions with other variables. The association of dysuria with higher risk for rUTI-associated VUR may seem counter-intuitive.³⁰ However, since our primary outcome was VUR with recurrent febrile UTI, we hypothesized that dysuria alone was more likely an indicator in older children for BBD or lower tract non-febrile UTI instead of pyelonephritis. This hypothesis was supported by the relatively weak association between rUTI-associated VUR and symptom-only non-febrile UTI in our study. These findings demonstrate that a robust machine learning model can potentially detect biologically plausible, but previously under-appreciated, clinical associations.

The findings of our study should be viewed in the context of its design limitations. RIVUR and CUTIE enrolled children between 2 months and 6 years old with heavy female predominance. Children with congenital anatomical abnormalities were also excluded.¹² This may impact our model’s generalizability, especially for patients out of the range of the cohort. However, we believe the original cohort did capture the majority of children of interest, and RIVUR/CUTIE represented the most comprehensive, best described cohort available. Many anatomical abnormalities such as ureterocele would likely be detected via screening ultrasound and would therefore be diverted out of the algorithm, since decicison-making for VCUG in such patients is not addressed by the AAP guidelines. Additionally, the cohort enrolled a few children after their second UTI, if they otherwise met inclusion criteria. Such subjects represented a small fraction of the cohort (9%, n=47), and having a second “index” UTI was not significantly associated with recurrent UTI(p=0.19) or VUR(p=0.43) therefore we were not surprised that this variable did not make it into the model.

Furthermore, the power of our model was limited by the relatively small sizes of the cohorts. This limitation further restricted the evaluation of factors such as BBD (non-toilet trained could not be evaluated) and ureteral dilation (n=17). We were able to stabilize the model and provide good performance with the combination of weighting and oversampling techniques. Nevertheless, the model would be more accurate and useful with broader generalizability if the cohort were larger, supplying additional data to train and validate the algorithm. It is important to recognize that this prediction model is imperfect, primarily due to the inherent size limitation.

One of the most essential insights this study provided was that building such a prediction model using robust data and state-of-the-art analytics is achievable and potentially impactful for current practice. We have built a free webpage and app (pending Appstore approval) that permit rUTI-associated VUR prediction (Appendix 5). The tool will aid clinical counseling and decision-making for clinicians seeing children after initial UTI, and may also facilitate data collection for further studies. We hope that this study can serve as an impetus to incorporate machine learning techniques to advance our ability to care for our patients in a more individualized, quantitative, and data-driven fashion.

CONCLUSIONS

We developed a predictive model using novel machine learning algorithms and robust trial data that estimates risks of rUTI-associated VUR after intial UTI in children. This information may facilitate more selective use of VCUG after initial UTI than current guidelines permit, reserving VCUG for those in whom it is likely to be high-yield, while allowing VCUG to be safely deferred in those determined to be at very low risk. Further validation especially for novel factors such as urine ACR is warranted and will require additional data collection.

Abbreviations:

RIVUR

Randomized Intervention for Children with Vesico-Ureteral Reflux

CUTIE

Careful Urinary Tract Infection Evaluation

VUR

Vesico-Ureteral Reflux

csVUR

clinically significant Vesico-Ureteral Reflux

OCT

Optimal Classification Tree

UTI

urinary tract infection

AAP

American Academy of Pediatrics

VCUG

Voiding Cystourethrogram

DMSA

Dimercaptosuccinic Acid

SBP

Systolic Blood Pressure

BBD

Bladder and Bowel Dysfunction

ROC

Receiver Operating Characteristics

AUC

Area Under Curve

Urine ACR

urine albumin to creatinine ratio