External Validation of Brief Resolved Unexplained Events Prediction Rules for Serious Underlying Diagnosis

Nassr Nama; Ye Shen; Jeffrey N Bone; Zerlyn Lee; Kara Picco; Falla Jin; Jessica L Foulds; Josée Anne Gagnon; Chris Novak; Brigitte Parisien; Matthew Donlan; Ran D Goldman; Anupam Sehgal; Joanna Holland; Sanjay Mahant; Joel S Tieder; Peter J Gill

doi:10.1001/jamapediatrics.2024.4399

. 2024 Dec 16;179(2):188–196. doi: 10.1001/jamapediatrics.2024.4399

External Validation of Brief Resolved Unexplained Events Prediction Rules for Serious Underlying Diagnosis

Nassr Nama ^1,^✉, Ye Shen ², Jeffrey N Bone ², Zerlyn Lee ³, Kara Picco ⁴, Falla Jin ², Jessica L Foulds ⁵, Josée Anne Gagnon ⁶, Chris Novak ⁷, Brigitte Parisien ⁸, Matthew Donlan ⁹, Ran D Goldman ², Anupam Sehgal ¹⁰, Joanna Holland ¹¹, Sanjay Mahant ⁴, Joel S Tieder ¹², Peter J Gill ⁴, for the Canadian BRUE Collaboration (C-BRUE-C) and the Canadian Paediatric Inpatient Research Network (PIRN)

¹Division of Hospital Medicine, Department of Pediatrics, University of Washington, Seattle Children’s Hospital, Seattle

²BC Children’s Hospital Research Institute, Vancouver, British Columbia, Canada

³Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada

⁴Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada

⁵Stollery Children’s Hospital, Division of Pediatric Hospital Medicine, Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada

⁶CHU de Québec, Université Laval, Québec City, Québec, Canada

⁷Department of Pediatrics, Alberta Children’s Hospital, University of Calgary, Calgary, Alberta, Canada

⁸Department of Pediatrics, CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada

⁹MUHC-The Montreal Children’s Hospital, McGill University, Montreal, Quebec, Canada

¹⁰Department of Paediatrics, Kingston General Hospital, Queen’s University, Kingston, Ontario, Canada

¹¹Division of General Pediatrics, Department of Pediatrics, IWK Health Centre, Halifax, Nova Scotia, Canada

¹²Division of Hospital Medicine, Department of Pediatrics, Seattle Children’s Hospital and the University of Washington, Seattle

Group Information: Members of the Canadian BRUE Collaboration (C-BRUE-C) and the Canadian Paediatric Inpatient Research Network (PIRN) appear in Supplement 2.

Accepted for Publication: August 10, 2024.

Published Online: December 16, 2024. doi:10.1001/jamapediatrics.2024.4399

^✉

Corresponding Author: Nassr Nama, MD, MSc, Division of Hospital Medicine, Department of Pediatrics, University of Washington, Seattle Children’s Hospital, 4800 Sand Point Way, NE, FA.2.115, Seattle, WA 98105 (nassr.nama@seattlechildrens.org).

Author Contributions: Dr Nama had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Nama, Picco, Jin, Gagnon, Tieder, Gill.

Acquisition, analysis, or interpretation of data: Nama, Shen, Bone, Lee, Picco, Jin, Foulds, Gagnon, Novak, Parisien, Donlan, Goldman, Sehgal, Holland, Mahant, Gill.

Drafting of the manuscript: Nama, Lee, Picco, Novak, Goldman, Tieder, Gill.

Critical review of the manuscript for important intellectual content: Nama, Shen, Bone, Picco, Jin, Foulds, Gagnon, Novak, Parisien, Donlan, Goldman, Sehgal, Holland, Mahant, Tieder, Gill.

Statistical analysis: Nama, Shen, Bone, Gill.

Obtained funding: Nama, Lee, Gill.

Administrative, technical, or material support: Nama, Jin, Foulds, Novak, Donlan.

Supervision: Nama, Foulds, Gagnon, Parisien, Donlan, Sehgal, Tieder.

Conflict of Interest Disclosures: Dr Nama reported receiving grants from the American Academy of Pediatrics and Canadian Paediatric Society during the conduct of the study and being a prior shareholder of insightScope outside the submitted work. Dr Lee reported receiving grants from the Canadian Pediatric Society during the conduct of the study. Dr Tieder reported receiving personal fees from Children’s Hospital Association outside the submitted work. Dr Gill reported receiving grants from the Canadian Institutes of Health Research and the PSI Foundation outside the submitted work. No other disclosures were reported.

Funding/Support: Dr Nama was supported by the Seattle Children’s Research Institute/Career Development Award 2023-2026, the American Academy of Pediatrics/Resident Research Grant Program 2021, and the BC Children’s Hospital Research Institute/Evidence to Innovation/Early Investigator Seeds Grant 2022. Dr Lee was supported by the BC Children’s Hospital/Resident Research Funding 2022 and the Canadian Paediatric Society/Hospital Pediatrics Resident Grant 2022, Dr Novac was supported by the Department of Pediatrics at Alberta Children’s Hospital and the Alberta Children’s Hospital Foundation.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: The members of the Canadian BRUE Collaboration (C-BRUE-C) and the Canadian Paediatric Inpatient Research Network (PIRN) appear in Supplement 2.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We thank Polina Kyrychenko, Nardin Kirolos, MD, Ioulia Opotchanova, BSc, Émilie Harnois, MD, Elisa Frizon-Peresa, Praveen Rajasegaran, MSc, Parnian Hosseini, MD, Melody Wyslobicky, BHSc, Susan Akbaroghli, MD, MPH, Prathiksha Nalan, BSc, HBA, Marie-Pier Goupil, Shawn Lee, Emy Philibert, Juliette Dufresne, Raman Chawla, MD, and Martin Ogwuru, MD for their help in data collection. Beyond usual salary, no one was financially compensated for their contribution.

^✉

Corresponding author.

PMCID: PMC11791710 PMID: 39680379

This multicenter cohort study evaluates the effectiveness of the clinical prediction rules for brief resolved unexplained events (BRUE) compared with the higher-risk criteria of the American Academy of Pediatrics in estimating risk for serious underlying diagnosis.

Key Points

Question

Are recently derived clinical prediction rules for brief resolved unexplained events (BRUE) better than the American Academy of Pediatrics (AAP) higher-risk criteria in estimating risk for serious underlying diagnosis?

Findings

In a multicenter cohort study of 1042 infants with BRUE, the BRUE prediction rules demonstrated superior prediction performance over the AAP higher-risk criteria. The BRUE prediction rules showed significant improvements in discrimination, with area under the curve of 0.71, as opposed to a value of 0.53 for the AAP.

Meaning

The validated BRUE prediction rules offer a superior prediction performance of risk for serious underlying diagnosis than the widely used AAP higher-risk criteria.

Abstract

Importance

The American Academy of Pediatrics (AAP) higher-risk criteria for brief resolved unexplained events (BRUE) have a low positive predictive value (4.8%) and misclassify most infants as higher risk (>90%). New BRUE prediction rules from a US cohort of 3283 infants showed improved discrimination; however, these rules have not been validated in an external cohort.

Objective

To externally validate new BRUE prediction rules and compare them with the AAP higher-risk criteria.

Design, Setting, and Participants

This was a retrospective multicenter cohort study conducted from 2017 to 2021 and monitored for 90 days after index presentation. The setting included infants younger than 1 year with a BRUE identified through retrospective chart review from 11 Canadian hospitals. Study data were analyzed from March 2022 to March 2024.

Exposures

The BRUE prediction rules.

Main Outcome and Measure

The primary outcome was a serious underlying diagnosis, defined as conditions where a delay in diagnosis could lead to increased morbidity or mortality.

Results

Of 1042 patients (median [IQR] age, 41 [13-84] days; 529 female [50.8%]), 977 (93.8%) were classified as higher risk by the AAP criteria. A total of 79 patients (7.6%) had a serious underlying diagnosis. For this outcome, the AAP criteria demonstrated a sensitivity of 100.0% (95% CI, 95.4%-100.0%), a specificity of 6.7% (95% CI, 5.2%-8.5%), a positive likelihood ratio (LR+) of 1.07 (95% CI, 1.05-1.09), and an AUC of 0.53 (95% CI, 0.53-0.54). The BRUE prediction rule for discerning serious diagnoses displayed an AUC of 0.60 (95% CI, 0.54-0.67; calibration intercept: 0.60), which improved to an AUC of 0.71 (95% CI, 0.65-0.76; P < .001; calibration intercept: 0.00) after model revision. Event recurrence was noted in 163 patients (15.6%). For this outcome, the AAP criteria yielded a sensitivity of 99.4% (95% CI, 96.6%-100.0%), a specificity of 7.3% (95% CI, 5.7%-9.2%), an LR+ of 1.07 (95% CI, 1.05-1.10), and an AUC of 0.58 (95% CI, 0.56-0.58). The AUC of the prediction rule stood at 0.67 (95% CI, 0.62-0.72; calibration intercept: 0.15).

Conclusions and Relevance

Results of this multicenter cohort study show that the BRUE prediction rules outperformed the AAP higher-risk criteria on external geographical validation, and performance improved after recalibration. These rules provide clinicians and families with a more precise tool to support decision-making, grounded in individual risk tolerance.

Introduction

Brief resolved unexplained events (BRUE) are common in infants¹ and involve transient symptoms such as apnea, color change, muscle tone alteration, or altered breathing patterns, resolving spontaneously without a clear cause.² These events, often benign, raise substantial caregiver and health care professional anxiety due to their unexplained nature,^3,4,5 potential recurrence in 9% to 14% of infants, and the small yet concerning possibility that a BRUE might be the first indication of a more serious underlying condition, seen in approximately 5% of cases.^6,7

The 2016 clinical practice guidelines by the American Academy of Pediatrics (AAP) newly defined BRUE and categorized them into lower- and higher-risk groups, based on patient factors, to help clinicians identify infants at risk of serious conditions.^2,8 These higher-risk criteria were based on the apparent life-threatening events literature available at the time. Although the higher-risk criteria were later demonstrated to have an excellent negative predictive value (NPV, 98%), they demonstrated a poor positive predictive value (PPV, 5%), overclassifying the vast majority (92%) of patients as higher risk when only 5% were eventually diagnosed with a serious underlying problem.⁶

In response, new BRUE prediction criteria were developed from a US retrospective study involving 3283 patients experiencing BRUE that showed improved discrimination in identifying serious underlying diagnoses (area under the curve (AUC, 0.61 vs 0.52) and predicting event recurrence (AUC, 0.68 vs 0.54) compared with the AAP higher-risk criteria.⁶ External geographical validation of these rules will confirm their generalizability and applicability across diverse populations and health care settings, by determining if the predictive accuracy observed in the initial study remains consistent in an independent sample.^9,10,11

Therefore, this study was focused on externally validating the BRUE prediction rules within a Canadian cohort of infants experiencing BRUE. This is a critical step to verify the rules’ utility in clinical practice and improve risk estimation for 2 important outcomes of serious underlying diagnosis and event recurrence.

Methods

Study Design

This retrospective cohort study was conducted in the Canadian Paediatric Inpatient Research Network (PIRN). We identified patients from 11 hospitals, including a mix of tertiary pediatric centers and general hospitals. These sites were located across 5 of Canada’s 10 provinces, including 8 of the country’s 10 largest pediatric centers. The primary site and each participating center granted ethical approval for this study with waiver of consent (University of British Columbia Children’s and Women’s Research Ethics Board [H21-02357]). The study’s protocol was published a priori.¹² The validation of the clinical prediction rules conducted in this study adheres to the principles outlined in the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines (eTable 1 in Supplement 1).¹³ This constitutes geographical validation of the BRUE prediction rules, as it evaluates their performance in a distinct country from where they were derived.¹⁴

Cohort Identification

Infants aged 1 to 365 days, presenting to the emergency department (ED) for a BRUE between January 1, 2017, and December 31, 2021, were identified through a validated method based on admission and discharge codes.¹⁵ Eligible patients had to meet the diagnostic criteria for BRUE,² with the event remaining unexplained after a comprehensive history and physical examination. Trained research assistants at each participating site reviewed medical records for eligibility and data extraction (details in published protocol).¹² Participant race and ethnicity data were not collected, as this is not reliably collected across Canadian hospitals.

Outcomes

The study evaluated 2 outcomes of interest. The primary outcome was the identification of a serious underlying diagnosis as the causative factor for the BRUE-like event. This category included conditions where a delay in diagnosis could lead to increased morbidity or mortality. Criteria for classification as serious or nonserious were determined a priori.¹² Diagnoses with a range of presentations, like viral respiratory tract infections, were classified as serious if they necessitated medical interventions (eg, administration of oxygen, high-flow therapy, mechanical ventilation, or intensive care unit admission). The secondary outcome was the recurrence of BRUE either during the index hospital stay or based on representation to the ED. Monitoring for both outcomes was limited to 90 days after the index ED visit.

Comparing Derivation and Validation Cohorts

A comparative analysis was conducted between the derivation and validation cohorts to assess the frequency of the outcomes of interest and the prevalence of predictors outlined by the AAP higher-risk criteria and the BRUE prediction rules. The derivation cohort included 3283 patients diagnosed with BRUE, identified across 15 children’s hospitals within the US. These patients presented for care from October 1, 2015, to September 30, 2018, and from April 1, 2019, to June 30, 2020.⁶ The validation cohort, as described herein, was identified across 11 Canadian hospitals (2017-2021). Standardized mean differences (SMDs) were used to compare variables across cohorts.¹⁶ Variability observed in the predictors and outcomes between the cohorts is considered beneficial, as it facilitated external geographical validation within a distinct population, rather than one that identically mirrored the derivation cohort.^14,17

AAP Higher-Risk Criteria

Predictors

According to the AAP clinical practice guidelines, infants with BRUE are stratified into lower-risk and higher-risk categories.² An infant is considered higher risk if they meet any of the following: (1) age 60 days or younger, (2) gestational age younger than 32 weeks or corrected younger than 45 weeks, (3) event lasting 1 minute or longer, (4) multiple events or event clusters, (5) cardiopulmonary resuscitation by a medical professional, (6) concerning features in the medical history (eTable 2 in Supplement 1), or (7) concerning signs on physical examination.

Performance

We evaluated the AAP higher-risk criteria sensitivity, specificity, PPV, NPV, positive likelihood ratio (LR+), negative LR (LR−), and the AUC in relation to the study outcomes.

BRUE Prediction Rules

Predictors

The BRUE prediction rule for the outcome of serious underlying diagnosis included the following predictors: (1) history of a similar event, (2) abnormal medical history, and (3) age in days. The predictors for event recurrence included the following: (1) prematurity (defined in line with the AAP higher-risk criteria), (2) multiple events or event clusters, (3) color change, (4) abnormal respiratory pattern, and (5) change in muscle tone. The predictors of abnormal respiratory pattern and change in muscle tone were negatively associated with the outcome. Any patient or BRUE characteristics not documented in the patient’s chart were presumed absent.

Recalibration

For each outcome, we estimated the risk using the coefficients and intercept obtained from the derivation study original model. The BRUE prediction rule was then updated using 3 methods that progressively modified the model parameters^18,19:

Recalibration in the large: this initial step involved adjusting the prediction rule’s intercept to mirror the difference in outcome prevalence between the derivation and validation cohorts.
Logistic recalibration: here, the intercept was re-estimated as mentioned previously, and a uniform (same for all variables) scaling factor was applied to modify all coefficients within the prediction rule.
Model revision: the final recalibration involved re-estimating the prediction rule parameters through logistic regression analysis within the validation cohort. This step involved updating both intercept and coefficients, keeping the predictor variables unchanged.

Model Comparisons

LR tests were used sequentially to compare the fits of the previously mentioned nested models. We used the DeLong test to quantify the difference in AUC between the original and revised models and the corresponding 95% CI.

Performance

In assessing the performance of clinical prediction rules, 2 aspects were evaluated^{11,20,21,22,23}:

Discrimination: the capability of the prediction rule to distinguish between patients with and without each outcome was assessed using a receiver operating characteristic curve.¹⁹ The AUC provides a summary measure, with values ranging from 0.5, denoting random chance, to 1.0, indicating perfect discrimination. We additionally generated precision-recall curves comparing the original and the revised models (eFigure 1 in Supplement 1).
Calibration: the concordance between predicted and observed outcomes. Calibration was assessed visually using calibration plots and quantitatively through estimation of the calibration slope and intercept and corresponding 95% CIs.²⁴

Sensitivity Analysis

To ensure robustness and reliability of the findings, a preplanned sensitivity analysis was conducted. First, to address potential variations in patient characteristics between those admitted and those discharged from the ED, the sensitivity analysis excluded data from 3 hospitals. These specific hospitals only provided data on admitted patients. Second, transferred patients were also excluded, as they may represent a more severely ill subset.

Sample Size

The study’s sample size was calculated a priori, using data from the derivation study.⁶ With an AUC of 0.61 for the primary outcome and a prevalence rate of 4.57%, a sample size of 1182 patients experiencing BRUE, including 54 with serious underlying diagnoses, was estimated to be necessary to achieve an 80% power at a 5% significance level to detect an AUC above 0.5.⁶ A post hoc analysis of the validation cohort demonstrated a prevalence of the primary outcome at 7.58%. With this higher than anticipated prevalence, a smaller sample size of 736 patients, including 56 cases, was required to achieve the same power.

Statistical Analysis

Statistical analyses were carried out using R software, version 4.3.2 (R Project for Statistical Computing), with the pROC, version 1.18.5, and the CalibrationCurves, version 2.0.3, packages.^24,25 Hypothesis testing was 2-sided, with a threshold of .05 set for statistical significance. Study data were analyzed from March 2022 to March 2024.

Results

Study Population

The validation cohort included 1042 eligible infants (median [IQR] age, 41 [13-84] days; 529 female [50.8%]; 513 male [49.2%]) with BRUE (Table 1).⁶ The majority of the cohort were infants aged 60 days or younger (658 [63.1%]). There was documentation of an abnormal medical history in 320 infants (30.7%), concerning family history in 133 infants (12.8%), and concerning social history in 48 infants (4.6%). Hospital admission was common, in 665 infants (63.8%). The primary outcome of serious underlying diagnosis was observed in 79 patients (7.6%) (eTable 3 in Supplement 1).²⁶ The secondary outcome of event recurrence occurred in 163 patients (15.6%).

Table 1. Demographic and Clinical Characteristics of Patients in the Brief Resolved Unexplained Events (BRUE) Group in the Derivation and Validation Cohorts.

Characteristic	Derivation cohort (US 2015-2020)^a	Validation cohort (Canada 2017-2021)	Standardized mean difference^b
No. of patients	3283	1042	NA
Age (IQR), d	48 (18-116)	41 (13-84)	NA
Sex, No. (%)
Female	1705 (51.9)	529 (50.8)	0.023
Male	1578 (48.1)	513 (49.2)	0.023
Patient risk factors
Gestational age, No. (%)
Term (≥37 wk)/not indicated	2429 (74.0)	890 (85.4)	0.322
Late preterm (34-36+ wk)	541 (16.5)	102 (9.8)
Moderate preterm (32-33+ wk)	246 (7.5)	26 (2.5)
Very preterm (28-31+ wk)	67 (2.0)	24 (2.3)
Prematurity (<32 wk) or corrected <45 wk,^c No. (%)	636 (19.4)	180 (17.3)	0.054
Age ≤60 d, No. (%)	1874 (57.1)	658 (63.1)	0.124
Family history concerning for serious condition, No. (%)	312 (9.5)	133 (12.8)	0.104
Social history concerning for abuse, No. (%)	129 (3.9)	48 (4.6)	0.034
Abnormal medical history, No. (%)	993 (30.2)	320 (30.7)	0.010
BRUE characteristics,^d No. (%)
Color change	1645 (50.1)	514 (49.3)	0.016
Abnormal breathing	2278 (69.4)	686 (65.8)	0.076
Tone change	1481 (45.1)	607 (58.3)	0.265
Altered responsiveness	1167 (35.5)	377 (36.2)	0.013
Event duration ≥1 min	958 (29.2)	354 (34.0)	0.103
History of similar event	1192 (36.3)	343 (32.9)	0.071
History of multiple events or event clusters	996 (30.3)	325 (31.2)	0.018
CPR performed and indicated	60 (1.8)	44 (4.2)	0.140
Higher-risk BRUE as defined by the AAP guidelines	3005 (91.5)	977 (93.8)	0.086
Clinical outcomes, No. (%)
Hospital admission	2063 (62.8)	665 (63.8)	0.020
Serious underlying diagnosis^e	150 (4.6)	79 (7.6)	0.126
Recurrent event^f	469 (14.3)	163 (15.6)	0.038
Rehospitalization^e	281 (8.6)	64 (6.1)	0.093
Death^e	1 (0.03)	1 (0.1)	0.026

Open in a new tab

Abbreviations: AAP, American Academy of Pediatrics; CPR, Cardiopulmonary Resuscitation.

^{^a}

Data from source.⁶

^{^b}

Comparison between the derivation and validation cohorts using standardized mean differences. Age as a continuous variable was not compared, as we do not have access to the raw data for the derivation cohort.

^{^c}

Defined as per the AAP higher-risk criteria.

^{^d}

Patient may present with BRUE episodes including multiple characteristics.

^{^e}

At or within 90 days of index presentation.

^{^f}

Recurrent events were limited to the index hospitalization in the derivation cohort, whereas in the validation cohort, they included representations to the emergency department within 90 days.

When compared with the 3283 patients in the derivation cohort, the 1042 patients in the validation cohort demonstrated a higher prevalence of term births (890 [85.4%] vs 2429 [74.0%]; SMD = 0.322), age 60 days or younger (658 [63.1%] vs 1874 [57.1%]; SMD = 0.124), and concerning family history (133 [12.8%] vs 312 [9.5%]; SMD = 0.103). Events in the validation cohort were more likely to be associated with changes in muscle tone (607 [58.3%] vs 1481 [45.1%]; SMD = 0.265). The incidence of serious underlying diagnoses was higher in the validation cohort (79 [7.6%] vs 150 [4.6%]; SMD = 0.126), whereas event recurrence rate was similar between cohorts (163 [15.6%] vs 469 [14.3%]; SMD = 0.038).

Serious Underlying Diagnosis

For the primary outcome, the AAP higher-risk criteria demonstrated high sensitivity at 100.0% (95% CI, 95.4%-100.0%) but poor specificity at 6.7% (95% CI, 5.2%-8.5%) (Table 2)⁶ and an LR+ of 1.07 (95% CI, 1.05-1.09). The AUC for discrimination was poor, at 0.53 (95% CI, 0.53-0.54) (Figure 1). Improved discrimination was achieved with the BRUE prediction rules, yielding an AUC of 0.60 (95% CI, 0.54-0.67; P = .02), comparable with an AUC of 0.61 (95% CI, 0.49-0.72) in the derivation cohort (Table 2).⁶ However, the BRUE prediction rule’s calibration was suboptimal (Figure 2 and eTable 4 in Supplement 1), tending to underestimate the risk of a serious underlying diagnosis (intercept: 0.60, slope: 0.81). After model revision, both the discrimination, with an AUC of 0.71 (95% CI, 0.65-0.76; P < .001), and calibration (intercept: 0.00, slope: 1.00) were considered satisfactory. Age in days had a coefficient of −0.0046 (SE = 0.0021) in the revised model, suggesting that the increasing age is associated with lower risk of a serious underlying diagnosis.

Table 2. Performance Metrics of the American Academy of Pediatrics (AAP) Higher-Risk Criteria and the Brief Resolved Explained Events (BRUE) Prediction Rules.

Metric	Estimate (95% CI)
Metric	Serious underlying diagnosis	Event recurrence
Patients, No. (%)	79 (7.6)	163 (15.6)
AAP higher-risk criteria
Sensitivity, %	100.0 (95.4-100.0)	99.4 (96.6-100.0)
Specificity, %	6.7 (5.2-8.5)	7.3 (5.7-9.2)
PPV, %	8.1 (6.5-10.0)	16.6 (14.3-19.1)
NPV, %	100.0 (94.5-100.0)	98.5 (91.7-100.0)
LR+	1.07 (1.05-1.09)	1.07 (1.05-1.10)
LR−	0.00 (NA^a)	0.08 (0.01-0.60)
AUC	0.53 (0.53-0.54)	0.53 (0.52-0.54)
BRUE Prediction Rules (AUC)
Performance in the derivation cohort^b	0.61 (0.49-0.72)	0.68 (0.62-0.74)
Performance in the validation cohort
Original model	0.60 (0.54-0.67)	0.67 (0.62-0.72)
Recalibration in the large	0.61 (0.54-0.67)	0.67 (0.63-0.71)
Logistic recalibration	0.61 (0.55-0.67)	0.67 (0.62-0.72)
Model revision	0.71 (0.65-0.76)	0.69 (0.66-0.74)

Open in a new tab

Abbreviations: AUC, area under the curve; LR, likelihood ratio; NA, not applicable; NPV, negative predictive value; PPV, positive predictive value.

^{^a}

Calculation of the 95% CI for LR− cannot be performed when LR− is equal to 0.

^{^b}

Data from source.⁶

Figure 1. — The Brief Resolved Unexplained Events (BRUE) prediction rules (original model [blue] and after model revision [gray] are compared with the American Academy of Pediatrics [AAP] higher-risk criteria [orange]).

Figure 2. — Calibration of the prediction rules began with the original model and underwent 3 incremental modifications. First, recalibration in the large adjusted the original model’s intercept to align with the prevalence of the outcome in the Canadian validation dataset. Next, logistic recalibration modified both the intercept and slope. The final adjustment was a model revision, resulting in a new logistic regression with updated coefficients and intercept.

Event Recurrence

Event recurrence was noted in 163 patients (15.6%). For this secondary outcome, the AAP higher-risk criteria were highly sensitive at 99.4% (95% CI, 96.6%-100.0%), the specificity remained low at 7.3% (95% CI, 5.7%-9.2%) (Table 2), LR+ was 1.07 (95% CI, 1.05-1.10), and AUC was 0.58 (95% CI, 0.56-0.58). The estimated AUC for discrimination was 0.53 (95% CI, 0.52-0.54). The BRUE prediction rule’s discrimination was higher, with an AUC of 0.67 (95% CI, 0.62-0.72; P < .001), which was on par with the derivation cohort’s AUC of 0.68 (95% CI, 0.62-0.74). Calibration for the BRUE prediction rule was deemed acceptable (intercept: 0.15, slope: 0.82) (Figure 3), and subsequent adjustments through the 3-step model refinement did not markedly enhance discrimination (AUC = 0.69; 95% CI, 0.66 to 0.74; P = .05).

Figure 3. — Calibration of the prediction rules began with the original model and underwent three incremental modifications. First, recalibration in the large adjusted the original model’s intercept to align with the prevalence of the outcome in the Canadian validation dataset. Next, logistic recalibration modified both the intercept and slope. The final adjustment was a model revision, resulting in a new logistic regression with updated coefficients and intercept.

Sensitivity Analysis

In sensitivity analysis, transferred patients as well as those from 3 hospitals that only had access to admitted patients were excluded. A serious underlying diagnosis was noted in 33 of 691 patients (4.8%), and the discrimination of the BRUE prediction rules remained similar to the overall cohort, with an AUC of 0.60 (95% CI, 0.51-0.69) and 0.66 (95% CI, 0.58-0.75) after model revision (eTable 5 and eFigure 2 in Supplement 1). Calibration of the BRUE prediction rule was improved without evidence of underestimation of the risk (intercept: 0.14) (eFigure 3 in Supplement 1). Event recurrence was noted in 92 of 691 infants (13.3%), and the discrimination of the clinical prediction rules remained similar to the overall cohort, with an AUC of 0.67 (95% CI, 0.60-0.73) (eTable 5 and eFigure 2 in Supplement 1).

Discussion

In this large, multicenter cohort study of patients with BRUE, we found that the BRUE prediction rules outperformed the AAP higher-risk criteria on external geographical validation. Specifically, the BRUE prediction rule consistently underestimated risk to diagnose serious underlying condition. This underestimation was predominantly due to the higher incidence of these outcomes within our cohort than in the cohort used for model derivation. After appropriate model revisions, both discrimination and calibration metrics reached acceptable performance levels. For event recurrence, the original model demonstrated good performance not requiring revision. These BRUE prediction rules offer a valuable tool for clinicians, enabling the provision of individualized risk assessments. Consequently, health care professionals and families are better equipped to engage in informed discussions regarding potential risks, thereby guiding management decisions in alignment with their risk tolerance.

The findings from this study provide compelling evidence to reconsider the continued application of the AAP higher-risk criteria within the context of BRUE. Notably, the AAP criteria classified a significant majority (94%) of patients as higher risk, a trend that aligns with observations from various BRUE studies, where 74% to 93% of participants were similarly categorized as higher-risk.^{6,27,28,29,30,31} Furthermore, the discriminative ability of this stratification approach was markedly poor, approximating the efficacy of random chance for both evaluated outcomes. This observation echoes the results from a large US-based cohort study, which also reported notably inadequate discrimination by the AAP criteria.⁶ The propensity of the criteria to designate most individuals as higher risk may inadvertently escalate anxiety levels among health care providers and caregivers. Studies have shown that patients identified as higher risk under these criteria are more frequently subjected to hospital admissions and undergo diagnostic testing.^32,33 This trend reinforces the argument for discontinuing the use of the AAP stratification.³⁴

Regression models and clinical prediction rules often demonstrate optimal performance on the datasets from which they are derived.^9,21 Unfortunately, this performance does not always generalize to other datasets, frequently resulting in diminished accuracy.^23,35 The exponential increase in the derivation of these rules across clinical practices, and specifically in pediatric settings, has been accompanied by a concerning lack of external validation.^36,37,38,39 In our external geographical validation, the original BRUE prediction rules performed similarly in terms of discrimination to what was observed in the derivation cohort and even better for the primary outcome after model revision. Notably, the original model for serious underlying diagnosis initially underestimated risk. This underestimation was resolved after recalibration, likely due to the higher prevalence in our cohort, influenced by including patients from 3 hospitals that only had access to admitted patients. This inclusion likely led to a selection bias toward a population with a higher risk profile, thereby increasing the prevalence of serious underlying diagnoses. A sensitivity analysis, excluding these hospitals, demonstrated minimized risk underestimation. For clinical application concerning serious underlying outcomes, reliance on the revised BRUE prediction rule is recommended due to its improved performance. However, the original BRUE prediction rule is advised for event recurrence, given the minimal gains observed with revision. Despite its fair discrimination, the original model represents the best available evidence, significantly outperforming the AAP stratification. When compared with other commonly used pediatric prediction tools, the performance of the BRUE prediction rules was similar. For instance, the urinary tract infection (UTI) calculator, UTICalc (University of Pittsburgh), a tool for assessing UTI risk in febrile children, demonstrated an AUC of 0.73 (95% CI, 0.68-0.79) in recent external validation.⁹ Similarly, the AAP guidelines for febrile infants yielded an AUC of 0.67 (95% CI, 0.65-0.69) for the identification of invasive bacterial infections.⁴⁰

Incorporating precise risk percentages, as determined by clinical prediction rules, into clinical practice significantly enhances the nuanced understanding of patient risk, facilitating a more informed shared decision-making (SDM) process.⁴¹ This nuanced approach allows for a more tailored discussion between caregivers and health care providers, accommodating individual risk tolerance levels that broad risk categories cannot capture. For instance, a 5% risk of event recurrence might prompt some caregivers to opt for inpatient observation, whereas others may consider it low enough to monitor at home.^3,4 There has been a growing focus on incorporating this risk tolerance variability into clinical guidelines and practices. A notable example is the AAP’s guidelines for managing febrile infants.⁴² These guidelines exemplify how SDM is integral in making informed decisions about antibiotic use, the necessity of lumbar punctures, and hospital admissions, all while respecting the varied risk tolerances among caregivers. Embracing the wide range of risk tolerance enables health care professionals to create a more collaborative decision-making environment, ensuring that care plans are closely aligned with the unique needs of each family.

Strengths and Limitations

Our study presents several key strengths. Primarily, it provides the first, to our knowledge, external validation of the BRUE prediction rules. Contrasting with the AAP higher-risk criteria, which lacked prior validation, our findings highlight prior risk stratification’s limitations and underscore the superior performance of our clinical prediction rules. This validation draws on a large multicenter cohort across diverse Canadian health care settings, from tertiary to community hospitals, enhancing the generalizability of our results. Furthermore, our approach facilitates obtaining more precise risk estimates, offering a more nuanced assessment than mere risk stratification.

However, our study is not without limitations. First, the discrimination of the BRUE prediction rules, while the best evidence available to date, was only fair. Future prospective studies with better ascertainment of the predictors and outcomes may improve the BRUE prediction rules discrimination. Additionally, our sample size (and the number of events) was not sufficient to conduct more sophisticated machine-learning techniques. Second, the diverse and heterogeneous nature of serious underlying diagnoses presents a challenge, particularly for patients identified as higher risk, complicating the determination of specific tests required. Third, the practical application of these clinical prediction rules can be complex due to the necessity for logistic regression analysis. To address this, we have developed a user-friendly calculator on MDCalc, and future integration into the electronic health record could further facilitate their use.⁴³

Conclusions

In conclusion, in this large, multicenter cohort study, our BRUE prediction rules demonstrated significantly improved performance in identifying risks for serious conditions and event recurrence, compared with the widely applied AAP higher-risk criteria. Future studies should investigate if the use of the BRUE prediction rules, when integrated into SDM processes, can reduce unnecessary hospital admissions and excessive testing, while improving patient-centered outcomes like reducing caregiver anxiety and decisional conflict.

Supplement 1.

eTable 1. Checklist From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) Statement

eTable 2. Examples of Findings for Abnormal Medical History, Concerning Family History, and Concerning Social History

eTable 3. Primary Outcome—Serious Underlying Diagnosis Among Patients With a Brief Resolved Unexplained Event (BRUE)

eTable 4. BRUE Prediction Rules With Regression Coefficients and Model Intercept for Each of the Two Outcomes of Interest

eTable 5. Sensitivity Analysis—Area Under the Curve for the AAP Higher-Risk Criteria and the BRUE Prediction Rules

eFigure 1. Precision-Recall Curves

eFigure 2. Sensitivity Analysis—Receiver Operating Characteristic (ROC) Curves

eFigure 3. Sensitivity Analysis—Calibration Plots for the BRUE Prediction Rules

jamapediatr-e244399-s001.pdf^{(779.8KB, pdf)}

Supplement 2.

Nonauthor Collaborators. Canadian BRUE Collaboration (C-BRUE-C) and the Canadian Paediatric Inpatient Research Network (PIRN).

jamapediatr-e244399-s002.pdf^{(66.1KB, pdf)}

Supplement 3.

Data Sharing Statement.

jamapediatr-e244399-s003.pdf^{(16.6KB, pdf)}

References

1.Nama N, DeLaroche AM, Neuman MI, et al. Epidemiology of brief resolved unexplained events and impact of clinical practice guidelines in general and pediatric emergency departments. Acad Emerg Med. 2024;31(7):667-674. doi: 10.1111/acem.14881 [DOI] [PubMed] [Google Scholar]
2.Tieder JS, Bonkowsky JL, Etzel RA, et al. ; Subcommittee on Apparent Life-Threatening Events . Brief resolved unexplained events (formerly apparent life-threatening events) and evaluation of lower-risk infants. Pediatrics. 2016;137(5):e20160590. doi: 10.1542/peds.2016-0590 [DOI] [PubMed] [Google Scholar]
3.Khan A, Wallace SS, Sampayo EM, Falco C. Caregivers’ perceptions and hospital experience after a brief resolved unexplained event: a qualitative study. Hosp Pediatr. 2019;9(7):508-515. doi: 10.1542/hpeds.2018-0220 [DOI] [PubMed] [Google Scholar]
4.Maksimowski K, Haddad R, DeLaroche AM. Pediatrician perspectives on brief resolved unexplained events. Hosp Pediatr. 2021;11(9):996-1003. doi: 10.1542/hpeds.2021-005805 [DOI] [PubMed] [Google Scholar]
5.Nama N, DeLaroche AM, Gremse DA. Brief resolved unexplained event (BRUE): is reassurance enough for caregivers? Hosp Pediatr. 2022;12(12):e440-e442. doi: 10.1542/hpeds.2022-006939 [DOI] [PubMed] [Google Scholar]
6.Nama N, Hall M, Neuman M, et al. ; Brief Resolved Unexplained Event Research and Quality Improvement Network . Risk prediction after a brief resolved unexplained event. Hosp Pediatr. 2022;12(9):772-785. doi: 10.1542/hpeds.2022-006637 [DOI] [PubMed] [Google Scholar]
7.DeLaroche AM, Haddad R, Farooqi A, Sapién RE, Tieder JS. Outcome prediction of higher-risk brief resolved unexplained events. Hosp Pediatr. 2020;10(4):303-310. doi: 10.1542/hpeds.2019-0195 [DOI] [PubMed] [Google Scholar]
8.Merritt JL II, Quinonez RA, Bonkowsky JL, et al. A Framework for evaluation of the higher-risk infant after a brief resolved unexplained event. Pediatrics. 2019;144(2):e20184101. doi: 10.1542/peds.2018-4101 [DOI] [PubMed] [Google Scholar]
9.Smith AG, Kshetrapal A, Boles L, et al. External validation of the UTICalc with and without race for pediatric urinary tract infection. J Pediatr. 2023;263:113681. doi: 10.1016/j.jpeds.2023.113681 [DOI] [PubMed] [Google Scholar]
10.Adibi A, Sadatsafavi M, Ioannidis JPA. Validation and utility testing of clinical prediction models: time to change the approach. JAMA. 2020;324(3):235-236. doi: 10.1001/jama.2020.1230 [DOI] [PubMed] [Google Scholar]
11.Steyerberg EW, Harrell FE Jr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016;69:245-247. doi: 10.1016/j.jclinepi.2015.04.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Nama N, Hosseini P, Lee Z, et al. ; Canadian Paediatric Inpatient Research Network (PIRN) . Canadian infants presenting with Brief Resolved Unexplained Events (BRUEs) and validation of clinical prediction rules for risk stratification: a protocol for a multicenter, retrospective cohort study. BMJ Open. 2022;12(10):e063183. doi: 10.1136/bmjopen-2022-063183 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-73. doi: 10.7326/M14-0698 [DOI] [PubMed] [Google Scholar]
14.Moons KGM, Kengne AP, Grobbee DE, et al. Risk prediction models: II. external validation, model updating, and impact assessment. Heart. 2012;98(9):691-698. doi: 10.1136/heartjnl-2011-301247 [DOI] [PubMed] [Google Scholar]
15.DeLaroche AM, Hall M, Mittal MK, et al. ; Brief Resolved Unexplained Events Research and Quality Improvement Collaborative . Accuracy of diagnostic codes for identifying brief resolved unexplained events. Hosp Pediatr. 2021;11(7):726-749. doi: 10.1542/hpeds.2020-005330 [DOI] [PubMed] [Google Scholar]
16.Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res. 2011;46(3):399-424. doi: 10.1080/00273171.2011.568786 [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Antoon JW, Nian H, Ampofo K, et al. Validation of childhood pneumonia prognostic models for use in emergency care settings. J Pediatric Infect Dis Soc. 2023;12(8):451-458. doi: 10.1093/jpids/piad054 [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Cooray SD, De Silva K, Enticott JC, et al. Temporal validation and updating of a prediction model for the diagnosis of gestational diabetes mellitus. J Clin Epidemiol. 2023;164:54-64. doi: 10.1016/j.jclinepi.2023.08.020 [DOI] [PubMed] [Google Scholar]
19.Vergouwe Y, Nieboer D, Oostenbrink R, et al. A closed testing procedure to select an appropriate method for updating prediction models. Stat Med. 2017;36(28):4529-4539. doi: 10.1002/sim.7179 [DOI] [PubMed] [Google Scholar]
20.Cowley LE, Farewell DM, Maguire S, Kemp AM. Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature. Diagn Progn Res. 2019;3(1):16. doi: 10.1186/s41512-019-0060-y [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Bleeker SE, Moll HA, Steyerberg EW, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003;56(9):826-832. doi: 10.1016/S0895-4356(03)00207-5 [DOI] [PubMed] [Google Scholar]
22.Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19(4):453-473. doi: [DOI] [PubMed] [Google Scholar]
23.Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. doi: 10.1136/bmj.b605 [DOI] [PubMed] [Google Scholar]
24.Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167-176. doi: 10.1016/j.jclinepi.2015.12.005 [DOI] [PubMed] [Google Scholar]
25.Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12(1):77. doi: 10.1186/1471-2105-12-77 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Nama N, Lee Z, Picco K, et al. ; Canadian BRUE Collaboration (C-BRUE-C) and the Canadian Paediatric Inpatient Research Network (PIRN) . Identifying serious underlying diagnoses among patients with Brief Resolved Unexplained Events (BRUEs): a Canadian cohort study. BMJ Paediatr Open. 2024;8(1):e002525. doi: 10.1136/bmjpo-2024-002525 [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Evers KS, Wellmann S, Donner BC, Ritz N. Apparent life-threatening events and brief resolved unexplained events: management of children at a Swiss tertiary care center. Swiss Med Wkly. 2021;151(41-42):w30026. doi: 10.4414/SMW.2021.w30026 [DOI] [PubMed] [Google Scholar]
28.Jiménez TC, Romero PC, Sánchez IG. Replacing “Apparent Life Threatening Event” (ALTE) with “Brief Resolved Unexplained Event” (BRUE). A retrospective review of the ALTEs that meet the criteria of a BRUE. Pediatria (Napoli). 2020;93(4):261-262. doi: 10.1016/j.anpede.2019.12.010 [DOI] [PubMed] [Google Scholar]
29.Haddad R, Parker S, Farooqi A, DeLaroche AM. Diagnostic evaluation low yield for patients with a lower-risk brief resolved unexplained event. Glob Pediatr Heal. 2021;8:2333794X20967586. doi: 10.1177/2333794X20967586 [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Vigne MH, Moreau M, Gascoin G, Darviot E. Descriptive analysis of infant population younger than 1 year admitted for BRUE. Arch Pediatr. 2023;30(5):271-276. doi: 10.1016/j.arcped.2023.02.006 [DOI] [PubMed] [Google Scholar]
31.Duncan DR, Liu E, Growdon AS, Larson K, Rosen RL. A prospective study of brief resolved unexplained events: risk factors for persistent symptoms. Hosp Pediatr. 2022;12(12):1030-1043. doi: 10.1542/hpeds.2022-006550 [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Mittal MK, Tieder JS, Westphal K, et al. Diagnostic testing for evaluation of brief resolved unexplained events. Acad Emerg Med. 2023;30(6):662-670. doi: 10.1111/acem.14666 [DOI] [PubMed] [Google Scholar]
33.Tieder JS, Sullivan E, Stephans A, et al. ; Brief Resolved Unexplained Event Research and Quality Improvement Network . Risk factors and outcomes after a brief resolved unexplained event: a multicenter study. Pediatrics. 2021;148(1):e2020036095. doi: 10.1542/peds.2020-036095 [DOI] [PubMed] [Google Scholar]
34.Pitetti R. Defining risk factors for children following a BRUE: the need to revisit the AAP BRUE guideline. Pediatrics. 2021;148(1):e2021049933. doi: 10.1542/peds.2021-049933 [DOI] [PubMed] [Google Scholar]
35.Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375. doi: 10.1136/bmj.b375 [DOI] [PubMed] [Google Scholar]
36.Wang S, Huang H, Hou M, et al. Risk-prediction models for intravenous immunoglobulin resistance in Kawasaki disease: risk-of-bias assessment using PROBAST. Pediatr Res. 2023;94(3):1125-1135. doi: 10.1038/s41390-023-02558-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Rankin DA, Peetluk LS, Deppen S, et al. Diagnostic models predicting pediatric viral acute respiratory infections: a systematic review. BMJ Open. 2023;13(4):e067878. doi: 10.1136/bmjopen-2022-067878 [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Mangold C, Zoretic S, Thallapureddy K, Moreira A, Chorath K, Moreira A. Machine learning models for predicting neonatal mortality: a systematic review. Neonatology. 2021;118(4):394-405. doi: 10.1159/000516891 [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Ogero M, Sarguta RJ, Malla L, et al. Prognostic models for predicting in-hospital pediatric mortality in resource-limited countries: a systematic review. BMJ Open. 2020;10(10):e035045. doi: 10.1136/bmjopen-2019-035045 [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Yaeger JP, Richfield C, Schiller E, et al. Performance of AAP clinical practice guideline for febrile infants at 1 pediatric hospital. Hosp Pediatr. 2023;13(3):e47-e50. doi: 10.1542/hpeds.2022-006820 [DOI] [PMC free article] [PubMed] [Google Scholar]
41.Dudley N, Ackerman A, Brown KM, Snow SK; American Academy of Pediatrics Committee on Pediatric Emergency Medicine; American College of Emergency Physicians Pediatric Emergency Medicine Committee; Emergency Nurses Association Pediatric Committee . Patient- and family-centered care of children in the emergency department. Pediatrics. 2015;135(1):e255-e272. doi: 10.1542/peds.2014-3424 [DOI] [PubMed] [Google Scholar]
42.Pantell RH, Roberts KB, Adams WG, et al. ; Subcommittee on Febrile Infants . Evaluation and Management of Well-Appearing Febrile Infants 8 to 60 Days Old. Pediatrics. 2021;148(2):e2021052228. doi: 10.1542/peds.2021-052228 [DOI] [PubMed] [Google Scholar]
43.Tieder JS, Nama N, Hall M. Brief Resolved Unexplained Events 2.0 (BRUE 2.0) criteria for infants. Accessed July 5, 2024. https://www.mdcalc.com/calc/10400/brief-resolved-unexplained-events-2.0-brue-2.0-criteria-infants

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement 1.

eTable 1. Checklist From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) Statement

eTable 2. Examples of Findings for Abnormal Medical History, Concerning Family History, and Concerning Social History

eTable 3. Primary Outcome—Serious Underlying Diagnosis Among Patients With a Brief Resolved Unexplained Event (BRUE)

eTable 4. BRUE Prediction Rules With Regression Coefficients and Model Intercept for Each of the Two Outcomes of Interest

eTable 5. Sensitivity Analysis—Area Under the Curve for the AAP Higher-Risk Criteria and the BRUE Prediction Rules

eFigure 1. Precision-Recall Curves

eFigure 2. Sensitivity Analysis—Receiver Operating Characteristic (ROC) Curves

eFigure 3. Sensitivity Analysis—Calibration Plots for the BRUE Prediction Rules

jamapediatr-e244399-s001.pdf^{(779.8KB, pdf)}

Supplement 2.

Nonauthor Collaborators. Canadian BRUE Collaboration (C-BRUE-C) and the Canadian Paediatric Inpatient Research Network (PIRN).

jamapediatr-e244399-s002.pdf^{(66.1KB, pdf)}

Supplement 3.

Data Sharing Statement.

jamapediatr-e244399-s003.pdf^{(16.6KB, pdf)}

[poi240078r1] 1.Nama N, DeLaroche AM, Neuman MI, et al. Epidemiology of brief resolved unexplained events and impact of clinical practice guidelines in general and pediatric emergency departments. Acad Emerg Med. 2024;31(7):667-674. doi: 10.1111/acem.14881 [DOI] [PubMed] [Google Scholar]

[poi240078r2] 2.Tieder JS, Bonkowsky JL, Etzel RA, et al. ; Subcommittee on Apparent Life-Threatening Events . Brief resolved unexplained events (formerly apparent life-threatening events) and evaluation of lower-risk infants. Pediatrics. 2016;137(5):e20160590. doi: 10.1542/peds.2016-0590 [DOI] [PubMed] [Google Scholar]

[poi240078r3] 3.Khan A, Wallace SS, Sampayo EM, Falco C. Caregivers’ perceptions and hospital experience after a brief resolved unexplained event: a qualitative study. Hosp Pediatr. 2019;9(7):508-515. doi: 10.1542/hpeds.2018-0220 [DOI] [PubMed] [Google Scholar]

[poi240078r4] 4.Maksimowski K, Haddad R, DeLaroche AM. Pediatrician perspectives on brief resolved unexplained events. Hosp Pediatr. 2021;11(9):996-1003. doi: 10.1542/hpeds.2021-005805 [DOI] [PubMed] [Google Scholar]

[poi240078r5] 5.Nama N, DeLaroche AM, Gremse DA. Brief resolved unexplained event (BRUE): is reassurance enough for caregivers? Hosp Pediatr. 2022;12(12):e440-e442. doi: 10.1542/hpeds.2022-006939 [DOI] [PubMed] [Google Scholar]

[poi240078r6] 6.Nama N, Hall M, Neuman M, et al. ; Brief Resolved Unexplained Event Research and Quality Improvement Network . Risk prediction after a brief resolved unexplained event. Hosp Pediatr. 2022;12(9):772-785. doi: 10.1542/hpeds.2022-006637 [DOI] [PubMed] [Google Scholar]

[poi240078r7] 7.DeLaroche AM, Haddad R, Farooqi A, Sapién RE, Tieder JS. Outcome prediction of higher-risk brief resolved unexplained events. Hosp Pediatr. 2020;10(4):303-310. doi: 10.1542/hpeds.2019-0195 [DOI] [PubMed] [Google Scholar]

[poi240078r8] 8.Merritt JL II, Quinonez RA, Bonkowsky JL, et al. A Framework for evaluation of the higher-risk infant after a brief resolved unexplained event. Pediatrics. 2019;144(2):e20184101. doi: 10.1542/peds.2018-4101 [DOI] [PubMed] [Google Scholar]

[poi240078r9] 9.Smith AG, Kshetrapal A, Boles L, et al. External validation of the UTICalc with and without race for pediatric urinary tract infection. J Pediatr. 2023;263:113681. doi: 10.1016/j.jpeds.2023.113681 [DOI] [PubMed] [Google Scholar]

[poi240078r10] 10.Adibi A, Sadatsafavi M, Ioannidis JPA. Validation and utility testing of clinical prediction models: time to change the approach. JAMA. 2020;324(3):235-236. doi: 10.1001/jama.2020.1230 [DOI] [PubMed] [Google Scholar]

[poi240078r11] 11.Steyerberg EW, Harrell FE Jr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016;69:245-247. doi: 10.1016/j.jclinepi.2015.04.005 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r12] 12.Nama N, Hosseini P, Lee Z, et al. ; Canadian Paediatric Inpatient Research Network (PIRN) . Canadian infants presenting with Brief Resolved Unexplained Events (BRUEs) and validation of clinical prediction rules for risk stratification: a protocol for a multicenter, retrospective cohort study. BMJ Open. 2022;12(10):e063183. doi: 10.1136/bmjopen-2022-063183 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r13] 13.Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-73. doi: 10.7326/M14-0698 [DOI] [PubMed] [Google Scholar]

[poi240078r14] 14.Moons KGM, Kengne AP, Grobbee DE, et al. Risk prediction models: II. external validation, model updating, and impact assessment. Heart. 2012;98(9):691-698. doi: 10.1136/heartjnl-2011-301247 [DOI] [PubMed] [Google Scholar]

[poi240078r15] 15.DeLaroche AM, Hall M, Mittal MK, et al. ; Brief Resolved Unexplained Events Research and Quality Improvement Collaborative . Accuracy of diagnostic codes for identifying brief resolved unexplained events. Hosp Pediatr. 2021;11(7):726-749. doi: 10.1542/hpeds.2020-005330 [DOI] [PubMed] [Google Scholar]

[poi240078r16] 16.Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res. 2011;46(3):399-424. doi: 10.1080/00273171.2011.568786 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r17] 17.Antoon JW, Nian H, Ampofo K, et al. Validation of childhood pneumonia prognostic models for use in emergency care settings. J Pediatric Infect Dis Soc. 2023;12(8):451-458. doi: 10.1093/jpids/piad054 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r18] 18.Cooray SD, De Silva K, Enticott JC, et al. Temporal validation and updating of a prediction model for the diagnosis of gestational diabetes mellitus. J Clin Epidemiol. 2023;164:54-64. doi: 10.1016/j.jclinepi.2023.08.020 [DOI] [PubMed] [Google Scholar]

[poi240078r19] 19.Vergouwe Y, Nieboer D, Oostenbrink R, et al. A closed testing procedure to select an appropriate method for updating prediction models. Stat Med. 2017;36(28):4529-4539. doi: 10.1002/sim.7179 [DOI] [PubMed] [Google Scholar]

[poi240078r20] 20.Cowley LE, Farewell DM, Maguire S, Kemp AM. Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature. Diagn Progn Res. 2019;3(1):16. doi: 10.1186/s41512-019-0060-y [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r21] 21.Bleeker SE, Moll HA, Steyerberg EW, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003;56(9):826-832. doi: 10.1016/S0895-4356(03)00207-5 [DOI] [PubMed] [Google Scholar]

[poi240078r22] 22.Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19(4):453-473. doi: [DOI] [PubMed] [Google Scholar]

[poi240078r23] 23.Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. doi: 10.1136/bmj.b605 [DOI] [PubMed] [Google Scholar]

[poi240078r24] 24.Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167-176. doi: 10.1016/j.jclinepi.2015.12.005 [DOI] [PubMed] [Google Scholar]

[poi240078r25] 25.Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12(1):77. doi: 10.1186/1471-2105-12-77 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r26] 26.Nama N, Lee Z, Picco K, et al. ; Canadian BRUE Collaboration (C-BRUE-C) and the Canadian Paediatric Inpatient Research Network (PIRN) . Identifying serious underlying diagnoses among patients with Brief Resolved Unexplained Events (BRUEs): a Canadian cohort study. BMJ Paediatr Open. 2024;8(1):e002525. doi: 10.1136/bmjpo-2024-002525 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r27] 27.Evers KS, Wellmann S, Donner BC, Ritz N. Apparent life-threatening events and brief resolved unexplained events: management of children at a Swiss tertiary care center. Swiss Med Wkly. 2021;151(41-42):w30026. doi: 10.4414/SMW.2021.w30026 [DOI] [PubMed] [Google Scholar]

[poi240078r28] 28.Jiménez TC, Romero PC, Sánchez IG. Replacing “Apparent Life Threatening Event” (ALTE) with “Brief Resolved Unexplained Event” (BRUE). A retrospective review of the ALTEs that meet the criteria of a BRUE. Pediatria (Napoli). 2020;93(4):261-262. doi: 10.1016/j.anpede.2019.12.010 [DOI] [PubMed] [Google Scholar]

[poi240078r29] 29.Haddad R, Parker S, Farooqi A, DeLaroche AM. Diagnostic evaluation low yield for patients with a lower-risk brief resolved unexplained event. Glob Pediatr Heal. 2021;8:2333794X20967586. doi: 10.1177/2333794X20967586 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r30] 30.Vigne MH, Moreau M, Gascoin G, Darviot E. Descriptive analysis of infant population younger than 1 year admitted for BRUE. Arch Pediatr. 2023;30(5):271-276. doi: 10.1016/j.arcped.2023.02.006 [DOI] [PubMed] [Google Scholar]

[poi240078r31] 31.Duncan DR, Liu E, Growdon AS, Larson K, Rosen RL. A prospective study of brief resolved unexplained events: risk factors for persistent symptoms. Hosp Pediatr. 2022;12(12):1030-1043. doi: 10.1542/hpeds.2022-006550 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r32] 32.Mittal MK, Tieder JS, Westphal K, et al. Diagnostic testing for evaluation of brief resolved unexplained events. Acad Emerg Med. 2023;30(6):662-670. doi: 10.1111/acem.14666 [DOI] [PubMed] [Google Scholar]

[poi240078r33] 33.Tieder JS, Sullivan E, Stephans A, et al. ; Brief Resolved Unexplained Event Research and Quality Improvement Network . Risk factors and outcomes after a brief resolved unexplained event: a multicenter study. Pediatrics. 2021;148(1):e2020036095. doi: 10.1542/peds.2020-036095 [DOI] [PubMed] [Google Scholar]

[poi240078r34] 34.Pitetti R. Defining risk factors for children following a BRUE: the need to revisit the AAP BRUE guideline. Pediatrics. 2021;148(1):e2021049933. doi: 10.1542/peds.2021-049933 [DOI] [PubMed] [Google Scholar]

[poi240078r35] 35.Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375. doi: 10.1136/bmj.b375 [DOI] [PubMed] [Google Scholar]

[poi240078r36] 36.Wang S, Huang H, Hou M, et al. Risk-prediction models for intravenous immunoglobulin resistance in Kawasaki disease: risk-of-bias assessment using PROBAST. Pediatr Res. 2023;94(3):1125-1135. doi: 10.1038/s41390-023-02558-6 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r37] 37.Rankin DA, Peetluk LS, Deppen S, et al. Diagnostic models predicting pediatric viral acute respiratory infections: a systematic review. BMJ Open. 2023;13(4):e067878. doi: 10.1136/bmjopen-2022-067878 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r38] 38.Mangold C, Zoretic S, Thallapureddy K, Moreira A, Chorath K, Moreira A. Machine learning models for predicting neonatal mortality: a systematic review. Neonatology. 2021;118(4):394-405. doi: 10.1159/000516891 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r39] 39.Ogero M, Sarguta RJ, Malla L, et al. Prognostic models for predicting in-hospital pediatric mortality in resource-limited countries: a systematic review. BMJ Open. 2020;10(10):e035045. doi: 10.1136/bmjopen-2019-035045 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r40] 40.Yaeger JP, Richfield C, Schiller E, et al. Performance of AAP clinical practice guideline for febrile infants at 1 pediatric hospital. Hosp Pediatr. 2023;13(3):e47-e50. doi: 10.1542/hpeds.2022-006820 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi240078r41] 41.Dudley N, Ackerman A, Brown KM, Snow SK; American Academy of Pediatrics Committee on Pediatric Emergency Medicine; American College of Emergency Physicians Pediatric Emergency Medicine Committee; Emergency Nurses Association Pediatric Committee . Patient- and family-centered care of children in the emergency department. Pediatrics. 2015;135(1):e255-e272. doi: 10.1542/peds.2014-3424 [DOI] [PubMed] [Google Scholar]

[poi240078r42] 42.Pantell RH, Roberts KB, Adams WG, et al. ; Subcommittee on Febrile Infants . Evaluation and Management of Well-Appearing Febrile Infants 8 to 60 Days Old. Pediatrics. 2021;148(2):e2021052228. doi: 10.1542/peds.2021-052228 [DOI] [PubMed] [Google Scholar]

[poi240078r43] 43.Tieder JS, Nama N, Hall M. Brief Resolved Unexplained Events 2.0 (BRUE 2.0) criteria for infants. Accessed July 5, 2024. https://www.mdcalc.com/calc/10400/brief-resolved-unexplained-events-2.0-brue-2.0-criteria-infants

PERMALINK

External Validation of Brief Resolved Unexplained Events Prediction Rules for Serious Underlying Diagnosis

Nassr Nama, MD, MSc

Ye Shen, MPH

Jeffrey N Bone, MSc

Zerlyn Lee, MD, BSN

Kara Picco, MD

Falla Jin, BA

Jessica L Foulds, MD

Josée Anne Gagnon, MD

Chris Novak, MD, MEd

Brigitte Parisien, MD, MSc

Matthew Donlan, MD

Ran D Goldman, MD

Anupam Sehgal, MD

Joanna Holland, MD

Sanjay Mahant, MD

Joel S Tieder, MD, MPH

Peter J Gill, MD, DPhil

Key Points

Question

Findings

Meaning

Abstract

Importance

Objective

Design, Setting, and Participants

Exposures

Main Outcome and Measure

Results

Conclusions and Relevance

Introduction

Methods

Study Design

Cohort Identification

Outcomes

Comparing Derivation and Validation Cohorts

AAP Higher-Risk Criteria

Predictors

Performance

BRUE Prediction Rules

Predictors

Recalibration

Model Comparisons

Performance

Sensitivity Analysis

Sample Size

Statistical Analysis

Results

Study Population

Table 1. Demographic and Clinical Characteristics of Patients in the Brief Resolved Unexplained Events (BRUE) Group in the Derivation and Validation Cohorts.

Serious Underlying Diagnosis

Table 2. Performance Metrics of the American Academy of Pediatrics (AAP) Higher-Risk Criteria and the Brief Resolved Explained Events (BRUE) Prediction Rules.

Figure 1. Receiver Operating Characteristic Curves.

Figure 2. Calibration Plots for the Brief Resolved Unexplained Events (BRUE) Prediction Rule for Serious Underlying Diagnosis.

Event Recurrence

Figure 3. Calibration Plots for the Brief Resolved Unexplained Events (BRUE) Prediction Rule for Event Recurrence.

Sensitivity Analysis

Discussion

Strengths and Limitations

Conclusions

References

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases