Prospective Evaluation of Clinical Prediction Rules for Brief Resolved Unexplained Event

Abstract

Objectives:

Published clinical prediction rules propose to determine risks of event recurrence and serious underlying condition in infants with brief resolved unexplained event (BRUE). The objective of this study was to evaluate these rules in a prospectively recruited cohort of infants with BRUE to test the hypothesis that the rules may be helpful in predicting risk but their test characteristics may differ from those previously reported.

Methods:

To determine validity of prediction rules for risk of recurrent events and serious diagnoses, we applied the calculators to patients with BRUE who were prospectively recruited at our institution. Infants were recruited during their index hospitalization, caregiver-reported symptoms were obtained by questionnaires during the 6-month follow-up period, and charts reviewed for clinical data. Receiver operating characteristic (ROC) analyses were used to calculate values for the sensitivity and specificity of each calculator.

Results:

The cohort included 178 subjects and 73% (130) completed questionnaires. Overall, 78% (101/130) had persistent caregiver-reported symptoms, 12% (16/130) recurrent BRUE, and 28% (50/178) a serious condition diagnosis. The most common serious condition was oropharyngeal dysphagia with aspiration (78% of serious diagnoses, representing 22% of the cohort). On ROC analysis, area under the curve was 0.7 (95% CI 0.57–0.83, p=0.004) for recurrent BRUE and 0.59 (95% CI 0.49–0.68, p=0.08) for serious underlying condition. The calculators provided 81% sensitivity, 52% specificity, 20% positive predictive value, and 96% negative predictive value for predicting recurrent BRUE and 54% sensitivity, 68% specificity, 40% positive predictive value, and 79% negative predictive value for predicting a serious condition in our cohort.

Conclusions:

Our results suggest that published clinical prediction rules may accurately predict event recurrence but may have relatively low discriminatory power and tend to underestimate risk of serious underlying diagnoses, which may limit their clinical impact.

Introduction

A key challenge in caring for infants with brief resolved unexplained event (BRUE) is determining which patients might be at risk for clinically significant underlying pathology, recurrent symptoms or need for hospital admission^1–6. A recent large retrospective cohort study derived clinical prediction rules that considered patient and BRUE characteristics to determine risks of event recurrence and serious underlying condition⁷. Furthermore, the authors created a shared decision-making tool publicly available at mdcalc.com to guide clinicians and families in planning clinical care after a BRUE^{8, 9}. These clinical prediction rules were proposed to help hospitalists “provide education around BRUE and reassurance, engage in shared decision-making with the families if testing and hospital admission is a consideration, and consider the need for diagnostic testing”^{7, 8, 10}.

Pediatric gastroenterologists are frequently involved in caring for these children in the hospital or in follow-up after their hospitalization, and frequently guide testing and treatment related to gastroesophageal reflux disease (GERD) and oropharyngeal dysphagia for these patients^{11, 12}. Although prior studies have demonstrated high rates of aspiration in infants with BRUE, clinicians have not been able to reliably predict an aspiration diagnosis without videofluoroscopic swallow study (VFSS) due to the low sensitivity of observed clinical feeding evaluations resulting from the high rate of silent aspiration in infants^{11, 13–16}. Improved risk prediction in this patient population would be of significant benefit to help gastroenterologists differentiate between GERD and more serious diagnoses such as aspiration.

Recently, the original creators of the mdcalc.com risk calculator sought to provide external validation of these prediction rules using retrospectively collected data across centers in the US and Canada^{10, 17}. While there is a great need for reliable and straightforward approaches for determining clinical risk after BRUE, it is not yet clear if this risk calculator will provide valid results in a prospective cohort with patient reported outcomes, not subject to the biases of the electronic medical record review. Furthermore, the calculator is limited by: 1) the low frequency of reported serious diagnoses in the cohort that defined the predictions, 2) the limited definition of event recurrence as events taking place only during (but not after) the hospitalization period, and 3) lack of caregiver-reported symptom outcomes.

In the present study, we sought to evaluate the validity of these prediction rules by performing a secondary analysis of a prospective longitudinal cohort study of infants with BRUE. We hypothesized that the clinical prediction rules may be helpful in predicting risk of event recurrence and serious underlying condition in our cohort.

Methods

To determine the validity of the prediction rules for risk of recurrent events and serious diagnoses, we performed an interim analysis of an ongoing prospective longitudinal cohort study by applying the calculators to patients with BRUE who were recruited as part of this natural history study, the overall aims of which are to identify risk factors for persistent symptoms and repeat hospitalization^{12, 18}. Enrollment took place between April 2017 and November 2023. Infants were recruited at our institution if they were hospitalized with an admitting diagnosis of BRUE and no other clear explanation for their symptoms at the time of hospital admission. History of BRUE was determined based on the American Academy of Pediatrics (AAP) definition and verified by the clinical team caring for each subject⁵. Exclusion criteria included previous BRUE hospitalization or any preexisting medical diagnosis. Clinical data were obtained from chart review and caregiver-reported symptoms were obtained by questionnaires administered during the 6-month follow-up period as previously described^{12, 18}. This was an observational, non-blinded cohort study.

Risk prediction calculators were retrospectively accessed at mdcalc.com and utilized following provided instructions⁸. Formulas are reported to have been derived from and refined by two different cohort studies and are shown in the supplement^{7, 10}. Each calculator uses clinical characteristics as inputs to calculate percent chance of each outcome including percentage risk of serious underlying condition and percentage risk of recurrent event. These clinical characteristics include age, prematurity, history of similar events, event clusters, color change, tone change, abnormal respiratory pattern and abnormal medical history, as shown in Supplemental Digital Content 1.

We entered values for subjects in our cohort to obtain their calculated risks. We first compared risk of serious underlying condition obtained by the calculator with actual serious diagnoses made in the prospective cohort. Serious underlying condition was defined as in prior studies as “a condition for which a delay in diagnosis or treatment could potentially increase morbidity or mortality”^{6, 7, 10}. We next compared the risk of recurrent symptoms obtained by the calculator with actual recurrence of BRUE reported by caregivers on questionnaires following the index hospitalization. Subjects missing follow-up questionnaire data were excluded from the recurrent symptom analysis.

We next performed exploratory analyses to evaluate secondary outcomes in our cohort, including 1) parent-reported persistent BRUE symptoms, 2) repeat hospital visit, and 3) diagnosis of the most frequent serious condition in our cohort, aspiration. Parent-reported persistent symptoms were defined as recurrent BRUE spells in addition to coughing and choking that were concerning to caregivers as reported in prior studies^{12, 18}. Repeat hospital visit was determined by chart review and defined as any hospital visits (admissions or emergency room visits) for recurrent BRUE or related symptoms (e.g. respiratory distress) as previously described¹⁸. Aspiration diagnoses were determined solely by an abnormal VFSS and requiring intensive feeding therapy as defined by Nama et al^{10, 11, 13, 18}. VFSS were considered abnormal if aspiration or laryngeal penetration was observed for any liquid consistency; laryngeal penetration was considered abnormal because patients with laryngeal penetration have similar symptoms and outcomes to those with overt aspiration and respond to feeding therapies such as thickening^19–22. Subjects that did not undergo VFSS were assumed not to aspirate.

In our primary analysis, calculated prediction results were compared to parent-reported recurrent BRUE and serious diagnoses using the student’s t test. Receiver operating characteristic (ROC) analyses were used to determine optimal cut-off values based on Youden’s index and to calculate values for the sensitivity, specificity, positive predictive value and negative predictive value of each calculator against the gold standard actual rates in our prospective cohort^{23, 24}. ROC analyses were also used for our secondary outcomes. All statistical tests were 2-sided with p<0.05 considered statistically significant. Data were analyzed using SPSS version 27.

Ethics Statement

This study was approved by the Boston Children’s Hospital Institutional Review Board and registered on ClinicalTrials.gov (NCT04477460).

Results

Our cohort included 178 subjects with cohort characteristics shown in Table 1. Ninety-five percent of infants (n=173) were under 6 months of age at enrollment. Follow-up questionnaires were completed for 73% (130/178) of subjects. A flow diagram of the cohort is shown in Supplemental Figure S1. Baseline characteristics did not significantly differ for the subjects that did not complete follow-up assessments (p=0.8 for age, p=0.6 for sex, p=0.7 for race, p=0.4 for ethnicity, p=0.9 for prematurity, p=0.93 for weight, p=0.67 for weight z-score, p=0.09 for NICU/ICU admission, p=0.4 for length of stay).

Table 1:

Cohort Characteristics

	Cohort (n=178) Median (IQR), Mean ± SE or n (%)
Subject Characteristics
Age at Presentation	29.5 (50.3) days
Male Sex	80 (45%)
Race
White	104 (58%)
Not Reported	26 (15%)
Other	22 (12%)
Black	21 (12%)
Asian	5 (3%)
Ethnicity
Hispanic	18 (12%)
Non-Hispanic	160 (88%)
History of Prematurity	49 (28%)
Gestational Age at Birth	34.8 ± 0.3 weeks
Weight	4.4 ± 0.1 kg
Weight Z-Score	−0.7 ± 0.1
Admitted to NICU/ICU	25 (14%)
Length of Hospital Stay	1.6 (1.7) nights

IQR: interquartile range; SE: standard error

Overall, 12% (16/130) had recurrent BRUE and 28% of the total cohort (50/178) were diagnosed with a serious condition (Table 2). Additionally, 78% (101/130) had persistent caregiver-reported symptoms and 16% (28/178) had repeat hospital visit. The most common serious condition diagnosed was oropharyngeal dysphagia with aspiration (78% of serious diagnoses, representing 22% of the cohort). Of note, 60% (30/50) of serious diagnoses were made during the index hospitalization, as shown in Table 2.

Table 2:

Recurrent Event and Serious Condition Outcomes

	Cohort (n=178) Mean ± SE or n (%)
Recurrent Events
Calculated Risk of Recurrent Event	19.8 ± 0.8%
Actual Recurrent BRUE	16/130 (12%)
Actual Persistent Symptoms	101/130 (78%)
Actual Repeat Hospital Visit	28/178 (16%)
Serious Conditions
Calculated Risk of Serious Condition	7.7 ± 0.3%
	Overall	Diagnosed at Index Hospitalization
Actual Serious Condition Diagnosed	50/178 (28%)	30/50 (60%)
Gastrointestinal
Oropharyngeal Dysphagia with Aspiration	39/50 (78%)	25/39 (64%)
Feeding Difficulties Requiring NG	3/50 (6%)	1/3 (33%)
FPIES	2/50 (4%)	1/2 (50%)
Neurological
Seizure	3/50 (6%)	0/3 (0%)
Pulmonology
Central Apnea	2/50 (4%)	2/2 (2%)
Otolaryngology
Laryngomalacia Requiring Supraglottoplasty	1/50 (2%)	1/1 (100%)

SE: standard error; BRUE: brief resolved unexplained event; NG: nasogastric tube; FPIES: food protein-induced enterocolitis syndrome. Note: denominators for persistent symptoms and recurrent BRUE represent participants that responded to relevant question.

We first evaluated predictions from the recurrent event calculator. The mean calculated risk of recurrent event was 19.8 ± 0.8% in our cohort. The recurrent event calculator predicted 26.5 ± 2.9% risk of recurrent events for the 16 subjects who all went on to have a recurrent BRUE and 19.0 ± 1.0% risk for the 114 subjects who did not have recurrent BRUE (p=0.008) as shown in Figure 1. On ROC analysis, area under the curve (AUC) was 0.7 (95% CI 0.57–0.83, p=0.004) for recurrent BRUE as in Figure 2. A cut-off of 18.75 from the calculator provides 81% sensitivity, 52% specificity, 20% positive predictive value, and 96% negative predictive value for predicting recurrent BRUE.

Figure 1:

Distribution of Calculated Clinical Risks For Subjects with Each Outcome

Figure 2:

ROC Curves Comparing Calculated Risks with Actual Outcomes

Notes: AUC: area under the curve; BRUE: brief resolved unexplained event; CI: confidence interval; ROC: receiver operative characteristic

We next evaluated predictions from the serious underlying condition calculator. The mean calculated risk of a serious condition was 7.7 ± 0.3% in our cohort. When entering values for our subjects, the calculator predicted 8.8 ± 0.7% risk of serious condition for the 50 subjects who all went on to have a serious condition diagnosed and 7.3 ± 0.4% for the 128 subjects who did not have a serious condition diagnosed (0.05) as shown in Figure 1. We also evaluated the serious condition calculator with exclusion of the aspiration; however, results were similar even with exclusion of the aspiration diagnosis (9.2 ± 1.4 vs 7.6 ± 0.4, p=0.17). On ROC analysis, AUC was 0.59 (95% CI 0.49–0.68, p=0.08) as in Figure 2. A cut-off of 5.4 from the serious condition calculator provides 54% sensitivity, 68% specificity, 40% positive predictive value, and 79% negative predictive value for predicting a serious condition. Figure 3 shows a stacked bar graph with frequency of actual serious diagnoses compared by calculated risk score.

Figure 3:

Visualizations of Calculated Risks of Serious Diagnoses with Actual Outcomes

For our exploratory analyses, we then evaluated 1) the recurrent event calculator with caregiver reported persistent symptoms and repeat hospital visits and 2) the serious underlying condition calculator with the most frequent serious condition in our cohort, aspiration. On ROC analysis, AUC was 0.56 (95% CI 0.46–0.67, p=0.24) for persistent symptoms. A cut-off of 30.6 from the recurrent event calculator provides 26% sensitivity, 93% specificity 81% positive predictive value, and 25% negative predictive value for predicting persistent symptoms. On ROC analysis with repeat hospital visit, AUC was 0.58 (95% CI 0.47–0.69, p=0.15). A cut-off of 11.5 from the recurrent event calculator provides 82% sensitivity, 34%, 19% positive predictive value, and 87% negative predictive value specificity for predicting repeat hospital visit. On ROC analysis for the aspiration diagnosis, AUC was 0.6 (95% CI 0.5–0.7, p=0.05) A cut-off of 5.4 from the serious condition calculator provides 57% sensitivity, 68% specificity, 36% positive predictive value, and 83% negative predictive value for predicting aspiration. ROC curves from our exploratory analyses are shown in Supplemental Figure S2. Figure 3B shows a scatter plot of calculated risk of serious underlying diagnosis scores with aspiration diagnoses highlighted.

Discussion

We used subjects enrolled in our ongoing prospective longitudinal cohort study to evaluate clinical prediction rules designed to estimate risk of serious diagnoses and recurrent events after BRUE. Improving clinicians’ ability to provide an accurate assessment of these risks would greatly inform discussions with caregivers and add crucial data to support shared decision making for infants with BRUE^{9, 25}. Gastroenterologists are frequently involved in caring for these children and are uniquely positioned to help guide testing and treatment related to gastroesophageal reflux disease and oropharyngeal dysphagia for these patients, particularly as they related to risk of recurrent events and serious underlying conditions that may require gastroenterologist expertise^{11, 12}. Our objective in this prospective validation study was to refine our understanding of how well these rules perform when applied to an independent, prospective cohort of patients that included patient-reported outcomes. Our results suggest the calculator tool may accurately predict recurrent event, but it may significantly underestimate the risk of serious conditions in our cohort.

We first evaluated risk of recurrent event using the online calculator. We found modest sensitivity (81%) and low specificity (52%) with an area under the curve that closely matches that reported by Nama et al (AUC 0.7 in our cohort vs 0.69 in theirs) when using a cut-off value of 18.75¹⁰. While there may be statistically significant differences between values, the actual numbers (i.e. 26% vs 19%) may not appear clinically different to clinicians and families when interpreting risk values as currently presented and therefore the calculator may have limited clinical utility as currently presented. Future versions of the calculator may need to consider presenting categories of risk or cut-offs to aid in interpretation.

We next evaluated the risk of a serious diagnosis using the online calculator. In our cohort, we found lower sensitivity (54%) and specificity (68%) compared to those reported by Nama et al (AUC 0.59 in our cohort vs 0.71 in theirs) when using a cut-off value of 5.4¹⁰. We acknowledge that our results include wide confidence intervals on these estimates, likely due to our sample size, and may be unable to show if discrimination is adequate for the serious underlying diagnosis outcome. However, given the clinical importance of this outcome and the extent to which it may inform clinician and caregiver discussions regarding any need for additional diagnostic evaluations, the risk of serious diagnosis calculator may need to be tested by larger prospective cohorts and further refined before incorporation into clinical practice. We note that when the aspiration diagnosis is excluded, the actual risk of serious diagnosis in our cohort is similar to other cohorts and that which is predicted by the calculator (7–8%); however, as discussed further below, missing the diagnosis of aspiration is a significant limitation of the calculator as currently presented to clinicians.

We acknowledge that the calculators were designed for prediction of recurrent BRUE and serious diagnosis. However, our prior work suggests that there are other common clinical outcomes that have clinical relevance when setting families’ expectations following BRUE: 1) persistent symptoms of coughing and choking and 2) repeat hospital visit^{12, 18}. For these reasons, we performed exploratory analyses by evaluating the clinical prediction rules for these outcomes. Our results suggest the recurrent event calculator does not effectively discriminate infants likely to have persistent symptoms or repeat hospital visit. Therefore, prediction results do not appear to have relevance to these outcomes.

We additionally performed an exploratory analysis using the serious diagnosis calculator to predict the aspiration diagnosis since this was the most common serious diagnosis in our cohort. Our results suggest that the sensitivity of the tool is too low for reliably predicting an aspiration diagnosis without VFSS, findings consistent with other reports on the low sensitivity of clinical feeding evaluations and the high rate of silent aspiration in infants with BRUE^{12–16, 18, 26}. For these reasons, aspiration and feeding-related pathology may be less likely to be considered in the work-up of BRUE patients and underrepresented in the risk prediction models, despite being found to be relatively high risk condition in our cohort. Clinicians may need to consider the potentially serious consequences of missing the aspiration diagnosis when relying on the calculator as currently presented, as treatment for aspiration has been shown to improve outcomes and decrease repeat hospitalization^{11, 12, 18, 19, 27–29}. Therefore, infants with a low calculated risk of serious diagnosis may still have aspiration that needs to be evaluated and treated to improve symptoms. Future work may identify parameters that could improve the performance of the tool in predicting aspiration, but at this point, this calculator lacks the sensitivity needed to identify patients at risk for aspiration who may need an urgent referral for VFSS.

Strengths of our study include our prospective longitudinal cohort design which included caregiver-reported outcomes and a focus on additional clinically-relevant outcomes beyond recurrent BRUE, including persistent concerning symptoms after discharge, repeat hospital visits, and the aspiration diagnosis. Our cohort was comparable to that used to derive the clinical prediction rules with a similar proportion of patients that went on to have recurrent BRUE. However, our results suggest an event higher proportion of infants likely have other persistent symptoms that may be concerning to caregivers and a meaningful outcome for clinicans^{18, 30, 31}.

In their use of the calculator, families and clinicians may reasonably interpret “low risk of recurrence” and “low risk of serious diagnosis” as essentially meaning a given infant will not have any ongoing symptoms or diagnoses. This is also suggested by the language in the “visualize risk” section of the online tool, which describes low risk infants as “most are healthy” which may be misleading and lead to symptom misattribution^{8, 11, 26, 32–34}. This may be a limitation in generalizability of the dataset from which these rules were derived. Future versions of the calculator may need to acknowledge these limitations and incorporate other clinically relevant outcomes including persistent concerning symptoms and repeat hospital visit. In the present study, we utilized Youden’s index to obtain test cutoffs and sensitivity and specificity values; however, this approach may not reflect all clinicians’ perspectives on the clinical value of these calculators³⁵. Future studies may include developing a multivariable prediction model using variables from this and prior studies, including exploratory analysis to identify new predictors, evaluating whether any score elements are associated with timing of recurrence or adverse outcomes, and assessing whether the prediction score correlates with response to commonly used interventions, such as feed thickening. Based on the results presented in the present study, a sample size of 382 subjects would be required to provide 90% power to develop a test with 90% sensitivity and 85% specificity for diagnosing a serious condition, assuming 8% prevalence and 20% dropout³⁶.

Our study has several limitations that should be acknowledged. These primarily relate to differences between our study design and the cohorts from which the clinical prediction rules were designed, including our smaller cohort size and the use of the calculators for exploratory analyses that were not their intended use. We acknowledge that some of our results include wide confidence intervals due to sample size and statistical power and therefore our findings in regards to risk of persistent symptoms, repeat hospital visit, and the aspiration diagnosis should be interpreted with caution. Additionally, the low prevalence of other diagnoses apart from aspiration may have limited our assessment of the serious diagnosis risk calculator. We also note that the calculators were derived from repeat events taking place during the index BRUE hospitalization, but recurrent concerning symptoms after discharge may also be a meaningful outcome for clinicians and families. Lastly, we acknowledge that AAP risk stratification may still be in use clinically but evaluating these criteria was not an aim of this study as these have already been extensively evaluated^{5, 7, 10}.

Conclusion

In summary, our results suggest clinicians should be mindful of the limitations of proposed prediction rules. The recurrent event calculator can inform risk of recurrent BRUE but cannot reliably predict which patients might have higher or lower risk of other concerning symptoms or higher or lower risk of having symptoms severe enough to need to return to the hospital. The serious diagnosis calculator has relatively low discriminatory power, tends to underestimate risk of serious underlying diagnoses, and infants with a low calculated risk of serious diagnosis may still have aspiration that needs to be evaluated and treated. In particular, gastroenterologists should be aware of these limitations as they relate to potential GI explanatory diagnoses when asked to provide consultation on infants with BRUE. Future work may be needed in order to improve their prognostic value in infants with BRUE. Refinements to include feeding-related variables may also improve future prediction tools. Therefore, despite their potential, these calculators and their proposed utilization in shared decision-making should be used with caution.

Supplementary Material

Supplementary Material

Supplemental Digital Content 1: Formulas for Calculating Risk of Serious Underlying Condition and Risk of Recurrent Event

Supplemental Figure S1: Cohort Flow Diagram

Supplemental Figure S2: ROC Curves Comparing Calculated Risks with Actual Outcomes for Exploratory Analyses, Showing that the BRUE Calculator Did Not Strongly Predict Persistent Symptoms, Repeat Hospitalization Or Aspiration

A. Calculated Risk of Recurrent Event vs Persistent Symptoms: AUC 0.56 (95% CI 0.46–0.67, p=0.24) for persistent symptoms. A cut-off of 30.6 from the recurrent event calculator provides 26% sensitivity and 93% specificity for predicting persistent symptoms.

B. Calculated Risk of Recurrent Event vs Repeat Hospital Visit: AUC 0.58 (95% CI 0.47–0.69, p=0.15). A cut-off of 11.5 from the recurrent event calculator provides 82% sensitivity and 34% specificity for predicting repeat hospital visit.

C. Calculated Risk of Serious Condition vs Aspiration Diagnosis: AUC 0.6 (95% CI 0.5–0.7, p=0.05). A cut-off of 5.4 from the serious condition calculator provides 57% sensitivity and 68% specificity for predicting aspiration.

AUC: area under the curve; CI: confidence interval; ROC: receiver operative characteristic

What is Known/What is New.

What is Known:

• A key challenge in caring for infants with brief resolved unexplained event (BRUE) is determining which patients might be at risk for clinically significant underlying pathology and recurrent symptoms

• Recent clinical prediction rules were designed to determine risks of event recurrence and serious underlying condition after BRUE but the validity of these prediction rules has not been evaluated in a prospective cohort

What is New:

• Clinical prediction rules may accurately predict event recurrence but have relatively low discriminatory power, underestimate risk of serious underlying diagnoses, and in particular underestimate the risk of aspiration that needs to be evaluated and treated