Validation of a scale to assess the severity of bronchiolitis in a population of hospitalized infants

Abstract

Background

Although assessment of the severity of bronchiolitis using severity scores is important both in daily practice and as an outcome measure in clinical trials, many of these scores have not been formally validated or have been only partially validated.

Methods

We conducted a prospective cohort study on a sample of children diagnosed with bronchiolitis. Two physicians independently assessed all of the children on the modified Wood’s Clinical Asthma Score (M-WCAS) and on the Tal et al. severity score and collected the information required to assess the criterion validity, construct validity, inter-rater agreement, sensitivity to change, and usability of the M-WCAS.

Results

The median (interquartilic range [IQR]) of the age of the 54 patients included in the study was 5 (2–9) months. Thirty (55.6%) of the patients were males and 24 (44.4%) were female. The scores of the M-WCAS correlated positively with the scores of the Tal et al. severity score (ρ = 0.761, p < 0.001). The scores of the M-WCAS in patients who required subsequent admission to the PICU were significantly higher than those in patients who required admission only to the pediatric medical floor (PMF) [4.5 (3.6–5.2) vs. 2.5 (1.5–2.5), p < 0.001]. The inter-rater agreement for the raters was found to be κ = 0.897 (p < 0.001), 95% CI (0.699–1.000). The scores of the M-WCAS in patients at admission to the PMF were significantly higher than those obtained immediately before discharge from the hospital [2.5 (1.9–2.5) vs. 1.0 (0.5–1.6), p < 0.001).

Conclusions

Our results suggest that the M-WCAS severity score has adequate criterion validity, adequate construct validity, adequate inter-rater agreement, adequate sensitivity to change, and appropriate usability for infants hospitalized for acute bronchiolitis.

Introduction

Acute bronchiolitis is the leading cause of lower respiratory tract infection in the first year of life and the most common cause of admission to the hospital for infants beyond the neonatal period [1]. Although the majority of affected children have a self-limited condition and can be managed as outpatients, over the last 20 years the rate of hospitalization for patients with bronchiolitis has reportedly increased [2]. Even though it is still not clear whether a greater severity of the disease plays any significant role in the increase in hospitalization rates over time [3], the assessment of the severity of bronchiolitis is important both in daily practice (for clinical decision-making and for evaluation of treatment effectiveness) and as an outcome measure in clinical trials. Although pulmonary function testing is helpful in determining the severity of airway obstruction, it is often neither feasible nor easily accessible in infants. For these reasons, validated respiratory severity scores consisting of a combination of clinical symptoms and physical signs can be helpful in assessing the severity of bronchiolitis [4,5]. One of these severity scores that has been already used for assessing disease severity and response to therapy in infants with bronchiolitis is the modified Wood’s Clinical Asthma Score (M-WCAS) [6], an apparently appropriate instrument for the assessment of the severity of acute bronchiolitis.

The validation of severity scores (and other outcome measures) is an essential process for assessing their measurement properties to ensure that they measure what it is intended to be measured [7]. This validation process should include assessment of reliability (internal consistency, test–retest reliability, and inter-rater agreement), validity (face validity, construct validity, and criterion validity), responsiveness, and usability [7]. However, despite the undoubted importance of this validation process and the widespread use of severity scores in bronchiolitis not only for clinical purposes but also in a research context, many of them have not been formally validated or have been only partially validated. The validation process of those that have been validated have often been limited to assessing the inter-rater agreement [8,9]. Additionally, many of these severity scores have not been applied to different populations and clinical settings (the emergency room, inpatient, outpatient clinics).

The aim of this study is to assess the reliability, validity, responsiveness, and usability of the M-WCAS in a population of infants hospitalized with a clinical diagnosis of bronchiolitis.

Methods

Study population

The study was undertaken at the Hospital Santa Clara (HSC), a third-level University-based multidisciplinary teaching hospital located in Bogota, Colombia. Children admitted via the Emergency Department (ED) of the HSC from April 2010 to July 2011 for acute bronchiolitis were eligible for the study. Bronchiolitis was operationally defined as the first episode of lower airway obstruction (chest wall retractions, wheezing, or ronchi) preceeded by or associated with upper respiratory infection symptoms. Children known to have any condition other than bronchiolitis that might affect the cardiopulmonary status (e.g. chronic lung disease, congenital heart disease, or metabolic acidosis), those with mild disease but admitted mainly for social factors (e.g. a lack of recognition by parents of the early signs of deterioration in their child’s clinical status, geographical factors, and transport difficulties), those admitted only for a history of apnea, and those who required intubation on arrival to the ED were excluded from the study.

Study design and procedures

We conducted a prospective cohort and a validation study by following a convenience sample of children aged younger than 24 months admitted to HSC who fulfilled the eligibility criteria (inclusion and exclusion criteria), and whose parents agreed to participate and signed the informed consent. Criteria for hospitalization were determined by the ED physicians independently of the study team. In general, these children had signs of respiratory distress of variable intensity, required frequent nebulization treatments (more frequent than every 4 h) or oxygen supplementation, or both.

For each child, standardized forms were used to collect baseline data, including demographic, medical history, and clinical information. Children were assessed with the M-WCAS. Although the M-WCAS has been used previously mainly in the context of the pediatric intensive care unit (PICU), in this study we chose to evaluate this respiratory severity score because we considered it to be a simple, well-categorized, and a clinically appropriate tool for the assessment of the severity of acute bronchiolitis. Moreover, the M-WCAS includes “mild” categories of 0.5 points to better define the clinical response to therapy [6]. To assess the M-WCAS’s criterion validity, children were also assessed with the Tal et al. severity score, because it has been previously validated in patients with bronchiolitis [10]. Two physicians independently assessed all of the children on both respiratory scores upon admission to the pediatric medical floor (PMF), and one of them also assessed all of the children on both respiratory scores immediately before discharging each one from the hospital. Raters were instructed not to share their findings with each other, in order to remain blind to the other rater’s assessments. All assessments were done between15 min and 2 h of each other, with no change in medical treatment between assessments, reducing the probability of an important change in the clinical condition of the child. The first rater was a pediatric pulmonologist, and the second rater was either a clinical resident in pediatric pulmonology or a pediatrician. The severity scores were not used in place of routine care or to make treatment decisions for patients.

The 5-item M-WCAS has been previously used for assessing the severity and the clinical response to therapy in patients with bronchiolitis [11]. The M-WCAS is a clinical scoring system that takes into account oxygen saturation, inspiratory breath sounds, expiratory wheezing, use of accessory muscles, and mental status, with scores for each sign ranging from zero to two, including “mild” categories of 0.5 points (Table 1). All of these scores were added to individual item scores to generate a summary unweighted bronchiolitis severity score. The possible total score ranged from 0 to 10, with a higher score indicating more respiratory distress. A score of 0–3 represents a mildly ill infant, 4–6 moderately ill, and greater than 6 severely ill. All measurements were taken when the child was awake and not crying.

Table 1.

Modified Wood’s Clinical Asthma score (M-WCAS).

	0	0.5	1	2
SatO2	SatO2 ≥ 95% in room air	95% > SatO2 > 90% in room air	SatO2 ≥ 90%, with FiO2 > 0.21	SatO2 < 90, with FiO2 > 0.21
Inspiratory breath sounds	Normal	Slightly unequal	Markedly unequal	Decreased/absent
Expiratory wheezing	None	Mild	Moderate	Marked
Accessory muscles	None	Mild	Moderate	Maximal
Cerebral function	Normal	Agited when disturbed	Depressed/agitated	Markedly depressed, coma

The 4-item Tal et al. severity score consists of respiratory rate, wheeze, cyanosis, and accessory respiratory muscle utilization, with scores for each sign ranging from 0 to 3 (Table 2). The score was in the range 0–12, with a higher score indicating more respiratory distress. Again, all measurements were taken when the child was awake and not crying.

Table 2.

Tal et al. severity score.

Score	Respiratory rate (breath/min)	Wheeze	Cyanosis	Accessory respiratory muscle utilization
0	<30	None	None	None (no chest in-drawing), i.e. absence of lower part of the chest moves in or retracts when inhalation occurs
1	30–45	Terminal expiratory only	Peri-oral on crying only	Presence of mild intercostal in-drawing (just visible), no head bobbing or tracheal tug
2	46–60	Entire expiration and inspiration with stethoscope only	Perioral in rest	Moderate amount of intercostal in-drawing, no head bobbing or tracheal tug
3	>60	Entire expiration and inspiration without stethoscope only	Generalized in rest	Moderate or marked intercostal in-drawing with presence of head bobbing or tracheal tug

The raters who assessed children on both respiratory scores were briefed on the scoring system of each severity score and the purpose of the study, but no formal training about the use of the severity scores was provided, in order to more closely approximate a “real-life” scenario.

Saturation of oxygen (SatO₂) was measured with a pulse-oximeter (Nellcor N-595 Pulse Oximeter) with the patient quietly awake. The maximum SatO₂ was read for each child after the pulse-oximeter was attached to the patient for at least 60 s.

Study methods were approved by the hospital’s Ethics Committee, and children were enrolled after informed consent was obtained from their parents.

Assessment of the psychometric characteristics of the severity score

To assess the M-WCAS’s criterion validity (i.e. the degree to which the measurement correlates with some other measure of the specific construct of the severity of bronchiolitis, such as another validated severity instrument or another “gold standard” for the severity of bronchiolitis), we correlated scores of the M-WCAS with scores of the Tal et al. severity score. Raters were asked to assess each patient’s respiratory status using both severity scores at the same time.

To assess the M-WCAS’s construct validity (i.e. the degree to which the measurement corresponds to other variables and measures that are not identical to the construct of the severity of bronchiolitis but to which the construct of the severity of bronchiolitis should be related), scores of the M-WCAS in patients who required subsequent admission to the Pediatric Intensive Care Unit (PICU) were compared to scores in the patients who required admission only to the PMF.

To assess the M-WCAS’s inter-rater agreement (i.e. the degree to which two or more observers independently obtain similar results), two raters independently assessed all of the children’s respiratory status at admission to the PMF using the M-WACS, and each physician scored each patient independently and without knowledge of the other’s scores. All assessments were done between 15 min and 2 h of each other, with no change in medical treatment between assessments. We did not evaluate the test–retest reliability, because it has been reported that this psychometric characteristic may have limited meaning in the context of scores developed to measure change in acute clinical conditions [12].

To assess the M-WCAS’s sensitivity to change (i.e. the ability of a score to detect a clinically important change over time), we compared scores of the M-WCAS at admission to the PMF with scores immediately before discharge from the hospital.

To assess the M-WCAS’s usability, the raters were requested to qualify the ease of scoring of the M-WCAS as easy to score, moderately easy to score, or difficult to score. Additionally, the time to complete the score was reported.

Statistical analysis

To assess the M-WCAS’s criterion validity, we used Spearman’s correlation coefficient to correlate the scores of the M-WCAS with the scores of the Tal et al. severity score. To assess the M-WCAS’s construct validity, the scores of the M-WCAS in patients who required subsequent admission to the PICU were compared to scores in the patients who required admission only to the PMF, using the Mann–Whitney U test. An inter-rater agreement analysis of the M-WCAS scores at admission to the PMF using the unweighted κ statistics was performed in order to determine consistency among raters. Inter-rater agreement was also analyzed through the construction of the Bland and Altman plot [13]. Additionally, we compared the scores of the M-WACS provided by the two raters using the Mann–Whitney U test.

The M-WCAS’s sensitivity to change was determined by using a Wilcoxon signed-rank test to compare scores of the M-WCAS at admission to the PMF with scores obtained immediately before discharge from the hospital.

The use of the method proposed by Walter et al. [14] yielded a sample size of 54 patients, two methods to be reported in the diagnosis, a κ for the null hypothesis of 0.5, a κ for the alternative hypothesis of 0.7, a statistical significance level of 0.05, and a power of 80%. Statistical analysis was done with Stata 12.0 (Stata Corporation, College Station, TX).

Results

Of the total number of children admitted to the hospital due to bronchiolitis during the study period (n = 89), 56 (62.9%) met our eligibility criteria, and 54 (60.7%) were finally enrolled in the study. Of the remaining 35 patients, 20 (57.1%) were excluded because of a pre-existing chronic lung disease, 13 (37.1%) because of a previous history of congenital heart disease and 2 (5.7%) because their parents declined to participate in the study. All of the 54 participants were able to be followed until the end of the study. The median (inter-quartilic range [IQR]) of the age of the 54 patients included in the study was 5 (2–9) months. The age group distribution was: 28 (51.8%) <6 months, 23 (42.6%) between 6 and 12 months and 3 (5.6%) between 13 and 24 months. Thirty (55.6%) of the patients were males and 24 (44.4%) were female. The most common signs and symptoms at presentation were cough in all 54 (100%) patients, wheezing in 45 (83.3%), fever in 35 (64.8%), poor food intake in 28 (51.9%), inspiratory crackles in 10 (18.5%) and apnea in 3 (5.6%) patients. Out of the total patients included, 6 (11.1%) required admission to the PICU. The median (IQR) of the scores of the M-WCAS at admission to the PMF for both raters was 2.5 (1.9–2.5) and 2.5 (2.0–3.0), and the score obtained immediately before discharge from the hospital was 1.0 (0.5–1.6). The median (IQR) of the elapsed time between assessments of the two raters were 32 (19–43) min. The median (IQR) of the scores of the Tal et al. severity score at admission to the PMF for both raters was 4.0 (3.0–5.0) and 4.0 (3.0–5.0), and the score obtained immediately before discharge from the hospital was 1.0 (1.0–2.0).

The scores of the M-WCAS at admission to the PMF correlated positively with the scores of the Tal et al. severity score at admission to the PMF (Spearman correlation coefficient ρ = 0.761, p < 0.001 for the first rater, and ρ = 0.809, p < 0.001 for the second rater). Likewise, the scores of the M-WACS immediately before discharge from the hospital correlated positively with the scores of the Tal et al. severity score immediately before discharge from the hospital (Spearman correlation coefficient ρ = 0.712, p < 0.001) (Figure 1).

Figure 1.

Box plots of the scores of the M-WCAS and of the Tal et al. severity score obtained by both raters at admission to the pediatric medical floor (PMF) and immediately before discharge from the hospital*.

The scores of the M-WCAS in patients who required subsequent admission to the PICU were significantly higher than those in patients who required admission only to the PMF [4.5 (3.6–5.2) vs. 2.5 (1.5–2.5), p < 0.001 for the first rater and 4.7 (3.6–5.1) vs. 2.5 (2.0–2.5), p < 0.001 for the second rater] (Figure 2).

Figure 2.

Box plots of the scores of the M-WCAS obtained by both raters in patients who required subsequent admission to the PICU and in patients who only required admission to the pediatric medical floor*.

No statistically significant difference [2.5 (1.9–2.5) vs. 2.5 (2.0–3.0), p = 0.48] was found in the scores of the M-WCAS at admission to the PMF between the two raters (p = 0.48). The inter-rater agreement for the raters was found to be κ = 0.897 (p < 0.001), 95% CI (0.699–1.000). The Bland and Altman plot shows the agreement of the M-WCAS score between the two raters. Figure 3 shows that the mean difference in the M-WCAS score between the two raters was −0.08, and their corresponding 95% limit of agreement was −1.44 to 1.27. Five outliers were found, and the points in the plot show random distribution.

Figure 3.

Bland and Altman plot displaying the difference in the M-WCAS measurements plotted against the mean M-WCAS measurements.

The scores of the M-WCAS in patients at admission to the PMF were significantly higher than those obtained immediately before discharge from the hospital [2.5 (1.9–3.0) vs. 1.0 (0.5–1.6), p < 0.001]. All raters qualified the M-WCAS as easy to score, and they reported that the time required to complete the score ranged from 1 to 3 min.

Discussion

The present study shows that in infants with acute bronchiolitis the M-WCAS severity score has an adequate criterion validity when correlated with another validated severity instrument, an adequate construct validity when comparing the scores obtained in patients who required admission to the PICU with those obtained in patients who required admission only to the PMF, an adequate inter-rater agreement when compared the scores obtained by two independent raters at admission of the patients to the PMF, an adequate sensitivity to change when comparing the scores obtained at admission to the PMF with those obtained immediately before discharge from the hospital, and an appropriate usability when raters evaluated its ease of scoring and the time to complete the score.

The findings of the present study are important because they support the use of this tool to assess the severity of bronchiolitis for clinical decision-making, for evaluation of treatment effectiveness in daily clinical practice, and as a useful outcome in clinical trials. For example, in the clinical practice guideline on the diagnosis and management of bronchiolitis from the Subcommittee on Diagnosis and Management of Bronchiolitis of the American Academy of Pediatrics, it is stated that although there is no evidence to justify routine use of bronchodilators, it may be reasonable to administer a nebulized bronchodilator and document pre-therapy and post-therapy changes using an objective means of evaluation [15]. To the best of our knowledge, no other severity scoring system for bronchiolitis has been examined for sensitivity to change.

Our results agree with those reported by Martinón-Torres F. et al., who reported that the inter-rater agreement for the raters of the M-WCAS in infants with moderate-to-severe bronchiolitis was found to be κ = 0.831. Moreover, in this study, compared to patients in the control group who received supportive care and nebulized epinephrine, patients who received additional treatment with heliox therapy presented significant reductions in the M-WCAS score, heart and respiratory rates, and length of stay in the PICU. These findings support the sensitivity to change and construct validity of the M-WCAS in patients suffering from bronchiolitis [11].

Although to the best of our knowledge no other studies have assessed the psychometric characteristics of the M-WCAS, some other papers have reported the inter-rater agreement of other bronchiolitis severity scores, such as Lowell et al. scores [16], Tal et al. score [5], modified Tal et al. score [10], Wang et al. scores [17], and Liu et al. score [7]. The assessment components of these bronchiolitis severity scores vary considerably among these studies. While some severity scores include only clinical signs of respiratory distress assessment, such as wheezing and retractions, others include cyanosis, general appearance, and SatO₂. In general, the rates of agreement for respiratory parameters reported in these studies are similar to those obtained in our study, except for Wang et al. and Liu et al. severity scores studies, which reported κ values for the inter-rater agreement that varied between 0.189 and 0.43. The reasons for these discrepancies are not clear, but they are probably related to the nature of bronchiolitis itself, which can cause the respiratory rate of young children to vary quickly over a few breaths and which probably contributes to lowering the overall inter-rater agreement of some of the severity scores that contain this component, the time elapsed between assessments for both raters, the level of training in the use of the severity scores, and the rater’s professional experience with infants. Although in our study the time elapsed between assessments for both raters ranged between 15 minutes and 2 hours, and the raters were briefed on the scoring system of each severity score but had no formal training about their use, our high inter-rater agreement is likely to be related to the absence of the respiratory rate component of the M-WCAS severity score and to the high level of professional experience with infants of the physicians who scored the M-WCAS.

The main strength of our study is the assessment of all of the recommended psychometric characteristics in the validation process of severity scores and other outcome measures, and not only the inter-rater agreement, as was done in previous studies. As mentioned, we did not evaluate the test–retest reliability, because it has been reported that this psychometric characteristic may have limited meaning in the context of scores developed to measure change in acute clinical conditions.

The main limitations of our study comprise the small number of patients included and that the study was performed in a single center and in a unique clinical setting (inpatients). Although it is probable that the M-WCAS severity score shows similar psychometric properties in other populations and other clinical settings, it is necessary to conduct additional studies to determine the reliability, validity, responsiveness and usability of the M-WCAS severity score in different populations based on a larger number of patients and in different settings with a more representative sample of the general population of infants with bronchiolitis.

In summary, our results suggest that the M-WCAS severity score has adequate criterion validity, adequate construct validity, adequate inter-rater agreement, adequate sensitivity to change, and appropriate usability for infants hospitalized for acute bronchiolitis. Additional research is needed in different populations based on a larger number of patients and in different settings with a more representative sample of the general population of infants with bronchiolitis.