Diagnostic accuracy of the bronchodilator response in children

Abstract

Background

The bronchodilator response (BDR) reflects the reversibility of airflow obstruction and is recommended as an adjunctive test to diagnose asthma. The validity of the commonly used definition of BDR, a 12% or greater change in FEV₁ from baseline, has been questioned in childhood.

Objectives

We sought to examine the diagnostic accuracy of the BDR test by using 3 large pediatric cohorts.

Methods

Cases include 1041 children with mild-to-moderate asthma from the Childhood Asthma Management Program.

Control subjects (nonasthmatic and nonwheezing) were chosen from Project Viva and Home Allergens, 2 population-based pediatric cohorts. Receiver operating characteristic curves were constructed, and areas under the curve were calculated for different BDR cutoffs.

Results

A total of 1041 cases (59.7% male; mean age, 8.9 ± 2.1 years) and 250 control subjects (46.8% male; mean age, 8.7 ± 1.7 years) were analyzed, with mean BDRs of 10.7% ± 10.2% and 2.7% ± 8.4%, respectively. The BDR test differentiated asthmatic patients from nonasthmatic patients with a moderate accuracy (area under the curve, 73.3%).

Despite good specificity, a cutoff of 12% was associated with poor sensitivity (35.6%). A cutoff of less than 8% performed significantly better than a cutoff of 12% (P = .03, 8% vs 12%).

Conclusions

Our findings highlight the poor sensitivity associated with the commonly used 12% cutoff for BDR. Although our data show that a threshold of less than 8% performs better than 12%, given the variability of this test in children, we conclude that it might be not be appropriate to choose a specific BDR cutoff as a criterion for the diagnosis of asthma.

In asthmatic patients the response to bronchodilators reflects the reversibility of airway airflow obstruction. High bronchodilator response (BDR) in asthmatic patients has been associated with poor clinical outcomes,¹ increased airway inflammation,² and increased response to inhaled corticosteroids,^1,3 supporting that BDR has prognostic and therapeutic relevance. The BDR test provides additional information to the clinical history and is recommended by international guidelines as an adjunct to the clinical history in the diagnosis of asthma.⁴

BDR can be expressed by using different methods, with the 3 most common being the percentage of FEV₁, the percentage of the initial predicted value for FEV₁, and absolute change in FEV₁ after administration of a short-acting bronchodilator. A significant BDR is commonly defined as a 12% or greater and 200 mL or greater change in FEV₁ from baseline.⁵ This 12% criterion, which was recently reaffirmed in a large international study,⁶ approximates the 95th percentile for percentage change in FEV₁ after bronchodilator administration in general population studies, which mainly consist of adults.⁷ The forced oscillation technique, which provides information on airway resistance and reactance, has also been used to measure response to bronchodilators and to differentiate between asthmatic and nonasthmatic patients,^8,9 especially in younger children who are unable to cooperate during spirometric testing. However, this technique is not widely used because standardized guidelines are lacking.

Although the National Asthma Education and Prevention Program’s Expert Panel Report 3 uses the 12% cutoff as evidence of airway reversibility in establishing the diagnosis of asthma⁴ and this cutoff is used to include subjects in several childhood asthma trials,^10,11 the validity of the 12% cutoff has been questioned in the pediatric population.¹² BDR tends to increase with decreasing baseline FEV₁.¹³ In subjects with low baseline FEV₁, small changes in absolute FEV₁ in response to a bronchodilator translate into large percentage changes in FEV₁. Because most children with asthma have baseline FEV₁ within the normal reference range, the increase in FEV₁ after bronchodilator administration is limited. Dundas et al¹⁴ suggest that a 9% or greater increase in FEV₁ provided the most acceptable sensitivity (50%) and specificity (86%) to detect previous wheeze in 5- to 10-year-old children. These results were corroborated by a more recent study showing that a BDR of 9% or greater was optimal at differentiating asthmatic from nonasthmatic children with similar sensitivity and specificity.¹² This study was performed in a predominantly Hispanic cohort and thus might have limited generalizability to children of other ethnicities.

In this study we determined the BDR cutoff that best differentiates between children with mild-to-moderate asthma and children without asthma in a large and predominantly white population. We hypothesized that a BDR cutoff of 12% would be specific but not sufficiently sensitive for diagnosing asthma in children.

METHODS

Subjects

Participants from 3 pediatric cohorts were included in this study. Asthma cases consisted of children enrolled in the Childhood Asthma Management Program (CAMP). The demographics of the CAMP subjects and the study design have been previously reported.¹⁵ Briefly, this multicenter trial randomized 1041 children with mild-to-moderate asthma aged 5 to 13 years to budesonide, nedocromil, or placebo. Children were included if they had asthma symptoms 2 or more times per week, used an asthma medication daily, or used an inhaled bronchodilator twice per week for 6 or more months and had a positive methacholine challenge test result. The long-term effects of these treatments on lung growth were evaluated.¹⁵ Follow-up visits occurred at 2 and 4 months after randomization and every 4 months thereafter for an average of 4.3 years.

The control patients for this study were selected from 2 pediatric general population cohorts. Project Viva is a prospective cohort study examining the effect of prenatal and perinatal factors on maternal and child health. Details of the study design have been described previously.¹⁶ This prebirth cohort consisted of 2128 singleton infants, the mothers of whom were recruited from Harvard Vanguard Medical Associates, a multispecialty group practice in eastern Massachusetts. Of these children, 1116 attended Project Viva’s 7-year in-person visit, 819 of whom attempted the BDR test and 468 had valid results. The Home Allergens cohort is a birth cohort of children with a parental history of allergy or asthma. Details of the study design have been published.¹⁷ Five hundred five infants were recruited within 48 hours of birth in the metropolitan Boston area. Telephone questionnaires about symptoms and diagnoses of atopic disease were administered every 2 months for the first 2 years of life and then every 6 months. A total of 284 children were followed until age 12 years, when 250 of them performed the bronchodilator test. Subjects from these 2 population-based cohorts were selected as control subjects if they had never been given a diagnosis of asthma or had never wheezed, both assessed through a questionnaire or interview (n = 197 in Project Viva and n = 53 in Home Allergens).

In all 3 cohorts the children’s parents or guardians provided informed consent, and the study was approved by the respective local institutional review board.

Spirometry and BDR

In CAMP the BDR test was performed at randomization and at subsequent visits during which a methacholine challenge was not administered.¹⁸ Spirometry was performed at least 4 hours after the last use of a short-acting bronchodilator and at least 24 hours after the last use of a long-acting bronchodilator. For the purpose of this analysis, we used the BDR test at randomization, at which point subjects had been off their regular asthma medications for at least 28 days but were allowed to use a rescue bronchodilator and prednisone, if needed. The BDR test was performed around age 7 years for Project Viva (range, 6.6-10.9 years) and at age 12 years for Home Allergens (range, 11.1-12.7 years). In Project Viva spirometry was performed with the EasyOne Spirometer (NDD Medical Technologies, Andover, Mass). In Home Allergens spirometry was performed by using the Eagle+ spirometer (Collins Medical, Louisville, Colo). In all 3 cohorts postbronchodilator spirometric measures were obtained at least 15 minutes after administration of 2 puffs (90 μg per puff) of albuterol. Spirometric performance was required to meet American Thoracic Society criteria for acceptability and reproducibility, with each subject producing at least 3 acceptable spirograms, 2 of which must have been reproducible.¹⁹

In this study we defined BDR as a percentage change in absolute FEV₁ after albuterol administration, as follows:

$$\begin{array}{cc}\hfill ({\text{Postbronchodilator FEV}}_1-& {\text{Prebronchodilator FEV}}_1)∕\text{Prebronchodilator}\hfill \\ \hfill & {\text{FEV}}_1\times 100.\hfill \end{array}$$

Given the lack of consensus in the definition of BDR, we have also calculated BDR as a percentage of the initial predicted value for FEV₁ in CAMP, as follows:

$$\begin{array}{cc}\hfill ({\text{Postbronchodilator FEV}}_1-& {\text{Prebronchodilator FEV}}_1)∕\text{Prebronchodilator}\hfill \\ \hfill & {\text{percent predicated FEV}}_1\times 100\hfill \end{array}$$

and compared it with the former definition.

Statistical analysis

A descriptive analysis of baseline characteristics was performed. A Pearson correlation test was used to evaluate the association between the 2 definitions of BDR in CAMP. The study population was divided into a training set, consisting of a random selection of 50 cases and 50 control subjects, and a validation set, consisting of the remaining subjects, to evaluate the diagnostic accuracy of the BDR test. This allows for validation of results within our cohorts. Receiver operating characteristic curves were constructed, and areas under the curve (AUCs) were calculated for different BDR thresholds (R package pROC).²⁰ The AUCs for different thresholds were compared by using the DeLong test. A sensitivity analysis was performed to assess how the diagnostic accuracy of the BDR test varies across different severities of asthma. AUCs were examined for the CAMP subjects with and without evidence of baseline airflow obstruction (FEV₁ percent predicted <80% and ≥80%, respectively). P values are 2-sided. All analyses were performed with R software, version 2.12.1 (www.r-project.org).

RESULTS

Patient demographics

A total of 1041 children with mild-to-moderate asthma were included from the CAMP cohort (cases). Control subjects consisted of 250 children from the Project Viva and Home Allergens cohorts who had no history of wheezing and asthma at the time of the BDR test. Baseline characteristics of the study population by asthma status are presented in Table I. Although their baseline FEV₁ percent predicted values were similar and within normal limits (93.7% ± 14.3% in cases and 98.4% ± 12.2% in control subjects), the mean BDR differed between the 2 groups, as expected. Both groups consisted of predominantly white subjects. The control subjects from Project Viva were younger than those from Home Allergens (7.9 ± 0.8 vs 11.7 ± 0.5 years, respectively), but other baseline characteristics were similar, including race/ethnicity, baseline FEV₁ percent predicted, and BDR (see Table E1 in this article’s Online Repository at www.jacionline.org). Baseline characteristics of subjects who performed a BDR test and those who did not are shown in Table E2 in this article’s Online Repository at www.jacionline.org. In both Project Viva and Home Allergens subjects who underwent the BDR test had a higher prevalence of asthma and wheezing, whereas other measures of atopy were similar to those of the subjects who did not perform the test. However, the control subjects selected for this analysis were not different from the control subjects who did not undergo the BDR test in terms of age, anthropometric measures, and race (see Table E3 in this article’s Online Repository at www.jacionline.org).

TABLE I.

Baseline characteristics

	Cases (n = 1041)	Control subjects (n = 250)
Male sex, no. (%)	621 (59.7)	117 (46.8)
Age (y), mean (SD)	8.9 (2.1)	8.7 (1.7)
Height (cm), mean (SD)	133.6 (13.8)	133.0 (11.5)
Weight (kg), mean (SD)	33.4 (12.1)	31.0 (8.9)
BMI (%), median (IQR)	68.0 (41.3-89.2)	62.4 (32.1-82.8)
Race, no. (%)
White	711 (68.3)	190 (76.0)
Black	138 (13.3)	20 (8.0)
Hispanic	98 (9.4)	7 (2.8)
Other	94 (9.0)	33 (13.2)
FEV1 (L), mean (SD)	1.65 (0.48)	1.72 (0.49)
FEV1 (% predicted), mean (SD)	93.7 (14.3)	98.4 (12.2)
BDR (%), mean (SD)	10.7 (10.2)	2.7 (8.4)

BMI, Body mass index.

Comparison of different definitions of BDR

Given the lack of consensus on how to define BDR, we compared BDR defined as (1) percentage of initial FEV₁ and (2) percentage of initial predicted value for FEV₁. In CAMP asthmatic patients these 2 definitions were almost perfectly correlated (Pearson correlation, r = 0.958; Fig 1). Therefore we used BDR as a percentage of initial FEV₁ in the subsequent analyses because the commonly accepted 12% cutoff is based on this definition.

FIG 1.

Correlation between BDR expressed as percentage of baseline FEV₁ and as percentage of baseline predicted value for FEV₁ in CAMP (n = 1041).

Diagnostic accuracy of the BDR test

For the purposes of this analysis, the study population was divided into a training set and a validation set. In the training set (n = 100) the BDR test differentiated children with and without asthma with a moderate accuracy (AUC, 74.5%; 95% CI, 69.0% to 80.0%). The AUC was replicated in the validation set at 73.3% (95% CI, 69.2% to 77.2%; Fig 2). Similar AUCs were obtained when the analysis was stratified by sex (AUC, 72.5% in male and 74.0% in female subjects). Table II shows the sensitivity and specificity for different BDR cutoff values. Despite good specificity (89.5%), a cutoff of 12% offered low sensitivity (35.6%). A BDR cutoff of 8% (sensitivity, 54.4%; specificity, 76.5%) or less performed significantly better than a cutoff of 12% (comparing AUCs: P = .031, 8% vs 12%). A cutoff of 5% offered the highest AUC (68.6%), with a sensitivity of 73.6% and specificity of 63.5% (Table II).

FIG 2.

Receiver operating characteristic curve in validation set comparing 991 patients with mild-to-moderate asthma from CAMP and 200 control subjects from Project Viva and Home Allergens.

TABLE II.

A BDR cutoff of 8% or less performed significantly better than a cutoff of 12%

	Sensitivity	Specificity	AUC (95% CI)	P value (comparing with AUC for 12%)
BDR cutoff (%)
12	35.6	89.5	62.5 (59.9-65.1)	—
11	40.1	86.5	63.3 (60.5-66.1)	.261
10	45.2	81.0	63.1 (60.0-66.2)	.613
9	49.1	78.0	63.6 (60.3-66.8)	.414
8	54.4	76.5	65.4 (62.1-68.8)	.031
7	61.1	73.0	67.0 (63.6-70.5)	.003
6	67.6	66.5	67.1 (63.5-70.7)	.007
5	73.6	63.5	68.6 (64.9-72.2)	.001

Sensitivity analysis

A total of 148 CAMP subjects had an initial FEV₁ percent predicted value of less than 80%. Their mean BDR was 23.6% ± 14.6%. The BDR test differentiated these subjects from control subjects with a high accuracy (AUC, 93.1%; 95% CI, 90.4% to 95.7%; Fig 3, A). In contrast, the diagnostic accuracy of the BDR test was lower for CAMP subjects with an initial FEV₁ percent predicted value of 80% or greater (n = 881; AUC, 71.0%; 95% CI, 67.2% to 74.8%; Fig 3, B). This subset of children also had a lower BDR (mean, 8.6 ± 7.4).

FIG 3.

Separate receiver operating characteristic curves comparing 148 asthmatic patients with a baseline FEV₁ of less than 80% of predicted value and 250 control subjects (A) and 881 asthmatic patients with a baseline FEV₁ of 80% or greater and 250 control subjects (B). The best threshold based on the AUC, along with its specificity and sensitivity, are shown.

DISCUSSION

In this analysis of the diagnostic accuracy of the BDR test, we found that the commonly used BDR cutoff of 12% was associated with a poor sensitivity and that a BDR cutoff of 8% or lower performed better than 12% at differentiating between children with and without asthma using 3 predominantly white populations.

The BDR test is recommended as an adjunct in the diagnosis of asthma, along with the clinical history and signs of inflammation, such as eosinophilia and increased IgE levels. This test assesses the degree of reversible airway obstruction with the administration of a bronchodilator and has been shown to provide independent information in the assessment of asthma.²¹ The widely accepted BDR cutoff of 12% is based on general population and asthmatic patient studies consisting of mostly adults, who might have a higher interindividual variability in baseline FEV₁. A recent international study examined the BDR test in more than 3900 healthy adult never smokers participating in the population-based Burden of Obstructive Lung Disease study.⁶ The authors reaffirmed that the 95th percentile for BDR was 12%, with BDR expressed as a percentage of absolute baseline FEV₁. Although this study provides confirmation to current guidelines in adults, our findings suggest that this cutoff might not be appropriate for the diagnosis of asthma in children given its poor sensitivity, which could result in missed cases of asthma. Studies have shown that the BDR tends to increase with decreasing baseline lung function,¹³ and therefore BDR might be higher in asthmatic adults. In the Burden of Obstructive Lung Disease study cohort, the mean percent predicted FEV₁ values in the entire cohort were generally lower than in our study. In contrast, in asthmatic children baseline lung function tends to remain normal,^22,23 even in a subset of patients with severe asthma²⁴; hence a 12% change from baseline with bronchodilator might be difficult to reach. This inference is in accordance with our sensitivity analysis, in which only a minority of CAMP subjects had a baseline FEV₁ of less than 80% of predicted value.

The choice of an optimal BDR cutoff is complex. Maximizing sensitivity and specificity is key in the evaluation of any diagnostic test. In the case of asthma, there is no clear consensus on what is an acceptable tradeoff between these 2 test properties because this decision depends on the treatment threshold. The treatment threshold is in turn determined by the costs and benefits of the treatment and consideration of the harm caused by the treatment of false-positive cases and the nontreatment of false-negative cases. Furthermore, the positive and negative predictive values will depend on the prevalence of asthma, and these differ from a general pediatrician’s office to a pulmonologist’s office, where the BDR test is commonly performed. In patients with mild-to-moderate asthma, as with this study population, treatment can consist of a bronchodilator on an as-needed basis or a daily inhaled corticosteroid, both with different pharmacologic profiles. Hence choosing an optimal cutoff based on prespecified acceptable values of sensitivity and specificity is difficult. Rather, we have used the area under the receiver operating characteristic curve to guide our selection of an appropriate cutoff.

Although a BDR cutoff of 12% provided good specificity (89.5%), it offered poor sensitivity (35.6%). A cutoff of 8%, which had a significantly better AUC than 12%, was associated with a sensitivity of 54.4% and a specificity of 76.5%. The higher performance of a BDR cutoff of less than 12% is in accordance with a previous study of the BDR test in a mainly Hispanic pediatric population, which identified a BDR cutoff of 9% as the optimal value, with a sensitivity of 42.5% and a specificity of 86.3%.¹² This suggests that the BDR test is robust enough to withstand variations in ethnicity.

Although our findings identified that a BDR cutoff of 8% performed significantly better than a cutoff of 12%, it is important to note that changes in FEV₁ of less than 8% are generally considered to be within measurement variability.^5,25 This variability is likely higher in children given their imperfect technique. Thus using a cutoff of 8% as a criterion for the diagnosis of asthma might not be appropriate either. For these reasons, we suggest using caution when picking a BDR cutoff to aid in the diagnosis of asthma. Overall, the BDR test performs poorly in diagnosing asthma in children and might be a better reflection of changes in bronchomotor tone in response to bronchodilators, which in turn might reflect the frequency of asthma symptoms or exacerbations.

Based solely on the AUC, our data suggest a BDR cutoff of 5% as the optimal cutoff (Table II) with a sensitivity of 73.6% and specificity of 63.5%. This threshold would not be clinically useful given that it is within the range of measurement variability. Interestingly, our sensitivity analysis showed that the diagnostic accuracy of the BDR test was higher for subjects with lower baseline lung function. The optimal BDR cutoff for children with an FEV₁ of 80% of predicted value or less based on the AUC was 11%, reflecting the high mean BDR in these persons compared with control subjects. In contrast, the “optimal” 5% cutoff derived from the overall data likely reflects that the majority of the cases have mild asthma, with a significant overlap of the BDR between cases and control subjects (Table I).

Although not the focus of this study, it is interesting to consider the consistency with which children with asthma meet the criterion for a BDR cutoff of less than 12% over time. Using the CAMP cohort, Sharma et al²⁶ reported that, similarly to a cutoff of 12%, a consistent BDR of 10% or greater over 4 years predicted nighttime awakenings, oral steroid bursts, and emergency department visits or hospitalizations. Future studies should explore whether a consistent BDR response at lower thresholds would also predict future clinical outcomes and diagnoses of asthma.

There is no clear consensus on how the BDR should be expressed. Some authors suggest an advantage of using BDR as a percentage of the initial predicted value for FEV₁ over a percentage of initial FEV₁ because this definition is less dependent on baseline lung function.^25,27 In our population of children with mild-to-moderate asthma, these 2 definitions of BDR were highly correlated (r = 0.958). This agreement is in accordance with the fact that most of these children had normal baseline FEV₁ percent predicted values. On the basis of these results, the use of either definition of BDR might be acceptable in children with mild-to-moderate asthma, whereas BDR as a percentage of predicted FEV₁ should be considered in populations with a high heterogeneity in baseline lung function, such as children with severe asthma.²⁴

Our findings can have varied and wide-ranging implications, depending on the clinical situation. Clinically, the BDR test is an imperfect diagnostic test, as suggested by the AUC of less than 70% for any cutoff, and it should be used only as an adjunct for the diagnosis of asthma. Using a cutoff of 12% or greater as “significant” bronchodilation might also result in underdiagnosis and undertreatment of asthma, given the poor associated sensitivity. In asthma research several studies use the 12% BDR cutoff to select patients for inclusion in the study. In children this criterion will lead to a subset of patients selected for a higher degree of airway airflow obstruction, which might limit the generalizability of the results. In addition, it might be more difficult to recruit the number of patients required given the restriction by this criterion. This consideration is especially important in asthma genetic and pharmacogenetic studies, in which large sample sizes are often required. On the other hand, in intervention studies a higher cutoff could select for children who have a higher chance of showing an intervention effect.

Several limitations of our study need to be addressed. This study was designed to examine how well the BDR test differentiates children with mild-to-moderate asthma from children without asthma. This is also a predominantly white population. The diagnostic accuracy of the BDR test was derived from these populations and might not be generalizable to other situations, such as more severe asthma. However, previous studies found similar results,¹² including one comparing subjects with mild intermittent wheeze with control subjects.¹⁴ This supports the use of the BDR test as an adjunct in patients with different degrees of symptom severity. Asthma is a heterogeneous disease, and in addition to pharmacogenetic predisposition in the response to bronchodilators,^28-30 there is evidence that patients with the allergic asthma phenotype have higher BDR,³¹ whereas others have a lower response.³² Thus the diagnostic value of the BDR test might depend on the asthma phenotype. Finally, a minority of subjects in CAMP received rescue prednisone in the weeks before randomization. Although these are short courses of corticosteroids, this could have affected the participants’ responsiveness to a bronchodilator and made cases more similar to control subjects, which would lead to a bias toward a decreased accuracy of the BDR test in this study.

In conclusion, this study demonstrated that the commonly used BDR cutoff of 12% in adults offers poor sensitivity as an adjunct test in the diagnosis of asthma in children. Our results from populations of children with mild-to-moderate asthma suggest that a BDR cutoff of 8% or less performed significantly better than a cutoff of 12%. However, given the variability of this test in children, it might not be appropriate to choose 8% as the optimal cutoff. Thus we advise against choosing a specific BDR cutoff for the diagnosis of asthma but rather to use the BDR test as a guide in the treatment and diagnosis of asthma because the persistence of a positive response might be associated with worse clinical outcomes in asthmatic patients over time.

Supplementary Material

01

TABLE E1. Baseline characteristics of control subjects

TABLE E2. Characteristics of subjects who did and did not perform BDR in Project Viva and Home Allergens

TABLE E3. Characteristics of control subjects who did and did not perform the BDR test in Project Viva and Home Allergens

Clinical implications.

The BDR reflects the reversibility of airflow obstruction. The commonly used cutoff of 12% is associated with poor sensitivity in the diagnosis of asthma in children and might result in missed cases of asthma.

Abbreviations used

AUC

Area under the curve

BDR

Bronchodilator response

CAMP

Childhood Asthma Management Program