Abstract
Rationale: Asthma is a variable condition with an apparent tendency for a natural decline in asthma symptoms and health care use occurring as children age. As a result, asthma interventions using a pre-post design may overestimate the intervention effect when no proper control group is available.
Objectives: Investigate patterns of natural decline over time with increasing age in asthma symptoms and health care use of children. Develop a statistical procedure that enables adjustment that accounts for expected declines in these outcomes and is useable when intervention evaluations must rely solely on pre-post data.
Methods: Mixed-effects models with mixture distributions were used to describe the pattern of symptoms and health care use in 3,021 children aged 2 to 15 years in a combined sample from three controlled trials. An adaptive least squares estimation was used to account for overestimation of intervention effects and make adjustments for pre-post only data. Termed “Adjustment for Natural Declines in Asthma Outcomes (ANDAO),” the adjustment method uses bootstrap sampling to create control cohorts comparable to subjects in the intervention study from existing control subjects. ANDAO accounts for expected declines in outcomes and is beneficial when intervention evaluations must rely solely on pre-post data.
Measurements and Main Results: Children under 10 years of age experienced 18% (95% confidence interval, 15–21%) fewer symptom days and 28% (95% confidence interval, 24–32%) fewer symptom nights with each additional year of age. The decline was less than 10% after age 10 years, depending on baseline asthma severity. Emergency department visits declined regardless of baseline symptom frequency (P = 0.02). The adjustment method corrected estimates to within 2.4% of true effects through simulations using control cohorts.
Conclusions: Because of the declines in symptoms and health care use expected with increasing age of children with asthma, pre-post comparisons will greatly overestimate intervention effects. The ANDAO provides means to adequately estimate treatment effects when a control group design is not possible.
Keywords: research design, program evaluation, statistical models, asthma, self-management
Together with the significant increase in asthma programs and in asthma coalitions around the country (the number of coalitions alone exceeds 200) has come a proliferation of interventions for children provided in clinics, homes, and schools (1). A large number of these efforts are an extension of or are in some way connected to clinical service. Their goals usually include reduction of asthma symptoms and health care use associated with the condition (2). Generally, evaluation of these service-related interventions is modest, as they are not intended to be research investigations. Rather, the goal is to reach and assist children as a dimension of quality care and to assess impact to ensure accountability and effective use of resources. Evaluation of these programs, in the main, takes the form of pre- and post-intervention data collection from parents and children and/or from clinical, medical, or financial records (3). Such assessments are usually believed to be manageable within the scope and budget of the program provider. Randomized controlled trial (RCT) designs are generally not used, as they may be considered inappropriate in the service setting, require additional outlay of resources, create a greater time burden on the provider organization, and be undervalued by patients and families, especially those selected for the control condition.
However, an inherent flaw in any pre-post only evaluation is lack of a comparison against which to measure outcomes. This problem is amplified in assessment of asthma interventions. The pattern of asthma in childhood appears to fluctuate with time. Some children, for example, with mild infrequent asthma have been shown to have complete resolution of symptoms by adulthood (4–9). Alternatively, some children, especially those with persistent disease, have been shown to experience continuing symptoms into adult life (5, 6). A prospective study of a group of randomly selected children with wheeze showed that half of those with frequent wheezing had considerable amelioration of their symptoms by 21 years of age (9). Another longitudinal population-based study found that about one-fourth of participating children had wheezing that persisted into adulthood (7). In longitudinal cohort studies, the relapse rate or complete remission rate for asthma has been reported to range between 12 and 22% by adulthood (7, 8). In short, the condition is variable, with an apparent tendency for symptoms in children to decline over time. If this is a general tendency, studies that do not use a control or comparison group might mistake program impact for what actually may be the natural development toward fewer symptoms as a child matures. Reduction of symptoms has been shown to be associated with less need for health services (10).
The first objective of this study was to investigate patterns of natural decline over time with increasing age in asthma symptoms and health care use of children. The second was to develop a statistical procedure that would enable adjustment (which we will refer to as the Adjustment for Natural Declines in Asthma Outcomes [ANDAO]) of intervention effect size in evaluations absent a control group. This objective, if realized, would allow program providers to differentiate outcomes associated with an intervention from a decline in symptoms and health care use that might be expected to occur over time without intervention.
Methods
All study procedures were approved by the University of Michigan Institutional Review Board. The research approach was to determine the pattern of childhood asthma through standardized, coordinated examination of findings from existing cohort studies. Data from three RCTs that included participants in different age ranges were combined to generate a large sample for analysis.
Participants
The combined sample from three cohorts of subjects comprised: (a) children aged 7 to 10 years with asthma symptoms in Detroit, Michigan attending elementary schools (n = 833) (11), (b) children aged 10 to 13 years with asthma symptoms attending Detroit middle schools (n = 1292) (12), and (c) children aged 2 to 12 years seeking care from pediatric practices in 10 cities across the United States (n = 896) (13). Datasets (a) and (b) are referred to as the School cohorts, and (c) is referred as the Pediatric Practice cohort. The three cohorts were established in respective RCTs reported in the literature, in which participants were followed from baseline through 2 years of follow-up. Data collection was primarily through telephone interviews with the child’s primary caretaker and verification in school and medical records. To investigate the development pattern of asthma outcomes over time, children who had received a study intervention in addition to usual care were excluded from follow-up analysis. That is, the development pattern analysis presented here was based on data consisting of all baseline and two annual follow-up observations for children in the control groups from the three studies. Baseline data collected before the intervention from those children in the intervention arms were also included in the sample. In this way, we generated a larger sample size with more variations and hence produced a more efficient estimation of the asthma development pattern that obviates bias in the estimation of intervention effect. The resulting combined sample comprised 3,021 children with a total of 5,168 observations.
Outcome Measures
The outcomes of interest in the combined sample were asthma symptom days, symptom nights, days of limited activities, and health care use. Asthma symptom frequency was categorized into four levels: intermittent, mild persistent, moderate persistent, and severe persistent, according to the National Asthma Education and Prevention Program Classification (14). Health care use included overnight hospitalizations, emergency department (ED) visits, and physician (MD) office visits (both regularly scheduled and urgent visits) on an annual basis.
Statistical Analysis
Table E1 in the online supplement provides a summary of missing data rates for the RCTs composing the study sample and step-by-step guidance for applying the statistical approach. Potential causes for missing data (e.g., moving away from the study schools, parent unable to attend the interviews) were not directly associated with the underlying measurement process for asthma symptoms, so it was reasonable to assume that data were missing at random. We used the classic likelihood approach in mixed-effects models to analyze the data (15).
Covariates used for development of models for symptom days, symptom nights, and days with limited activity were: child’s age, symptom frequency at baseline, and sex, and the interaction between age and frequency of baseline asthma symptoms. Age, baseline asthma symptoms, annual household income, use of antiinflammatory medicine, and use of a bronchodilator were included in the models for examining health care use outcomes. Polynomial terms of age were also included in the models initially to determine if any nonlinear pattern in the presence of asthma symptoms was evident. A P value of less than 0.05 was deemed statistically significant.
Given that the Detroit Elementary School cohort and the Detroit Middle School cohort were similar (the majority of children were African American from low-income families), we combined them into one dataset. Initial findings from individual cohort-specific regression analysis showed that the estimated slopes of time were distinct between the two school cohorts (namely, older children tended to have a slower decline rate than younger ones). Therefore, we adopted a “broken stick” regression model (16) and applied the Akaike Information Criterion (17) to select the optimal age point of change in the decline rate among many candidate change-points. Subjects in the Pediatric Practice cohort were more heterogeneous than the School cohorts, with children at different age ranges, with different characteristics (e.g., higher income levels), and with the majority (70%) being white. Thus, these data were analyzed separately and served as a comparison for the School data analysis.
As shown in Figure 2, some children were found to have no symptoms in the data collection windows; therefore we modeled repeated measures data and addressed the probability of excess zero counts by a zero-inflated mixture-distribution approach using the hurdle model (18, 19). We considered a mixed-effects logistic model for the occurrence component and a mixed-effects log-linear model for the nonzero component. Random effects were used to account for unobserved heterogeneity among participants and the within-subject longitudinal correlation. The random effects for occurrence and nonzero counts were allowed to be correlated by assuming a bivariate normal distribution, which reflects the fact that some correlated variations were shared due to unobserved confounding factors.
Figure 2.
Proportions of zero counts, nonzero counts, and missing data for the six asthma outcomes: symptom days, symptom nights, days with limited activity, hospitalizations, emergency department (ED) visits, physician (MD) visits in the Elementary School, Middle School, and Pediatric Practice datasets at baseline.
This analytical method was implemented using SAS software, Version 9.2 of the SAS System by SAS Institute Inc. (Cary, NC) (20) with the PROC COUNTREG and PROC NLMIXED procedures. First, PROC COUNTREG with a zero-inflated Poisson distribution was used to estimate parameters for the occurrence component and the nonzero count component without any random effects. Second, these parameter estimates were used as the starting values in PROC NLMIXED for separate occurrence and nonzero count models with uncorrelated random effects. Third, the parameter estimates from the second step were used as the starting values in PROC NLMIXED to obtain estimated effect sizes of covariates in the mixed-distribution model with correlated random effects. Final models were selected according to the likelihood ratio tests and Akaike Information Criterion.
Most health care use data (hospitalizations, ED visits, and MD visits) used in the analysis were predominated by excess zeros. For example, more than 80% of the baseline observations for number of hospitalizations were zeros (see Figure 2). The remaining 20% of children with nonzero counts of health care use had only a few possible events. Thus, possible count outcomes were grouped into four ordered categories (0, 1, 2, and 3 or more). A proportional odds model with random effects was used to model the relationship between health care use and age (SAS PROC NLMIXED) (19). Given that the School sample and the Pediatric Practice sample had different demographic characteristics and potentially different asthma management, the two samples were analyzed separately. We also applied the Breslow-Day test to examine the homogeneity of odds ratios between the School and Pediatric Practice cohorts for all three health care use outcomes (21). As a result of significantly different odds ratios (P < 0.001), separate sets of estimates were reported for these two samples.
Application of a Comparison for Pre-Post Intervention Studies: The ANDAO
The ultimate goal of the above analysis was to develop a way to correct the hypothesized overestimated effect of an intervention when a control group is unavailable, that is, adjusting for an expected natural decline rate in asthma symptoms with increased age. We applied a least squares estimation (LSE) based on linear regression to achieve this adjustment. A key assumption of applying the adjustment is that the subjects in a pre-post study are from the approximately same study population as the existing control cohort. Under the assumption, the two cohorts would have the same underlying model for the natural decline rate of asthma symptoms.
We applied an adaptive LSE (ALSE) to estimate the natural decline rate (denoted as δ) of asthma symptoms for the intervention cohort evident in pre-post data compared with control cohort data. With the availability of a corrected estimate of δ, the difference in the outcome variable of interest between pre- and postintervention findings minus the estimated δ gives an adjusted estimate of true intervention effect in a pre-post only study. The LSE of δ can be expressed as a function of covariances between covariates and between the outcome variable and covariates. To make the estimation of δ adaptive to the cohort under investigation, computations were made using the intervention cohort data and information from both the control cohort and the intervention cohort. This enabled us to optimally use intervention cohort data and minimize any unbalanced impact of control data.
To deal with a potential sample size imbalance between the control cohort and the intervention cohort, which could lead to biased estimation, we applied the bootstrap resampling technique (22) to both cohorts. The idea behind bootstrap is to use the available data as a “surrogate population.” A repeated resampling can approximate the sampling distribution of a statistic. We drew samples from both cohorts (with replacement) repeatedly to form equally sized intervention and control groups with the group size set as the minimum size of samples. Details for implementation of the bootstrap resampling technique are provided in the online supplement. Figure 1 gives a brief description of the schematic process of bootstrap resampling and computation of the adjustment estimate. Further step-by-step details of how to apply the ALSE method are also presented in the online supplement.
Figure 1.
This diagram illustrates a schematic procedure of calculating a corrected estimate of intervention effect in a longitudinal pre-post study with no control subjects. The intervention cohort has a smaller sample size than that of the existing control cohorts. In the adjustment (adaptive least squares estimation approach), control subjects from the Detroit School cohorts are used as references to “clone” a group of control subjects matched with the intervention group. This procedure is repeated many times (e.g., 1,000) using the virtual resampling technique to remove sampling bias in the process of matching.
We used a simulation to compare results obtained through ALSE using virtual control subjects with those obtained from the “gold standard approach,” that is, using the true control cohort comprising combined data from three clinical trials. The control cohort was simulated from the distributions of baseline covariates using the three existing cohort datasets. ANDAO mimics an actual pre-post only study without the corresponding matched control group and uses different distributions of baseline covariates (e.g., different age distributions) to simulate the intervention cohort. In particular, we incorporated these covariates in the simulation model: sex, age, asthma symptom frequency, time, and intervention by time interaction. Two outcome variables were examined in the simulations: symptom days (log transformed) and health care use (ordinal categorical variables). Estimates of intervention effect using ALSE were compared with unadjusted estimates and those obtained by the gold standard approach through 1,000 rounds of simulation under each scenario of pre-post data.
Results
Baseline demographic and asthma-related characteristics of the 3,021 study participants are summarized in Table 1. Approximately 50% of the children in the Elementary School and the Pediatric Patient studies were assigned to the control group, whereas about one-third in the Middle School study were assigned to the control group (an intentional feature of the RCTs). Sex was distributed evenly in the two School datasets, and slightly more boys (65%) were present in the Pediatric Practice dataset. More than 90% were African American in the School samples; 73% were Caucasian, 12% were African American, and 15% Latino/Hispanic, Asian, and other in the Pediatric Practice sample. The sample of Elementary School children appeared to have more frequent asthma than the Middle School sample, followed by the Pediatric Practice cohort that comprised more children of younger age. All children in the Pediatric Practice sample and approximately one-half of the children with symptoms of probable asthma in the Schools cohort had been diagnosed by a doctor.
Table 1.
Characteristics of 3,021 study children from the three studies at baseline
| Elementary School |
Middle School |
Pediatric Practice |
|
|---|---|---|---|
| (N = 833) | (N = 1,292) | (N = 896) | |
| Child is in control group | 418 (50) | 408 (32) | 464 (52) |
| Age, mean (SD) | 8.85 (1.29) | 11.61 (0.63) | 7.69 (2.94) |
| Male | 441 (53) | 671 (52) | 584 (65) |
| African American | 810 (97) | 1,197 (93) | 106 (12) |
| Annual household income | |||
| ≤$10,000 | 230 (28) | 147 (11) | 42 (5) |
| $10,001 ∼ $40,000 | 321 (39) | 602 (47) | 218 (24) |
| >$40,000 | 216 (26) | 149 (12) | 577 (64) |
| Asthma severity | |||
| Intermittent | 300 (36) | 613 (47) | 535 (60) |
| Mild persistent | 147 (18) | 324 (25) | 185 (21) |
| Moderate persistent | 159 (19) | 210 (16) | 122 (14) |
| Severe persistent | 227 (27) | 142 (11) | 51 (6) |
| Doctor diagnosed | 486 (58) | 585 (45) | 896 (100) |
| Use of spacer with inhaler* | 90 (11) | 503 (39) | 442 (49) |
| Use of antiinflammatory medications† | 139 (17) | 191 (15)‡ | 641 (72) |
| Use of bronchodilator | 360 (43) | 491 (38)‡ | 268 (30) |
| Use of antihistamine | 160 (19) | 82 (6) | 310 (35) |
Data are presented as n (%) except as otherwise noted. Note that the percentages do not sum to one because of missing observations.
Near 49 and 29% of parents did not report whether the child used a spacer in the Elementary School and the Middle School data, respectively.
Antiinflammatory medications include any prescribed antiinflammatory medications, including oral steroids and leukotriene modifiers.
The middle school data had 27% missing on antiinflammatory and bronchodilator uses.
Figure 2 shows the proportions of zero counts, nonzero counts, and missing data for the six asthma outcomes in the Elementary School, Middle School, and Pediatric Practice data. A larger proportion of the children in Pediatric Practice sample had nonzero counts for yearly symptom days (98%), symptom nights (91%), days of limited activity (74%), and doctor office visits (91%) than those in the Schools sample. Pediatric Practice children also had higher levels of health care use.
Expected Decline in Asthma Symptoms and Health Care Use
Results for symptom days, symptom nights, and days with limited activity in the School data set are displayed in Table 2. The top section of Table 2 lists results of the occurrence component of the model. Cross-sectionally, older age was significantly associated with greater probability of having symptom days (P = 0.002) and symptom nights (P < 0.0001). In contrast, longitudinally, for children who were younger than 10 years, the estimated odds ratios for having more symptom days and symptom nights were exp(−0.58) = 0.56 (95% CI, 0.39–0.81) and exp(−0.61) = 0.54 (95% CI, 0.38–0.77) with each 1-year increase in age, adjusting for sex and baseline age. The estimated odds ratios of having one or more days with limited activity in a year were exp(−0.36) = 0.70 (95% CI, 0.52–0.94) with each year of increased age. Specifically, among those older than 10 years, distinct slopes of age were estimated for children with severe persistent asthma and for children with intermittent, mild persistent, or moderate persistent asthma. Children with severe persistent asthma at baseline were estimated to experience greater reduction in their likelihood of having symptom days and nights as well as days with limited activity each year, compared with children with intermittent, mild persistent, or moderate persistent asthma at baseline. The bottom section of Table 2 shows results for the nonzero count component, which are consistent with the occurrence component of the model. Given a child having a positive count, symptom days, symptom nights, and days of limited activity significantly declined with increasing years of age regardless of baseline age and severity (P < 0.01). Estimated mean nonzero counts and the associated 95% confidence bands for yearly symptom days, yearly symptom nights, and yearly days with limited activity against age for children in the School and the Pediatric Practices samples are plotted in Figure 3. In both datasets, asthma symptoms gradually decreased with age with a change point at age 10 years.
Table 2.
Parameter estimates and SE estimates for the final hurdle mixed-distribution models of yearly symptom days, symptom nights, and days with limited activity (using Elementary School and Middle School data)
| Parameter | Symptom Days |
Symptom Nights |
Days of Limited Activity |
||||||
|---|---|---|---|---|---|---|---|---|---|
| Est | SE | P Value | Est | SE | P Value | Est | SE | P Value | |
| Occurrence | |||||||||
| Intercept | 0.15 | 0.59 | 0.80 | −3.18 | 0.57 | <0.0001 | −0.14 | 0.44 | 0.76 |
| Female vs. male | 0.02 | 0.16 | 0.89 | −0.54 | 0.15 | <0.001 | −0.13 | 0.13 | 0.29 |
| Mild persistent vs. intermittent | 1.03 | 0.22 | <0.0001 | 1.42 | 0.22 | <0.0001 | 0.83 | 0.17 | <0.0001 |
| Moderate persistent vs. intermittent | 1.52 | 0.26 | <0.0001 | 2.62 | 0.28 | <0.0001 | 1.51 | 0.19 | <0.0001 |
| Severe persistent vs. intermittent | 2.14 | 0.31 | <0.0001 | 3.98 | 0.35 | <0.0001 | 1.70 | 0.21 | <0.0001 |
| Age at baseline interview | 0.17 | 0.05 | <0.01 | 0.35 | 0.05 | <0.0001 | −0.002 | 0.04 | 0.96 |
| Age (<10 yr) | −0.58 | 0.19 | <0.01 | −0.61 | 0.18 | <0.001 | −0.36 | 0.15 | 0.02 |
| Age (≥10 yr): not severe persistent | −0.56 | 0.08 | <0.0001 | −0.42 | 0.08 | <0.0001 | −0.28 | 0.07 | <0.0001 |
| Age (≥10 yr): severe persistent | −0.75 | 0.19 | <0.0001 | −1.24 | 0.18 | <0.0001 | −0.51 | 0.14 | <0.001 |
| Nonzero count | |||||||||
| Intercept | −0.42 | 0.27 | 0.12 | −1.54 | 0.31 | <0.0001 | −0.62 | 0.27 | 0.02 |
| Female vs. male | 0.18 | 0.06 | <0.01 | −0.38 | 0.07 | <0.0001 | 0.12 | 0.07 | 0.08 |
| Mild persistent vs. intermittent | 0.69 | 0.11 | <0.0001 | 0.66 | 0.12 | <0.0001 | 0.63 | 0.11 | <0.0001 |
| Moderate persistent vs. intermittent | 0.98 | 0.11 | <0.0001 | 1.24 | 0.13 | <0.0001 | 1.04 | 0.11 | <0.0001 |
| Severe persistent vs. intermittent | 1.22 | 0.11 | <0.0001 | 1.94 | 0.13 | <0.0001 | 1.26 | 0.12 | <0.0001 |
| Age at baseline interview | 0.24 | 0.02 | <0.0001 | 0.29 | 0.03 | <0.0001 | 0.19 | 0.02 | <0.0001 |
| Age (<10 yr) | −0.20 | 0.02 | <0.0001 | −0.33 | 0.03 | <0.0001 | −0.34 | 0.03 | <0.0001 |
| Age (≥10 yr): not severe persistent | −0.24 | 0.01 | <0.0001 | −0.07 | 0.01 | <0.0001 | −0.16 | 0.01 | <0.0001 |
| Age (≥10 yr): severe persistent | −0.07 | 0.01 | <0.0001 | −0.03 | 0.01 | <0.01 | −0.14 | 0.02 | <0.0001 |
Definition of abbreviation: Est = parameter estimate.
Figure 3.
Estimated mean nonzero counts and the associated 95% confidence bands for yearly symptom days, yearly symptom nights, and yearly days with limited activity versus age for children in the Detroit School dataset (left panels) and Pediatric Practice dataset (right panels).
In summary, longitudinally, children not having severe persistent asthma at baseline were estimated to experience 18 to 21% fewer symptom days with each year of increased age. Children with severe persistent asthma at baseline were estimated to experience 18% fewer symptom days per year increase of age when they were under age 10 years; the rate reduced to 7% fewer symptom days per year when they were older than 10 years. The final models for symptom nights and days with limited activity both indicate two slope estimates for younger children and older children, indicating that symptom nights decreased more rapidly against age for younger children. After age 10 years, classification of asthma severity makes little difference in the relationship between symptom night counts or days with limited activity and age.
Table 3 shows the results of the Pediatric Practice sample data compared with the School data. For the occurrence component of the model, increase in age was significantly associated with the reduced probability of symptom days and symptom nights in children under or above age 10 years (both P < 0.0001). Increase in age was significantly associated with reduced probability of having days with limited activity only for children older than 10 years (P < 0.001). In the nonzero count component of the model, mean counts of yearly symptom days, symptom nights, and days with limited activity were significantly decreased with age (P < 0.0001).
Table 3.
Parameter estimates and SE estimates for the final hurdle mixed-distribution models of yearly symptom days, symptom nights, and days with limited activity (using Pediatric Practice data)
| Parameter | Symptom Days |
Symptom Nights |
Days of Limited Activity |
||||||
|---|---|---|---|---|---|---|---|---|---|
| Est | SE | P Value | Est | SE | P Value | Est | SE | P Value | |
| Occurrence | |||||||||
| Intercept | 4.42 | 0.63 | <0.0001 | 3.26 | 0.41 | <0.0001 | 1.00 | 0.27 | <0.001 |
| Female vs. male | −0.36 | 0.29 | 0.21 | 0.02 | 0.23 | 0.94 | −0.10 | 0.18 | 0.58 |
| Age at baseline interview | 0.05 | 0.06 | 0.38 | −0.04 | 0.04 | 0.39 | 0.06 | 0.03 | 0.06 |
| Age (<10 yr) | −1.07 | 0.16 | <0.0001 | −0.79 | 0.12 | <0.0001 | −0.12 | 0.09 | 0.19 |
| Age (≥10 yr) | −0.89 | 0.15 | <0.0001 | −1.03 | 0.12 | <0.0001 | −0.32 | 0.09 | <0.001 |
| Nonzero count | |||||||||
| Intercept | 3.00 | 0.08 | <0.0001 | 2.56 | 0.07 | <0.0001 | 1.94 | 0.13 | <0.0001 |
| Female vs. male | −0.08 | 0.05 | 0.12 | −0.06 | 0.04 | 0.15 | 0.11 | 0.08 | 0.18 |
| Mild persistent vs. intermittent | 0.84 | 0.07 | <0.0001 | 1.03 | 0.05 | <0.0001 | 0.87 | 0.10 | <0.0001 |
| Moderate persistent vs. intermittent | 1.43 | 0.08 | <0.0001 | 1.88 | 0.06 | <0.0001 | 1.39 | 0.11 | <0.0001 |
| Severe persistent vs. intermittent | 2.06 | 0.11 | <0.0001 | 2.91 | 0.08 | <0.0001 | 2.04 | 0.16 | <0.0001 |
| Age at baseline interview | 0.001 | 0.01 | 0.59 | −0.02 | 0.01 | 0.01 | 0.03 | 0.01 | 0.03 |
| Age (<10 yr) | −0.21 | 0.01 | <0.0001 | −0.17 | 0.01 | <0.0001 | −0.04 | 0.01 | <0.0001 |
| Age (≥10 yr) | −0.29 | 0.01 | <0.0001 | −0.12 | 0.01 | <0.0001 | −0.18 | 0.01 | <0.0001 |
Definition of abbreviation: Est = parameter estimate.
Analysis results of yearly counts of hospitalizations, ED visits, and MD visits (sum of regularly scheduled and urgent office visits) for the School data are shown in Table 4. Only ED visits were significantly reduced with age regardless of baseline asthma severity. The common odds of having a higher count of ED visits (e.g., one or more vs. zero ED visits, two or more vs. one or fewer ED visits, and so forth) are estimated to be 0.83 (95% CI, 0.71–0.96) and 0.66 (95% CI, 0.50–0.87) times lower for each year increase in age in children with intermittent, mild, or moderate persistent asthma and in children with severe persistent asthma at baseline, respectively. Hospitalizations and MD visits were not significantly associated with age.
Table 4.
Odds ratio estimates and the corresponding 95% confidence intervals from the results of proportional odds models for yearly hospitalizations, emergency department visits, physician office visits (using Elementary School and Middle School data)
| Parameter | Hospitalizations |
ED Visits |
MD Visits |
||||||
|---|---|---|---|---|---|---|---|---|---|
| OR | 95% CI | OR | 95% CI | OR | 95% CI | ||||
| Severe persistent vs. not severe persistent | 2.26 | 1.29 | 3.96 | 2.58 | 1.77 | 3.76 | 1.72 | 1.26 | 2.36 |
| Age at baseline interview | 0.99 | 0.87 | 1.13 | 1.02 | 0.94 | 1.11 | 1.12 | 1.05 | 1.20 |
| Age, yr: not severe persistent | 0.86 | 0.65 | 1.13 | 0.83 | 0.71 | 0.96 | 1.01 | 0.90 | 1.13 |
| Age, yr: severe persistent | 0.81 | 0.54 | 1.20 | 0.66 | 0.50 | 0.87 | 1.06 | 0.85 | 1.34 |
| Household income, $10,000* | 0.93 | 0.84 | 1.03 | 0.88 | 0.83 | 0.93 | 0.96 | 0.92 | 1.00 |
| Antiinflammatory medications† | 2.67 | 1.70 | 4.21 | 3.08 | 2.26 | 4.21 | 3.30 | 2.53 | 4.31 |
| Bronchodilator | 7.68 | 4.11 | 14.35 | 6.12 | 4.47 | 8.37 | 7.35 | 5.71 | 9.47 |
Definition of abbreviations: CI = confidence interval; ED = emergency department; MD = physician; OR = odds ratio.
The probabilities that the number of health care use is greater than 0, 1, and 2 are predicted.
Household income has 10 levels ($10,000 difference between adjacent levels), with the lowest income level as ≤$10,000 and the highest level as >$100,000, and is considered as a continuous variable in analysis.
Antiinflammatory medications include any prescribed antiinflammatory medications, including oral steroids and leukotriene modifiers.
Table 5 shows the results of the same models using the Pediatric Practice sample data. Results in this dataset are a little different from the findings in the School data. Older age is significantly associated with decreased hospitalizations, ED visits, and MD visits for children with intermittent, mild, or moderate persistent asthma at baseline. For children with severe persistent asthma at baseline, increased age is only significantly associated with fewer MD visits.
Table 5.
Odds ratio estimates and the corresponding 95% confidence intervals from the results of proportional odds models for yearly hospitalizations, emergency department visits, physician office visits (using Pediatric Practice data)
| Parameter | Hospitalizations |
ED Visits |
MD Visits |
||||||
|---|---|---|---|---|---|---|---|---|---|
| OR | 95% CI | OR | 95% CI | OR | 95% CI | ||||
| Severe persistent vs. not severe persistent | 0.59 | 0.15 | 2.26 | 2.26 | 0.91 | 5.57 | 1.82 | 0.81 | 4.10 |
| Age at baseline interview | 0.86 | 0.78 | 0.95 | 0.90 | 0.84 | 0.97 | 0.96 | 0.91 | 1.00 |
| Age, yr: not severe persistent | 0.64 | 0.48 | 0.86 | 0.66 | 0.55 | 0.78 | 0.48 | 0.42 | 0.54 |
| Age, yr: severe persistent | 0.58 | 0.17 | 1.91 | 0.94 | 0.54 | 1.63 | 0.61 | 0.37 | 1.00 |
| Household income, $10,000* | 0.78 | 0.70 | 0.87 | 0.85 | 0.79 | 0.92 | 0.94 | 0.89 | 0.99 |
| Antiinflammatory medications† | 4.88 | 2.28 | 10.42 | 3.00 | 1.96 | 4.60 | 3.52 | 2.66 | 4.65 |
| Bronchodilator | 1.51 | 0.85 | 2.68 | 1.83 | 1.23 | 2.70 | 2.01 | 1.47 | 2.74 |
Definition of abbreviations: CI = confidence interval; ED = emergency department; MD = physician; OR = odds ratio.
The probabilities that the number of health care use is greater than 0, 1, and 2 are predicted.
Household income has 10 levels ($10,000 difference between adjacent levels), with the lowest income level as ≤$10,000 and the highest level as >$100,000, and is considered as a continuous variable in analysis.
Antiinflammatory medications include any prescribed antiinflammatory medications, including oral steroids and leukotriene modifiers.
Results of the Adjustment for the Natural Decline in Asthma Symptoms and Health Care Use
Table 6 shows the mean estimated effect sizes obtained by, respectively, the gold standard and ALSE adjusted and unadjusted approaches under several parameter settings and more than 1,000 simulations. Parameter settings were based on the outcomes of log symptom days and categorical health care use similar to those in the existing cohort data. Comparing the true intervention effect size to those estimated by the RCT cohorts, it is evident that the adjustment yielded very satisfactory results. It produced estimates very close to the true effects under all simulation scenarios and very marginally different from those given by the gold standard (true control group) approach. In contrast, the unadjusted estimates are biased toward overestimation up to 295%, when only pre-post intervention data were used in the estimation. Such overestimation of effects is the consequence of ignoring the natural decline with increased age of the child and thus is misleading regarding the impact of an intervention.
Table 6.
Application of adaptive least squares estimation to intervention effect accounting for natural growth effect on log symptom days and categorical health care use in various simulation settings (1,000 simulations, 1,000 bootstrap replications)
| Simulation Setting | Ncon | Nint | Var(b) | Var(w) | True Effect | Gold | Adjusted | Unadjusted |
|---|---|---|---|---|---|---|---|---|
| Elementary School and Middle School | 400 | 200 | 1.50 | 1.50 | −0.2 | −0.2084 | −0.1918 | −0.5009 |
| 400 | 400 | 1.50 | 1.50 | −0.2 | −0.2003 | −0.2025 | −0.5007 | |
| 400 | 600 | 1.50 | 1.50 | −0.2 | −0.1983 | −0.2005 | −0.4981 | |
| 400 | 200 | 3.00 | 3.00 | −0.1 | −0.0977 | −0.0884 | −0.2712 | |
| 400 | 400 | 3.00 | 3.00 | −0.1 | −0.1005 | −0.1035 | −0.2710 | |
| 400 | 600 | 3.00 | 3.00 | −0.1 | −0.0976 | −0.1008 | −0.2673 | |
| Pediatric Practice | 500 | 250 | 0.26 | 1.10 | −0.4 | −0.4006 | −0.4013 | −0.8506 |
| 500 | 500 | 0.26 | 1.10 | −0.4 | −0.4008 | −0.4012 | −0.8511 | |
| 500 | 750 | 0.26 | 1.10 | −0.4 | −0.4002 | −0.3996 | −0.8501 | |
| 500 | 250 | 0.60 | 2.20 | −0.2 | −0.2010 | −0.2002 | −0.7909 | |
| 500 | 500 | 0.60 | 2.20 | −0.2 | −0.2012 | −0.2018 | −0.7916 | |
| 500 | 750 | 0.60 | 2.20 | −0.2 | −0.2003 | −0.1995 | −0.7901 |
Definition of abbreviations: Ncon = subject number in existing control cohort data; Nint = subject number in new intervention cohort data; Var(b) = between-subject variance; Var(w) = within-subject variance.
True effect is the true intervention effect (i.e., intervention by time interaction effect); negative sign indicates that in the simulation the intervention group was assumed to have a greater decline in symptom days or health care use over time than the control group.
Discussion
This study demonstrates that among African American and white children declines in childhood asthma symptom frequency and asthma-related health care use in ages from 2 to 15 years must be accounted for in the evaluation of intervention effects. The findings are in agreement with those reported in the literature (4–9). The new contribution of this study is the proposed ALSE method that enables adjustment for the natural decline not accounted for in a pre-post study by borrowing data from existing control cohorts.
The ANDAO has limitations. First, the adjustment based on the resampling technique is applicable when a similar age range and outcome and control variables are available. Of course, the ideal application condition for the adjustment would be a pre-post intervention study that recruits subjects from a study population highly similar to the available control cohorts (e.g., our school samples are mostly African Americans from low-income families). Second, as noted, the adjustment is limited to the presence of common covariates in the control and intervention databases. Variables not in databases may be important to asthma control. Third, the reliability of the estimated rate for symptom reduction may depend on the study design, in which the time interval window to collect repeated measurements in the control cohort is of particular importance. Such window may be related to the underlying development of children.
Clinical Implications for Pre-Post Program Evaluations
The existence of natural decline in asthma symptoms and health care use is confirmed in our analysis of the study cohorts. Merely comparing the outcomes before and after intervention without accounting for the fact of natural decline will result in an overestimated intervention effect. As illustrated here, by using an existing comparison cohort, such as the Michigan cohorts, such bias in estimation can be corrected. Two clinical implications are evident. First is to avoid acceptance of pre-post study results in assessing interventions aiming for reduction in asthma symptoms and health care use of children. Second is to recognize that there are ways to adjust data and uncover the real effects of intervention.
Methodological Implications for Researchers and Program Evaluators
In general, an RCT design is recommended for an unbiased estimate of intervention effect. When a control group is not available in an evaluation design, the proposed adjustment may provide an alternative to correct potential overestimation of intervention effect. The adjustment entails a resampling technique using the existing control cohorts. Further studies may build on this research to explore other means that provide program evaluators with options for identifying true effects when control groups are not possible in their evaluations.
Acknowledgments
Acknowledgment
The authors thank Amy Friedman Milanovich, M.P.H., and Margaret Wilkin, M.P.H., for their assistance with the study presented here.
Footnotes
Supported by the NHLBI for: the School-Based Approaches to Help Preteens Manage Asthma project grant R01 HL068654; the Interventions to Improve Asthma Management and Prevention at School project grant HR-56028; and the MD/Family Partnership: Education in Asthma Management, grant HL-44976 of the Lung Division of NHLBI. Also supported by the Robert Wood Johnson Foundation.
Author Contributions: Y.-A.K., P.X.S., and N.M.C. contributed to development of study, data analysis and interpretation, and writing of the manuscript.
This article has an online supplement, which is accessible from this issue’s table of contents at www.atsjournals.org
Author disclosures are available with the text of this article at www.atsjournals.org.
References
- 1.Clark N, Lachance L, Milanovich AF, Stoll S, Awad DF. Characteristics of successful asthma programs. Public Health Rep. 2009;124:797–805. doi: 10.1177/003335490912400606. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Davis A, Savage Brown A, Edelstein J, Tager IB. Identification and education of adolescents with asthma in an urban school district: results from a large-scale asthma intervention. J Urban Health. 2008;85:361–374. doi: 10.1007/s11524-008-9266-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mosnaim GS, Li H, Damitz M, Sharp LK, Li Z, Talati A, Mirza F, Richardson D, Rachelefsky G, Africk J, et al. Evaluation of the Fight Asthma Now (FAN) program to improve asthma knowledge in urban youth and teenagers. Ann Allergy Asthma Immunol. 2011;107:310–316. doi: 10.1016/j.anai.2011.07.008. [DOI] [PubMed] [Google Scholar]
- 4.To T, Gershon A, Wang C, Dell S, Cicutto L. Persistence and remission in childhood asthma: a population-based asthma birth cohort study. Arch Pediatr Adolesc Med. 2007;161:1197–1204. doi: 10.1001/archpedi.161.12.1197. [DOI] [PubMed] [Google Scholar]
- 5.Koh MS, Irving LB. The natural history of asthma from childhood to adulthood. Int J Clin Pract. 2007;61:1371–1374. doi: 10.1111/j.1742-1241.2007.01426.x. [DOI] [PubMed] [Google Scholar]
- 6.Piippo-Savolainen E, Korppi M. Wheezy babies—wheezy adults? Review on long-term outcome until adulthood after early childhood wheezing. Acta Paediatr. 2008;97:5–11. doi: 10.1111/j.1651-2227.2007.00558.x. [DOI] [PubMed] [Google Scholar]
- 7.Sears MR, Greene JM, Willan AR, Wiecek EM, Taylor DR, Flannery EM, Cowan JO, Herbison GP, Silva PA, Poulton R. A longitudinal, population-based, cohort study of childhood asthma followed to adulthood. N Engl J Med. 2003;349:1414–1422. doi: 10.1056/NEJMoa022363. [DOI] [PubMed] [Google Scholar]
- 8.Vonk JM, Postma DS, Boezen HM, Grol MH, Schouten JP, Koëter GH, Gerritsen J. Childhood factors associated with asthma remission after 30 year follow up. Thorax. 2004;59:925–929. doi: 10.1136/thx.2003.016246. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Martin AJ, McLennan LA, Landau LI, Phelan PD. The natural history of childhood asthma to adult life. BMJ. 1980;280:1397–1400. doi: 10.1136/bmj.280.6229.1397. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Guilbert TW, Garris C, Jhingran P, Bonafede M, Tomaszewski KJ, Bonus T, Hahn RM, Schatz M. Asthma that is not well-controlled is associated with increased healthcare utilization and decreased quality of life. J Asthma. 2011;48:126–132. doi: 10.3109/02770903.2010.535879. [DOI] [PubMed] [Google Scholar]
- 11.Clark NM, Brown R, Joseph CL, Anderson EW, Liu M, Valerio M, Gong M. Issues in identifying asthma and estimating prevalence in an urban school population. J Clin Epidemiol. 2002;55:870–881. doi: 10.1016/s0895-4356(02)00451-1. [DOI] [PubMed] [Google Scholar]
- 12.Clark NM, Shah S, Dodge JA, Thomas LJ, Andridge RR, Little RJ. An evaluation of asthma interventions for preteen students. J Sch Health. 2010;80:80–87. doi: 10.1111/j.1746-1561.2009.00469.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Cabana MD, Slish KK, Evans D, Mellins RB, Brown RW, Lin X, Kaciroti N, Clark NM. Impact of physician asthma care education on patient outcomes. Pediatrics. 2006;117:2149–2157. doi: 10.1542/peds.2005-1055. [DOI] [PubMed] [Google Scholar]
- 14.National Asthma Education and Prevention Program. Expert Panel Report 3: Guidelines for the Diagnosis and Management of Asthma. Bethesda, MD: National Institutes of Health; National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program; 2007.
- 15.Little RJA, Rubin DB. Wiley series in probability and statistics. Hoboken, NJ: Wiley; 2002. Statistical analysis with missing data, 2nd ed. [Google Scholar]
- 16.Bacon DW, Watts DG. Estimating transition between two intersecting straight lines. Biometrika. 1971;58:525–534. [Google Scholar]
- 17.Akaike H. Factor analysis and AIC. Psychometrika. 1987;52:317–332. [Google Scholar]
- 18.Mullahy J. Specification and testing of some modified count data models. J Econom. 1986;33:341–365. [Google Scholar]
- 19.Min YY, Agresti A. Random effect models for repeated measures of zero-inflated count data. Statistical Modelling. 2005;5:1–19. [Google Scholar]
- 20.Base SAS 9.2 Procedure Guide. Cary, NC: SAS Institute Inc.; 2009. [Google Scholar]
- 21.Breslow N. Statistical methods for censored survival data. Environ Health Perspect. 1979;32:181–192. doi: 10.1289/ehp.7932181. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Efron B, Tibshirani R. Monographs on statistics and applied probability. New York: Chapman & Hall; 1993. An introduction to the bootstrap. [Google Scholar]