Daily Diaries vs Retrospective Questionnaires to Assess Asthma Control and Therapeutic Responses in Asthma Clinical Trials

Abstract

Background:

Presently, there is insufficient information to compare the value of daily diaries vs retrospective questionnaires for assessing symptoms in relationship to asthma control in clinical trials. Daily symptom diaries are often burdensome to gather, incomplete, susceptible to fabrication, and of questionable reliability. There is also concern that retrospective symptom questionnaires may be subject to poor recall and may be insensitive.

Methods:

To compare these two methods of assessing symptoms reporting, we analyzed data collected during the Best Add-on Therapy Giving Effective Responses (BADGER) trial. During the trial, asthma control in 182 children aged 6 to 17 years was assessed in two ways: (1) by asthma control days (ACDs) determined by manually recorded daily diary symptom and rescue medication use scores and (2) by monthly retrospective report of symptoms embedded within the age-appropriate version of the Asthma Control Test (ACT). Correlations between ACDs and ACT scores were analyzed, and the sensitivity of each method for measuring asthma control and determining the differential response among the three BADGER treatments was evaluated.

Results:

Although validated using a 4-week recall period, ACT correlated better with daily diary information from the last 2 weeks of the 4-week recall (r = 0.46) than from the first 2 weeks (r = 0.34). In addition, clinically significant differential treatment responses were detected using ACDs but not ACT scores .

Conclusions:

The results of this study indicate that daily diaries used to determine ACDs can be a more sensitive tool than ACT for assessing differential treatment responses with respect to asthma control.

One of the major outcomes in asthma clinical research is the measurement of participants’ symptoms. Symptom burden is essential to determine intervention effectiveness. However, obtaining an accurate measurement is challenging. Because there is variation in asthma symptom assessment methods, standardization of symptom measures is important for both internal validity of individual trials and cross-trial comparisons.

Two common methods of assessing symptom burden are real-time recordings in daily diaries and retrospective symptom recall through questionnaire, usually over the previous 1 to 4 weeks. Real-time reporting requires participants or caregivers to record daily asthma symptoms between research visits. Diaries are burdensome to complete, often illegible or incomplete, and occasionally lost. Written daily diaries may be unreliable because of lack of compliance.¹ Retrospective questionnaires ask participants or caregivers to recall asthma symptoms over the prior few weeks at the start of the research visit with the study coordinator present. Although several validated questionnaires are available, they have inherent limitations because of the use of recall periods that are less likely to precisely capture fluctuations in asthma symptoms during more remote times.

At the March 2010 National Institutes of Health-supported Asthma Outcomes workshop,² an expert committee reviewed the research instruments used to measure asthma symptoms and recently published its recommendations.² The report specifically discussed symptom measures and the current assumption, without formal evidence, that more accurate and precise trial data are obtained when both daily diaries and monthly questionnaires are used. The lack of data highlights the need to answer two important questions: Do we really need both instruments to measure outcomes? Are there certain trial designs in which both tools are required to determine a particular outcome and others in which use of only the less-burdensome retrospective questionnaires would be adequate?²

These questions inspired the present analysis, which used data from the Best Add-on Therapy Giving Effective Responses (BADGER) trial.³ The comparative utility of two approaches for measuring symptoms to determine a differential treatment response was assessed first by the asthma control days (ACDs) outcome, which was determined by participant diary-recorded symptom frequency, severity, and rescue medication use (e-Figs 1, 2 ^{(764KB, pdf)} ), and second, by the Asthma Control Test (ACT) (ages ≥ 12 years) or Childhood Asthma Control Test (C-ACT) (ages < 12 years) monthly retrospective questionnaires (e-Figs 3, 4 ^{(764KB, pdf)} ).

Materials and Methods

Study Participants

From March 2007 through July 2008, children aged 6 to 17 years were recruited at Childhood Asthma Research and Education (CARE) Network centers to participate in the BADGER trial.³ Each center’s institutional review board approved the study, and parents or guardians provided written informed consent. In addition, children aged < 7 years provided oral consent, and older children provided written consent (Health Science IRB No. 2006-0137).

Study Design

Details regarding the BADGER protocol design have been published.³ In brief, children whose asthma was uncontrolled after ≥ 2 weeks of treatment with fluticasone 100 μg bid (Flovent Diskus; GlaxoSmithKline plc) entered a randomized, 48-week, double-blind, placebo-controlled, three-treatment, three-period, crossover trial. During each 16-week period, children received one of three step-up treatments.

The primary aim of the BADGER trial was to determine whether a differential response to the three step-up treatments (fluticasone 250 μg bid, fluticasone 100 μg bid plus the long-acting β-agonist salmeterol 50 μg bid [Advair Diskus; GlaxoSmithKline plc], and fluticasone 100 μg bid plus the leukotriene receptor antagonist montelukast 5 or 10 mg daily [Singulair; Merck & Co, Inc]) existed on the basis of an assessment of two components from the impairment domain (FEV₁ and ACDs) and one from the risk domain (exacerbations).⁴ For the current analysis, only ACDs are used as the gold standard to define the differential response.

Evaluation of Symptoms

Children were evaluated every 4 weeks (Fig 1). The ACT was administered at the beginning of each visit to avoid bias from additional medical information sharing during the visit regarding the level of asthma control (eg, FEV₁). We administered the validated ACT for children aged ≥ 12 years,⁵ with higher scores (range, 5-25) indicating greater control (minimally important difference [MID], 3.0⁶), and the C-ACT for children aged 6 to 11 years,^7,8 with higher scores (range, 0-27) indicating greater control (no validated MID published) (e-Figs 3, 4 ^{(764KB, pdf)} ).

Figure 1.

Study design. *During each period, patients received ICS plus one of three add-on treatments: ICS, long-acting β-agonist (LABA), or leukotriene receptor antagonist (LTRA). ACT = Asthma Control Test; c-ACT = Childhood Asthma Control Test; ICS = inhaled corticosteroid.

Although it has yet to be validated because of the challenge of diary validation procedures,^9‐11 the BADGER asthma diary was created and used in prior CARE Network published trials.^12,13 According to BADGER procedures, coordinators comprehensively reviewed the diary details and the proper reporting of symptoms with the participant and caregiver during the first visit, with reinforcement provided at subsequent visits. Daily procedure adherence was emphasized. Diary information was subsequently entered into the study database.

Daily components were recorded in this diary instead of an overall composite (e-Figs 1, 2 ^{(764KB, pdf)} ). Entries were used to determine an ACD on the basis of a composite of these symptoms. An ACD was defined as a day without use of albuterol rescue (excluding preexercise use of albuterol), use of nonstudy asthma medications, daytime or nighttime symptoms, an unscheduled health-care provider visit for an asthma exacerbation, and school absenteeism for an asthma exacerbation. Peak expiratory flow (PEF) measurements of < 80% of the predetermined reference value, although used to define ACDs in the BADGER trial, were not used in the present analysis to facilitate better harmonization for comparisons between ACD and ACT because lung function measures, such as PEF, are not included in the ACT instrument. If no diary information was recorded on a specific day, that day was not included in the determination of ACDs; 89% of days encompassed by ACT measurements had corresponding diary data. Annualized asthma control days (AACDs) were calculated as 365 times the proportion of ACDs during the final 12 weeks of each 16-week BADGER treatment period, which were adjusted for seasonal differences.

Statistical Analysis

The correlations between ACDs and ACT scores were examined as well as the sensitivity of ACT as a determinant of differential response compared with ACD. Initially, Pearson correlations between ACT scores and ACDs were calculated. For the initial analysis, ACDs were determined by only the diary entries corresponding to the time frame covered by the ACT (4 weeks). Correlations were analyzed separately for each study visit and plotted serially across visits. Secondary analyses examined correlations between the ACT score and ACDs determined over two other time periods that partially covered the time frame of the ACT (ie, the first and last 2 weeks).

As indicated previously, ACDs have been used as the primary outcome or part of the primary composite outcome in two published CARE Network trials^12,13 to determine differential response between treatments. Therefore, sensitivity analyses were based on the comparison of ACT with ACDs for determining BADGER differential response. We did not attempt to further determine the specificity of either measure against external reference standards. Thresholds for determining differential response with respect to ACDs and the ACT were based on published results and recommendations. For ACDs, the differential response threshold was 31 AACDs.³ This threshold was vetted by the Protocol Review Committee and Data and Safety Monitoring Board for the CARE Network. For ACT, the published MID is 3.0.⁶

As previously described, the ACD definition used for these analyses was different from that of previously published trials. To harmonize with ACT, PEF was eliminated as a criterion for determining an ACD because it is not used to determine an ACT score. The analysis was carried out using the original ACD definition to assess the impact of eliminating PEF; no significant differences were noted (data not shown).

Results

Of the 480 children enrolled after the BADGER run-in phase, 182 underwent randomization, and 157 completed all three study periods (Table 1). A total of 165 children completed at least two study periods, which permitted determination of a differential response. Table 1 presents age-stratified relevant baseline demographic and physiologic data. Participants completed 90% of study visits and provided sufficient data in their daily diaries to determine control status on 96% of days.

Table 1.

—Baseline Participant Characteristics

	Age Group
Characteristic	6-11 y	12-17 y
No. children	126	56
Age, y	9.1 ± 1.5	14.7 ± 1.7
Male sex	83 (66)	36 (64)
Self-reported race/ethnicity
Hispanic or Latino	38 (30)	22 (39)
Non-Hispanic white	54 (43)	20 (36)
Black	37 (29)	12 (21)
Hispanic white	28 (22)	15 (27)
Other	7 (6)	9 (16)
Height, cm	134.3 ± 10.8	164.2 ± 11.0
Weight, kg	36.1 ± 12.7	63.4 ± 17.2
BMI, kg/m2	19.6 ± 4.5	23.3 ± 4.8
ACDs during worst 2 wk of run-in period, %	30 ± 21	36 ± 23
ACT or C-ACT scorea	20.5 ± 3.8	19.8 ± 3.4

Data are presented as mean ± SD or No. (%), unless otherwise indicated. Percentages may not total 100 because of rounding. ACD = asthma control day; ACT = Asthma Control Test; C-ACT = Childhood Asthma Control Test.

^aScores on the ACT (for patients aged ≥ 12 y) are measured on a scale of 5 to 25, with higher scores indicating greater control. Scores on the C-ACT (for children aged 4-11 y) are measured on a scale from 0 to 27, with higher scores indicating greater control.

Correlation of ACD and ACT Assessments

Figure 2 shows the correlations between ACDs determined over three different time periods and the ACT score at each visit. The ACT score correlated significantly better with ACDs determined over the previous 4 weeks (r = 0.45) and most recent 2 weeks (r = 0.46) than over the farthest 2 weeks (r = 0.34, P < .05 for comparison against previous 4 weeks and most recent 2 weeks). The validated ACT recall period is 4 weeks.

Figure 2.

Correlations between percent asthma control days and ACT. See Figure 1 legend for expansion of abbreviation.

No significant differences in patterns of correlations were found between the composite ACD and the retrospective questionnaire scores when stratified by age group (6-11 years vs 12-17 years), sex, or season. Results were also independent of whether the C-ACT or the ACT was administered. Whether the strength of the correlations differed according to underlying symptom burden as measured by ACT was also analyzed. When the ACT scores were stratified above and below the accepted control reference score of 19, similar correlations with ACDs were found for both strata (data not shown).

The Use of ACT Scores to Evaluate Differential Treatment Responses

Figure 3 depicts the sensitivity of ACT to ascertain differential treatment responses compared with ACDs. Each data point represents the largest differential treatment response for each child (ie, the difference in ACDs during the treatment having best response vs worst response). Only data points where AACDs detected a significant differential response (> 31) are included. The top portion of Figure 3 represents children in whom ACT detected a differential response that agreed with the ACDs (31%). The lower portion represents children in whom the ACT score detected a differential response discordant (ie, in the opposite direction) from that detected by ACDs (3%). For the majority of children with an ACD-defined differential response, the ACT did not detect a differential response, as shown in the middle portion of Figure 3 (66%).

Figure 3.

Association between greatest ACD differential response and ACT differential response in all children. Each data point is a difference between two treatment period averages. On the y-axis, the reference lines at −3 and +3 reflect the published clinically minimally important differences for the ACT. All points to the right of the vertical line represent a significant differential response on the basis of criteria used in the original Best Add-on Therapy Giving Effective Responses (BADGER) trial (ie, > 31 annualized ACD, which corresponds to 8.5% ACD). For threshold values of < 31 d, the observed relationships did not change significantly; therefore, the final analyses were based on the 31-d threshold. Correlation = 0.24. ACD = asthma control day. See Figure 1 legend for expansion of other abbreviation.

Discussion

The purpose of this analysis was to investigate whether certain asthma clinical trial designs require use of both daily diaries and retrospective questionnaires to determine a particular outcome. To our knowledge, this is one of only two studies^14,15 to have conducted such a comparison and the only analysis that used the ACT, a commonly applied tool in research and clinical practice. Both the ACT and the C-ACT instruments used in BADGER have been meticulously validated.^5,7,8,16 The tools demonstrate good receiver operating characteristic values relative to the specialists’ ratings of asthma control as well as good performance of scores in their ability to discriminate various levels of clinical variables, including spirometry and quality-of-life parameters. This post hoc analysis precludes our ability to directly compare two methods of assessing symptom reporting but provides significant novel information on the topic.

Although the ACT is a validated measure of asthma control over the prior 4 weeks, the present analysis suggests that the ACT correlates more strongly with ACDs determined by daily symptom diaries over the last 2 weeks of the validated recall period than when determined over the first 2 weeks. One would intuit that this stronger correlation with the most recent time period is due to hampered recall but would acknowledge that the discrepancy may be inherent to the design of the ACT.

Although the ACT and the diary-calculated ACDs showed a positive correlation, ACT scores were not as sensitive in detecting a differential response. Because both means of assessment are designed to evaluate various aspects of control, the reasons for the observed discrepancy are of interest. The ACT was designed for use in a clinical setting, and scores were referenced for validity to a clinician impression of global asthma control status. Being a restrospective questionnaire, the ACT depends on accurate recall of symptoms and asthma control over time (e-Fig 3 ^{(764KB, pdf)} ). In contrast, the ACD outcome has largely been used in research and functions more as a real-time (daily) evaluation of asthma control. The ACT entails a more global assessment of control, including questions beyond strict query of symptoms, such as the patient’s rating of asthma control in the past 4 weeks, whereas daily diary data provide a more granular assessment of symptoms. The variable nature of asthma symptoms and the need to recall only 12 h (ACD calculated through diary entries) vs 4 weeks (ACT determined) of symptoms, therefore, would potentially favor the ACD determination to be more sensitive to detect a differential treatment response.

Another explanation is the difference in determining the MID. The ACD threshold of 31 AACDs used in this analysis was determined by consensus opinion of the CARE Network Steering Committee,³ whereas the ACT threshold score of 3 was determined by a prospective study purposefully designed to evaluate longitudinal changes in asthma control.⁶ In the present analysis, ACT had low sensitivity compared with AACD (31%) when the validated MID of 3 was applied. As expected, however, the sensitivity of ACT increases when lower thresholds for MID are applied (ie, an MID of 2 yields 46% sensitivity, and an MID of 1 yields 68% sensitivity).

The present results diverge from a published observational study reporting that although both the Asthma Control Questionnaire and the Asthma Control Diary were valid instruments for measuring asthma control, the questionnaire had slightly better discriminative and evaluative measurement properties than the diary.¹⁵ Differences between methods used in that study compared with those in the present analysis were a smaller number of patients (n = 50), an older study population (aged 17-70 years), shorter study duration (9 weeks), and diary use for only 1 week prior to questionnaire administration. The present study encompassed a longer duration (12 months) in a younger population (aged 6-17 years) and with a much larger sample size. In addition, diary collection over a longer period (4 weeks) prior to retrospective questionnaire administration provides more robust results. Moreover, this is the only analysis known to us with the unique component of evaluating symptom measures in the context of determining a differential treatment response.

We know that trial diary data are of questionable reliability because they are burdensome to gather, often incomplete because of illegibility or loss, and susceptible to fabrication. The estimated staff and family time burden for ACT is ≤ 5 min. The initial study staff time to teach a family to complete the diary varied from 10 to 20 min. Follow-up training and diary review took 20 min per study visit, and electronic data entry took 5 min. However, technological advancements in the past decade leading to the development of electronic diaries may both increase patient adherence and decrease the errors and shortcomings associated with paper diaries.^17‐20 However, electronic diaries also have disadvantages, such as cost, operator error, limited user interfaces necessary for data entry and downloading, and equipment reliability.^1,17,21,22 Nonetheless, a joint American Thoracic Society/European Respiratory Society task force concluded that overall, their many benefits outweigh their disadvantages.²³

In conclusion, this analysis demonstrates similarities and differences between daily diaries and retrospective questionnaires used to measure symptoms in asthma clinical trials and indicates that diaries generally should not be used interchangeably with retrospective questionnaires. For example, study populations where a higher degree of adherence to completion of daily diaries is achievable, such as noted in the present analysis, may be better suited to daily diaries. Another consideration would be the nature of the hypotheses that mandate the use of specific outcome measures. For example, for longitudinal trials where the primary outcome may not require a high degree of day-to-day precision in terms of variable symptoms but a more global view of asthma control, the retrospective questionnaire may be sufficient. On the other hand, in trials designed to determine differential treatment responses, the incorporation of daily diary data and repeated measures (preferably electronically to address the limitations of paper diaries) in addition to retrospective questionnaires may be necessary for greater power and precision.

Supplementary Material

Online Supplement

Abbreviations

AACD

annualized asthma control day

ACD

asthma control day

ACT

Asthma Control Test

BADGER

Best Add-on Therapy Giving Effective Responses

C-ACT

Childhood Asthma Control Test

CARE

Childhood Asthma Research and Education

MID

minimally important difference

PEF

peak expiratory flow