Longitudinal Validation of a Tool for Asthma Self-Monitoring

Abstract

OBJECTIVES:

To establish longitudinal validation of a new tool, the Asthma Symptom Tracker (AST). AST combines weekly use of the Asthma Control Test with a color-coded graph for visual trending.

METHODS:

Prospective cohort study of children age 2 to 18 years admitted for asthma. Parents or children (n = 210) completed baseline AST assessments during hospitalization, then over 6 months after discharge. Concurrent with the first 5 AST assessments, the Asthma Control Questionnaire (ACQ) was administered for comparison.

RESULTS:

Test–retest reliability (intraclass correlation) was moderate, with a small longitudinal variation of AST measurements within subjects during follow-ups. Internal consistency was strong at baseline (Cronbach’s α 0.70) and during follow-ups (Cronbach’s α 0.82–0.90). Criterion validity demonstrated a significant correlation between AST and ACQ scores at baseline (r = −0.80, P < .01) and during follow-ups (r = −0.64, −0.72, −0.63, and −0.69). The AST was responsive to change over time; an increased ACQ score by 1 point was associated with a decreased AST score by 2.65 points (P < .01) at baseline and 3.11 points (P < .01) during follow-ups. Discriminant validity demonstrated a strong association between decreased AST scores and increased oral corticosteroid use (odds ratio 1.13, 95% confidence interval, 1.10–1.16, P < .01) and increased unscheduled acute asthma visits (odds ratio 1.23, 95% confidence interval, 1.18–1.28, P < .01).

CONCLUSIONS:

The AST is reliable, valid, and responsive to change over time, and can facilitate ongoing monitoring of asthma control and proactive medical decision-making in children.

What’s Known on This Subject:

To prevent asthma exacerbations, asthma guidelines recommend ongoing monitoring of patients’ asthma symptoms to promote timely adjustments of therapy to achieve and maintain optimal control. Existing tools, validated for ongoing monitoring, have significant limitations in children.

What This Study Adds:

Our study established longitudinal validation of the Asthma Symptom Tracker, a novel tool designed for use by children or their parents to facilitate ongoing monitoring of patients’ asthma symptoms and proactive medical decision-making to prevent acute exacerbations.

Asthma is the most common pediatric chronic illness, with a lifetime prevalence of 13.8% in the United States¹ and a significant impact on the child’s and family’s quality of life and on health care costs.² In 2009, ∼7.1 million children under age 18 years had asthma, and 4.1 million had an asthma exacerbation.² The total costs for pediatric asthma in the United States are estimated at $15.9 billion annually.^3,4

Asthma is one of the most preventable conditions necessitating frequent use of acute care services, including emergency department (ED) and hospital admissions in children.^5–8 Overall, use of acute care services in children accounts for 54% of the direct asthma medical expenditures,⁹ and the major reason is poor asthma control in the ambulatory setting.^10–13 Poor control leads to recurrent asthma exacerbations, poor quality of life, increased school absenteeism, and increased ED and hospital admissions.^14–16

To prevent asthma exacerbations, the National Asthma Education and Prevention Program Expert Panel Report 3¹⁷ guidelines recommend ongoing monitoring of patients’ asthma symptoms to promote timely adjustments of controller therapy to achieve and maintain optimal control. Yet engaging children and their parents in ongoing monitoring of asthma symptoms remains challenging.¹⁸ Parents often underestimate their child’s risk for exacerbations and lack effective tools to help them monitor their child’s asthma.^19,20 Similarly, primary care providers (PCPs) often lack the tools, incentives, time, and resources to adequately monitor their patients outside clinical encounters.²¹

Several tools for assessing asthma control have been developed, but few have been validated for ongoing monitoring, and studies have reported significant limitations in children.^22–27 We have developed a new tool, the Asthma Symptom Tracker (AST) or Asthma Tracker, a paper-based, patient-centered tool designed to facilitate ongoing monitoring of asthma control through weekly assessments of asthma symptoms. The objective of this article is to establish longitudinal validation of the AST for use by children and their parents.

Methods

Setting

The study took place at Primary Children’s Medical Center (PCMC), an academic children’s hospital in Salt Lake City, Utah affiliated with the Department of Pediatrics at the University of Utah School of Medicine. PCMC has 289 licensed beds and is owned and operated by Intermountain Healthcare.²⁸

Description of the AST

The AST was developed by a multidisciplinary pediatric care team, including physicians, nurses, health services researchers, graphic designers, and experts in quality improvement from the University of Utah and Intermountain Healthcare, with input from a national asthma expert (M.S.) and children with asthma and their parents. The AST was designed for use at home by older children (age ≥11 years), with or without initial assistance from their parents, or by parents of younger children (age <11 years). The AST is printed on 17 × 11″ format card-stock paper (Fig 1) that can be placed on a family’s refrigerator as a reminder for regular use. The tool has 4 sections: user instructions, a questionnaire, a color-coded graphic display, and color-coded decision support.

FIGURE 1.

Sample AST with longitudinal data. Red line and data points illustrate how patients can monitor their levels of asthma control over time.

The user instruction section gives a brief justification of AST use and its complementary role with the asthma action plan. This section provides directions on how to answer the questionnaire to determine the child’s current level of asthma control, how to calculate and plot the total score on the graph, and how to interpret results.

The questionnaire section includes a modified Asthma Control Test (ACT) questionnaire,²⁷ which was adapted for weekly use by changing the survey question header to “During the past week” rather than “During the past month,” as in the standard ACT.

The graphic display section provides space for plotting weekly scores (or AST scores) over time with color-coded zones (green indicates well controlled, yellow indicates not well controlled, and red indicates poorly controlled) using the standard ACT cutoff points previously validated for monthly assessments.²⁷

The decision support section includes 3 color-coded boxes that congratulate and encourage the patient or parent when asthma control scores fall within the green zone or prompt the patient or parent to schedule a follow-up PCP visit and bring the AST with them to that visit if the scores are in the yellow zone for 2 consecutive weeks or in the red zone once.

Study Design and Population

This was a prospective cohort study from March 2011 through May 2012 of children 2 to 18 years old admitted to PCMC with a primary diagnosis of asthma.

Patient Identification

An electronic system²⁹ automatically generated a daily list of eligible patients and e-mailed the list to study coordinators, who then enrolled assenting or consenting patients into the study.

Study Procedures and Data Collection

After assent or consent was signed and instructions given on use of the AST, the study coordinator assisted patients and parents in completing the baseline assessment to determine the patient’s asthma control level during the week before hospitalization. To test for reliability of the AST, the study coordinator concurrently determined the patient’s asthma control level by using a validated tool for weekly assessment, the Asthma Control Questionnaire (ACQ).²⁶ Subsequent AST assessments were completed at home by patients or parents, and the information was mailed weekly to the research team; concurrent ACQ assessments were completed over the phone^30,31 by the study coordinator because the tool was designed for use by a trained person.²⁶

As part of the validation testing, at each assessment patients or parents completed 5 additional questions assessing how many days in the previous week they used controller medications or oral corticosteroids; whether they used any nonprescription drugs to ease symptoms; whether the child had any unscheduled acute clinic, ED, or hospital visits; and who (patient or parent) completed the questionnaire.

To facilitate data collection, patients and parents were provided a package to take home including the AST with questions on 26 detachable preaddressed postcards for completing and mailing their responses to the study coordinator on a weekly basis. During a patient’s first month in the study, after a postcard was returned, the study coordinator phoned the family the same or next day to obtain a concurrent ACQ score for that week. Overall, we obtained 5 paired AST and ACQ scores during the first month, 1 during hospitalization (baseline) and 4 weekly assessments after hospital discharge. Patients or parents then completed weekly AST assessments alone until the end of the 6-month study period and mailed scores to the study coordinator. Patients received $40 in compensation for their participation. The PCMC privacy board and the University of Utah Institutional Review Board approved the study.

Statistical Analysis

Statistical analyses consisted of test–retest reliability and longitudinal variation, internal consistency, criterion validity, responsiveness, and discriminant validity of the AST scores. The Supplemental Information provides additional information.

Test–retest reliability was assessed by estimating the intraclass correlation (ICC) of the AST scores on a subset of patients whose asthma control was stable between the baseline and first follow-up ACQ assessment and between the baseline and first similar ACQ score obtained during the first 4 follow-up assessments. Stable patients included those for whom a subsequent follow-up ACQ score was within 0.25 points of the baseline ACQ score, because this was below the minimal important difference value necessary to detect clinical changes in asthma control status.²⁶ For estimation of the ICC, 1 assessment with an outlier AST score was excluded from the analysis because the patient incorrectly reversed the response scale.

Internal consistency among all 5 items of the AST was quantified by using Cronbach’s α. Because there is a known learning curve with AST use, Cronbach’s α values were computed separately at baseline and each of the 4 follow-up weeks.

Criterion validity was assessed by cross-sectional correlations between paired AST and ACQ scores at baseline and at the first 4 follow-up assessments. We applied a mixed effects model to separately estimate a baseline regression coefficient to represent the cross-sectional association between AST and ACQ scores at baseline and a pooled regression coefficient to characterize the cross-sectional association between AST and ACQ scores across the first 4 follow-ups. Lastly, in pooled analyses across the first 4 follow-up assessments, we produced receiver operating characteristic (ROC) curves to assess the capability of the AST to diagnose inadequate asthma control by using the ACQ as the gold standard, with inadequate control defined as an ACQ score ≥1.

We assessed responsiveness of the AST by characterizing the association between changes in AST and ACQ scores over the baseline and first 4 follow-up assessments. We used a mixed effects model to relate changes in AST scores as dependent variable to contemporaneous changes in the ACQ scores. We first evaluated changes from baseline to the 4 follow-up assessments; subsequently, we repeated the analysis after excluding the baseline to evaluate changes in AST scores between the first follow-up and the second to fourth follow-up assessments.

Discriminant validity was characterized by the association between change in AST scores (both as a continuous variable and dichotomized as poor control [<15] or better control [≥15]) and the use of oral corticosteroids and unscheduled acute care visits reported by patients over the full study period. Use of oral corticosteroids and unscheduled visits were selected as surrogate measures for inadequate asthma control. We used separate mixed effects logistic regression analyses with random individual effects to relate the occurrence of unscheduled visits and the use of oral corticosteroids to AST scores over the full study period.

Results

Study Population

Overall, 210 children with asthma (60% males) with a median age of 5.0 years (interquartile range 3.0–8.5 years) were enrolled in the study. Among patients, 2912 individual AST assessments were completed over the 6-month study period, with a total of 911 paired AST and ACQ assessments obtained during the first month. Of the individual AST assessments, 6.5% (188/2912) were completed by children alone, and 73.9% (2154/2912) were completed by parents. In 19.6% (570/2912) of assessments, no information was provided about who completed the form. Overall, frequency of AST use per week decreased over time. Participant attrition, including the percentage of patients remaining in the study at each assessment interval, is reported in the Supplemental Information.

Test–Retest Reliability

The ICC of AST scores for the subset of patients with stable asthma control was .65 between the first and the second ACQ follow-up assessment (n = 36) and .50 between the first and any subsequent ACQ assessment obtained in the second and fourth follow-up (n = 58). Among assessments categorized as well-controlled asthma by ACQ <1 (n = 152), the within-subject SD of the AST was 2.4 points (scale, 5–25) during follow-ups.

Internal Consistency

Internal consistency of AST scores was strong at baseline, with a Cronbach’s α value of 0.70, and increased (to 0.82, 0.83, 0.85, and 0.90, respectively) during follow-ups. We also found consistency among items of the AST through a direct association of the AST subscores q1 to q4 with improvement in overall patient’s self-perception of asthma control (q5) (see Supplemental Information).

Criterion Validity

The cross-sectional correlation between AST and ACQ scores was strong at baseline (R = −0.80, P < .01) and remained strong (R = −0.64, −0.72, −0.63, and −0.69, with P < .01), although to a lesser degree, during follow-ups. In the cross-sectional mixed effects analysis (Table 1), a 1-point higher ACQ at baseline was associated with a 2.65 (95% confidence interval [CI], 2.24–2.89, P < .0001) lower mean baseline AST score, and a 1-point higher ACQ during follow-up was associated with a 3.11 (95% CI, 2.88–3.34, P < .0001) lower mean follow-up AST score. The ROC curve (Fig 2) depicts a strong relationship between AST and ACQ scores over the first 4 follow-up assessments. The area under the curve of the relationship between AST and ACQ scores at baseline was 0.96 and remained high (0.86, 0.86, 0.75, and 0.90) during each follow-up.

TABLE 1.

Results of Cross-Sectional and Longitudinal Mixed Effects Analyses

Type of Analysis	Covariates	Coefficient	Estimate (95% CI)	P
Cross-sectional association of AST with ACQa	Indicator for follow-up versus baseline visit, follow-up week	Relation of AST and ACQ at baseline	−2.65 (−2.89 to –2.24)	<.001
		Relation of AST and ACQ at follow-up assessments	−3.11 (−3.34 to −2.88)	<.001
Longitudinal association of change in AST with change in ACQ	Follow-up week	Relation of change in AST from baseline to follow-up versus contemporaneous change in ACQ	−2.53 (−2.80 to –2.26)	<.001
		Relation of change in AST at follow-up versus contemporaneous change in ACQ	−2.08 (−2.52 to −1.64)	<.001

^aRegression coefficients for the cross-sectional relationship of the AST with the ACQ differed significantly between follow-up and baseline (P = .007). Therefore, each 1-unit increment in the ACQ was associated with a greater difference in the AST at follow-up than at baseline.

FIGURE 2.

ROC curve at baseline. The ROC curve relates the proportion of true-positive and false-positive inadequate asthma control cases for difference cutoff points of the AST score. An ACQ score ≥1 was used as the reference standard for defining inadequate asthma control. Black diamonds indicate the cutoff points of the AST score.

Responsiveness

We found a significantly strong association between changes in AST scores and changes in ACQ scores (Fig 3). In the longitudinal analysis relating change in AST to change in ACQ scores from baseline to follow-up assessments (Table 1), an increase in ACQ score by 1 point was associated with a decrease in AST score by 2.53 (95% CI, 2.26–2.80, P < .001). During follow-ups, a 1-point increase in the ACQ was associated with a 2.08 (95% CI, 1.64–2.52, P < .001) decrease in the AST.

FIGURE 3.

Relationship between changes in AST and changes in ACQ scores. Each symbol and number represent an assessment schedule (eg, 1, baseline; 2, first follow-up assessment; 3, second follow-up assessment etc).

Discriminant Validity

A decrease in AST scores was significantly associated with an increase in the use of oral steroids (odds ratio [OR] 1.13 per 1-point decrease in the AST, 95% CI, 1.10–1.16, P < .001) and an increase in frequency of unscheduled acute care visits (OR 1.23 per 1-point decrease in the AST, 95% CI, 1.18–1.28, P < .001) over the full study period. Poor AST control (AST score <15 or red zone) was associated with an increase in the risk of unscheduled hospital visits (OR 10.8, 95% CI, 6.5–18.0, P < .001) and an increase in the use of oral steroids (OR 2.6, 95% CI, 1.8–3.8, P < .001).

Discussion

Our study validates use of the AST as a tool for weekly monitoring of asthma control in children. We found a moderate test–retest reliability, small longitudinal variation, high internal consistency, and good criterion validity of the AST. We also found an improvement in the participant learning curve over time, as demonstrated by an increasing trend in internal consistency from baseline to follow-up assessments. The AST was very responsive to changes in asthma status, as reflected by a significant increase in AST scores as patients’ ACQ scores decreased. Furthermore, the AST had clear discriminant validity; decreases in AST scores were significantly associated with increased use of oral corticosteroids and unscheduled acute care visits.

Most asthma exacerbations are preventable because they rarely occur without warning. Specifically, patients with asthma often have warning signs or evidence of declining asthma control that occurs days to weeks before an asthma exacerbation.³² The AST allows frequent (weekly) assessments and effective monitoring to facilitate a fast recognition and rapid response to warning signs of asthma control deteriorations. The AST also provides patients or parents (and PCPs) with easy visualization of changes in asthma control status over time to facilitate appropriate and timely response to deteriorations. Ongoing monitoring with the AST can provide objective data to determine the effectiveness of controller therapy and facilitate timely adjustments of therapy to achieve and maintain optimal asthma control.¹⁷

Several symptom-based short questionnaires have been validated for assessing and monitoring asthma control, but studies have reported significant limitations in children.^22–27 For instance, the ACQ has been validated with and without the inclusion of peak flow and has been used for weekly assessment of asthma control.^25,26 The ACT is also a short, easy-to-complete questionnaire and is the most commonly used symptom-based tool to assess asthma control.²⁷ Both the ACQ and ACT are responsive to change, but a recent study found that the ACT was preferable to the ACQ in clinical practice.³³ Yet weekly use of the ACT has never been validated, and no evidence exists to support its use by proxies (parents acting for their young children). Through the AST and by extension, our study provides evidence of the ACT’s reliability, validity, and responsiveness as a weekly tool and its use by parents. Other symptom-based questionnaires have been validated for use in various settings,^34–37 but longitudinal validation has not been evaluated, or studies included only brief follow-up periods of 2 or 3 different assessments.^25,34,38

Although the AST was useful in engaging patients and parents in ongoing monitoring, there were several limitations. First, our ICC values were moderate, somewhat lower than previously reported.³⁸ This may be a result of the longer time interval between test and retest in some patients and a smaller between-subject variation in our sample. Yet we found a small within-subject SD in the well-controlled subjects, demonstrating the longitudinal stability of repeated AST measurements. Second, the attrition rate over time was high, with approximately half (52%) of patients still providing assessments at 10 weeks. Third, criterion validity was assessed against the ACQ rather than an objective assessment of patients’ level of asthma control. Fourth, parents often completed the ASTs, particularly for young children who lacked the cognitive capacity to answer the questionnaire independently. Studies have reported that using parents as proxies can introduce bias because they tend to underestimate their children’s asthma symptoms. We believe that because the AST focuses on tracking trends in asthma control, even if bias is introduced by a proxy, as long as the bias remains constant by having the same parent complete all assessments, the AST can provide valuable information about changes in the child’s asthma control status that can be used for decision-making. Fifth, we did not analyze separately assessments made by children and parents because most assessments were completed by parents, and only 6.5% were completed by children alone. Sixth, we were unable to assess the tool’s utility for children independent of parental oversight, although independent use did occur successfully in some children ≥11 years of age. Although our results show that changes in AST scores were associated with changes in surrogate measures of asthma control, it is still necessary to evaluate whether using the AST will reduce asthma exacerbations. Lastly, the AST is a paper-based tool and has its own limitations, including lack of active prompt to patients or parents. Patients or parents had to interpret results themselves, based on preset recommendations. After completing an assessment, patients and parents had to send their responses back to the research team on a postcard. This process often delayed identification of asthma control deteriorations early to provide active feedback. Another limitation was the lack of patient or parent incentives for sustained AST use. This may explain the high degree of attrition, which may be related to improved asthma control or a child with intermittent asthma. Our future plan includes assessing the AST’s effectiveness in improving asthma outcomes, understanding and addressing critical barriers of long-term AST use by patients and parents, developing and implementing an electronic AST to address several limitations of the paper AST, and creating an incentive mechanism to increase patient and parent engagement in long-term use.

Conclusions

Our study demonstrates that the AST is a reliable, valid, and responsive tool for weekly, ongoing monitoring of asthma control. We hope that use of the AST can facilitate care continuity and shift asthma care from the current reactive, acute model to a preventive, proactive model where assessments are made weekly and treatment decisions are tailored to patients’ individual needs.

Glossary

ACQ

Asthma Control Questionnaire

ACT

Asthma Control Test

AST

Asthma Symptom Tracker

CI

confidence interval

ED

emergency department

ICC

intraclass correlation

OR

odds ratio

PCMC

Primary Children’s Medical Center

PCP

primary care provider

ROC

receiver operating characteristic