Abstract
Background
The Asthma Impact on Quality of Life Scale (A-IQOLS) assesses patient-perceived negative effect of asthma on quality of life (QoL). Its standard error of measurement is known, it has strong construct, convergent, and divergent validity, and provides information that is unique among asthma outcome measures.
Objective
To characterize A-IQOLS’ psychometric properties and suitability for use in demographically and clinically diverse adult asthma populations.
Methods
Data from participants in five independent asthma studies, whose samples ranging from patients with well-controlled moderate asthma to patients with severe, poorly controlled asthma were pooled to determine the psychometric performance of A-IQOLS scores, overall and in multiple demographic, disease status, and study subgroups.
Results
Pooled sample (n = 597) age averaged 45 years; 66% were female, 65% were White, 22% African American, 11% Hispanic, and 11% had ≤ high school education. The rated importance of its underlying life dimensions and the associations between A-IQOLS scores and lung function, symptoms, ACT, Juniper Mini-AQLQ, and Marks AQLQ scores was very similar regardless of patient demographic and clinical characteristics. A-IQOLS scores discriminated among the individual study samples as well as other patient-reported symptom and functional status measures. Distribution and anchor-based considerations suggest an A-IQOLS minimum clinically important difference (MID) in the vicinity of 0.50, and not smaller than 0.33, scale score units.
Conclusions
A-IQOLS is valid for research, and potentially clinical, use in demographically and clinically diverse patients.
Keywords: Asthma, Quality of life, Measurement/Standardized measures, Clinical outcomes, Patient-centered outcomes
INTRODUCTION
Standardized core outcome measures used in asthma research include spirometry, asthma control, exacerbations, and health care utilization.1 No measure of patients’ asthma-related physical, social, emotional, or role functioning or quality of life has been recommended as a core outcome. Recently, a measure is available that assesses the patient’s perspective on the impact of asthma and its treatment on their quality of life.
The Asthma Impact on Quality of Life Scale (A-IQOLS) asks patients to rate the negative effect of their asthma on each of 16 research-based dimensions of quality of life on a 5-point scale.2 A-IQOLS summary scores (the average of the dimension ratings) are interpretable on the original rating scale. In the AQOLIS test-retest study, A-IQOLS has strong content, convergent and divergent validity, and its standard error of measurement (SEM) was determined to be ±0.27. Participants in AQOLIS had diverse levels of asthma severity, a wide range in level of asthma control, and moderate racial and ethnic diversity. However, its size (n = 147) and composition did not support subgroup analyses to determine the generalizability of the A-IQOLS’ psychometric properties to specific patient populations, such as those with low education, from specific racial/ethnic backgrounds, or with severe asthma.
A-IQOLS also was administered as an ancillary measure in four additional, independent asthma clinical trials that differed widely in target population, eligibility criteria, clinical intervention, and study design. Pooled baseline data from all five studies were analyzed to: 1) determine whether the rated personal importance of the Flanagan quality of life dimensions and psychometric performance of the A-IQOLS support its use in diverse demographic groups and clinical populations, 2) evaluate A-IQOLS’s sensitivity for differences in asthma severity and clinical status, 3) estimate the minimum within-patient score change that is clinically important, and 4) provide information to inform sample size and power estimates when planning future studies.
METHODS
This research was supported by Grant No. HL119845 (PI: SRWilson) from the NIH/NHLBI and is approved by the Sutter Health IRB (SHIRB No. 14-06-327). All studies contributing data for the present analyses were approved by their respective IRBs. All subjects provided written consent.
Studies
The following studies contributed baseline data for pooled analysis:
“Asthma Quality of Life Impact Study” (AQOLIS)2
“DASH Diet for Asthma” (DASH)3
“Long-acting Beta Agonist Step-Down Study” (LASST)4
“A Prospective Observational Cohort Study of Biopredictors of Bronchial Thermoplasty Response in Patients with Severe Refractory Asthma” (BTR)5
“Effect of [Continuous] Positive Airway Pressure on Reducing Airway Reactivity in Patients with Asthma” (CPAP)6
Eligibility criteria of the LASST, AQOLIS, and BTR studies (Table E1, Online Repository) ensured that enrolled patients had persistent asthma and were prescribed asthma treatment at or above step 2.7 LASST patients’ asthma was well-controlled and relatively stable on combination inhaled corticosteroid (ICS) and long-acting beta agonist (LABA) (step 4 treatment). AQOLIS participants had persistent asthma (treatment steps 2–6), without regard to level of asthma control.2 DASH targeted obese patients with poorly-controlled asthma and CPAP targeted patients with stable asthma and airway reactivity, without regard to treatment step. BTR study participants had severe asthma, defined as asthma that was uncontrolled despite high dose inhaled corticosteroids plus LABA and/or other asthma controller medications (i.e., step 5 or 6 anti-asthma treatment), and were approved to undergo bronchial thermoplasty.
Data
Baseline data were available for n = 597 patients (AQOLIS: n = 152, LASST: n = 227, CPAP: n = 92, and BTR: n = 38).
Treatment step
Pharmacotherapy regimens were documented in all studies except CPAP and classified, by step, per U.S. treatment guidelines.7
Lung Function
Baseline lung function was assessed pre- and post-bronchodilator using ATS-recommended procedures and equipment.8 Race-specific norms were used to determine FEV1 percent predicted values (PPFEV1).
Asthma Control
All studies administered the Asthma Control Test.9
Symptoms
The Asthma Symptom Utility Index (ASUI)10 was administered in LASST, CPAP, and AQOLIS. Juniper Mini-Asthma Quality of Life Symptom Sub-scale11 scores were available for AQOLIS, DASH, and BTR.
Disease-specific health status
Asthma symptoms and impairment in patients’ physical, social, emotional, and/or role-functioning were assessed by the Juniper Mini-Asthma Quality of Life Scale Total (AQLQ) score11 in AQOLIS, DASH, and CPAP, the Juniper Standardized AQLQ in BTR, and the Marks AQLQ12 in AQOLIS, LASST, and CPAP.
Asthma Impact on QoL Scale (A-IQOLS)
The A-IQOLS2 was self-administered by patients, requiring less than 5 minutes (typically, 3–4). It asks: “Over the past four weeks, how much did your asthma negatively affect your life in each of the following areas?” Respondents were instructed to “Consider the effects of the asthma itself, the asthma medications you use, and anything you did to avoid, treat, or get medical care for asthma symptoms.” Ratings were obtained on each of 16 life dimensions,13,14 on a 5-point, unidirectional Likert-type scale: 1, No negative effect at all; 2, Slightly negative effect; 3, Moderately negative effect; 4, Very negative effect; and 5, Extremely negative effect.
One study (AQOLIS) added the following question assessing overall effects of asthma: “In the past four weeks, how much did your asthma negatively affect your life overall” at both test and retest assessments. Responses used A-IQOLS’ 5-point negative effect rating scale.
Flanagan QOL dimension importance questionnaire
The personal importance of each of the Flanagan QoL dimensions,13 plus Burckhardt’s addition, Independence, was determined in all five asthma studies. Respondents were asked: “At this time in your life, how important is each of the following areas to you?” Ratings were obtained on a 5-point scale: 1, Not at all Important; 2, Only Slightly Important; 3, Moderately Important; 4, Important; and 5, Very Important. The importance questionnaire has no summary score.
Statistical Analysis
Sample characteristics
Means±SD or proportions, as appropriate, were calculated for patient characteristics for the pooled sample, each constituent study, and the demographic sub-groups defined by age, sex, race, ethnicity, education, and ACT score category. Sub-group differences were evaluated using t-tests and linear regression methods.
QoL dimension importance
The mean (SD) rated importance of each QoL dimension, and the proportion of patients rating each dimension as Important/Very Important was determined for the pooled sample and demographic sub-groups.
Asthma outcome measure scoring
All standardized asthma outcome measures were scored and scaled using their published algorithms: ACT,9 ASUI,15 Marks AQLQ,16 Juniper AQLQ,11,17,18 and A-IQOLS.2
The A-IQOLS summary score, , where rd is the individual’s asthma negative effect rating for each dimension, d, and the number of dimensions actually rated, n.2 Dimension importance ratings do not have to be obtained in order to score the A-IQOLS. Importance weighted and unweighted A-IQOLS scores were highly correlated (r = O.99) and had similar correlations with other asthma outcome measures.
Convergent and divergent validity
Pearson correlations between A-IQOLS summary scores and other asthma outcome measures were determined for the pooled sample and demographic subgroups. The strength of the correlations (r) was interpreted as follows: 0.00–0.19, very weak; 0.20–0.39, weak; 0.40–0.59, moderate; 0.60–0.79, strong; and 0.80–1.0, very strong.19 R2 estimates the proportion of common variance between any two measures.
Internal consistency reliability
The standardized coefficient alpha (α) was calculated to characterize the internal consistency of the A-IQOLS dimension ratings.
Sensitivity to differences in asthma status
An overall test of the differences among the five studies on each asthma outcome was performed, as well as the various pairwise comparisons. No specific study differences were hypothesized, hence no adjustment was made for multiple pairwise comparisons.
Association between change in overall asthma effect rating and change in A-IQOLS score
Linear regression and descriptive analyses were used to estimate the size of the change in A-IQOLS score associated with a 1-category change in the rated overall negative effect of asthma in the AQOLIS study sample.
Estimation of the Minimum Clinically Important Difference (MID)
The MID is the smallest change in a score that patients perceive as beneficial (or detrimental) that would mandate, in the absence of other considerations (e.g., side effects, excessive cost), a change in the patient’s management.20 The MID defines cut-points in change scores that distinguish between improvement, stability, and worsening. The definition of the MID presumes that the difference is large enough to be a true difference, not the result of measurement error.
A triangulation approach, as recommended by Leidy, et al, 2005, was used to estimate the MID of the A-IQOLS from the following: 1) distribution-based considerations21,22 using a) Wyrwich et al’s (1999)23 suggested value of 1 standard error of measurement (SEM), b) McHorney & Tarlov’s (1995)24 suggested value of 1.96 SEM, and c) intermediate values that represent the smallest score changes that constitute a true difference with 80% and 67% probability -- 1.81 SEM and 1.15 SEM, respectively, and d) one-half the measure’s standard deviation (SD),25 and 2) an anchor-based estimation approach -- the magnitude of the (A-IQOLS) score change that was associated, using linear regression, with a 1-category change in patients’ rating of the overall negative effect of asthma between test and retest in the AQOLIS study.
All statistical analyses were performed using SAS v 9.2 (SAS Institute, Inc., Cary, NC). A significance level of 0.05 was used throughout.
RESULTS
Sample demographic characteristics
Participants’ mean age was approximately 45±14 years; 66% were female; 65% were White, 22% Black, 9% Asian, and 4% another race; and 11% were Hispanic (Table 1). Approximately 11% had a high school education or less, 36% some college, and 53% a 4-year college or advanced degree.
Table 1.
Demographic, health status, and quality of life characteristics: pooled and individual study samples.
Pooled Sample (n=597) | LASST (n=227) | AQOLIS (n=152) | CPAP (n=92) | DASH (n=88) | BTR (n=38) | P-value: overall test of study differences | |
---|---|---|---|---|---|---|---|
Measures | N (%) or Mean ± SD (range) | N (%) or Mean ± SD (range) | N (%) or Mean ± SD (range) | N (%) or Mean ± SD (range) | N (%) or Mean ± SD (range) | N (%) or Mean ± SD (range) | |
Demographic | |||||||
Age, yrs.* | 44.8 ± 13.5 (18–84) | 42.6 ± 13.2 (18–84) | 49.3 ± 12.3 (21–70) | 35.5 ± 11.0 (18–59) | 51.5 ± 12.4 (20.5–70.5) | 47.3 ± 12.2 (21.4–67.5) | <0.0001 |
18–44 years old | 291 (48.7) | 125 (55.1) | 51 (33.6) | 73 (79.3) | 26 (29.5) | 16 (42.1) | <0.0001 |
45–59 years old | 208 (34.8) | 74 (32.6) | 64 (42.1) | 19 (20.7) | 37 (42.0) | 14 (36.8) | |
60 and older | 98 (16.4) | 28 (12.3) | 37 (24.3) | 0 (0.0) | 25 (28.4) | 8 (21.1) | |
Sex | 0.45 | ||||||
Female | 396 (66.3) | 153 (67.4) | 97 (63.8) | 58 (63.0) | 58 (65.9) | 30 (78.9) | |
Male | 201 (33.7) | 74 (32.6) | 55 (36.2) | 34 (37.0) | 30 (34.1) | 8 (21.1) | |
Race | <0.0001 | ||||||
White | 386 (64.9) | 139 (61.2) | 114 (75.0) | 56 (60.9) | 44 (50.6) | 33 (89.2) | |
Black/African American | 129 (21.7) | 70 (30.8) | 22 (14.5) | 24 (26.1) | 10 (11.5) | 3 (8.1) | |
Asian | 54 (9.1) | 9 (4.0) | 13 (8.6) | 5 (5.4) | 27 (31.0) | 0 (0) | |
Other | 26 (4.4) | 9 (4.0) | 3 (2.0) | 7 (7.6) | 6 (6.9) | 1 (2.7) | |
Missing | 2 (0.3) | 0 (0) | 0 (0) | 0 (0) | 1 (1.1) | 1(2.6) | |
Ethnicity | 0.64 | ||||||
Hispanic | 67 (11.2) | 24 (10.6) | 17 (11.2) | 11 (12.0) | 13 (14.8) | 2 (5.4) | |
Non-Hispanic | 529 (88.8) | 203 (89.4) | 135 (88.8) | 81 (88.0) | 75 (85.2) | 35 (94.6) | |
Missing | 1 (0.2) | 1 (2.6) | |||||
Education | 0.02 | ||||||
≤ High school | 64 (10.8) | 31 (13.7) | 8 (5.3) | 13 (14.1) | 7 (8.0) | 5 (13.9) | |
Some college | 213 (35.8) | 80 (35.2) | 46 (30.3) | 41 (44.6) | 33 (37.5) | 13 (36.1) | |
≥ 4-yr college degree | 318 (53.4) | 116 (51.1) | 98 (64.5) | 38 (41.3) | 48 (54.5) | 18 (50) | |
Missing | 2 (0.3) | 2 (5.3) | |||||
Treatment Step† | <0.0001 | ||||||
Step 1 | 39 (7.7) | 0 (0) | 0 (0) | - | 39 (44.3) | 0 (0) | |
Step 2 | 24 (4.8) | 0 (0) | 16 (10.5) | - | 8 (9.1) | 0 (0) | |
Step 3 | 35 (6.9) | 0 (0) | 23 (15.1) | - | 12 (13.6) | 0 (0) | |
Step 4 | 296 (58.6) | 227 (100) | 64 (42.1) | - | 5 (5.7) | 0 (0) | |
Step 5 | 92 (18.2) | 0 (0) | 48 (31.6) | - | 24 (27.3) | 20 (52.6) | |
Step 6 | 19 (3.8) | 0 (0) | 1 (0.7) | - | 0 (0) | 18 (47.4) | |
Asthma status | |||||||
Percent FEV1 Predicted‡ | 88.6 ± 17.0 (34.9–154.3) | 91.6 ± 13.4 (69.3–154.3) | 87.2 ± 18.7 (37.0–151.0) | 89.3 ± 10.9 (74.6–134.6) | 89.8 ± 19.6 (41.5–129.9) | 69.2 ± 24.4 (34.9–137.6) | <0.0001 |
ACT § | 19.6 ± 4.8 (5–25) | 22.9 ± 1.7 (20–25) | 18.8 ± 4.0 (10–25) | 20.8 ± 3.3 (9–25) | 14.9 ± 3.7 (7–22) | 10.5 ± 4.3 (5–22) | <0.0001|| |
ASUI | 0.86 ± 0.15 (0.20–1)¶ | 0.94 ± 0.07 (0.64–1) | 0.81 ± 0.17 (0.20–1) | 0.89 ± 0.12 (0.28–1) | 0.81 ± 0.14 (0.44–1)¶ | 0.60 ± 0.13 (0.36–0.89)¶ | <0.0001 |
Juniper Mini-AQLQ: Symptom Score** | 5.4 ± 1.1 (1.0–7) ¶ | 6.0 ± 0.42 (4.2–6.3) ¶ | 5.2 ± 1.2 (2.2–7) | 5.6 ± 0.7 (1.9–6.3)¶ | 5.2 ± 1.3 (1.8–7) | 3.2 ± 1.2 (1–5.9) | <0.0001 |
Juniper Mini-AQLQ: Total Score** | 5.5 ± 1.1 (1.2–7) †† | 6.0 ± 0.5 (3.5–6.4) †† | 5.4 ± 1.1 (2.9–6.9) | 5.4 ± 0.9 (2.5–6.4)†† | 5.2 ± 1.1 (1.9–7) | 3.2 ± 1.2 (1.2–6.1) | <0.0001 |
Marks AQLQ‡‡ | 12.2 ± 12.0 (0–65) †† | 5.8 ± 6.5 (0–40) | 14.2 ± 12.1 (0–65) | 13.5 ± 12.5 (0–54) | 15.3 ± 9.6 (0.1–44.1)†† | 33.0 ± 10.8 (7.9–50.6)†† | <0.0001 |
Quality of life | |||||||
A-IQOLS§§ | 1.43 ± 0.68 (1–4.94) | 1.22 ± 0.56 (1–4.94) | 1.36 ± 0.45 (1–3.94) | 1.43 ± 0.73 (1–4.50) | 1.56 ± 0.60 (1–3.25) | 2.67 ± 0.87 (1.13–4.38) | <0.0001|| || |
Abbreviations: FEV1, forced expiratory volume in one second; ACT, Asthma Control Test (Well-controlled, 20–25; poorly controlled, 16–29; very poorly controlled, 5–15); ASUI, Asthma Symptom Utility Index (0-Worst possible symptoms to 1-Best state/no symptoms); Marks AQLQ, Asthma Quality of Life Questionnaire (0-Less negative impact on functional status to 80-Very severe negative impact on functional status); Juniper Mini-AQLQ, Juniper mini-Asthma Quality of Life Questionnaire, total score 1-Totally limited to 7-Not at all limited and symptom sub-scale (1-Symptoms all of the time to 7-Symptoms none of the time); A-IQOLS, Asthma Impact on Quality of Life Scale (1-No negative effect at all to 5-Extremely negative effect).
For AQOLIS, age was as of the patient’s last birthday. For DASH, age was the difference between the date of the baseline visit and the patient’s birth date, divided by 365.25. For LASST, CPAP, and BTR, age at enrollment was available to the nearest tenth of a year.
Patient’s pharmacotherapy regimens were documented in all except the CPAP study.
In CPAP, lung function was measured 0–2 weeks before the baseline questionnaire was administered. One AQOLIS, two DASH, and five BTR participants were missing a lung function value.
In DASH, ACT was used for phone eligibility screening.
After controlling for (age, sex, race, ethnicity, and education, the overall test of group ACT differences remained significant (p<0.0001), as did all pairwise study comparisons (all p<0.0001). Pooled sample size in these analyses was n=593. The resultant least square ACT score means (LASST: 23.0; AQOLIS: 18.7; CPAP: 20.9; DASH: 14.9; BTR, 10.5) were virtually the same as the unadjusted means.
The Pearson correlation between the ASUI and the Juniper Mini-AQLQ Symptom Score in the AQOLIS sample was 0.81. Linear regression was used to impute ASUI scores from Juniper Symptom Scores in the DASH and BTR samples and to impute Juniper Mini-AQLQ Symptom Scores in the LASST sample and CPAP samples from their ASUI scores.
BTR used the full-length “standardized” Juniper AQLQ questionnaire.
The Pearson correlation between the Juniper Mini-AQLQ Total Score and the Marks AQLQ score in the AQOLIS sample was -0.80. Linear regression was used to impute Marks AQLQ scores for DASH and BTR samples from their Juniper Total scores and to impute Juniper Total scores in the LASST and CPAP studies from their Marks scores.
One CPAP participant was missing a Marks AQLQ score.
An individual’s standard A-IQOLS summary score, S, is defined as , where rd is their rating of the impact of asthma on dimension d, and n = number of dimensions rated, typically 16.
After controlling for age, sex, race, ethnicity, and education, p<0.0001. After further controlling for ACT score, p<0.0001; Least-square means for A-IQOLS scores were: LASST: 1.4; AQOLIS: 1.3; CPAP: 1.5 DASH: 1.3; BTR, 2.2; All pairwise study comparisons with BTR were significant (p<0.0001). Other pairwise study comparisons were not significant.
As expected based on eligibility criteria, the five studies had diverse asthma outcome measures (p < 0.0001) (Table 1) and, constituted a very heterogeneous population of patients with persistent asthma. Demographically, they differed in age, education level, and race, but not sex or ethnicity. BTR patients, who all had severe refractory asthma, had significantly worse lung function (PPFEV1) than the other samples -- similar to values reported for adult patients categorized as exacerbation prone in the SARP-3 severe asthma cohort.26 BTR patients also had higher symptom levels (ASUI and Juniper AQLQ Symptom scores), worse asthma control (ACT), greater impairment in functional status (Juniper Mini-AQLQ Total and Marks AQLQ scores), and higher A-IQOLS scores (greater negative effect) compared to the other four studies (Table 2; and see graphic comparisons, Online Repository, Figure E1). Conversely, LASST patients, selected for good asthma control, had better status on all asthma measures than did the other four studies. Results for the combined AQOLIS and CPAP samples also are presented (Table 2) since they are the most similar samples and results for this composite may be useful for planning future studies with similar eligibility requirements.
Table 2.
Statistical significance (p values) of tests of pairwise mean differences between studies* on all asthma measures.
Measures | LASST vs. BTR |
AQOLIS vs. BTR |
CPAP vs. BTR |
AQOLIS /CPAP vs. BTR |
DASH vs. BTR |
LASST vs. AQOLIS |
LASST vs. CPAP |
LASST vs. AQOLIS /CPAP |
LASST vs. DASH |
AQOLIS vs. DASH |
CPAP vs. DASH |
AQOLIS /CPAP vs. DASH |
CPAP vs. AQOLIS |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
PPFEV1† | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | 0.009 | 0.25 | 0.02 | 0.37 | 0.24 | 0.85 | 0.38 | 0.32 |
ACT | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 |
ASUI‡ | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | 0.0003 | <0.0001 | <0.0001 | 0.94 | <0.0001 | 0.06 | <0.0001 |
Juniper Mini-AQLQ: Symptom Score§ | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | 0.004 | <0.0001 | <0.0001 | 0.95 | 0.001 | 0.13 | 0.0003 |
Juniper Mini-AQLQ: Total Score|| | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | 0.29 | 0.18 | 0.19 | 0.66 |
Marks AQLQ¶ ** | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | 0.40 | 0.22 | 0.26 | 0.58 |
A-IQOLS†† | <0.0001 | <0.0001 | <0.0001 | <0.0001 | <0.0001 | 0.02 | 0.003 | 0.002 | <0.0001 | 0.01 | 0.16 | 0.02 | 0.35 |
Study sample size: LASST: n=227; AQOLIS: n=152; CPAP: n=92; DASH: n=88; BTR: n=38.
One AQOLIS, two DASH, and five BTR participants were missing a lung function value.
ASUI scores imputed from Juniper Mini-AQLQ: Symptom scores for DASH and BTR samples.
Juniper Mini-AQLQ: Symptom scores were imputed from ASUI scores for LASST and CPAP samples.
Juniper Mini-AQLQ: Total score were imputed from Marks AQLQ scores for LASST and CPAP samples.
Marks AQLQ score is missing for one CPAP participant.
Marks AQLQ scores were imputed from Juniper Total scores for DASH and BTR samples.
An individual’s standard A-IQOLS summary score, , where rd is their rating of the impact of asthma on dimension d, and n = number of dimensions rated, typically 16.
The AQOLIS, CPAP and DASH samples fell between the LASST and BTR samples on all seven asthma outcomes. Of the three, DASH was the most distinct, with numerically, but not necessarily significantly, worse status on all asthma outcome measures except symptoms. The AQOLIS and CPAP samples varied in their positions relative to the DASH sample depending on which asthma outcome measure was considered. A-IQOLS scores differed significantly between the AQOLIS and DASH samples and between the combined AQOLIS/CPAP and DASH samples, but ASUI, Juniper Mini-AQLQ Total and Symptom, and Marks AQLQ scores did not. By contrast, ASUI and Juniper Mini-AQLQ Symptom scores differed significantly between the CPAP sample and both DASH and AQOLIS, whereas the Juniper Mini-AQLQ Total, Marks AQLQ, and A-IQOLS scores did not.
Perceived importance of the QoL dimensions
From 60%–97% of patients in the pooled sample rated 15 of the individual quality of life dimensions as personally Important/Very Important, and 56% rated 12 or more of the 16 dimensions as personally Important/Very Important (Figure 1a). Only dimension #8, Participation in activities relating to local and national government and public affairs, was considered Important/Very Important by fewer than half of the patients (23%). No importance ratings were missing.
Figure 1.
Figure 1a. Mean ratings of the importance of each quality of life dimension: pooled sample (n=597)
Figure 1b. Mean A-IQOLS ratings of negative effect of asthma on each dimension: pooled and by study sample.
The dimension importance patterns were very similar for men and women and patients of all ages, races, ethnicity, and education level (Online Repository, Figure E2), with the exception that younger subjects, on average, considered Having and Raising Children (#4) to be less important than older subjects.
Despite similar mean importance rating patterns across demographic subgroups, there was substantial individual variability. To characterize this, the absolute difference between each individual’s importance rating on a given dimension and the dimension’s overall mean importance rating was calculated and summed across the 16 dimensions. The mean (SD) of the individual deviations was 12.8(3.7). Individuals’ summed differences ranged from 4.9 (an individual with importance ratings similar to the sample average ratings) to 35.0 (an individual with a ≥ 2 pt. average deviation from the sample mean dimension importance ratings). Demographic characteristics together accounted for only 2% of the variance in deviations; 98% was due to other (undetermined) individual differences and measurement error. Only increasing age was significantly associated with greater deviation (p = 0.03)
Negative impact of asthma: A-IQOLS Summary Scores
A-IQOLS scores and dimension ratings
Missing A-IQOLS ratings were rare – only three of 597 patients omitted one, and only one patient omitted two, dimension negative effect ratings.
In the pooled sample and the individual study samples, the dimensions of Health/Safety and Active Recreation were most negatively affected by the patient’s asthma and its treatment, with Work, Material Well-being, Independence, and Socializing somewhat more negatively affected than the remaining 10 dimensions (Figure 1b). BTR (severe asthma) patients reported substantially greater negative effects on all QoL dimensions than the other study samples, and greater negative effects on the Work, Socializing, Health/Safety, and Active Recreation dimensions, relative to the other 10 dimension, than the other studies. There was substantial individual variation in the dimensions perceived to be most negatively affected by their asthma (e.g., 59.1% of patients considered the negative effect to be as great or greater on one or more of the other dimensions than on any of the six most commonly affected dimensions).
Demographic subgroup A-IQOLS score comparisons
A-IQOLS scores did not differ significantly as a function of age, sex, race, or ethnicity (Online Repository, Table E2) but differed significantly by level of education (i.e., greater negative effect of asthma in those with high school education or less), and treatment step (greater negative effect at higher treatment step). These associations (p = 0.02 and p < 0.0001, respectively), persisted after controlling for ACT score (p = 0.008 and p = <0.0001). Thus, the greater perceived negative effects of asthma and its treatment among patients with more intense treatment regimens and/or lower education are not explained by subgroup differences in level of disease control, but may be due to their functional impairments or the cost, inconvenience, or side effects of asthma treatment.
Correlations between A-IQOLS scores and other outcomes (convergent and divergent validity)
A-IQOLS scores were significantly correlated with all other asthma outcome measures (Table 3). Although statistically significant, the correlation between PPFEV1 and A-IQOLS was very weak (r=-0.13). For all other asthma outcomes the correlations were highly significant (p < 0.0001) but ranged from very weak (ASUI) to moderate (ACT, Marks AQLQ) and strong (Juniper Mini-AQLQ). In all cases, the shared/common variance (R2) between the A-IQOLS and the other asthma measures was ≤48%, confirming, as previously reported for the AQOLIS sample,2 that the A-IQOLS provides substantial unique information not provided by the other measures.
Table 3.
Correlations and shared variances between A-IQOLS scores and other asthma measures: Pooled sample (n = 597).
A-IQOLS score | |||
---|---|---|---|
Variable | r* | p | R2 |
PPFEV1‡ | −0.13 | 0.002 | 0.02 |
ACT | −0.50 | <0.0001 | 0.25 |
ASUI§ | −0.33 | <0.0001 | 0.11 |
Juniper Mini-AQLQ: Symptom Score|| | −0.64 | <0.0001 | 0.41 |
Juniper Mini-AQLQ: Total Score|| | −0.70 | <0.0001 | 0.48 |
Marks AQLQ§ ¶ | 0.50 | <0.0001 | 0.25 |
Abbreviations: PPFEV1, percent predicted forced expiratory volume in one second; ACT, Asthma Control Test; ASUI, Asthma Symptom Utility Index; Marks AQLQ, Asthma Quality of Life Questionnaire; Juniper Mini-AQLQ, Juniper Mini-Asthma Quality of Life Questionnaire, total score and symptom sub-scale;
r = Pearson product-moment correlation.
Pooled sample size, n = 589; one AQOLIS, two DASH, and five BTR participants were missing a baseline PPFEV1 value.
Pooled results for LASST, AQOLIS, and CPAP samples only. When ASUI scores are imputed from Juniper Mini-AQLQ Symptom scores, r = −0.51 and R2 = 0.26. When Marks AQLQ scores are imputed from Juniper Mini-AQLQ Total scores, r = 0.62 and R2 = 0.38.
Pooled Juniper Mini-AQLQ results for AQOLIS, DASH, and BTR samples only. AQOLIS and DASH used the Juniper Mini-AQLQ; BTR used the Standardized Juniper AQLQ. When Juniper Mini-AQLQ Total scores are imputed from Marks AQLQ scores, r = −0.62 and R2 = 0.38. When Juniper Mini-AQLQ Symptom scores are imputed from ASUI scores, r = −0.54 and R2 = 0.29.
One CPAP participant was missing the Marks AQLQ score.
In all age groups, male and females, Hispanic and non-Hispanics, all racial groups, and at different levels of education, the associations between A-IQOLS scores and other asthma outcome measures were statistically significant and their absolute and relative magnitudes, as well as their R2 values, were similar to those in the pooled sample as a whole (Online Repository, Table E3). The only exception was that the correlations between the A-IQOLS and PPFEV1 tended to be lower and not statistically significant in numerically smaller sub-groups, which also tended to be less heterogeneous than the pooled sample as a whole on both measures.
Estimation of the Minimum Clinically Important Difference (MID)
The SEM of the A-IQOLS, averaged across all score levels, was previously determined to be 0.27.2 One recommended estimate of the MID based on its distributional properties (i.e., its SEM) is ±1.96 SEM24, which for A-IQOLS yields an MID = 0.54. This criterion gives 95% probability that a change of this magnitude or greater is a true change (i.e., statistically reliable). An alternative criterion (±1 SEM) yields an estimated MID = 0.27,23 but less assurance that the change is real. Other levels of assurance might be considered as well. For example, the smallest A-IQOLS change that constitutes a true change with 80% probability would be ±1.81 SEM = MID = ±0.49, or, if only 67% probability were acceptable, ±1.15 SEM = MID = ±0.31.
Another distributional approach to estimating the MID is to use one-half the SD of the measure. The A-IQOLS’ SD in the pooled sample (0.68; Table 1) yields an estimated MID = 0.34.
An anchor-based approach to estimating the MID uses self-reported change (e.g., a patient’s perception that the negative effects of asthma had increased, decreased, or remained the same) over a given time period. Reliance on a single item reporting perceived change has been criticized on methodologic grounds, and may not provide new information since the results of many such studies turn out to yield an MID value close to one-half the standard deviation of the measure.25 In AQOLIS, however, patients were asked for an overall rating of the negative effect of asthma on their life at both test and retest assessments, allowing direct determination of change rather than retrospective report. Using simple linear regression, a one category change in patients’ overall assessment was associated with a 0.23 change in A-IQOLS summary score.
Considering these various estimates, it appears reasonable to consider the A-IQOLS’ MID, conservatively, to be approximately one-half scale score point (0.50), and possibly somewhat less if one is willing to accept less assurance that the change exceeds statistical variability.
A-IQOLS’ internal consistency reliability
Standardized coefficient alpha is commonly reported, even though it is not informative regarding a measurement reliability. The standardized coefficient alpha (α) values of the A-IQOLS were high in the pooled sample as a whole (α = 0.97) and in all demographic sub-groups (α = 0.94–0.98).
DISCUSSION
We evaluated the psychometric properties of the A-IQOLS in five asthma studies constituting a large heterogeneous population, representing multiple demographic subgroups and asthma severities. The dimensions underlying the A-IQOLS were personally important for substantial proportions of patients in all demographic subgroups. These findings reinforce the content and construct validity of the A-IQOLS for use in diverse populations. Similar ratings of which dimensions are important, similar associations between A-IQOLS scores and other asthma outcome measures, and similar standardized coefficient alpha values in all subgroups suggest that patients, including those with low education, had a similar understanding of the dimension descriptions and rating task.
A strength of the A-IQOLS is that individuals can characterize the negative effects of asthma in light of how important each dimension is to them. The similarity of the mean dimension importance ratings across gender, race, and ethnic subgroups shows the general relevance of the dimensions. However, the validity and utility of the A-IQOLS do not depend on whether an individual’s priorities are similar to others’ priorities, or whether the individual’s priorities remain stable over time.
The question posed to respondents by the A-IQOLS differs substantially from the questions posed in instruments conventionally referred to as “asthma-related quality of life” measures. A number of the A-IQOLS dimensions also are not represented by items in other instruments, nor in the item bank of a newer measure, the RAND-IAQL.27 When choosing items to include in “disease-specific QoL” measures, it has been common practice to pretest a pool of items and eliminate those that relate to domains that are affected in smaller proportion of respondents and/or are less strongly correlated with other items.27,28 This practice reduces instrument length and administration time, and tends to increase internal consistency reliability. However, it can compromise content validity by narrowing the range of disease effects that are assessed. The modest shared variances between the A-IQOLS and the Juniper and Marks AQLQ measures are likely due to the smaller number of life dimensions assessed by the latter measures. The construct validity of the IQOLS template requires including all dimensions of life, and since the A-IQOLS administration time is already brief, there is no need to exclude dimensions less commonly affected.
BTR patients, all of whom had severe asthma refractory to intensive asthma pharmacotherapy and who had been approved to undergo the BT procedure, typically felt their asthma was having an effect between Slightly negative” and “Moderately negative” on most dimensions and a very negative effect on their overall health and safety (A-IQOLS mean ± SD = 2.67 ± 0.87). Patients with moderate persistent asthma that was stable and well controlled (LASST) typically felt their asthma had very little negative effect on their life (mean = 1.22 – only slightly above “No negative effect at all”). Those with asthma that was not well controlled (e.g., DASH patients) typically felt that its negative effect was about half way between “No negative effect at all” and a “Slightly negative effect.” The relatively low mean A-IQOLS scores of the DASH patients may seem surprising, since the functional consequences and risks of poorly controlled asthma can be important. However, A-IQOLS scores indicate that how much the patient feels asthma affects their life, which perspective may be influenced by their overall quality of life or other circumstances. Other life problems may overshadow the effects of asthma. In all these groups, however, the situation of individual patients varies substantially. Scores ≥3.25 were observed in all study groups and scores ≥4.38 in BTR, CPAP and LASST.
In summary, the A-IQOLS, which is based on a comprehensive set of empirically-derived dimensions of QoL, accommodates variation in individual priorities, yields reliable information on how and how much asthma negatively affects patients’ lives, is associated with other asthma outcomes but provides unique information, has a score range that makes it useful across the full spectrum of asthma severity and control, requires only a few minutes to administer, and is not copyrighted. The underlying IQOLS template (dimensions, stem question, and rating scale) can be easily modified for use in other diseases/conditions, assuming their psychometric performance, when evaluated, supports such use.
Measurement reliability and the minimum clinically important difference
Lydick and Epstein21 noted that any change in a QoL measure might be considered clinically significant in that it represents the patient’s perception of an altered health outcome. However, this proposition needs to be tempered. The evidence presented here suggests that the minimum clinically important within- person change (MID) of the A-IQOLS is approximately 0.50, but that somewhat smaller changes (between one-third and one-half a scale score point) may be informative, while smaller changes are more likely to be due measurement error. As with any new measure, it will take time and experience with its response to clinical interventions to clarify the clinical importance of within-person A-IQOLS score changes.21
Use of the results to inform sample size and power calculations
When planning studies in which the A-IQOLS may be a primary or secondary outcome measure, the information provided in Table 1 and Online Repository Table E1 can be used to estimate sample size requirements or determine statistical power.
Limitations and Future Directions
A-IQOLS scores are sensitive to sample differences and are correlated with other asthma outcome measures. Changes in asthma control are significantly associated with changes in A-IQOLS scores2 providing grounds for optimism regarding its suitability as an outcome measure in clinical research and regarding the suitability of the IQOLS approach, more generally, for assessing the patient-perceived negative impact of other medical conditions. However, A-IQOLS sensitivity to experimental group differences has not yet been directly evaluated.
The present study populations do not exhaust the demographic or the clinical diversity of patients in which the A-IQOLS, or any measure using the IQOLS template, might be used. The basic IQOLS stem question, dimension descriptions, and response scale already have been translated into many languages and national variants of those languages for an international clinical trial in another respiratory disease. Simply inserting the relevant translations of the word asthma into these language versions will yield a corresponding translation of the A-IQOLS. Further information is available from the corresponding author.
Conclusions
The A-IQOLS measures the patient-perceived impact of asthma and its treatment on the patient’s life. The dimensions of life underlying the A-IQOLS are important across adults of both genders, all age groups, of Caucasian, African American, and Asian ancestry, Hispanics and non-Hispanics, and all levels of education, but their importance varies greatly between individuals. A-IQOLS scores are significantly associated with standardized measures of symptoms and disease control in all of these demographic subgroups, and can be recommended for use in those groups. A-IQOLS scores discriminate between asthma patients with different levels of disease control and severity at least as well as functional status or “asthma-specific QoL” measures, but yield unique information.
Supplementary Material
Clinical Implications.
The Asthma Impact on Quality of Life Scale assesses the negative impact of asthma on quality of life from the patient’s perspective and is suitable as an asthma outcome measure in diverse demographic and clinical populations.
Acknowledgments
Funding source: National Heart, Lung and Blood Institute (R01HL119845, PI: SWilson)
The authors gratefully acknowledge the contributions of the investigators, staff, participants, oversight committees, funders, and institutions/research networks involved in the LASST, AQOLIS, CPAP, DASH, and BTR studies that provided the data used in the present analyses. We also acknowledge the helpful suggestions regarding the data analyses and their interpretation provided by Lauress Wise, PhD (educational measurement and statistics) and Lan Xiao, PhD (biostatistics). Ultimately, responsibility for the research and this report rests solely with the authors.
Abbreviations used
- ACT
Asthma Control Test
- A-IQOLS
Asthma Impact on Quality of Life Scale
- ASUI
Asthma Symptom Utility Index
- AQOLIS
Asthma Quality of Life Impact Study
- BTR
Bronchial Thermoplasty Responder
- CPAP
Effect of [Continuous] Positive Airway Pressure on Reducing Airway Reactivity in Patients with Asthma
- CR
Coefficient of Repeatability
- DASH
DASH Diet for Asthma study
- FEV1
Forced Expiratory Volume in 1 second
- LASST
Long-acting Beta Agonist Step Down Study
- Marks AQLQ
Marks Asthma Quality of Life Questionnaire
- Mini-AQLQ
Juniper Mini Asthma Quality of Life Questionnaire
- MID
Minimum Clinically Important Difference
- QoL
Quality of Life
- Flanagan QOLS
Flanagan Quality of Life Scale
- SD
Standard Deviation
- SEM
Standard Error of Measurement
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Busse WW, Morgan WJ, Taggart V, Togias A. Asthma outcomes workshop: Overview. J Allergy Clin Immunol. 2012;129:S1–8. doi: 10.1016/j.jaci.2011.12.985. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Wilson SR, Mulligan MJ, Ayala E, Chausow A, Huang Q, Knowles SB, et al. A new measure to assess asthma’s effect on quality of life from the patient’s perspective. J Allergy Clin Immunol [Internet] 2017 doi: 10.1016/j.jaci.2017.02.047. Available from: http://www.sciencedirect.com/science/article/pii/S0091674917308874. [DOI] [PMC free article] [PubMed]
- 3.Ma J, Strub, Peg, Lv N, Xiao L, Camargo C, Jr, Buist A, et al. Pilot randomized trial of a healthy eating behavioral intervention in uncontrolled asthma. Eur Respir J. doi: 10.1183/13993003.00591-2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Rogers L, Sugar EA, Blake K, Castro M, Dimango E, Hanania NA, et al. Step-Down Therapy for Asthma Well Controlled on Inhaled Corticosteroid and Long-Acting Beta-Agonist: A Randomized Clinical Trial. J Allergy Clin Immunol Pract [Internet] 2017 doi: 10.1016/j.jaip.2017.07.030. Available from: http://www.sciencedirect.com/science/article/pii/S221321981730555X. [DOI] [PubMed]
- 5.Washington University School of Medicine. ClinicalTrials.gov [Internet] Bethesda (MD): National Library of Medicine (US); 2000. A Prospective Observational Study of Biopredictors of Bronchial Thermoplasty Response in Patients With Severe Refractory Asthma (BTR Study) (BTR) [cited 2016 Apr 22]. Available from: https://clinicaltrials.gov/show/NCT01185275 NLM Identifier: NCT01185275. [Google Scholar]
- 6.Holbrook JT, Sugar EA, Brown RH, Drye LT, Irvin CG, Schwartz AR, et al. Effect of Continuous Positive Airway Pressure on Airway Reactivity in Asthma. A Randomized, Sham-controlled Clinical Trial. Ann Am Thorac Soc. 2016;13:1940–50. doi: 10.1513/AnnalsATS.201601-043OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.National Asthma Education and Prevention Program. Expert Panel Report 3 (EPR-3): Guidelines for the Diagnosis and Management of Asthma-Summary Report 2007. J Allergy Clin Immunol. 2007;120:S94–138. doi: 10.1016/j.jaci.2007.09.043. [DOI] [PubMed] [Google Scholar]
- 8.Miller MR. Standardisation of spirometry. Eur Respir J. 2005;26:319–38. doi: 10.1183/09031936.05.00034805. [DOI] [PubMed] [Google Scholar]
- 9.Nathan RA, Sorkness CA, Kosinski M, Schatz M, Li JT, Marcus P, et al. Development of the asthma control test: A survey for assessing asthma control. J Allergy Clin Immunol. 2004;113:59–65. doi: 10.1016/j.jaci.2003.09.008. [DOI] [PubMed] [Google Scholar]
- 10.Bime C, Wei CY, Holbrook JT, Sockrider MM, Revicki DA, Wise RA. Asthma Symptom Utility Index: Reliability, validity, responsiveness, and the minimal important difference in adult asthmatic patients. J Allergy Clin Immunol. 2012;130:1078–84. doi: 10.1016/j.jaci.2012.07.058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Juniper E, Guyatt G, Cox F, Ferrie P, King D. Development and validation of the mini asthma quality of life questionnaire. Eur Respir J. 1999;14:32–8. doi: 10.1034/j.1399-3003.1999.14a08.x. [DOI] [PubMed] [Google Scholar]
- 12.Marks GB, Dunn SM, Woolcock AJ. An evaluation of an asthma quality of life questionnaire as a measure of change in adults with asthma. J Clin Epidemiol. 1993;46:1103–11. doi: 10.1016/0895-4356(93)90109-e. [DOI] [PubMed] [Google Scholar]
- 13.Flanagan JC. A research approach to improving our quality of life. Am Psychol. 1978;33:138. [Google Scholar]
- 14.Burckhardt CS, Woods SL, Schultz AA, Ziebarth DM. Quality of life of adults with chronic illness: a psychometric study. Res Nurs Health. 1989;12:347–54. doi: 10.1002/nur.4770120604. [DOI] [PubMed] [Google Scholar]
- 15.Revicki DA, Leidy NK, Brennan-Diemer F, Sorensen S, Togias A. Integrating patient preferences into health outcomes assessment: the multiattribute Asthma Symptom Utility Index. Chest. 1998;114:998–1007. doi: 10.1378/chest.114.4.998. [DOI] [PubMed] [Google Scholar]
- 16.Marks G, Dunn S, Woolcock A. A scale for the measurement of quality of life in adults with asthma. J Clin Epidemiol. 1992;45:461–72. doi: 10.1016/0895-4356(92)90095-5. [DOI] [PubMed] [Google Scholar]
- 17.Juniper E, Guyatt G, Epstein R, Ferrie P, Jaeschke R, Hiller T. Evaluation of impairment of health related quality of life in asthma: development of a questionnaire for use in clinical trials. Thorax. 1992;47:76–83. doi: 10.1136/thx.47.2.76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Juniper E, Buist A, Cox F, Ferrie P, King D. Validation of a Standardized Version of the Asthma Quality of Life Questionnaire. CHEST J. 1999;115:1265. doi: 10.1378/chest.115.5.1265. [DOI] [PubMed] [Google Scholar]
- 19.Evans JD. Straightforward statistics for the behavioral sciences. Brooks/Cole; 1996. [Google Scholar]
- 20.Jaeschke R, Singer J, Guyatt GH. Measurement of health status: ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10:407–15. doi: 10.1016/0197-2456(89)90005-6. [DOI] [PubMed] [Google Scholar]
- 21.Lydick E, Epstein R. Interpretation of quality of life changes. Qual Life Res. 1993;2:221–6. doi: 10.1007/BF00435226. [DOI] [PubMed] [Google Scholar]
- 22.Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 2. Oxford University Press; 1995. (Oxford medical publications) [Google Scholar]
- 23.Wyrwich KW, Tierney WM, Wolinsky FD. Further Evidence Supporting an SEM-Based Criterion for Identifying Meaningful Intra-Individual Changes in Health-Related Quality of Life. J Clin Epidemiol. 1999;52:861–73. doi: 10.1016/s0895-4356(99)00071-2. [DOI] [PubMed] [Google Scholar]
- 24.McHorney CA, Tarlov AR. Individual-patient monitoring in clinical practice: are available health status surveys adequate? Qual Life Res. 1995;4:293–307. doi: 10.1007/BF01593882. [DOI] [PubMed] [Google Scholar]
- 25.Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care. 2003;41:582–92. doi: 10.1097/01.MLR.0000062554.74615.4C. [DOI] [PubMed] [Google Scholar]
- 26.Denlinger LC, Phillips BR, Ramratnam S, Ross K, Bhakta NR, Cardet JC, et al. Inflammatory and comorbid features of patients with severe asthma and frequent exacerbations. Am J Respir Crit Care Med. 2017;195:302–13. doi: 10.1164/rccm.201602-0419OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Stucky BD, Edelen MO, Sherbourne CD, Eberhart NK, Lara M. Developing an item bank and short forms that assess the impact of asthma on quality of life. Respir Med. 2014;108:252–63. doi: 10.1016/j.rmed.2013.12.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Eberhart NK, Sherbourne CD, Edelen MO, Stucky BD, Sin NL, Lara M. Development of a measure of asthma-specific quality of life among adults. Qual Life Res. 2014;23:837–48. doi: 10.1007/s11136-013-0510-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.