Impact of day-to-day variation in FEV1 on measures of change: a conceptual description

Abstract

Clinical trials often demonstrate treatment efficacy through change in forced expiratory volume in one second (FEV₁), comparing single FEV₁ measurements from post-versus pre-treatment timepoints. Day-to-day variation in measured FEV₁ is common for reasons such as diurnal variation and intermittent health changes, relative to a stable, monthly average. This variation can alter estimation of associations between change in FEV₁ and baseline in predictable ways, through a phenomenon called regression to the mean. We quantify and explain day-to-day variation in percent-predicted FEV₁ (ppFEV₁) from 4 previous trials, and we present a statistical, data-driven explanation for potential bias in ceiling and floor effects due to commonly observed amounts of variation. We recommend accounting for variation when assessing associations between baseline value and change in CF outcomes in single-arm trials, and we consider possible impact of variation on conventional standards for study eligibility.

Rationale

Ceiling and floor effects for changes in lung function are frequently reported in the literature for people with cystic fibrosis (pwCF). Described ceiling effects include reduced improvement with therapeutic interventions for pwCF having high pre-treatment ppFEV₁; and floor effects include diminished decline in pwCF with low baseline FEV₁, a consideration which is most relevant for observational studies (1, 2, 3, 4). We demonstrate here conceptually that individual day-to-day variation in ppFEV₁ has the potential to influence assessments of ceiling and floor effects, and we particularly focus on ceiling effects which are relevant to clinical trial eligibility. Concepts described are relevant to continuous outcomes (such as ppFEV₁) when assessing the association between baseline value as a covariate and change as an outcome; for example, evaluation of the association between pre-treatment ppFEV₁ and treatment effect based on change from a single pre-treatment ppFEV₁ to a single post-treatment ppFEV₁.

Methods

We evaluated multiple studies involving the collection of repeat spirometry over a short window and assessed the degree of variation present when measuring ppFEV₁ (see footnote to Table 1 for descriptions) (5, 6, 7, 8, 9). We considered windows no longer than 30 days, to identify a steady state period in which the natural decline of ppFEV₁ would be minimal. We selected 30 days as a steady state period, in which any potential natural decline in ppFEV₁ would be anticipated to be very small, only ~1/12^th as high as the recorded annual decline of ~1.5% per year; with decline likely further diminished with modulator use (10, 11). Spirometry measurements for all studies excluded pulmonary exacerbations (footnote to Table 1). To determine the relative amount of day-to-day variation, we performed variance components analysis (12). With a single linear mixed effects model incorporating only an intercept and a random term for individual, and using the R package ‘lme4’ and function ‘VarCorr’, we estimated between person $({\sigma }_{\mathsf{b}})$ and within-person, or day-to-day $({\sigma }_{\mathsf{w}})$ variation (13).

Table 1.

Estimates of variation in ppFEV₁ from several studies, when periods of exacerbations are excluded.

Study	Collection	Persons	Window (days)	Total measures/Possible (%)	Measures per person median (range)	${\sigma }_{\mathsf{w}}$	${\sigma }_{\mathsf{b}}$	${\sigma }_{\mathsf{w}}^2∕{\sigma }_{\mathsf{b}}^2$
Thornton (5)	Home	13	14	151/182 (83%)	12 (5, 14)	6.3	20.0	9.8%
Thornton (5)	Home	13	30	314/390 (81%)	25 (11, 30)	5.9	19.5	9.1%
e-ICE* (7)	Home	133	30	901/1140 (79%)	8 (1, 12)	8.7	21.6	16.3%
AZM-0001 (8)	Clinic	185	14	370/370 (100%)	2 (2, 2)	3.5	22.3	2.5%
PROMISE (6)	Home	202	30	404/404 (100%)	2 (2,2)	4.1	20.0	4.2%

Studies: Thornton conducted a long-term study of the airway microbiome in 13 persons. e-ICE (Early Intervention in Cystic Fibrosis Exacerbation) studied whether home spirometry would speed detection of exacerbations. AZM-0001 was a RCT of azithromycin treatment for chronic Pseudomonas infection. And PROMISE is an ongoing observational study of changes with CFTR modulator initiation. Window is the maximum time between all included measures per person, ${\sigma }_{\mathsf{b}}$ = between person standard deviation, ${\sigma }_{\mathsf{w}}$ = within person or day-to-day standard deviation. * indicates those persons randomized to do frequent home spirometry (early intervention arm). Study-specific definitions of exacerbations were use of intravenous antibiotics (e-ICE, AZM-0001) and clinical decision (Thornton, PROMISE).

Within one study of 13 participants with especially frequent, near-daily home spirometry, we also graphed ppFEV₁ during 30 contiguous days of observation (5, 9).

Results

A. Individual day-to-day variability is observed in CF home spirometry

Figure 1 shows all 13 participant’s measurements from the Thornton study over ~30 contiguous study days, along with their steady state means $(\mu )$ and day-to-day standard deviations about that mean $(\sigma )$ (5, 9). Participants contributed a mean of 25 measurements each (total 314 measurements), and individual level standard deviations $(\sigma )$ averaged 5.2 (range 2.4 to 10.7). Over all 4 studies, the day-to-day variation $({\sigma }_{\mathsf{w}}^2)$ from home spirometry was estimated at 4.2% to 16.3% as high as between-person variation $({\sigma }_{\mathsf{b}}^2)$ (Table 1). Day-to-day variation was lower using spirometry from the clinic (2.5%).

Figure 1.

The first 30 days (excluding exacerbations) from the 13 pwCF who participated in Thornton’s study of near-daily home spirometry, along with their individual-level steady state means $\mu$ and standard deviations $\sigma$. The best measurement from each day was used for the analyses, and spirometry collected during and 14 days before and after episodic antibiotic treatment for pulmonary exacerbation was excluded. Shaded areas show range of min and max ppFEV₁ over the interval. Average within-person standard deviation was 5.2 ppFEV₁ and ranged from 2.4 to 10.7.

B. Change to next measure is associated with current value

Figure 2a-c shows the day-to-day differences in ppFEV₁ from a single study participant from the above study, where differences were computed from current ppFEV₁ to the subsequent ppFEV₁. Overall, (Figure 2a) the average of the day-to-day differences depicted by the arrows is −0.9, close to zero. When only the ppFEV₁ measures above the mean of 94.1 are selected (filled circles), the differences from current to the subsequent ppFEV₁ average −6.0, indicating that, on average, values above the mean tend to decline (Figure 2b). Similarly, when only the ppFEV₁ measures below the mean of 94.1 are selected (open circles), the differences to the subsequent ppFEV₁ average +5.0, tending to increase. This statistical phenomenon, long observed, is called regression to the mean, where extreme values tend to resolve (14, 15, 16).

Figure 2.

Using spirometry from a single individual in figure 1, an example of how change from current to next is associated with current level. This is way of visualizing regression to the mean.

C. Measurements above a fixed lung function threshold are more likely to be above individual-level steady state

Next, we evaluated the 314 ppFEV₁ measurements comprising the first 30 days of all 13 participants and compared to a fixed ppFEV₁ threshold of 70. This threshold is often used to dichotomize baseline FEV₁ levels when examining their association with treatment (17, 18). Of the 67 measurements above the ppFEV₁ threshold of 70, 72% were also above the individual steady state mean for that person. In contrast, of the 247 measurements below 70, only 43% were above the individual steady state means. We repeated the computation at the 90% threshold (a common clinical trial cut point) and found very similar results to the 70% threshold.

D. Combining these concepts, an apparent ceiling effect is anticipated

In the presence of day-to-day variation in ppFEV₁ (paragraph A), persons having a high single ppFEV₁ measurement are likely to be above their individual steady state (paragraph C). In the absence of a change in health or an intervention, the subsequent ppFEV₁ measure is likely to be lower due to the inherent variability of lung function which leads to regression to the mean (paragraph B).

Now imagine the context of an effective therapeutic intervention, in which ppFEV₁ is measured once before and once after treatment, a common procedure in trials. An individual with a high pre-treatment ppFEV₁ is likely to experience these opposing influences over the subsequent treatment period: a) the treatment benefit (ppFEV₁ increase) as well as b) the regression to steady state (ppFEV₁ decrease). A high pre-treatment ppFEV₁ in the context of day-to-day variation results in a diminished estimated treatment effect, an apparent ceiling effect. This flattening is observed regardless of biological association between current lung health and treatment benefit and is a consequence of the associations previously described.

E. Does day-to-day variation impact study findings?

In related work (not shown here), some of us recently completed a simulation study to investigate the degree to which day-to-day variation impacts measures of association with the known variation described above (19). That simulation addresses potential bias in tests for association between baseline value and the outcome of change in any continuous measure of CF-related health (e.g., ppFEV₁, sweat chloride, etc.). The findings are relevant for uncontrolled study designs (single-arm, or observational), but not controlled studies, as the comparison between arms nullifies the variation-related bias in each. Even for ratios as low as 3% $({\sigma }_{\mathsf{w}}^2∕{\sigma }_{\mathsf{b}}^2)$, we found the potential for ceiling and floor effects to be exaggerated. Potential amelioratory approaches include averaging multiple baseline and follow-up FEV₁ measurements, as well as regression approaches (by Yanez et al. and Chambless et al.) that account for variation and remove bias (14, 20, 21).

F. Limitations

While ppFEV₁ is less variable in clinic than at home, both home and in-clinic spirometry data demonstrate that day-to-day variation in ppFEV₁ is ubiquitous. Measured variation is the result of a combination of factors, only some of which can be addressed in study design and are especially influential in-home spirometry: testing environment, device, coaching, diurnal variation, recency of treatment. Other sources of variation are pervasive and include sleep, exercise, effort, and mild infections. We know theoretically that we can reduce day-to-day variation by gathering more frequent samples pre- and post-intervention, as the standard deviation of the average of $m$ samples is $\sigma ∕\surd m$. However, clinical trials to date have not included frequent, repeated FEV₁ measurements over multiple days, either at home or in clinic. Therefore, we have not been able to directly test whether computing steady state ppFEV₁ over a longer sampling period, both before and after treatment, results in a diminishment of apparent ceiling effect.

G. Summary

PwCF with high ppFEV₁ values have historically been excluded from clinical trials of therapeutic interventions, with the anticipation that they have little room for improvement even if exposed to a drug with high effectiveness. Their exclusion has been founded, in part, on perceived ceiling effects that may have some basis in biological mechanisms, but which may also be influenced by the statistical phenomenon of regression to the mean. Should treatment benefit be, in truth, more equitable across baseline health, these eligibility exclusions may be unjustified. These trial exclusions for high ppFEV₁ may also increasingly limit study enrollment, as highly effective CFTR modulators improve lung function in pwCF. Caution and examination of day-to-day variation is needed when ascribing a biological foundation for ceiling effects. Bias-corrective measures for day-to-day variation are available and should be implemented (19, 20, 21).

Highlights.

• Ceiling effects in therapeutic benefit are commonly understood to impact cystic fibrosis.

• Day-to-day variation in measured FEV₁ is common.

• Floor and ceiling effects can be exaggerated in the presence of measurement error.

• Estimation of ceiling effects should account for biasing effects of measurement error.