Improving the external validity of Antenatal Late Preterm Steroids (ALPS) trial findings

Abstract

Background:

The external validity of randomised trials can be compromised when trial participants differ from real-world populations. In the Antenatal Late Preterm Steroids (ALPS) trial of antenatal corticosteroids at late preterm ages, participants had systematically younger gestational ages than those outside the trial setting. As risk of respiratory morbidity (the primary trial outcome) is higher at younger gestations, absolute benefits of corticosteroids calculated in the trial population may overestimate real-world treatment benefits.

Objectives:

To estimate the real-world absolute risk reduction and number-needed-to-treat (NNT) for antenatal corticosteroids at late preterm ages, accounting for gestational age differences between the ALPS and real-world populations.

Methods:

Individual participant data from the ALPS trial (which recruited 2831 women with imminent preterm birth at 34+0–36+5 weeks’) was appended to population-based data for 15,741 women admitted for delivery between 34+0 and 36+5 weeks’ from British Columbia, Canada, 2000–2013. We used logistic regression to calculate inverse odds of sampling weights for each trial participant, and re-estimated treatment effects of corticosteroids on neonatal respiratory morbidity in ALPS participants, weighted to reflect the gestational age distribution of the population-based (real-world) sample.

Results:

The real-world absolute risk reduction was estimated to be −2.2 (95% CI: −4.6, 0.0) cases of respiratory morbidity per 100, compared with −2.8 (95% CI: −5.3, −0.3) in original trial data. Corresponding NNTs were 46 in the real-world setting vs 35 in the trial. Our focus on absolute measures also highlighted that the benefits of antenatal corticosteroids may be meaningfully greater at 34 weeks vs 36 weeks (e.g., risk reductions of −3.7 vs −1.2, respectively).

Conclusions:

The absolute risk reductions and NNTs associated with antenatal corticosteroid administration at late preterm ages estimated in our study may be more appropriate for patient counselling as they better reflect the anticipated benefits of treatment when used in a real-world situation.

Background

Individuals who participate in randomised trials often differ from the population to whom the trial intervention will ultimately be offered (i.e., the real-world population).^1,2 Although these differences will not affect the validity of the estimated treatment effect within the study sample (i.e., its internal validity), they can introduce bias to its external validity.³ That is, the trial estimate may not reflect the anticipated treatment effect when used in the real-world setting. As a result, there is growing interest in extending randomised trial findings to estimate the trial intervention’s impact as applied in the real world setting.^3–8 Analytic methods exist that use individual patient data from the randomised trial, but re-weight them to make the distribution of characteristics in the trial population more similar to that of the usual-care population.^5,6 The effectiveness of the treatment is then re-calculated using the re-weighted trial pseudo-population. The approach thus takes advantage of the randomised trial’s high internal validity in estimating treatment effects, but improves external validity by making the distribution of characteristics in the trial population more similar to that of the usual-care population. The approach can be used to improve both generalisability (the extent to which trial findings are valid in the source population from which participants were recruited) and transportability (the extent to which trial findings are valid in a different population from which participants were drawn) of trial findings.⁹

In the Antenatal Late Preterm Steroids (ALPS) trial of antenatal corticosteroid administration among women with threatened imminent preterm birth at late preterm gestation (34–36 weeks),¹⁰ women recruited into the trial were disproportionately more likely to be randomized at 34 or 35 weeks of pregnancy compared with the gestational age at presentation of a real world population of women with imminent late preterm birth (66% vs. 41%, respectively).¹¹ As the risk of respiratory morbidity (the trial’s primary outcome) is significantly higher at 34 and 35 weeks than at 36 weeks,¹² the baseline risk of respiratory morbidity in the ALPS trial was likely higher than in a real-world population, and estimates of the absolute benefits of antenatal corticosteroid administration calculated from the ALPS findings (e.g., absolute risk reduction, numbers needed to treat) may be higher than experienced in outside the trial setting. This is concerning, because patients best understand the harms and benefits of a treatment when presented as an absolute risk reduction from a baseline risk.^13,14 This can be viewed as a problem of effect modification on the absolute scale: the magnitude of the absolute risk reduction associated with antenatal corticosteroids will differ according to the gestational age structure of the population in which they are being administered.¹⁵ We focus on gestational age because it is arguably the factor responsible for the greatest divergence in clinical recommendations regarding routine administration antenatal corticosteroid,^16–18 although it is theoretically possible that there could be other characteristics that modify the effect of antenatal corticosteroid administration and have a different distribution in the ALPS trial than in a real-world population.

As the routine use of antenatal corticosteroids among women with threatened preterm delivery at late preterm ages is controversial,^19–21 accurate estimates of treatment benefit are critical for informing patient-clinician decision-making and creation of clinical practice guidelines. The goal of this study was to estimate the absolute risk reduction in respiratory morbidity and NNT for antenatal corticosteroid administration expected when offered in a real-world setting at late preterm ages, accounting for differences in gestational age distributions between the ALPS and real-world populations. That is, we aimed to improve transportability by extending the ALPS findings to a population in which clinical guidelines informed by the trial are applied.

Methods

Study population

We obtained ALPS trial data through the NICHD data and specimen hub (DASH) data repository.²² Briefly, ALPS was a multicentre randomised trial conducted in 17 U.S. sites, October 2010 to February 2015.¹⁰ The trial recruited 2831 women between 34+0 (34 weeks, 0 days) and 36+5 weeks’ gestation at imminent risk of preterm delivery, and randomised them to 2 doses of 12 mg of betamethasone or placebo. Imminent risk of preterm delivery was defined as spontaneous rupture of membranes, preterm labor with at least 3cm dilation or 75% cervical effacement, or expected delivery for any other indication between 24 hours and 7 days after the planned randomization, based on the expert opinion of the obstetrical care provider. To ensure that an adequate proportion of women presenting at 34 to 35 weeks were included in the trial, it was planned that enrollment would be limited such that no more than 50% of the women in the trial would be 36 weeks, although ultimately, the quota was not needed as the trial only included 34% of participants presenting at 36 weeks, with 27% of participants presenting at 34 weeks and 39% at 35 weeks. Ethical review for these analyses was waived by the University of British Columbia/BC Children’s and Women’s Hospital research ethics board given that the ALPS data available through DASH are de-identified and publicly-available.

Gestational age distribution at presentation in a population-based sample

We used information on the gestational age distribution of 13,255 women presenting to hospital for delivery with a singleton pregnancy at late preterm gestation from a previously published population-based cohort with day-specific, ultrasound-confirmed estimates of gestational age from British Columbia, Canada, 2000–2013.¹¹ Gestational age at maternal admission (rather that at delivery) was used to approximate gestational age at randomisation, as decision-making on antenatal corticosteroid administration typically occurs at the time of initial presentation. In this population-based sample, 14% of women presented for admission at 34 weeks, 27% at 35 weeks, and 59% at 36 weeks. We did not exclude pregnancies complicated by comorbidities (such as diabetes), pregnancy complications (such as fetal anomalies) or in whom delivery was expected within 12 hours, since although these were exclusion criteria in the ALPS trial, these are not contra-indications for antenatal corticosteroid administration in Canadian clinical practice guidelines,¹⁶ and as result, these women would be included in a real-world population. Gestational age at delivery was also obtained from this cohort through hospital discharge records (12% at 34 weeks, 25% at 35 weeks, 56% at 36 weeks, and 8% at ≥37 weeks).

Outcomes

We used the original ALPS primary outcome, which was a composite neonatal outcome of perinatal death or treatment for respiratory morbidity within 72 hours after delivery (referred to hereafter as ‘respiratory morbidity’). In ALPS, the primary outcome occurred in 14.4% of infants in the placebo group, and 11.6% of infants in the betamethasone group (relative risk 0.80 [95% confidence interval (CI): 0.66, 0.97]). We also examined the ALPS secondary outcome of severe respiratory complications (CPAP or high-flow nasal cannula for ≥12 continuous hours, supplemental oxygen with a fraction of inspired oxygen of ≥0.30 for ≥24 continuous hours, ECMO or mechanical ventilation) or perinatal death within 72 hours after delivery.

Statistical analysis

We used previously-proposed methods for extending randomised trial findings to usual-care populations.^5,6,8 A first-principles schematic of the approach is shown in Figure 1, in which weights are assigned to randomised trial participants such that they contribute relatively more, or less, to the analysis based on whether individuals with similar characteristics to them make up a larger, or smaller proportion of the real-world population.

Figure 1.

First principles schematic illustrating re-weighting of Antenatal Late Preterm Steroids (ALPS) randomised trial participants to resemble the gestational age distribution of a real-world population of late preterm maternal delivery admissions from British Columbia, Canada, 2000–2013. For illustrative purposes, gestational age is shown in completed weeks rather than days. Adapted from: Am J Epidemiol 2017;186:1010–1014.

Specifically, we created a dataset that appended individual-level ALPS data with a dataset that contained one row per individual from the British Columbia data with information on gestational age at admission in days (re-created from published data). We used logistic regression to estimate each individual’s odds of being “sampled” into the ALPS cohort from the combined cohort, given their gestational age in days. Next, we calculated inverse odds of sampling weights for each trial participant as the inverse of the odds of being sampled into the ALPS cohort, given gestational age, multiplied by the marginal odds of trial sampling. As previously noted,⁷ inverse odds of sampling weights are calculated when extending trial results to a population that is different than the one from which trial participants were drawn (i.e., transportability), whereas inverse probability of sampling weights can be used when extending trial results to the source population from which trial participants were recruited (i.e., generalisability). Weights are calculated separately by treatment arm to account for chance imbalances during randomisation. Although this method could be used to account for differences in additional factors beyond gestational age, we focused on gestational age as this is the only characteristic of trial participants for which there are different recommendations in clinical practice guidelines.

We used log-binomial regression to estimate the effects of antenatal corticosteroid treatment within the ALPS cohort, before and after weighting models using the inverse odds of sampling weights. We used the ‘margins’ command to obtain marginal estimates of the absolute risk reductions in respiratory morbidity associated with corticosteroid treatment using the results of the log-binomial model.²³ We note that a logistic regression model could also have been used had there been issues with model convergence or implausible predicted probabilities produced by the log-binomial model. We calculated NNTs as the inverse of the absolute risk reduction. We bootstrapped 95% confidence intervals to account for sampling variation in our population-based cohort. Statistical code to apply and troubleshoot the methods used in this study are available at: https://github.com/CIRL-UNC/HybridDesignVisualsWorkshop.

To further explore the strong association between gestational age and baseline risks of respiratory morbidity, we additionally built unweighted and weighted log-binomial regression models that included gestational age at randomisation as a continuous variable (in addition to the treatment variable), enabling the estimation of population-based gestational age-specific absolute risk reductions and NNTs. Gestational age was included as a linear term after verifying that the assumption of linearity was reasonable in this model by visually inspecting the coefficients and 95% confidence intervals from a model regression our primary outcome on gestational age indicator variables and observing a linear pattern on the log scale.

Results

Of the 2,827 infants included in the published ALPS analysis, we excluded two with a gestational age at randomisation of <34+0 weeks, and two with a gestational age at randomisation of 36+6 weeks, as the limited amount of trial data for these gestational ages would make these observations overly influential in the re-weighted model.

Figure 2 shows the gestational age distribution in days of infants included in the ALPS trial, compared with the population-based real world population from British Columbia. Admissions younger than 36+0 weeks were over-represented in the ALPS cohort, while those ≥36+1 weeks were under-represented. Thus, admissions ≤36+0 were assigned inverse odds of sampling weights less than one (down-weighting their contribution), while those ≥36+1 were assigned weights above one (up-weighting their contribution).

Figure 2.

Distribution of gestational age at randomization of participants in the Antenatal Late Preterm Steroids (ALPS) trial compared with the gestational age at maternal admission in a real world population of late preterm births from British Columbia, Canada, 2000–2013.

As expected, the risks of respiratory morbidity were strongly associated with gestational age (Figure 3). At 34+0 weeks, over 30% of infants in the control arm experienced the trial primary respiratory outcome, while by 36+5 weeks, this risk was below 5%. Reweighting the ALPS cohort to reflect the older gestational age distribution of the real-world population resulted in lower average risks of adverse outcome (Table 1): 12.3 per 100 in the placebo arm and 10.1 per 100 in the intervention arm (vs 14.4 and 11.6 per 100 originally reported in ALPS). As a result of these lower baseline risks, the estimated absolute risk reductions associated with treatment were 21% smaller in the real-world population than the clinical trial population (2.2 vs 2.8 per 100, respectively), although 95% confidence intervals overlapped. Similarly, NNTs increased, from 35 in the trial population to 46 in a real-world population. Similar trends were observed for the secondary outcome of major respiratory morbidity.

Figure 3.

Antenatal Late Preterm Steroids (ALPS) primary outcome of neonatal respiratory morbidity in the control arm by gestational age at randomisation.

Table 1.

Antenatal Late Preterm Steroids trial data weighted to reflect the gestational age distribution of a real-world population within the late preterm period.

Outcome	Population	Antenatal corticosteroids risk per 100	Placebo risk per 100	Absolute risk per 100 (95% confidence interval)	Number Needed to Treat
Primary outcome	Original ALPS	11.6	14.4	−2.8 (−5.3, 0.3)	35
	ALPS reweighted	10.1	12.3	−2.2 (−4.6, 0.0)	46
Major respiratory morbidity	Original ALPS	7.9	12.0	−4.1 (−6.3, −1.9)	24
	ALPS reweighted	6.8	10.2	−3.5 (−5.4, −1.4)	29

Table 2 estimates the risk reductions and NNTs associated with antenatal corticosteroid administration for each week of gestation within the late preterm period in the original ALPS cohort and re-weighted to reflect the gestational age distribution in a real-world population. At 34 weeks, point estimates suggest that antenatal corticosteroid administration was associated with a 3.7 per 100 reduction in cases of the primary outcome and an NNT of 27, while by 36 weeks, this decreased to 1.2 fewer cases per 100 and an associated increase in NNT of 84. Similar trends were observed for the secondary outcome. These numbers were only modestly different than the unweighted week-specific ALPS values, suggesting that any differences in gestational age distribution within weeks between the trial and real-world population was modest (i.e., the primary benefit of re-weighting comes when estimating a single overall treatment effect).

Table 2.

Absolute risks and numbers needed to treat by week of gestation in the ALPS cohort, re-weighted to reflect the gestational age distribution at maternal admission for delivery of a population-based sample within the late preterm period from British Columbia, Canada, 2000–2013.

Outcome	Data	Gestational age	Antenatal corticosteroids risk per 100	Placebo risk per 100	Absolute risk per 100 (95% confidence interval)	Number Needed to Treat
Primary outcome	Original ALPS	34 weeks	19.3	23.3	−4.1 (−8.1, 0.1)	24
	Original ALPS	35 weeks	11.2	13.6	−2.4 (−4.7, 0.0)	42
	Original ALPS	36 weeks	6.6	7.9	−1.4 (−2.8, 0.0)	72
	ALPS re-weighted	34 weeks	19.7	23.4	−3.7 (−7.9, 0.5)	27
	ALPS re-weighted	35 weeks	11.1	13.2	−2.1 (−4.4, 0.3)	48
	ALPS re-weighted	36 weeks	6.3	7.5	−1.2 (−2.5, 0.1)	84
Major respiratory morbidity	Original ALPS	34 weeks	13.8	20.2	−6.4 (−10.1, −2.3)	16
	Original ALPS	35 weeks	7.6	11.1	−3.6 (−5.6, −1.5)	28
	Original ALPS	36 weeks	5.2	6.2	−2.0 (−3.2, −0.7)	51
	ALPS re-weighted	34 weeks	13.9	20.4	−6.5 (−10.1, −2.1)	15
	ALPS re-weighted	35 weeks	7.4	10.9	−3.5 (−5.5, −1.1)	29
	ALPS re-weighted	36 weeks	4.0	5.8	−1.9 (−3.2, −0.6)	54

Comment

Principal finding

In this study, we sought to improve the external validity of the ALPS trial findings by estimating the treatment effect of late preterm antenatal corticosteroids, had the trial recruited a population with a similar gestational age distribution to a real-world population. Although differences in point estimates after accounting for population differences were modest (e.g., 21% lower risk reductions of 2.2 vs 2.8 fewer cases per 100, NNTs of 46 vs 35), we argue that they represent better estimates for patient counselling and practice guideline creation. By formally accounting for the difference in gestational age distributions between the ALPS trial participants and a real-world population of women presenting to hospital at late-preterm ages, our estimates may also have greater face validity for physicians and other knowledge-users concerned about the external the validity of risk reductions and NNT calculated from the ALPS trial data. Our focus on absolute measures (absolute risk reductions and NNTs) also helps to highlight that the benefits of antenatal corticosteroids are likely meaningfully different even within the late-preterm period (e.g., risks reductions for major respiratory morbidity of 6.5 per 100 at 34 weeks vs 1.9 per 100 at 36 weeks, NNTs of 15 and 54, respectively).

Strengths

Our study’s novel application of quantitative methods to improve external validity of the treatment effect from a perinatal randomised trial, and our use of population-based, day-specific, ultrasound-confirmed estimates of gestation at maternal admission are study strengths.

Limitations

The real-world population to which we re-weighted trial findings was from Canada rather than the U.S. (where the ALPS trial was done), however, as our goal was to improve transportability of trial findings to an external population, rather than generalisability (i.e., the extent to which trial participants reflect the source population from which they were recruited), this is not a limitation per se, and reflects that randomised trial findings are often used to guide practice in jurisdictions other than that in which the trial was conducted. Analyses such as ours are one strategy to increase external validity in such situations. In the ALPS trial, infants in the antenatal corticosteroid arm were found to be at increased risk of hypoglycemia. We were unable to estimate re-weighted treatment effects for this harm associated with antenatal corticosteroid administration, as the variable for hypoglycemia was not included in the DASH dataset.

As when assessing the internal validity of a study seeking to establish the causal effect of an intervention, the validity of transporting trial findings to a different population requires that a number of assumptions be met. The assumptions required for external validity (both generalizability and transportability) under the potential outcomes framework are discussed in detail by Lesko and colleagues.²⁴ Two of these assumptions, exchangeability and no-multiple-versions of treatment, are most relevant to our study. The assumption that ALPS trial participants are exchangeable (conditional on gestational age) with those in our real-world population could be questioned because our real-world population included pregnancies with comorbidities or complications that were trial exclusion criteria (such as major congenital anomalies). However, the recommendations in clinical practice guidelines to administer antenatal corticosteroids to most pregnancies with threatened preterm delivery, irrespective of comorbidity status, speak to clinical consensus that violations of this assumption are likely not major. Likewise, the assumption that the distribution of treatment versions is the same in the ALPS trial and our real-world population may not hold because real-world administration of antenatal corticosteroids may involve more incomplete doses and sup-optimally timed administration. Nevertheless, similarities in the estimated effect size for antenatal corticosteroids on neonatal respiratory morbidity in ALPS and real-world populations¹¹ suggest that the practical impact of these violations may be minor.

Interpretation

Re-weighting methods have previously been applied to improve the external validity of randomised trial treatment effects in areas such as oncology⁶ and human immunodeficiency virus⁵. However, there have been very few studies to apply reweighting methods to a perinatal or paediatric trial.⁴ Our study serves as an example for other perinatal trials in which uptake of trial results has been limited by concerns over external validity. For example, external generalisability concerns with the ARRIVE trial of labour induction vs expectant management at term in nulliparous women²⁵ are cited as a reason for its poor adoption into clinical practice guidelines.^26,27 In the ARRIVE trial, participants were younger than non-participants, with lower than expected Cesarean delivery rates.²² The ARRIVE trial group published a separate study comparing the characteristics and pregnancy outcomes of participants vs non-participants in an effort to reduce external generalisability concerns;²⁶ the methodology used in our study demonstrates a more objective approach to assess external validity that formally accounts for any differences in treatment effect caused by differences between the trial population and a real world or source population. This more quantitative approach to bias assessment may better refute (or support) concerns about external validity when adopting trial findings into clinical practice and practice guidelines.

Re-weighting is not the only strategy available to increase the external validity of randomised trial findings.³ Stratification by the characteristic of interest-here, gestational age at admission-would also provide results that are more applicable outside the clinical trial population. However, this approach has the disadvantage that it does not provide a single summary measure of the effect of the intervention. In order to account for the day-by-day decrease in risk of adverse neonatal outcomes with advancing gestational age, even within each week of the late preterm period (as shown in Figure 3, where smoothed risks of neonatal respiratory morbidity are 27% at 34+0 weeks vs 15% at 34+6 weeks), 20 separate estimates of treatment effect (one for each day between 34+0 and 36+5 weeks) would need to be calculated, which may be less useful for clinicians and policy-makers. In contrast, re-weighting provides a single summary estimate of the effect of antenatal corticosteroids in the population that takes the daily change in risks into account.

An alternative approach known as post-stratification can produce a single summary estimate of the treatment effect.³ With this approach, estimated treatment effects are calculated within each strata of the characteristic(s) of interest in the trial population (e.g., gestational age), and averaged according to the distribution of the characteristic(s) in the real-world population (e.g., the gestational age distribution in the real world population). Similarly, direct standardisation,²⁸ can be used,⁵ in which the risks of an adverse outcome (rather than treatment effects) are calculated within strata of a characteristic(s) of interest, which are then applied to the population distribution of an external or standard population to produce a standardised risk. A treatment effect can then be estimated by comparing the standardised risks of two different groups. However, for both strategies, the number of adverse outcomes within each strata will become small as the number of strata increases, leading to potentially unstable estimates of risks or treatment effects. Thus, while straightforward, these approaches are less useful when multiple characteristics, characteristics measured as continuous variables, or characteristics with numerous categories such as day-specific gestational age, are being adjusted for. Stratification remains a useful first step to identify sparse data situations in which parametric regression modelling may not be appropriate.

Conclusions

The results presented in this study maximise the utility of the ALPS randomised trial for clinical counselling and guideline creation purposes by providing estimated treatment effects of antenatal corticosteroids at late preterm ages in absolute measures, as likely experienced in a real-world setting.

Social media quote:

Routine antenatal corticosteroid administration at late preterm ages in the real-world setting may provide lower absolute risk reductions and higher numbers-needed-to-treat than in the randomised trial setting.

Synopsis.

Study question:

What is the likely treatment effect of routine administration of antenatal corticosteroid administration at late preterm ages when administered outside of the clinical trial setting?

What’s already known:

Randomized trials show that antenatal corticosteroid administration at late preterm gestation reduces neonatal respiratory morbidity. However, external validity of trial findings may be limited by differences in characteristics between trial participants and real-world populations, particularly, the younger gestational age of trial participants.

What this study adds:

We re-analyzed individual participant data from the Antenatal Late Preterm Steroids (ALPS) trial to estimate treatment benefits of antenatal corticosteroids given a real-world gestational age distribution at late preterm. We found that absolute risk reductions may be moderately smaller, and numbers-needed-to-treat higher, with routine administration outside the trial setting.