Emulating randomized trials by observational database studies: the RCT-DUPLICATE initiative in COPD and asthma

Abstract

Observational studies are increasingly used to provide real-world evidence in regulatory decision-making. The RCT-DUPLICATE initiative conducted observational studies emulating 2 published randomized trials in patients with asthma and 3 in chronic obstructive pulmonary disease (COPD). For each trial, new-user cohorts were constructed from 2 US healthcare claims databases, comparing initiators of the study and comparator drugs, matched on propensity scores. Proportional hazards models were used to compare the treatments on study outcomes. The observational studies involved more subjects than the corresponding trials, with treatment arms well-matched on baseline characteristics. An asthma example involved emulation of the 26-week FDA-mandated D5896 trial. With 6494 asthma patients per arm, the hazard ratio (HR) of a serious asthma-related event with budesonide-formoterol vs budesonide was 1.29 (95% CI, 0.63-2.65) compared with 1.07 (95% CI, 0.70-1.65) in the trial. A COPD example is the emulation of the one-year IMPACT trial. With 4365 COPD patients per arm, the HR of a COPD exacerbation with triple therapy vs dual bronchodilators was 1.08 (95% CI, 1.00-1.17) compared with 0.84 (95% CI, 0.78-0.91) in the trial. We found mainly discordant results between observational analyses and their emulated randomized trials, likely from the forced discontinuation of treatments prior to randomization in the trials, not mimicable in the observational analyses.

This article is part of a Special Collection on Pharmacoepidemiology.

Introduction

Evidence-based approaches for the treatment of respiratory diseases such as asthma and chronic obstructive pulmonary disease (COPD) depend mainly on randomized controlled trials (RCTs). These trials form the basis of approval decisions by regulatory agencies, the content of labels, and treatment algorithms for clinical guidelines. Randomized controlled trials have been vital in establishing current pharmacologic approaches to asthma and COPD, which rely on long-acting bronchodilators and inhaled corticosteroids (ICS).^1,2

Observational studies using data from clinical practice have been used to complement RCTs, investigating the safety and effectiveness of drugs to treat several respiratory diseases, including asthma and COPD, in real-world settings. The role of observational studies to provide real-world evidence (RWE) in regulatory decision-making has received increasing attention.^3-5 Several initiatives were launched to analyze how findings from published RCTs track with well-conducted observational studies.^6-8 The RCT-DUPLICATE aimed to compare the findings of 32 RCTs for a wide range of diseases and drugs with corresponding analyses of claims data designed to emulate the selected RCTs, including 5 respiratory drug trials, 3 in COPD and 2 in asthma.^6,9

The American Thoracic Society recently called for rigorous real-world evidence to supplement data from RCTs in informing clinical practice guidelines.¹⁰ Understanding how the findings of carefully designed observational studies compare to those of RCTs is vital as clinician and professional societies rely on real-world evidence when RCT data are lacking or limited. We report the results of observational database analyses that emulated the 5 respiratory trials selected by the RCT-DUPLICATE initiative and provide an in-depth account of how they compare with the corresponding trials.

The 5 selected trials

The process for selecting trials to target for emulation and implementation process have been described previously.⁶ Briefly, the selection, carried out in collaboration with the Food and Drug Administration (FDA), included published RCTs designed to support regulatory decision-making and that could be emulated with the available claims databases.⁶ A trial was considered a good candidate for emulation if the primary outcome and the study treatments were measurable with sufficient specificity and if the inclusion/exclusion criteria were generally attainable. This prespecified selection process identified 3 COPD and 2 asthma trials that were used in a regulatory context.

The first trial in COPD was the Investigating New Standards for Prophylaxis in Reducing Exacerbations (INSPIRE) study.¹¹ It assessed the effectiveness of salmeterol-fluticasone propionate compared with tiotropium in 1323 patients with severe to very severe COPD on the incidence of exacerbations over 2 years. Upon discontinuing their current treatments, patients entered a 2-week run-in during which all received oral prednisolone and inhaled salmeterol, which were discontinued at randomization (example in Figure 1). The rate ratio of moderate or severe exacerbations was 0.97 (95% CI, 0.84-1.12) when comparing salmeterol-fluticasone propionate with tiotropium. The second trial was the Prevention of Exacerbations with Tiotropium in COPD (POET-COPD) study, a trial of 7376 patients with moderate to very severe COPD and a history of exacerbations.¹² Patients underwent randomization after a 2-week run-in during which prior tiotropium use was replaced by ipratropium and prior long-acting beta₂-agonist (LABA) use was permitted to continue, both discontinued at randomization. The hazard ratio of moderate or severe exacerbation with tiotropium was 0.83 (95% CI, 0.77-0.90) compared with salmeterol. The third trial was the Informing the Pathway of COPD Treatment (IMPACT) study, which compared 52 weeks of a single-inhaler triple therapy (fluticasone furoate-umeclidinium-vilanterol) with 2 dual therapies (fluticasone furoate-vilanterol and umeclidinium-vilanterol) on the incidence of exacerbation in 10 355 patients with a history of COPD exacerbations.¹³ The subjects discontinued all prior COPD medications and were immediately randomized with no run-in (example in Figure 2). The hazard ratio of a first exacerbation with triple therapy was 0.84 (95% CI, 0.78-0.91) compared with umeclidinium-vilanterol and 0.85 (95% CI, 0.80-0.91) compared with fluticasone furoate-vilanterol.

Figure 1.

Depiction of a trial with a run-in period: The example of the INSPIRE randomized trial in COPD, where patients discontinued their current COPD treatments and entered a 2-week run-in period during in which they all received oral prednisolone and inhaled salmeterol, which were discontinued at randomization.

Figure 2.

Depiction of a trial with no run-in period: The example of the FDA-mandated D5896 randomized trial in asthma, where patients discontinued their current asthma treatments and were randomized directly to the budesonide-formoterol combination or budesonide.

In asthma, the first trial was the P04334 study that investigated the efficacy and safety of mometasone furoate-formoterol combination treatment in 781 patients with persistent asthma previously receiving medium-dose ICS.¹⁴ Upon discontinuing their current treatment, the subjects entered a 2-3-week run-in, where all received mometasone furoate, and were then randomized to 26 weeks of treatment with the mometasone furoate-formoterol combination and each component or placebo. It found a reduction in asthma deteriorations with mometasone furoate-formoterol compared with formoterol (rate ratio 0.56; 95% CI, 0.44-0.72) and 0.90 (95% CI, 0.67-1.20) compared with mometasone. The second trial was an FDA-mandated study (D5896) to assess the safety of the budesonide-formoterol combination compared with budesonide over 26 weeks in 11 693 patients with persistent asthma, prior exacerbations, and treated with or requiring ICS.¹⁵ The patients were randomized directly after discontinuing their current treatments (Figure 2). The hazard ratio of a serious asthma-related event was 1.07 (95% CI, 0.70-1.65).

Methods

Data sources

We used 2 US healthcare claims data sources to emulate these trials by observational designs, Optum Clinformatics (2004 through 2020) and IBM MarketScan (2003 through 2017).⁶ These data sources contain deidentified information for all patients enrolled in participating health insurance plans, including demographics, enrollment dates, dispensed medication information, and procedures and medical diagnoses. Signed licensing agreements were in place, and approval was obtained from the Mass General Brigham Institutional Review Board.

Study design

The detailed protocols for the 5 studies were registered before the analysis was conducted and are available on clinicaltrials.gov (links provided in Table S1). Following an established practice for all trials,¹⁶ we applied a new-user cohort design for each of the 2 databases to compare initiators of the drug under study with initiators of the comparator treatments, with cohort entry defined as the day of treatment initiation. Subjects were required to have 6 months continuous enrollment in the database before cohort entry. Details of each trial emulation, including CONSORT diagrams for cohort formation, are available in Tables S2-S6 for the 5 trials.

Within each database, we replicated the trials using 1:1 propensity score (PS) nearest-neighbor matching with a caliper of 0.01 on the PS scale to control for >120 potential confounders measured during the 6 months before drug initiation. The 1:1 matching was used even if the trials used unequal randomization ratios. Although the trials reported data on fewer patient characteristics, the RWE emulation of randomization was based on a larger set of pre-exposure covariates to balance as many potential confounders or confounder proxies as possible.¹⁷ The covariates included demographics, calendar year, and proxy measures of asthma and COPD disease severity, including the number of prior moderate and severe COPD exacerbations and the use of other respiratory drugs during the baseline period. Comorbidity was also measured using diagnostic codes and prescribed medications.

For 2 COPD trials, namely IMPACT and POET, in which the primary outcome was the frequency of moderate or severe exacerbations, we instead analyzed and compared with the secondary outcome of time to the first exacerbation because of challenges with identifying recurrent exacerbations in databases. However, for the INSPIRE trial, which did not report on the outcome of time to first exacerbation, the reported rate ratio of the frequency of exacerbations was used to compare with the emulation, approximated with a hazard ratio from the database study. In all cases, a moderate exacerbation was defined as an outpatient visit requiring treatment with oral corticosteroids, while a severe exacerbation was defined as a hospitalization with a primary diagnosis of COPD. These diagnostic codes have been shown to have good accuracy in the databases.¹⁸ For the asthma P04334 trial, we did not replicate the placebo comparator and only present results for the active comparators. The outcomes of asthma deterioration were defined by an emergency department visit, a hospitalization, or an exacerbation treated with oral corticosteroids (P04334) and by a serious asthma-related event defined by intubation, hospitalization, or asthma-related death (D5896).^14,19

Data analysis

The propensity score of treatment initiation was estimated by logistic regression using covariates presented in Tables S2A-S6A. Standardized mean differences of covariates were computed to assess the comparability of the 2 treatment groups.

After matching, the Cox proportional hazard model was used in as-treated analyses to compare continuous exposure to these treatments during the trials’ given follow-up period. Continuing exposure was measured by successive prescriptions of the initial treatment, with 60-day grace period between the end of a prescription and the start of the next. The as-treated time ended with the discontinuation of the index drug, namely the end of the last treatment episode plus the 60-day grace period, or switch between compared therapies, or the end of the trial follow-up window, or death, whichever was first.

The concordance of the RWE results against the RCT results was evaluated using 3 predefined binary agreement metrics,⁶ namely: (1) statistical significance agreement, defined by estimates and confidence intervals on the same side of null; (2) estimate agreement, defined by whether estimates for the trial emulation fell within the 95% CI for the trial results; (3) standardized difference agreement between treatment effect estimates from trials and emulations, with a difference of 2 as the cutoff (corresponding to P value ≤ 0.05).

Several sensitivity analyses were performed. First, we restricted the follow-up of the observational analyses to that of the corresponding trials. Second, an intention-to-treat analysis was conducted, irrespective of treatment discontinuation or switching. Third, pneumonia was used as a control outcome event, in view of its known association with inhaled corticosteroids. Fourth, in both studies of asthma, we restricted subjects to those with no history of a COPD diagnosis. Finally, we looked at the difference in mean age between the RCT-database study pairs. For pairs where the difference in mean age was greater than 5 years, we reweighted the database study cohorts so that the mean age aligned with the mean age of the RCT participants using standardized morbidity weights. The reweighting was implemented by generating a simulated RCT population with age distributed using the observed mean and standardized difference for the RCT participants. After concatenating the simulated RCT data with the database study cohort, we fit a logistic regression model that predicted whether an observation came from the simulated RCT vs database cohort given age. Standardized morbidity ratio weights were then calculated to reweight the database cohort to the age distribution of the RCT. All analyses were conducted using the Aetion Evaluation Platform v4.40 (incl. R v3.4.2), which has been validated by accurately repeating 150 previously published studies,²⁰ and by replicating²¹ or predicting clinical trial findings²² (SAS version 9.4).

Results

The numbers of subjects, outcome event counts, and rates comparing the observational studies and the corresponding randomized trials are displayed in Table 1. The observational emulation of the 5 trials resulted in well-matched treatment and comparator arms on baseline characteristics (Tables S2B-S6B).

Table 1.

Comparison of first event rates (asthma or COPD exacerbation) and adjusted hazard ratios between the 5 RCTs and the duplicating observational claims database studies.

	Published RCT					Observational study
	No. of patients	No. with outcome event	Mean person- months	Rate per 100 per year	Hazard ratio (95% CI)	No. of patients	No. with outcome event	Mean person- months	Rate per 100 per year	Hazard ratio (95% CI)
COPD: INSPIRE study
Fluticasone-salmeterol	658	408	18.4	40.3	0.97 (0.84-1.12)b	49 405	6857	4.9	34.0	0.93 (0.90-0.96)
Tiotropium	665	392	17.0	41.5	1.00 (Reference)	49 405	7190	4.8	36.5	1.00 (Reference)
COPD: POET study
Tiotropium	3707	Not reported		46.2c	0.83 (0.77-0.90)	4358	950	4.3	60.3	1.02 (0.93-1.12)
Salmeterol	3669	Not reported		54.5c	1.00 (Reference)	4358	901	4.2	59.5	1.00 (Reference)
COPD: IMPACT study
Triple inhalerd	4151	Not reported		69.3c	0.84 (0.78-0.91)	4365	1276	4.3	82.3	1.08 (1.00-1.17)
Dual bronchodilatord	2070	Not reported		77.6c	1.00 (Reference)	4365	1148	4.1	77.3	1.00 (Reference)
Triple inhalerd	4151	Not reported		69.3c	0.85 (0.80-0.91)	4439	1311	4.3	83.2	1.13 (1.04-1.23)
LABA-ICS inhalerd	4134	Not reported		77.6c	1.00 (Reference)	4439	1112	4.1	73.9	1.00 (Reference)
Asthma: P04334 study
Mometasone-formoterol	191	5	5.5*	5.7	0.56 (0.44-0.72)	1054	114	4.0	32.7	0.78 (0.62-0.97)
Formoterol	202	31	4.7*	38.8	1.00 (Reference)	1054	139	3.9	40.5	1.00 (Reference)
Mometasone-formoterol	191	5	5.5*	5.7	0.90 (0.67-1.20)	2097	229	3.9	33.8	0.83 (0.70-0.99)
Mometasone	192	10	5.5*	11.4	1.00 (Reference)	2097	255	3.5	41.8	1.00 (Reference)
Asthma: D5896 study
Budesonide-formoterol	5846	43	6.0*	1.47	1.07 (0.70-1.65)	6494	24	5.1	0.87	1.29 (0.63-2.65)
Budesonide	5847	40	6.0*	1.36	1.00 (Reference)	6494	18	4.7	0.70	1.00 (Reference)

^aApproximated from flowchart.

^bThe hazard ratio was not reported in the INSPIRE trial, thus the reported rate ratio was used to approximate the hazard ratio.

^cApproximated from the Kaplan–Meier cumulative incidence, assuming an exponential distribution.

^dTriple inhaler is fluticasone furoate-umeclidinium-vilanterol, dual bronchodilator is umeclidinium-vilanterol, and LABA-ICS is fluticasone furoate-vilanterol.

For the COPD studies, the INSPIRE trial emulation included 49 405 patients with COPD in each of the salmeterol-fluticasone propionate and tiotropium arms (Table 1). The hazard ratio of a first COPD exacerbation with salmeterol-fluticasone was 0.93 (95% CI, 0.90-0.96) when compared with tiotropium (Table 1). The corresponding estimate from the trial was 0.97 (95% CI, 0.84-1.12; Table 1).

The emulation of the POET-COPD trial involved 4358 patients in each of the tiotropium and salmeterol arms (Table 1). The hazard ratio of a first COPD exacerbation with tiotropium was 1.02 (95% CI, 0.93-1.12) when compared with salmeterol (Table 1). The corresponding estimate from the trial was 0.83 (95% CI, 0.77-0.90; Table 1).

The emulation of the IMPACT trial included 4365 COPD patients in each of the triple and umeclidinium-vilanterol arms, and 4439 in each of the triple and fluticasone furoate-vilanterol arms (Table 1); these were all similar on baseline characteristics (Tables S4B and S4C). The hazard ratio of a first COPD exacerbation with triple therapy was 1.08 (95% CI, 1.00-1.17) when compared with umeclidinium-vilanterol and 1.13 (95% CI, 1.04-1.23) when compared with fluticasone furoate-vilanterol (Table 1). The corresponding estimates from the trial were 0.84 (95% CI, 0.78-0.91) and 0.85 (95% CI, 0.80-0.91; Table 1).

The emulation of the P04334 asthma trial involved 1054 subjects in each of the mometasone furoate-formoterol and formoterol arms, and 2097 in each of the mometasone furoate-formoterol and mometasone furoate arms (Table 1); these were similar on baseline characteristics (Tables S3B and S3C). The hazard ratio of a first event of asthma deteriorations in the 26-week follow-up with mometasone furoate-formoterol was 0.78 (95% CI, 0.62-0.97) compared with formoterol and 0.83 (95% CI, 0.70-0.99) compared with mometasone (Table 1). The corresponding estimates from the trial were 0.56 (95% CI, 0.44-0.72) and 0.90 (95% CI, 0.67-1.20; Table 1).

The FDA-mandated D5896 trial emulation included 6494 patients in each of the budesonide-formoterol and budesonide arms (Table 1). The hazard ratio of a first serious asthma-related event in the 26-week follow-up with budesonide-formoterol was 1.29 (95% CI, 0.63-2.65) compared with budesonide (Table 1). The corresponding estimate from the trial was 1.07 (95% CI, 0.70-1.65; Table 1).

Sensitivity analyses generally confirmed the robustness of the results (Tables S7-S9). In particular, the analyses based on the same follow-up as the corresponding trials and on the intention-to-treat analyses resulted practically in the same findings (Table S7A and S7B). The risk of pneumonia was, as expected, elevated with ICS use in all studies of COPD but not in the P4334 asthma study (Table S8). In both studies of asthma, excluding subjects with history of a COPD diagnosis did not alter the findings (Table S9). Finally, 2 of the RCT database study pairs had mean age differences greater than 5 years (P4334 and POET-COPD). After reweighting the database study cohorts to align the mean age with that of the RCTs, the results were virtually unchanged for P4334, while the results for POET-COPD moved slightly further away from the RCT result after aligning the distribution of age.

Discussion

In this respiratory component of the RCT DUPLICATE initiative, we found mainly discordant results between our observational analyses and those of the randomized trials they were emulating. These discrepancies were particularly notable for the 3 COPD studies with conflicting conclusions between the 2 approaches. Several reasons could explain these conflicting findings, besides the obvious fact of randomization since residual confounding cannot be ruled out in observational studies, despite propensity score matching which created groups highly comparable on available measures of patient characteristics.

A primary reason is the design aspects of RCTs that are impossible to emulate with clinical practice data, such as the forced discontinuation of baseline maintenance medications in RCTs. Indeed, the trials either had patients discontinue their baseline therapy at the time of randomization or introduced a run-in transition period after discontinuation of maintenance therapy, during which the subjects changed therapy, before being randomized to the study treatments.

An example of randomization immediately after discontinuation is the COPD IMPACT trial, where 70% of the subjects were already receiving ICS therapy, which was withdrawn at the time of randomization. Abrupt ICS discontinuation may have affected the outcomes observed for patients randomized to the LAMA-LABA arm, resulting in a confounded effect of triple therapy. Indeed, the trial clearly shows that the difference in exacerbation risk between triple therapy and LAMA-LABA is exclusively in the first month after discontinuation-randomization, with no difference in the subsequent 11 months.^23,24 Moreover, an analysis of the IMPACT trial among the patients who were not previously on ICS inhalers reported a nonsignificant rate ratio of moderate or severe COPD exacerbation comparing single-inhaler triple therapy with LAMA-LABA of 0.88 (0.76-1.03).^25,26

Another example is the FDA-mandated D5896 asthma trial comparing budesonide-formoterol and budesonide, for which 90% of enrolled subjects were using an ICS as part of their maintenance therapy leading up to the trial, probably most of which were in combination with a LABA. Thus, patients already on an LABA-ICS combination who were allocated to the budesonide (ICS) only arm likely had their outcome affected due to discontinuation of LABA at randomization. Indeed, LABA discontinuation in asthma has been associated with significant increases in emergency visits for asthma, in systemic steroid use, and in withdrawal due to lack of efficacy or loss of asthma control,²⁷ resulting in underestimating the relative risk of the combination arm in the randomized trials.^28,29 The mixing of randomization and discontinuation effects may explain the FDA-mandated D5896 asthma trial’s hazard ratio of 1.07 with budesonide-formoterol, lower than the 1.34 found in the observational emulation. Such forced discontinuation in an experimental trial setting, also affecting the IMPACT trial in COPD, cannot be found in clinical practice and thus is not capable of emulation with real-world data and nonexperimental designs.

The other 3 trials avoided the abrupt discontinuation of prior medications by introducing a run-in period before randomization. Thus, the INSPIRE trial patients entered a 2-week run-in period during which patients discontinued all existing COPD maintenance medications and received oral prednisolone and inhaled salmeterol to “standardize their clinical condition before randomization.” By standardizing treatments that will be discontinued equally for all patients, the effect of discontinuation of prior treatment is attenuated. As such, the relative risk (RR) of a COPD exacerbation with fluticasone-salmeterol vs tiotropiumin in our observational emulation (0.92; 95% CI, 0.90-0.96) was comparable to the trial’s (0.97; 95% CI, 0.84-1.12). Nonetheless, the treatment selected for the run-in period can also affect the results of randomized trials of the same class as one of the study treatments.³⁰ An interesting example is the P04334 asthma trial, where patients discontinued treatments and entered a 2- to 3-week run-in period during which they all received the ICS mometasone furoate. Thus, for the ICS-LABA vs LABA comparison, the run-in on the ICS means that ICS responders were selected, and those allocated to the LABA arm had to abruptly discontinue their ICS, potentially leading to more exacerbations, which could explain the larger difference in results between the trial and the observational study (RR, 0.81 vs 0.56). On the other hand, for the ICS-LABA vs ICS comparison, there was no effect of abrupt discontinuation of the ICS given during the run-in, with results that were much closer between the trial and the observational study (RR, 0.81 vs 0.90). Nonetheless, the P04334 asthma trial was very small, with around 200 patients per arm and 5 with an outcome exacerbation in the ICS-LABA arm—quite insufficient to provide reliable treatment effect estimates.

The POET-COPD trial, however, included a 2-week run-in period but with different treatments depending on a patient’s pretrial maintenance therapy. Indeed, patients already on tiotropium received its short-acting version, namely ipratropium, during the run-in, while those on a LABA simply continued their maintenance therapy until randomization, at which time these were discontinued. The effect of switching patients who are on a bronchodilator class at baseline to the other bronchodilator class by randomization is unclear. The observational analysis, which did not have such switching effects, resulted in a hazard ratio of a first COPD exacerbation with tiotropium of 1.03 (95% CI, 0.94-1.13) vs the trial’s estimate of 0.83 (95% CI, 0.77-0.90).

Other design aspects could help explain the differences between the RCT and the emulating observational studies. First, trials of longer duration can keep patients on the treatment for the duration of the study, but observational studies will typically reflect the shorter adherence of real-world clinical practice. For example, the 2-year INSPIRE trial followed patients for a mean of 561 days under salmeterol-fluticasone and 519 days under tiotropium compared with 149 and 145 days, respectively, in the observational emulation. Second, the observational studies did not consider the frequency of exacerbations occurring during follow-up as an outcome because of challenges with identifying recurrent exacerbations in databases. Third, the definitions of an exacerbation varied, with all COPD trials defining a moderate exacerbation as one that required corticosteroids and/or antibiotics, while we defined it as one that required corticosteroids because of our inability to determine whether the antibiotics were being used for COPD or some other indication. Additionally, a severe exacerbation was defined in the RCTs as one that “required hospitalization,” while we used a hospitalization with COPD in the primary diagnosis position, which could lead to some differences. Fourth, while adherence to inhaler treatments is particularly challenging, even in the controlled setting of randomized trials, the treatment exposure measures in our observational studies were based on dispensed prescriptions so that actual adherence to treatment was uncertain. Thus, the resulting exposure misclassification is potentially compounded by the less than optimal inhaler technique in the real-world setting, particularly since it is estimated that 50% of patients with asthma fail to take their medication as directed.³¹ Thus, the issue of adherence in the real-world context of clinical practice makes the study of respiratory medications particularly challenging. Fifth, we did not adjust for potentially informative censoring from treatment discontinuation, in line with the trials that also did not despite between arms differences in censoring. Nonetheless, there are few good predictors for censoring due to treatment discontinuation, which makes it challenging to build informative probability of censoring weights.³² Indeed, these weights require good predictors of discontinuation that are difficult to capture in clinical practice data (as well as trials). Consequently, the use of censoring weights derived from incorrectly specified models with poor discriminative ability has the potential to increase both variance and bias. Finally, the propensity score matching that we used in the observational studies estimates the average treatment effect for the treated (ATT), while randomized trials estimate the average treatment effect (ATE), which can also explain some of the differences between the estimates.

While the randomized trial is the gold standard for evidence of a treatment’s effectiveness, observational studies can be useful to evaluate situations not studied in RCTs. For example, while guidelines provide treatment recommendations for the initial treatment for patients newly diagnosed with COPD, there are no trials in these patients. Indeed, all trials in this disease have included patients with longstanding disease and treatment. Observational studies can address this question by identifying newly diagnosed patients and have been used to support guideline recommendations for initial treatment strategies of COPD management.^33,34 Another situation is that randomized trials enroll specific patient populations, such as patients with a history of COPD or asthma exacerbations. However, in real-world clinical practice, the study medications are prescribed to patients with no prior exacerbations, systematically excluded from the trials of these medications. For example, around 50% of patients initiating triple therapy in our observational study had no exacerbation in the year prior to initiation. Observational studies can provide valuable information on the effectiveness (or lack of) and safety of triple therapy for these patients excluded from the trials.^26,35-37

In all, while the goal of RCT DUPLICATE was to calibrate observational study results against randomized trials, we found that it was mostly unachievable in the context of asthma and COPD drug treatments. The main obstacle is in some design aspects of the randomized trials, such as the forced discontinuation of prior treatment, with or without a run-in, which cannot be emulated in an observational clinical context. These design aspects introduce a complexity in interpreting the resulting treatment effect, which inherently also includes an effect of prior treatment discontinuation. Clearly, the decision to enroll patients who are already treated in these trials and who are forced to discontinue their usual therapies is for practical reasons; a trial would take years to accrue only newly diagnosed or newly treated patients with asthma or COPD. The price to pay for this decision is, however, a biased treatment effect.

Future randomized trials that aspire to estimate treatment effects unaffected by the impact of prior treatment discontinuation and be complemented by real-world observational data could be designed accordingly using, for example, an adaptive selection trial design.³⁸ This design adapts the selection of patients for random allocation to study treatments according to the treatments already used by the patients. For example, an adaptive design for the IMPACT trial would only include patients on a LABA, a LAMA, or both when randomly allocating to the triple LAMA-LABA-ICS vs LAMA-LABA inhalers. Alternatively, when contrasting the triple LAMA-LABA-ICS and LABA-ICS inhalers, the adaptive design would only include patients on a LABA, ICS, or both. The resulting treatment effects would be unaffected by prior treatment discontinuations. Stratified results of the COPD IMPACT trial that correspond to such adaptive selection find unconfounded treatment effects that can be emulated more accurately by an observational study.²⁶ In the meantime, caution should be used in comparing results from RCTs and observational studies in regulatory decision-making in the context of asthma and COPD treatments, with careful consideration given to the design aspects of the randomized trials.

Supplementary material

Supplementary material is available at American Journal of Epidemiology online

Funding

This study was supported by contracts from the US Food and Drug Administration (HHSF223201710186C and HHSF223201810146C) to the Brigham and Women’s Hospital (PI S.Schneeweiss and S.W.). S.W. and S.Schneeweiss were further supported by funding from the National Institutes of Health RO1HL141505, R01AG053302, and R01AR080194. The content is solely the responsibility of the authors and does not necessarily represent the official views of the US Food and Drug Administration.

Conflict of interest

S.Suissa attended scientific advisory committee meetings or received speaking fees from AstraZeneca, Atara, Boehringer-Ingelheim, Bristol-Myers-Squibb, Merck, Novartis, Panalgo, Pfizer and Seqirus. S.Schneeweiss is principal investigator of the FDA Sentinel Innovation Center funded by the FDA, coprincipal investigator of an investigator-initiated grant to the Brigham and Women’s Hospital from Boehringer Ingelheim and UCB Pharma unrelated to the topic of this study. He is a consultant to Aetion Inc., a software manufacturer of which he owns equity. His interests were declared, reviewed, and approved by the Brigham and Women’s Hospital and MGB HealthCare System in accordance with their institutional compliance policies. W.F. reports serving as a consultant for Alosa Health and Aetion and has served as an expert witness in litigation against inhaler manufacturers. S.V.W reports ad hoc consulting for Cytel Inc, Exponent Inc and MITRE, a federally funded research and development center for the Centers for Medicare and Medicaid. All other authors have no disclosures.

Data availability

No data available. In line with HIPAA regulations, our data use agreements for MarketScan, Optum and Medicare/Medicaid do not permit us to share patient-level source data or data derivatives with individuals and institutions not covered under the agreements. The administrative and clinical research databases used in this study are accessible to other researchers by contacting the data providers and acquiring data use agreements/licenses. The research data and data derivatives cannot be shared outside of the terms of these agreements. The data providers we used are responsive to requests for contracting use of their patient data resources. However, the cost, timeframe, and process for completing the contract for authorized use of these data varies. Contacts and information on how to acquire access to source data: Medicare/Medicaid resdac@umn.edu, https://resdac.org/research-identifiable-files-rif-requests; Optum Clinformatics connected@optum.com, https://www.optum.com/business/solutions/life-sciences/real-worlddata/claims-data.html; MarketScan https://www.merative.com/contact, https://www.ibm.com/products/marketscan-research-databases/resources. Interested parties are also welcome to contact the PI to collaborate.