Illustrating an Adaptive Prespecification Framework for Observational Research: Target Trial Emulations Comparing Immunomodulator Treatments for COVID-19

Abstract

Rigid prespecification can be impractical for noninterventional studies using secondary datasets, where data-driven flexibility is often required. Using target trial emulations comparing immunomodulator treatments for COVID-19, we piloted an adaptive strategy that accommodates warranted mid-course refinements within a prespecified framework. Our preregistered protocol outlined an initial study plan along with predetermined diagnostic thresholds and contingencies. Implementation proceeded through sequential phases, allowing researcher decisions to be guided by prespecified criteria under varying degrees of blinding to results. The adaptive approach led to alterations in the underlying target trial and to the analysis plan used for emulation, strengthening the plausibility of causal assumptions and improving the relevance of findings. During the initial baseline phase, indicated contingencies included sample restrictions, redefining treatments from class-level to product-specific comparisons, a revised propensity score model, and weight truncation. In the subsequent postbaseline phase, diagnostic checks triggered a modified causal contrast, inverse probability of censor weighting to address noncompliance, cause-specific hazard estimation to contextualize competing events, and additional reporting of hazard ratios for progressively truncated follow-up periods. For a secondary study objective, the adaptive framework allowed for some iterative attempts to improve validity while providing a clear stopping point. Similar approaches could lend transparent structure to the process of learning what causal questions the data are equipped to support. Beyond guarding against researcher bias, prespecification of adaptive protocols may promote more robust designs by encouraging investigators to be explicit about their assumptions, strategies for interrogating those assumptions, and specific criteria for determining when and how deviations may be required.

Noninterventional (observational) studies are increasingly considered as a potential source of evidence regarding the safety and effectiveness of clinical interventions. Maintaining transparency and objectivity in design and analytic choices is essential to allow decision-makers to critically evaluate and interpret evidence derived from observational studies. A single research aim can be addressed through a number of alternative designs and analytical choices, and different choices can lead to different conclusions.¹ Use of different study design types,^1–3 exposure definitions,⁴ outcome algorithms,^1,5 covariate model specifications,^6,7 and follow-up criteria^1,8,9 have all been shown to contribute to variability in conclusions for the same exposure-outcome association. The practice of prespecifying detailed protocols and statistical plans can encourage independence between initial design and analysis choices and knowledge of how those choices influence study conclusions. However, rigid commitment to a study plan can present unique challenges for noninterventional studies using secondary datasets, where some degree of flexibility is often warranted.

The suitability of a given study plan is contingent on meeting statistical and causal identification assumptions, as well as on real-world treatment patterns and dataset features, which cannot all be known or anticipated a priori. Consequently, data-driven refinements are often needed to strengthen study validity or to aid with results interpretation. For instance, prespecified plans may be modified to improve feasibility, to adapt to unanticipated dataset features (e.g., patterns of missingness, data sparsity), or to strengthen the plausibility of key assumptions (e.g., conditional exchangeability, positivity). However, without prespecification of when and under what circumstances deviation from a prespecified protocol will occur, decisions may be influenced or perceived to be influenced by knowledge of study results.

Data-dependent decisions made before finalizing study protocols are subject to similar concerns. Within the target trial framework,¹⁰ for instance, the hypothetical target trial may be refined as researchers learn more precisely what causal questions (e.g., related to eligibility criteria, treatment strategies, causal contrasts) are best supported by the available data. Although often strengthening validity, such iterative explorations can be vulnerable to data dredging.¹¹ Further, this open-ended process, paired with the availability of alternate design choices, may inadvertently encourage multiple comparisons¹² or selective reporting of results. Thus, there is a natural tension between rigid prespecification to maintain objectivity in design and analysis choices, and the data-driven flexibility required to generate valid and interpretable inferences. More adaptive and transparent prespecification approaches are needed to adequately manage data-driven flexibility in observational research.

For studies that aim to emulate a hypothetical target trial, adaptive designs for randomized controlled trials may serve as a guiding model. Within adaptive clinical trials, design components can be modified during the study according to predetermined rules and scheduled interim checks of accumulating data.^13,14 Similarly, for non-interventional studies, some forms of planned flexibility can be incorporated as decision rules in a priori plans, reducing the need for unplanned and open-ended protocol deviations. Although non-interventional studies lack the prospective data accumulation of adaptive randomized controlled trials, this flexibility can still be staggered via phased access to study data.

Here, we demonstrate how analytic flexibility can be incorporated in a prespecified manner, using an adaptive design with predetermined diagnostic thresholds, contingency plans, and sequential study phases. We illustrate this adaptive approach with target trial emulations comparing immunomodulator treatments for COVID-19 for the outcome of 28-day in-hospital mortality.

METHODS

Overview of Adaptive Prespecification Approach

Development of the adaptive study protocol followed a multistep process (Figure 1). We first detailed a study design and analysis plan representing the optimal approach under what we expected—a priori—to be best supported by the available dataset (the “Base Case” approach). This Base Case plan served as the starting point for the adaptive strategy, termed an intentional multiphase approach (IMA). Next, to guide potential IMA refinements, we prespecified diagnostic thresholds for scenarios where deviations from the initial Base Case plan would be warranted, with contingency plans under different diagnostic circumstances. Diagnostic checks focused on assessing the plausibility of causal or statistical assumptions, informing dataset unknowns, and thresholds for alternative design choices. Corresponding contingency plans allowed for alterations to both the underlying target trial (e.g., via different eligibility criteria or causal contrasts) or adjustments to the data analysis plan (e.g., re-specify models with different covariate forms). Diagnostics and contingencies were separated into distinct study phases, according to checks using data from the baseline period (before time zero, inclusive) versus the postbaseline period (after time zero).

FIGURE 1.

Overview of adaptive intentional multiphase approach for prospective planning of flexibility. Elements of the general adaptive framework include prespecified decision rules (via predetermined diagnostics and contingency plans) and a phased study implementation. Figure 1 shows these elements as they were applied to the illustrative use cases in COVID-19, through what we termed an intentional multiphase approach.

For the applied use case, to assess the impact of different levels of diagnostic-driven flexibility on study conclusions, we compared results under: (1) a Base Case approach, which adhered to the fully prespecified plan; (2) IMA-1, which incorporated baseline flexibility, while maintaining blinding to postbaseline data; and (3) IMA-2, which built on IMA-1 by additionally considering postbaseline flexibility. Inferential statistics were only generated after pursuing all indicated contingency plans and satisfying all relevant diagnostic requirements in each phase.

Applied Use Case

The adaptive IMA strategy was illustrated with target trial emulations comparing 28-day in-hospital mortality after treatment with interleukin-6 receptor inhibitors (IL6) versus Janus kinase inhibitors (JAK), among patients hospitalized with COVID-19 receiving corticosteroids. Randomized trials have demonstrated survival benefits for the addition of either IL6^15,16 or JAK¹⁷ to systemic corticosteroids for persons with severe or critical COVID-19. However, given a paucity of head-to-head studies, there is limited evidence for which of these add-on immunomodulatory regimens is more effective.

We emulated separate target trials for those with severe and critical COVID-19, defined according to the National Institutes of Health (NIH) COVID-19 treatment guidelines¹⁸ updated on 1 February 2022. The severe COVID-19 emulation included patients for whom either IL6 or JAK were recommended by NIH, defined as those requiring supplemental oxygen, noninvasive ventilation, or high flow oxygen. The critical COVID-19 emulation targeted a subgroup for which only IL6 was recommended as of February 2022 (despite emerging evidence¹⁹ to support JAK use in critical COVID-19), comprising patients admitted directly to the intensive care unit who required invasive mechanical ventilation or extracorporeal membrane oxygenation. The severe COVID-19 emulation additionally characterized progression to invasive mechanical ventilation or extracorporeal membrane oxygenation as a secondary outcome.

The applied use case analyzed de-identified data from the HealthVerity Hospital Chargemaster and Administrative Claims database, which has been used for other published COVID-19 studies.^20–29 Use of these data was approved for institutional review board review exemption by the New England Independent Review Board. The study protocol was posted on clinicaltrials.gov³⁰ and followed the HARmonized Protocol Template to Enhance Reproducibility.³¹ The preregistered protocol describes two sensitivity analyses, which are outside the scope of the current manuscript. All analyses were conducted using Aetion Substantiate (version 2023) and R (version 4.1.2).

Base Case Design and Analysis Plan

The base population included adults hospitalized with a COVID-19 admitting diagnosis (International Classification of Diseases, Tenth Revision [ICD-10] UO07.1), and an admission date between 16 June 2020 (starting after publication of Randomised Evaluation of COVID-19 Therapy trial results that made dexamethasone³² the corticosteroid standard of care) to 1 February 2022 (end of available data). We used an active comparator, new-user design³³ to emulate assignment of treatment strategies for either IL6 (single intravenous dose of tocilizumab or sarilumab) or JAK (once-daily oral dose of baricitinib or tofacitinib) within 4 days of hospital admission, among subjects who required oxygen supplementation or ventilatory support and were receiving systemic corticosteroids. The index date (i.e., time zero) was assigned according to the date of observed IL6/JAK initiation for each subject.

Inverse probability of treatment weighting (IPTW) was used to control for measured baseline confounding. Unstabilized weights were constructed from propensity score (PS) models estimated with logistic regression as the probability of IL6 versus JAK exposure conditional on baseline covariates for select comorbidities, demographic factors, facility characteristics, concomitant treatments, and measures of COVID-19 severity. Covariates considered for confounding control were informed by clinical input and published studies,^{27,28,34–37} and were refined using causal diagrams. For the Base Case, all a priori selected covariates were included in the PS models (eFigure 1 and eTable 5; https://links.lww.com/EDE/C268).

Follow-up for the outcomes began 1 day after IL6/JAK initiation with censoring at the earliest of outcome, discharge, death (for secondary outcome), or a maximum of 28 days. The corresponding causal contrast was the observational analog of an intention-to-treat analysis, referred to herein as an “initial treatment” design, whereby any postbaseline noncompliance or crossover was ignored. Hazard ratios (HR) with 95% confidence intervals (CI) were estimated for time-to-event outcomes using weighted Cox proportional hazard models with robust standard variance estimators.

Intentional Multiphase Approach Diagnostics and Contingencies

In IMA-1, treatment definitions, eligibility criteria, and weight model specifications could be modified if predetermined baseline diagnostic criteria were not satisfied within the Base Case approach (eTable 1; https://links.lww.com/EDE/C268). Baseline diagnostics included measures of overlap and covariate balance between treatment groups, with imbalance defined as absolute standardized difference ≥0.1; inspection of weight distributions, with extreme weights defined as >6 SD from the mean value; PS model overfitting, defined as <12 exposed subjects per degree of freedom; and other diagnostics that indirectly assessed assumptions underlying valid estimation of average treatment effects.^38–41 Additional checks focused on specific unknowns regarding treatment patterns, including uncertain utilization of individual IL6 and JAK products, and uncertain relevance of contraindications listed within US labeling for non-COVID indications.^42,43 For each diagnostic check, we prespecified contingency plans if thresholds were not satisfied (eTable 1; https://links.lww.com/EDE/C268).

IMA-2 built upon IMA-1 by additionally considering checks in postbaseline data (eTable 1; https://links.lww.com/EDE/C268). The first set of these additional checks pertained to the handling of postbaseline events, including competing risks, crossover between treatment groups, and treatment discontinuation (for JAK only). As a greater proportion of subjects experience competing events^44–46 (discharge for the primary endpoint; discharge or death for the secondary endpoint), the Base Case survival analysis estimators may become increasingly biased. Similarly, the interpretation and relevance of the initial-treatment contrast may be diminished if a high proportion of subjects deviate from initial treatments during follow-up.⁴⁷ If this proportion exceeded 20% for a given postbaseline event in either treatment arm, we decided a priori that corresponding IMA-2 contingency plans would be warranted. The final postbaseline check entailed inspection of proportional hazards to determine if average 28-day HRs estimated with weighted Cox regression—as in the Base Case plan—were appropriate summaries of IMA-2 treatment effects.⁴⁸ Permitted IMA-2 contingencies for different postbaseline diagnostic scenarios are outlined in eTable 1; https://links.lww.com/EDE/C268.

RESULTS

Diagnostics and Contingencies Pursued Within the Intentional Multiphase Approach

Severe COVID-19

A total of 1,603 patients (835 new users of IL6 vs. 765 new users of JAK) met eligibility criteria for the Base Case target trial emulated in the sample with severe COVID-19. After applying IPTW and before considering IMA contingencies, this cohort had an effective sample size of 919 (511 IL6 vs. 408 JAK), a mean weight of 1.914 (SD 1.666), and a maximum weight of 20.711. Baseline and postbaseline diagnostic results are detailed in eAppendices 2–4; https://links.lww.com/EDE/C268 and summarized in Figure 2 below.

FIGURE 2.

Overview of diagnostic checks and corresponding contingencies for target trial emulations in severe and critical COVID-19. Panels (A,B) illustrate baseline and postbaseline diagnostics and corresponding IMA contingencies for the emulation in severe COVID-19. Postbaseline diagnostics in (B) are shown for the analysis using IMA-1 baseline components with Base Case postbaseline components, for the primary outcome. Panel (C) illustrates baseline diagnostics and attempted IMA-1 contingencies for the emulation in critical COVID-19. ASD indicates absolute standardized difference; BAR, baricitinib; HR, hazard ratio; IL6, interleukin-6 receptor inhibitors; IMA, intentional multiphase approach; IPCW/IPTW, inverse probability of censor/treatment weights; JAK, janus kinase inhibitors; PS, propensity score; TCZ, tocilizumab.

Overall, baseline diagnostics triggered five IMA-1 contingencies deviating from the Base Case approach ([2.1–2.5] in Figure 2A). Triggered contingencies (2.1–2.3) entailed changes to the hypothetical target trial of interest via modified treatment definitions and restricted eligibility criteria, including: (2.1) treatments re-defined as most commonly used drugs within each class, tocilizumab versus baricitinib; (2.2) restriction to hospitalizations beginning after December 2020, to address nonpositivity resulting from lack of JAK use from June to November 2020; and, (2.3) exclusion of subjects receiving inpatient dialysis, another likely structural cause of nonpositivity, given JAK cautions for severe renal impairment.⁴³ After implementing restrictions (2.1–2.3), 90.0% of the Base Case patients (1,442/1,603) were included in the base IMA-1 cohort.

Application of the Base Case weight model improved balance diagnostics overall, but did not yield balance across all measured covariates. Additionally, IMA-1 contingencies for PS model re-specification (2.4) and weight truncation (2.5) were applied in an iterative manner to address residual imbalance (absolute standardized differences ≥0.1) and to satisfy other predetermined diagnostic criteria (signs of model misspecification, extreme weights, random positivity violations, and overfit PS model). Several PS model specifications were pursued, with different covariates and functional forms, before satisfying all prespecified diagnostic requirements (2.4). Finally, after evaluating weight distributions, we chose to truncate weights (henceforth denoted “IPTW^T”) at the predetermined cutoff (10.04, 6 SD from the mean weight value) (2.5). Despite sample restrictions to the original cohort, the final IMA-1 IPTW^T population had a larger effective sample size (n = 1,037) than the analogous Base Case (n = 919) IPTW population, highlighting efficiency gains from diagnostic-driven refinements.

In the ensuing postbaseline phase, assessment of diagnostics within the IMA-1 approach led to four additional IMA-2 contingencies, detailed as (3.1–3.4) in Figure 2C. First, to address baricitinib noncompliance exceeding the 20% threshold, IMA-2 analyses additionally censored upon discontinuation of baricitinib before the recommended 14-day course or discharge (considered an “on-treatment” causal contrast, as patients were censored with no additional adjustment [3.1]). Assessment of postbaseline covariate balance relative to censoring⁴⁹ after contingency (3.1) suggested likely informative censoring, prompting use of inverse probability of censor weights (IPCW) to emulate the hypothetical scenario where all baricitinib initiators were fully compliant with their initial treatment during follow-up (i.e., “per-protocol” causal contrast; [3.2]). We estimated IPCW with pooled logistic regression as the inverse probability of remaining uncensored for discontinuation at each eligible time point, conditional on time-fixed (baseline) and time-varying (postbaseline) covariates. Accordingly, IMA-2 HRs were estimated from a marginal structural Cox model weighted by the product of time-fixed IPTW and time-varying IPCW, after truncation (IPTW^T × IPCW^T).⁴¹

The >20% diagnostic threshold was also surpassed for competing events, leading to IMA-2 contingency (3.3) to estimate cause-specific hazards for outcomes of interest and competing events, to generate cumulative incidence functions instead of Kaplan–Meier curves, and to define a composite secondary outcome combining the competing event of death with progression to invasive mechanical ventilation or extracorporeal membrane oxygenation.

Finally, we inspected log–log plots and Schoenfeld residuals. Departures from proportionality for both outcomes of interest were observed starting in the final week of follow-up (eAppendix 4; https://links.lww.com/EDE/C268). This triggered IMA-2 contingency (3.4): estimation of separate HRs for progressively longer durations of follow-up (0–7, 0–14, 0–21, 0–28). As an unplanned post hoc analysis, we also used bootstrapping in 1000 re-samples to obtain valid 95% CIs for the original 28-day HR (eAppendix 6; https://links.lww.com/EDE/C268).^48,50,51

Table 1 summarizes the final approaches used for Base Case and IMA target trial emulations within severe COVID-19. Further detail and rationale for contingencies, target trial components, and analysis plans are available in the Supplemental Digital Content.

TABLE.

Overview of Base Case and Intentional Multiphase Approach Target Trial Emulations in Hospitalized Patients With Severe COVID-19

	Base Case	Intentional Multiphase Approach (After Pursuing All Indicated IMA-1 and IMA-2 Deviations From Base Case)
Eligibility criteria	Hospitalized for severe COVID-19 from 16 Jun 2020 to 01 Feb 2022 and receiving corticosteroids	Hospitalized for severe COVID-19 from 01 Dec 2020 to 01 Feb 2022 and receiving corticosteroids, and not receiving inpatient dialysis
Treatment strategiesa	IL6: One dose of intravenous tocilizumab or sarilumab JAK: Once-daily oral baricitinib or twice-daily tofacitinib for 14 days or until discharge	IL6: One dose of intravenous tocilizumab JAK: Once-daily oral baricitinib for 14 days or until discharge
Treatment assignment	Emulate random assignment via new-user active comparator design with treatment strategies consistent with observed data and IPTW to address baseline confounding (randomization assumed conditional on covariates)	Same as Base Case, except randomization is assumed conditional on covariates within the refined PS/IPTW model
Time zero	Date of observed treatment initiation	Same as Base Case
Outcomes	Primary outcome of in-hospital death; Secondary outcome of invasive mechanical ventilation/extracorporeal membrane oxygenation	Primary outcome of in-hospital death; Secondary composite outcome of invasive mechanical ventilation/extracorporeal membrane oxygenation or in-hospital death
Follow-up period	Follow-up from 1 day after treatment initiation to the earliest of outcome, discharge, death (for secondary outcome), or 28th day of follow-up reached	Follow-up from 1 day after treatment initiation to the earliest of outcome, discharge, early baricitinib discontinuation, or end of maximum follow-up period (report separate estimates for maximum follow-up of 7, 14, 21, and 28 days)
Causal contrast	Observational analogue of intention-to-treat (initial-treatment contrast): effect of initiating IL6 vs JAK treatment strategies	Per-protocol: effect of sustained treatment with tocilizumab vs baricitinib treatment strategies, defined as initiating and adhering to specified treatment strategies
Identifying assumptions	Conditional exchangeability between treatment groups, given baseline covariates in Base Case IPTW model (eTable 5)	Conditional exchangeability between treatment groups, given baseline covariates in IMA IPTW model (eTable 5), AND conditional exchangeability between baricitinib compliers and noncompliers, given baseline and time-varying covariates in IMA IPCW model (eTable 8)
Statistical analysis	Baseline adjustment: IPTW used to adjust for baseline confounding. Unstabilized IPTW constructed from a fully prespecified PS model specification with 107 degrees of freedom and basic covariate forms. Postbaseline adjustment: None Inferential outputs: Initial-treatment HRs (95% CIs) for time-to-event outcomes from IPTW-adjusted Cox proportional hazard models and Kaplan-Meier survival curves	Baseline adjustment: Same as Base Case, except uses truncated weights constructed from refined PS model with 56 degrees of freedom Postbaseline adjustment: Adjust for selection bias from baricitinib discontinuation with IPCWT estimated with time-fixed (baseline) and time-varying (postbaseline) covariates Inferential outputs: Per-protocol HRs (95% CIs) for outcomes and competing events from IPTWT × IPCWT-adjusted cause-specific Cox regression models and cumulative incidence functions

For display of the target trial specifications underlying the Base Case and IMA emulation studies, see eTable 2.1. Bold type highlights the target trial emulation components that differ in the IMA relative to base case.

^aTreatment strategies defined according to NIH COVID-19 treatment guidelines.¹⁸ Crossover between treatment arms was permitted in either strategy.

CI indicates confidence interval; HR, hazard ratio; IL6, interleukin-6 receptor inhibitors; IMA, intentional multiphase approach; IPCW/IPTW, inverse probability ofcensor/treatment weights; IPTW^T/IPCW^T denote truncated weights; JAK, janus kinase inhibitors; PS, propensity score.

Critical COVID-19

Baseline diagnostic checks highlighted serious flaws for the attempted target trial emulation within critical COVID-19 (Figure 2C and eAppendix 3; https://links.lww.com/EDE/C268). No combination of permitted deviations from the Base Case approach was sufficient in satisfying predetermined baseline diagnostic criteria for this secondary study objective. Therefore, within the piloted IMA approach, emulating the target trial in a critical disease was deemed infeasible. Further detail on diagnostic findings and inferential results for the Base Case emulation in critical COVID-19 is provided in the supplementary appendix.

Use Case Inferential Results

Complete results across all outcomes and analyses are available in eAppendix 5; https://links.lww.com/EDE/C268. Here, we briefly highlight results for the primary outcome of in-hospital mortality for the treatment comparison in severe COVID-19.

The forest plot in Figure 3 presents HRs and 95% CIs across sequential baseline and postbaseline contingencies. The final results for Base Case, IMA-1, and IMA-2—bolded in Figure 3—are the fully adjusted estimates obtained after applying all indicated contingent plans within each approach.

FIGURE 3.

Forest plot for primary outcome of in-hospital death across sequential contingencies for target trial emulation in severe COVID-19. Bolded estimates indicate the final adjusted results for each of the Base Case, IMA-1, and IMA-2 approaches after applying all indicated contingency plans. Base Case results followed the fully prespecified study plan. IMA-1 and IMA-2 deviated from the Base Case plan according to predetermined diagnostic checks and contingency plans: IMA-1 incorporated baseline diagnostics, while IMA-2 incorporated both baseline and postbaseline diagnostics. BAR indicates baricitinib; CI, confidence interval; CW/TW, inverse probability of censor/treatment weights; CW^T/TW^T denote truncated weights; FUP, follow-up period; HR, hazard ratio; IL6, interleukin-6 receptor inhibitors; IMA, intentional multiphase approach; IT, initial treatment; JAK, janus kinase inhibitors; OT, on-treatment; PP, per-protocol; TCZ, tocilizumab.

Results From Fully Prespecified Plan (Base Case)

In the Base Case analysis, 23.8% of persons in the IL6 group and 19.1% in the JAK group experienced the outcome of death, with a median follow-up of 9 days (IQR, 6–16). Before adjustment, IL6 was associated with a greater risk of mortality compared with JAK (HR = 1.15; CI = 0.93, 1.42), which was attenuated after IPTW (HR = 1.05; CI = 0.80, 1.38) (Figure 3, [1]).

Results After Incorporating Baseline Flexibility (IMA-1)

Primary outcome estimates for IMA-1 followed similar trends as the Base Case (Figure 3, [2.1–2.5]). The unadjusted HR was 1.16 (CI = 0.92, 1.45) after IMA-1 sample restrictions (2.1–2.3), with attenuated estimates following adjustment with weights from the refined IMA-1 PS model (2.4) (HR = 1.07; CI = 0.92, 1.25) and after weight truncation (2.5) (HR = 1.10; CI = 0.94, 1.29). Relative to the Base Case, contingencies for sample restrictions (2.1–2.3) and modified weight models (2.4–2.5) yielded only minor changes to point estimates (Figure 3). However, the refined PS model (2.4) did improve precision, with narrower CIs than the Base Case both before and after truncation.

Results Incorporating Baseline and Postbaseline Flexibility (IMA-2)

In IMA-2, censoring for early baricitinib discontinuation without further adjustment led to a reduction in mortality for the baricitinib group (12.3%), as compared with initial treatment censoring within the same baricitinib initiator population (19.1%). The corresponding IMA-2 HRs with no adjustment or only baseline IPTW adjustment suggested a protective effect for baricitinib ([3.1] in Figure 3). However, after adjustment for potential selection bias with IPCW^T, the per-protocol estimate ([3.2] in Figure 3) was attenuated towards the null (HR = 1.01; CI = 0.75, 1.37).

All estimates reported to this point have been HRs during the protocol-defined follow-up period of up to 28 days. However, survival curves (Base Case and IMA-1) and cumulative incidence plots (IMA-2) (eAppendix S5; https://links.lww.com/EDE/C268) suggested that 28-day HRs were weighted averages of effects operating in different directions over the course of follow-up: an apparent protective effect for JAK/baricitinib during the first ~21 days, with an opposite trend (protective for IL6/tocilizumab) in the remaining ~7 days. In the final IMA-2 contingency (3.4), tocilizumab was associated with an increased hazard of death relative to baricitinib, with CIs excluding the null, for 0–7, 0–14, and 0–21 day follow-up durations. Point estimates trended towards the null as follow-up was progressively lengthened, culminating with an HR (95% CI) of 1.01 (0.75, 1.37) for the longest—and originally prespecified—28-day follow-up period. For the competing risk of discharge, HRs were more stable across different follow-up durations, with all 95% CIs containing the null ([3.4] in Figure 3).

DISCUSSION

Our Base Case plan in the prespecified protocol reflected a principled and rigorous approach. Nonetheless, when applied to the real-world dataset, the Base Case plan was found to be lacking, with implausible identification assumptions (e.g., nonpositivity and questionable exchangeability) for emulations in both severe and critical COVID-19.

For the emulation in severe COVID-19, the adaptive IMA approach addressed Base Case limitations with alterations to both the underlying target trial specification and the analysis plan used for emulation. Relative to the Base Case, IMA contingencies (2.1–3.3) did not lead to meaningfully different point estimates for primary or secondary outcomes. Regardless, these data-driven refinements improved upon the Base Case approach by strengthening the plausibility of assumptions necessary for estimates to be interpreted as causal and by reframing the original research question to be more pertinent for COVID-19 treatment in routine clinical practice.

Overall study conclusions for the applied use case were most influenced by the culminating contingency—IMA-2 contingency (3.4)—which suggested a protective effect for baricitinib relative to tocilizumab during the first ~21 days of follow-up. Similar time-varying trends have been observed in at least one other observational study,⁵² but residual bias from unmeasured confounding or differential depletion of susceptibles cannot be ruled out. Nonetheless, this finding highlights the need for transparent reporting of survival curves and other diagnostics to aid in the interpretation of summary effect measures, as well as the limitations of estimators relying on proportional hazards.⁴⁸ It also underscores the importance of prespecified guardrails to protect against selective reporting and other forms of results-driven design decisions, as different defensible choices (e.g., choice of follow-up duration) can lead to different conclusions regarding a drug’s effectiveness.

Our adaptive approach employed two complementary strategies for guarding against potential researcher bias, while still allowing researchers to tailor their analysis to the data: first, prespecified flexibility^53,54 via predetermined diagnostic checks and contingency plans; and second, separation of diagnostic checks and contingencies into study phases, during which decisions can be made under different degrees of blinding. Criteria for refining or modifying initial study plans are not routinely documented in prespecified protocols. When elements of this process can be anticipated, decisions that would otherwise be made ad-hoc (e.g., preliminary explorations, warranted protocol deviations, and post hoc analyses) can be incorporated in a prespecified manner. Many assumptions and unknowns cannot be informed by the observed data, and few can be encapsulated by a singular diagnostic metric. Indeed, several diagnostic thresholds within the applied use case were not informed by published precedent (e.g., 20% threshold for postbaseline events). Nonetheless, advanced commitment to justifiable benchmark(s) can instill trust that mid-course refinements are based on prespecified criteria.

In addition to promoting transparency and objectivity, the act of prespecifying decision rules may also encourage more principled research designs. Development of an adaptive protocol requires that researchers be explicit about not only their assumptions but also strategies for interrogating the plausibility of those assumptions, and specific criteria for determining when and how deviations may be required. Advanced consideration of these elements may also help investigators better anticipate potential challenges and identify relevant sensitivity analyses.

This study complements other work to promote transparency and reproducibility, including structured templates and protocols for planning and reporting real-world data studies,^31,55–57 a protocol registry built for such studies,⁵⁸ and stepwise processes such as the Causal Roadmap and PRINCIPLED framework that guide investigators to prespecify study design and analysis plans.^59,60 Our applied COVID-19 use case extends prior examples addressing challenges in identifying initial estimands^61–63 by incorporating a prespecified structure. While the specific study design and decision rules used in the COVID-19 example have important limitations, our intent is not to advocate for universal application of these elements in other research settings. Rather, we aimed to demonstrate how an adaptive prespecified structure can facilitate the prospective planning of flexibility, particularly in uncertain and high-stakes research environments.

Strengths of our general adaptive approach include its simplicity and versatility. For example, prespecified decision rules can be easily customized to meet study requirements, in terms of the types of diagnostics considered, the specificity of thresholds and contingency plans, and the timing of prespecification (i.e., all at once as in the Base Case, or before the beginning of each phase). For instance, we paired specific thresholds for overfitting and covariate balance with a more general list of permitted contingencies to achieve these thresholds (e.g., via adjustment to PS models, sample restriction, etc.). For the attempted target trial emulation in the critical COVID-19 use case, this allowed for some iterative attempts to address identified sources of bias while providing a clear and predetermined stopping point. Similar approaches could help lend transparency and structure to the often open-ended process of learning what causal questions are adequately supported by the available data.⁶⁴

Our applied use case followed sequential baseline and postbaseline phases, a common approach for studies evaluating simple point-in-time treatments within longitudinal datasets.^65,66 However, there are many ways to separate or phase data-driven decision making based on the degree of blinding to final results, and more complex approaches may be needed when baseline and postbaseline phases are not easily definable (e.g., dynamic treatment regimens). Investigator familiarity with the data and clinical context may also influence prespecification and phasing decisions. In our case, prior experience with a related version of the HealthVerity dataset—used in an earlier study of a different inpatient treatment—likely shaped the construction of our Base Case design and prespecified components. Other contexts may warrant a different starting point, a more or less prescriptive Base Case plan, or alternative phasing strategies. For example, phasing could include distinct feasibility stages or drug utilization studies before finalizing protocols,⁶⁷ using split samples,⁶⁸ staggered access to full datasets,^69,70 or techniques to formally mask or simulate data components.^54,71,72 Promising extensions to this simple adaptive design demonstration include simulation-based phasing strategies to compare different analytic strategies⁷³ as well as more comprehensive diagnostic metrics, such as negative control outcomes.

CONCLUSIONS

Mid-course refinements to study design and analysis plans, even if warranted, are typically not accommodated by rigid approaches to prespecification. The adaptive approach illustrated here represents a simple, practical, and versatile structure for accommodating learning from data activities within a prespecified framework.

ACKNOWLEDGMENTS

The authors thank David Lenis (Aetion, Inc.) for his critical feedback during the development of the study protocol and review of diagnostic findings. We would also like to thank Sachin Shah (FDA) and Melanie Wang (Aetion, Inc.) for their management and coordination of this collaborative research effort.