Evaluating treatment efficacy in hospitalized COVID-19 patients, with applications to Adaptive COVID-19 Treatment Trials

Abstract

Background

The current endpoints for therapeutic trials of hospitalized COVID-19 patients capture only part of the clinical course of a patient and have limited statistical power and robustness.

Methods

We specify proportional odds models for repeated measures of clinical status, with a common odds ratio of lower severity over time. We also specify the proportional hazards model for time to each level of improvement or deterioration of clinical status, with a common hazard ratio for overall treatment benefit. We apply these methods to Adaptive COVID-19 Treatment Trials.

Results

For remdesivir versus placebo, the common odds ratio was 1.48 (95% confidence interval (CI) = 1.23–1.79; p < 0.001), and the common hazard ratio was 1.27 (95% CI = 1.09–1.47; p = 0.002). For baricitinib plus remdesivir versus remdesivir alone, the common odds ratio was 1.32 (95% CI = 1.10–1.57; p = 0.002), and the common hazard ratio was 1.30 (95% CI = 1.13–1.49; p < 0.001). For interferon beta-1a plus remdesivir versus remdesivir alone, the common odds ratio was 0.95 (95% CI = 0.79–1.14; p = 0.56), and the common hazard ratio was 0.98 (95% CI = 0.85–1.12; p = 0.74).

Conclusions

The proposed methods comprehensively characterize the treatment effects on the entire clinical course of a hospitalized COVID-19 patient.

Introduction

A number of phase 3 clinical trials have been conducted to evaluate the efficacy of therapeutic agents for treating moderately or severely ill patients diagnosed with COVID-19.^1–6 The efficacy endpoints that have been used are the time to a specific change in clinical status or the clinical status on a particular day. Specifically, in the completed Adaptive COVID-19 Treatment Trials (ACTTs) 1, 2, and 3 and the ongoing Accelerating COVID-19 Therapeutic Interventions and Vaccines (ACTIV)-1 trial, the primary endpoints were time to recovery, defined as discharge from the hospital or hospitalization for infection-control purposes only, and the secondary endpoints included 28-day mortality or time to death, and clinical status at day 15 or day 28.^3–7 However, these endpoints do not fully represent important clinical outcomes or make efficient use of available data. Mortality is of utmost importance, but most clinical trials are not powered to detect a moderate difference in mortality, especially as COVID-19 deaths continue to decline. By contrast, with mortality as a secondary endpoint, success cannot be claimed if the treatment difference in mortality turns out to be statistically significant, but the treatment’s effect on time to recovery is not.

In clinical trials of hospitalized COVID-19 patients, such as ACTT and ACTIV, the clinical status of each patient is measured daily on an 8-point ordinal scale: 1, not hospitalized and no limitations of activities; 2, not hospitalized, with limitation of activities, home oxygen requirement, or both; 3, hospitalized, not requiring supplemental oxygen and no longer requiring ongoing medical care; 4, hospitalized, not requiring supplemental oxygen but requiring ongoing medical care; 5, hospitalized, requiring any supplemental oxygen; 6, hospitalized, requiring noninvasive ventilation or use of high-flow oxygen devices; 7, hospitalized, receiving invasive mechanical ventilation or extracorporeal membrane oxygenation; and 8, death.^3–7 A patient’s status may improve or deteriorate during the follow-up (Figure 1). Time to recovery is defined as the first day on which a patient reaches categories 1, 2, or 3.^3–7 This endpoint does not capture other types of improvement or provide any information about changes in clinical status for patients who have not recovered by the end of the study, as elaborated below.

Figure 1.

Clinical status of patients with COVID-19. Clinical status is rated on an 8-category ordinal scale: 1, not hospitalized and no limitations of activities; 2, not hospitalized, with limitation of activities, home oxygen requirement, or both; 3, hospitalized, not requiring supplemental oxygen and no longer requiring ongoing medical care (used if hospitalization was extended for infection-control or other nonmedical reasons); 4, hospitalized, not requiring supplemental oxygen but requiring ongoing medical care (related to COVID-19 or to other medical conditions); 5, hospitalized, requiring any supplemental oxygen; 6, hospitalized, requiring noninvasive ventilation or use of high-flow oxygen devices; 7, hospitalized, receiving invasive mechanical ventilation or extracorporeal membrane oxygenation; and 8, death. Transitions between adjacent categories are shown by arrows. Direct transitions over multiple categories (e.g. 3 to 5, 4 to 1) are possible but are not explicitly indicated. A transition to the left represents an improvement in clinical status, whereas a transition to the right represents a deterioration in clinical status.

Figure 2 displays the clinical-status trajectories over a 28-day period for four COVID-19 patients. Both patients 1 and 2 started at category 4 at enrollment. Both improved to category 3 on day 10 and thus had the same time to recovery. However, patient 1 continued to improve, reaching category 2 on day 20 and category 1 on day 26, whereas patient 2 remained at category 3 throughout the study. Thus, the clinical status of patient 1 improved much more than that of patient 2. Both patients 3 and 4 started at category 5 at enrollment. Patient 3 improved to category 4 on day 18, whereas patient 4 deteriorated to categories 6, 7, and 8 on days 8, 16, and 24, respectively. Clearly, patient 4 was much worse off than patient 3, and yet these two patients had the same value for time to recovery, that is, being censored on day 28. Patients 1 and 2 had the same clinical status at day 15, while patient 3 was better than patient 4 by only one category at day 15. Clearly, the treatment effect on clinical status at day 15 can be quite different from the treatment effect on clinical status on other days.

Figure 2.

Clinical-status trajectories for four patients with COVID-19. Clinical status is rated on an 8-point ordinal scale: 1, not hospitalized and no limitations of activities; 2, not hospitalized, with limitation of activities, home oxygen requirement, or both; 3, hospitalized, not requiring supplemental oxygen and no longer requiring ongoing medical care; 4, hospitalized, not requiring supplemental oxygen but requiring ongoing medical care; 5, hospitalized, requiring any supplemental oxygen; 6, hospitalized, requiring noninvasive ventilation or use of high-flow oxygen devices; 7, hospitalized, receiving invasive mechanical ventilation or extracorporeal membrane oxygenation; and 8, death. Patient 1 started at category 4 at enrollment and improved to category 3 on day 10. Patient 2 started at category 4 at enrollment and improved to category 3 on day 10, to category 2 on day 20, and to category 1 on day 26. Patient 3 started at category 5 at enrollment and improved to category 4 on day 18. Patient 4 started at category 5 at enrollment and deteriorated to categories 6, 7, and 8 on days 8, 16, and 24, respectively.

We propose two robust and powerful methods to assess the totality of evidence for treatment efficacy in a clinical trial: the first one considers the clinical status of each day, and the second one considers the time to each level of improvement or deterioration in clinical status. Both methods make full use of available data and allow investigators to make a single probability statement (one confidence interval, together with one p-value) about the treatment effect on the entire clinical course of a patient. These methods can be used for primary or secondary analysis. We demonstrate the advantages of these two methods over the currently used endpoints using ACTT-1, ACTT-2, and ACTT-3 data.

Methods

Data sources

ACTT-1 was a double-blind, randomized, placebo-controlled trial of intravenous remdesivir in adults who were hospitalized with COVID-19 (ClinicalTrials.gov registration number, NCT04280705). ³ Enrollment began on 21 February 2020 and ended on 19 April 2020. There were 60 trial sites and 13 subsites in the United States and 9 other countries. A total of 541 patients received remdesivir (200 mg loading dose on day 1, followed by 100 mg daily for up to 9 additional days), and 521 received placebo (for up to 10 days). Each patient was assessed daily on the 8-point ordinal scale during their hospital stay, from day 1 through day 29, as well as on days 15, 22, and 29 after discharge.

ACTT-2 was a double-blind, randomized, placebo-control trial evaluating baricitinib plus remdesivir versus remdesivir alone in adults hospitalized with COVID-19 (ClinicalTrials.gov registration number, NCT04401579). ⁴ Enrollment began on 8 May 2020 and ended on 1 July 2020. There were 67 trial sites in 8 countries. A total of 515 patients received baricitinib (a 4-mg daily dose for up to 14 days), and 518 received placebo; all 1033 patients received remdesivir. Each patient was evaluated daily on the 8-point ordinal scale during their hospital stay, from day 1 through day 29, as well as on days 15, 22, and 29 after discharge.

ACTT-3 was a double-blind, randomized, placebo-controlled trial evaluating interferon beta-1a plus remdesivir versus remdesivir alone in adults hospitalized with COVID-19 (ClinicalTrials.gov registration no. NCT04492475). ⁵ Between 5 August 2020 and 11 November 2020, a total of 969 patients from 63 hospitals across five countries entered the study and received remdesivir; 487 of them received interferon beta-1a (a 44-μg daily dose for up to 4 days) and 482 received placebo. Each patient was assessed daily on the 8-point ordinal scale during their hospital stay, from day 1 to day 29, as well as on days 15, 22, and 29 after discharge.

We obtained the ACTT-1, ACTT-2, and ACTT-3 data from AccessClinicalData@NIAID (https://accessclinicaldata.niaid.nih.gov/).

Statistical analysis

We characterize the treatment effect on the clinical status at a particular day by the odds ratio of lower severity (i.e. the odds of falling into or below a severity category versus falling above it for the treatment arm divided by that of the control arm) under the proportional odds model. We assume that the odds ratio of lower severity is constant over time and estimate or test this common odds ratio while accounting for the correlations of the repeated measures. Although the odds ratio may not be constant over time when treatment is effective, this formulation yields a nonparametric test of the null hypothesis of no treatment effect on the clinical status at any time and an estimator of the overall treatment effect on the trajectory of clinical status. To investigate time-varying treatment effects, we estimate a piecewise linear function for the log odds ratio, with change points placed at every week.

We define the clinical events by various levels of improvement or deterioration of clinical status over time relative to the status at enrollment. Suppose that the ordinal score at enrollment is 4, 5, or 6. Then, there are nine types of events: improvement by one, two, three, four, or five categories and deterioration by one, two, three, or four categories. Each patient can potentially experience seven events, and the types of events depend on the initial clinical status. For example, a patient initially placed in category 5 can potentially improve by one, two, three, or four categories and deteriorate by one, two, or three categories.

We formulate the treatment effects on the five levels of improvement and the four levels of deterioration through nine Cox proportional hazards models. We estimate a common hazard ratio for the five levels of improvement and a common hazard ratio for the four levels of deterioration. Although the hazard ratios may not be the same when the treatment is effective, this framework provides a valid test of the null hypothesis of no treatment effect on any level of improvement or any level of deterioration. We also summarize the treatment benefit by combining the log hazard ratio for improvement with the negative log hazard ratio for deterioration. This strategy yields a single probability statement (one confidence interval, with one p-value) about treatment efficacy in accelerating improvement and preventing deterioration of health.

The above two methods are described in greater detail in the Supplemental Appendix.

Results

ACTT-1

The protocol-specified primary endpoint was time to recovery, and the second endpoints included 28-day mortality and clinical status at day 15. ³ The hazard ratio of recovery for remdesivir versus placebo was estimated at 1.30 (95% confidence interval (CI) = 1.13–1.50; p < 0.001). The hazard ratio of death was estimated at 0.73 (95% CI = 0.52–1.02), and the odds ratio of lower severity at day 15 was estimated at 1.47 (95% CI = 1.18–1.82, after adjustment for baseline disease severity).

Figure 3(a) shows the estimates for the odds ratios of lower severity from day 1 to day 28 under separate proportional odds models, together with a smooth estimate of the odds ratio function over time by the proposed method. The common odds ratio over days 1–28 was estimated at 1.48 (95% CI = 1.23-1.79, after adjustment for baseline disease severity; p < 0.001), which would enable one to make a single probability statement that remdesivir significantly reduced disease severity. The confidence interval for the common odds ratio is narrower than that of the odds ratio at day 15, and it pertains to clinical status over the entire study period rather than on 1 day.

Figure 3.

Odds ratios of lower severity for treatment versus control in ACTTs. The estimates for the daily odds ratios from separate proportional odds models are shown by the dots, and the corresponding 95% confidence intervals are shown by the ticks. The estimates from a proportional odds model under which the log odds ratio is a piecewise linear function of time, with change points placed at every week, are shown by the solid curve, and the corresponding 95% confidence intervals are shown by the shaded bands. (a) ACTT-1: remdesivir versus placebo, (b) ACTT-2: baricitinib plus remdesivir versus placebo plus remdesivir, and (c) ACTT-3: interferon beta-1a plus remdesivir versus placebo plus remdesivir.

Figure S1 shows the cumulative incidence of each level of improvement or deterioration of clinical status for remdesivir versus placebo, and Table 1 (first column) shows the regression results for the effects of remdesivir on various clinical events. Remdesivir accelerated each level of improvement, although there were greater uncertainties associated with the estimates for higher levels of improvement due to smaller numbers of events. The common hazard ratio of improvement was estimated at 1.18 (95% CI = 1.03–1.35). In addition, remdesivir reduces the risk of each level of deterioration, with a common hazard ratio of 0.62 (95% CI = 0.46–0.82). Finally, the common hazard ratio of accelerating improvement and preventing deterioration was estimated at 1.27 (95% CI = 1.09–1.47; p = 0.002), which would enable one to claim that remdesivir was highly beneficial. This finding is more comprehensive about treatment efficacy on clinical events than the original finding on time to recovery.

Table 1.

Treatment effects on times to occurrences of clinical events in the ACTT studies.

Endpoint	ACTT-1 Remdesivir vs Placebo			ACTT-2 Baricitinib + Remdesivir vs Placebo + Remdesivir			ACTT-3 Interferon beta-1a + Remdesivir vs Placebo + Remdesivir
	HR	95% CI	P	HR	95% CI	P	HR	95% CI	P
Recovery	1.30	(1.13, 1.50)	< 0.001	1.17	(1.03, 1.33)	0.017	0.97	(0.86, 1.09)	0.58
Death	0.73	(0.52, 1.02)	0.064	0.64	(0.38, 1.06)	0.084	1.31	(0.69, 2.51)	0.41
Combined	1.32	(1.11, 1.57)	0.002	1.24	(1.05, 1.47)	0.012	0.93	(0.78, 1.12)	0.45
Improvement by
1 category	1.16	(1.02, 1.32)	0.025	1.24	(1.10, 1.40)	0.001	0.98	(0.87, 1.10)	0.73
2 categories	1.21	(1.06, 1.38)	0.004	1.20	(1.06, 1.36)	0.003	1.00	(0.89, 1.12)	0.98
3 categories	1.17	(1.02, 1.35)	0.023	1.18	(1.03, 1.34)	0.014	0.96	(0.85, 1.08)	0.48
4 categories	1.17	(0.99, 1.39)	0.061	1.24	(1.05, 1.47)	0.010	0.99	(0.83, 1.19)	0.95
5 categories	1.03	(0.78, 1.36)	0.835	1.44	(1.02, 2.02)	0.036	1.04	(0.46, 2.32)	0.93
6 categories	1.50	(0.92, 2.44)	0.105	1.56	(0.62, 3.96)	0.347
any categories	1.18	(1.03, 1.35)	0.017	1.24	(1.09, 1.41)	0.001	0.98	(0.88, 1.10)	0.77
Deterioration by
1 category	0.75	(0.61, 0.92)	0.006	0.73	(0.59, 0.91)	0.005	1.10	(0.85, 1.41)	0.47
2 categories	0.58	(0.41, 0.81)	0.002	0.58	(0.36, 0.93)	0.023	0.95	(0.59, 1.55)	0.85
3 categories	0.35	(0.18, 0.66)	0.001	0.40	(0.14, 1.13)	0.083	1.09	(0.52, 2.30)	0.82
4 categories	0.82	(0.17, 4.00)	0.803
any categories	0.62	(0.46, 0.82)	0.001	0.63	(0.47, 0.85)	0.002	1.05	(0.76, 1.47)	0.76
Overall Benefit	1.27	(1.09, 1.47)	0.002	1.30	(1.13, 1.49)	< 0.001	0.98	(0.85, 1.12)	0.74

ACTT-2

ACTT-2 adopted the same primary and second endpoints as ACTT-1. ⁴ The hazard ratio of recovery for baricitinib plus remdesivir versus remdesivir alone was estimated at 1.17 (95% CI = 1.03–1.33; p = 0.017). Although this result was statistically significant, baricitinib reduced the median time to recovery by only 1 day. The hazard ratio of death was estimated at 0.64 (95% CI = 0.38–1.06), and the odds ratio of lower severity at day 15 was estimated at 1.30 (95% CI = 1.04–1.61, after adjustment for baseline disease severity).

Figure 3(b) shows the estimated odds ratios of lower severity from day 1 to day 28 under separate models, together with a smooth estimate of the odds ratio function over time by the proposed method. The common odds ratio over days 1–28 was estimated at 1.32 (95% CI = 1.10–1.57, after adjustment for baseline disease severity; p = 0.002), which would enable one to conclude that baricitinib significantly reduced disease severity. The confidence interval for the common odds ratio is tighter than that of the odds ratio at day 15. Indeed, the lower limit of the latter interval was barely above the null value of 1. More important, the common odds ratio applies to clinical status over the entire study period rather than a single day.

Figure S2 shows the cumulative incidence of each level of improvement or deterioration of clinical status for baricitinib plus remdesivir versus remdesivir alone, and Table 1 (second column) shows the results for the effects of baricitinib on various clinical events. Baricitinib accelerated each level of improvement, with a common hazard ratio of 1.24 (95% CI = 1.09–1.41). In addition, baricitinib reduced the risk of each level of deterioration, with a common hazard ratio of 0.63 (95% CI = 0.47–0.85). Finally, the common hazard ratio of accelerating improvement and preventing deterioration was estimated at 1.30 (95% CI = 1.13–1.49; p < 0.001), which would enable one to claim that baricitinib was highly beneficial. This conclusion is clinically more relevant than the original conclusion on recovery.

ACTT-3

ACTT-3 adopted the same endpoints as ACTT-1 and ACTT-2, although the treatment effect on clinical status was assessed at 14 days instead of 15 days. ⁵ The hazard ratio of recovery for interferon beta-1a plus remdesivir versus remdesivir alone was estimated at 0.97 (95% CI = 0.86–1.09; p = 0.58). The hazard ratio of death was estimated at 1.31 (95% CI = 0.69–1.09), and the odds ratio of lower severity at day 14 was estimated at 0.97 (95% CI = 0.77–1.22, after adjustment for baseline disease severity).

Figure 3(c) shows the estimated odds ratios of lower severity from day 1 to day 28 under separate models, together with a smooth estimate of the odds ratio function over time by the proposed method. The common odds ratio over days 1–28 was estimated at 0.95 (95% CI = 0.79–1.14, after adjustment for baseline disease severity; p = 0.56).

Figure S3 shows the cumulative incidence of each level of improvement or deterioration of clinical status for interferon beta-1a plus remdesivir versus remdesivir alone, and Table 1 (last column) shows the results for the effects of interferon beta-1a on various clinical events. There was no evidence that interferon beta-1a accelerated any level of improvement, and the common hazard ratio of improvement was 0.98 (95% CI = 0.88–1.10). There was no evidence that interferon beta-1a reduced the risk of any level of deterioration either, and the common hazard ratio of deterioration was 1.05 (95% CI = 0.76–1.47). Finally, the common hazard ratio of accelerating improvement and preventing deterioration was estimated at 0.98 (95% CI = 0.85–1.12; p = 0.74).

The proposed methods yielded a conclusion that interferon beta-1a plus remdesivir was not superior to remdesivir alone by considering all important aspects of treatment actions. In addition, they provided more comprehensive summarization of the treatment effects than the original endpoints.

Discussion

The currently used endpoints for phase 3 clinical trials of hospitalized COVID-19 patients—the clinical status at a particular day and the time to a specific change in clinical status—have important shortcomings. First, these endpoints capture only part of the clinical course of a patient. Second, the available data on clinical outcomes are not fully utilized. Third, the robustness (to different scenarios of treatment effects) and statistical power are limited.

We proposed to evaluate the totality of evidence on treatment efficacy by considering the entire trajectory of clinical status or all major changes in clinical status over time. The first method is preferred to the second if there are substantial fluctuations of severity rating over time, whereas the second method is particularly appealing if severity-rating curves are largely monotone. Both methods enable investigators to make a single probability statement (one confidence interval and one p-value) about the overall treatment benefit and to identify the sources of treatment differences.

The approach taken here extends that of Lin et al. ⁸ by allowing the intercepts in the proportional odds models to vary freely over categories and over time (rather than as a parametric function of categories and time) and by estimating a piecewise linear function for the log odds ratio over time (in addition to a constant odds ratio over time). With fully saturated intercepts, the test for the null hypothesis of no treatment effect on the clinical-status trajectory is purely nonparametric.

Lin et al. conducted simulation studies under a wide variety of scenarios. Their results showed that the totality of evidence approach tends to be more powerful than the use of a single endpoint when the treatment is beneficial across several clinical outcomes and is more robust to different scenarios of treatment actions than the use of time to recovery or 28-day mortality. ⁸

The ACTIV-1 inpatient trial aimed to evaluate the safety and efficacy of three immune modulators as an add-on therapy to remdesivir in hospitalized adults with moderate to severe COVID-19 disease. ⁷ Compared to placebo, participants receiving infliximab or abatacept displayed no statistically significant improvement in the primary endpoint of time to recovery (as measured by day of discharge from hospital), but improvements on both key secondary endpoints of mortality and clinical status at 28 days were observed. ⁷ The proposed methods yielded positive conclusions about the efficacy of infliximab and abatacept. The findings will be communicated in a separate report.

In ACTT-1 and ACTT-2, statistical significance was achieved on the primary endpoint of time to recovery, although there was no evidence for treatment efficacy on the more important endpoint of mortality. In ACTIV-1, statistical significance was achieved on the important endpoint of mortality, but success could not be claimed because mortality was a secondary endpoint. These studies demonstrate the lack of robustness of the current endpoints and the need to incorporate the entire clinical course into the primary endpoint.

The ACTIV-4 inpatient trial was designed to evaluate the safety and efficacy of using varying doses of heparin, a blood thinner, to prevent the formation of blood clots and improve outcomes in hospitalized COVID-19 patients. ⁶ The primary endpoint was chosen to be time from randomization to sustained recovery, defined as being discharged from the index hospitalization, followed by being alive and home for 14 consecutive days prior to day 90. ⁶ This endpoint incorporates mortality into the endpoint of recovery. However, the number of patients who die within 14 days of recovery is likely small, so sustained recovery is not fundamentally different from the original definition of recovery. In addition, it would be difficult to construct a proportional hazards model for this composite endpoint.

We can determine the power and sample size for a future clinical trial at the design stage by Monte Carlo simulation. Specifically, we generate each patient’s clinical-status trajectory by simulating the transitions among the eight categories over time according to the multi-state model shown in Figure 1, with transition probabilities dependent on the treatment arm. We choose the simulation parameters using data from recently completed trials and by specifying the targeted sizes of the treatment effects. ⁸ We then analyze each simulated data set using the software program that implements the proposed methods (https://dlin.web.unc.edu/software/covid/). Finally, we calculate the empirical probability of rejecting the null hypothesis of no treatment difference. For the method based on proportional odds models, we can also calculate power and sample size analytically. ⁹

For the method based on proportional hazards models, we have focused on different levels of improvement or deterioration from the baseline clinical status. We may focus instead on some important clinical milestones, such as recovery, hospital discharge, death, and receiving invasive mechanical ventilation or extracorporeal membrane oxygenation. We can formulate the treatment effect on each clinical milestone by a proportional hazards model and combine the evidence for the treatment effects on all the milestones of interest. Indeed, the proposed approach can be applied to any set of event times.

The results of combining the evidence on time to recovery and time to death in ACTT-1, ACTT-2, and ACTT-3 are displayed in the top panel of Table 1. One would conclude that remdesivir was beneficial and baricitinib plus remdesivir was superior to remdesivir alone.