Discovering Treatment Effectiveness via Median Treatment Effects— Applications to COVID-19 Clinical Trials

Abstract

Comparing median outcomes to gauge treatment effectiveness is widespread practice in clinical and other investigations. While common, such difference-in-median characterizations of effectiveness are but one way to summarize how outcome distributions compare. This paper explores properties of median treatment effects as indicators of treatment effectiveness. The paper's main focus is on decisionmaking based on median treatment effects and it proceeds by considering two paths a decisionmaker might follow. Along one, decisions are based on point-identified differences in medians alongside partially identified median differences; along the other decisions are based on point-identified differences in medians in conjunction with other point-identified parameters. On both paths familiar difference-in-median measures play some role yet in both the traditional standards are augmented with information that will often be relevant in assessing treatments' effectiveness. Implementing either approach is straightforward. In addition to its analytical results the paper considers several policy contexts in which such considerations arise. While the paper is framed by recently reported findings on treatments for COVID-19 and uses several such studies to explore empirically some properties of median-treatment-effect measures of effectiveness, its results should be broadly applicable.

We might be anywhere but are in one place only…

Derek Mahon, A Garage in Co. Cork

1. Introduction

In a study published online on March 18, 2020, comparing lopinavir–ritonavir treatment for COVID-19 with standard care, Cao et al., 2020, report:

Patients assigned to lopinavir–ritonavir did not have a time to clinical improvement different from that of patients assigned to standard care alone in the intention-to-treat population (median, 16 days vs. 16 days; hazard ratio for clinical improvement, 1.31; 95% confidence interval [CI], 0.95 to 1.85; P=0.09)…

On April 29, 2020, the U.S. National Institute on Allergy and Infectious Diseases (NIAID, 2020) announced in a news release summarizing a separate study:

Hospitalized patients with advanced COVID-19 and lung involvement who received remdesivir recovered faster than similar patients who received placebo.… Specifically, the median time to recovery was 11 days for patients treated with remdesivir compared with 15 days for those who received placebo.¹

On the basis of that study the U.S. Food and Drug Administration (FDA) two days later issued an Emergency Use Authorization for remdesivir (U.S. FDA, 2020), noting:

While there is limited information known about the safety and effectiveness of using remdesivir to treat people in the hospital with COVID-19, the investigational drug was shown in a clinical trial to shorten the time to recovery in some patients.

The preliminary report of this study, the Adaptive Covid-19 Treatment Trial or ACTT Study (Beigel et al., 2020a), was subsequently published online on May 22, 2020.

In a third study, published online May 8, 2020, Hung et al., 2020, report on a trial comparing combination therapy for COVID-19 with lopinavir–ritonavir alone. The authors note:

The combination group had a significantly shorter median time from start of study treatment to negative nasopharyngeal swab (7 days [IQR 5–11]) than the control group (12 days [8–15]; hazard ratio 4·37 [95% CI 1·86–10·24], p=0·0010).

Apart from their focus on treating COVID-19, a common feature of these three studies is that they summarize their respective primary endpoints as medians of each arm's time-to-event (TTE) distribution, and then judge treatments' efficacy or effectiveness by the difference between those medians.² Such median-based comparisons are common in clinical research, so common perhaps that they and the information they convey may be taken for granted.

Much is at stake on the manner in which the effectiveness of candidate COVID-19 treatments and vaccines is assessed, using clinical trial data or otherwise. Loading this burden onto two parameters—median outcomes under treatment and comparator—is a high-stakes gamble. Consider for instance figure 1 whose top panel depicts by the two indicated points the results reported in the NIAID news release (NIAID, 2020); until the publication of the Beigel study over three weeks later this was the only information about the study's effectiveness known by the public. (The bottom panel of figure 1 reproduces the results reported by Beigel.) Is this information alone sufficient for a decisionmaker to conclude that remdesivir is effective since it will "shorten the time to recovery in some patients"? Analogously does the finding that median outcomes do not differ in the Cao study mean that decisionmakers ought not prefer one treatment (see figure 2)? Or might in both instances regulators, clinicians, patients, and others wish to know more about the distributions of outcomes before arriving at such conclusions?

Figure 1 —

Time-to-Recovery Outcomes in the Beigel et al., 2020, ACTT Remdesivir Study: Outcomes reported in NIAID, 2020 (top); Reproduction of Beigel's figure 2A (bottom)

Figure 2 —

Time-to-Improvement Outcomes in the Cao et al., 2020, Lopinavir–Ritonavir Study (Reproduction of Cao's figure 2)

The perhaps-obvious points are that there are many ways subject-level outcome data can be summarized or aggregated over a population or sample³ and that these alternatives may not yield the same conclusion about treatments' effectiveness, nor should they. Understanding the nature, merits, and shortcomings of the methods chosen and how those methods stack up against alternatives should thus be instructive for COVID-19 decisionmaking and beyond.

Numerous comparisons of candidate COVID-19 vaccines and treatments are being contemplated, studied, and reported. It is thus timely to assess the properties and merits of treatment effect (TE) measures defined by distributions' medians—like those featuring in the studies described above—as summaries of treatments' effectiveness: In a nutshell, what might decisionmakers learn about treatments' effectiveness by comparing two distributions' medians?⁴

The paper argues that a reasonable answer to this question is: "Probably something but often little." To learn more about treatments' effectiveness the paper suggests two paths decisionmakers might follow. Along path 1 decisions are based on point-identified differences in median outcomes—as is customary—in conjunction with medians of the distribution of differences in outcomes, i.e. the treatment effect distribution (section 2). The latter medians are typically partially identified since the distribution of differences may not be observable.⁵ Along path 2 decisions are again based on point-identified differences in medians in conjunction with additional point-identified parameters whose nature will be discussed in section 5. On either path it is proposed that the basis of decisions may be enhanced by considering features of distributions in addition to differences in median outcomes. Importantly, in both cases conventional differences in medians play a central role so that decisionmakers accustomed to relying on such metrics should find nothing particularly foreign in the proposed strategies.

Given the urgency of discovering effective treatments and vaccines for COVID-19 it is timely to consider measures of effectiveness that align with what is important to decisionmakers and those affected by their decisions. What is proposed here may be helpful to this end, particularly since the paper strives to offer intuitive and easily implemented strategies.

Section 2 offers definitions and other preliminaries. Section 3 discusses basic elements of decisions based on median treatment effects. Sections 4 and 5 present the key issues and results for paths 1 and 2. Section 6 illustrates the applicability of these ideas using three COVID-19 clinical studies. Section 7 summarizes. While the discussion throughout is motivated and framed by attempts to discover the effectiveness of COVID-19 treatments, the issues addressed should be broadly applicable in contexts where considerations of median treatment effects arise.⁶

2. Definitions and Other Preliminaries

Consider two subject-specific outcomes, y₀ and y₁ , which might be health status under treatment 0 ( T₀) and treatment 1 (T₁) and which may or may not—depending on particulars— be potential outcomes where only one of y₀ or y₁ is observed. Suppose throughout that each y_j is non-negative and has either continuous or discrete (integer) measurement. Whether or not both y_j are observed for each subject, the subject-level treatment effect is y₁ – y₀ .⁷ Unless noted otherwise smaller values of y will correspond to better health outcomes.

Let Pr(…) denote a probability mass or density and F(…) its corresponding cumulative. Define F_j(y) = Pr(y_j ≤ y) for j∈{0, 1}.⁸ F_j(y) will sometimes be abbreviated as F_j. Each y_j has support Y_j , which may be continuous or discrete. Y = Y₀ ∪ Y₁ is the common support of y₀ and y₁; often Y₀ = Y₁ . Define the α–quantile of F_j (y) as⁹

$${\mathrm{q}}_{{}_{\mathrm{j}}}\left(\alpha \right)={\text{inf}}_{\mathrm{y}\in \mathrm{Y}}\{\mathrm{y}\mid {\mathrm{F}}_{\mathrm{j}}\left(\mathrm{y}\right)\ge \alpha \}\phantom{\}}\text{for}\alpha \in \left(0,1\right).$$ (1)

m_j is shorthand used henceforth for q_j (.5) .¹⁰ Define the interquartile range (IQR) of each F_j as

$${\text{IQR}}_{{}_{\mathrm{j}}}=\left\{{\mathrm{q}}_{{}_{\mathrm{j}}}\left(.25\right),{\mathrm{q}}_{{}_{\mathrm{j}}}\left(.75\right)\right\}.$$ (2)

Note that IQR_j is defined as a two-element set not as an interval as perhaps more conventional.

It is assumed at this point that the F_j are point identified over relevant subsets of Y; requiring identification to be over all y ∈ Y or only over subsets will be considered in context.¹¹ Whether identification results from a randomized trial or some other method is unimportant; indeed each F_j can be learned from a different data source.

Define the difference-in-y-probabilities treatment effect as

$$\Delta \mathrm{F}\left(\mathrm{y}\right)={\mathrm{F}}_{{}_1}\left(\mathrm{y}\right)-{\mathrm{F}}_{{}_0}\left(\mathrm{y}\right),$$ (3)

a familiar endpoint in TTE and other studies (e.g. difference in 12-month survival probabilities). Define the difference-in- α -quantiles treatment effect as

$$\Delta \mathrm{q}\left(\alpha \right)={\mathrm{q}}_{{}_1}\left(\alpha \right)-{\mathrm{q}}_{{}_0}\left(\alpha \right).$$ (4)

A special case of (4) is the difference-in-medians treatment effect

$$\Delta \mathrm{m}={\mathrm{m}}_{{}_1}-{\mathrm{m}}_{{}_0}=\Delta \mathrm{q}\left(.5\right).$$ (5)

Δm is a prominent measure used to compare treatments' effectiveness in clinical studies, particularly albeit not exclusively for TTE outcomes. ¹² Δm is point identified under the assumption that both F_j(y) are point identified at least for {y∣F_j(y)≤.5}.

The treatment effect distribution is F(y₁ –y₀) , which is derived from the joint distribution Pr (y₀,y₁). In potential-outcomes contexts Pr(y₀,y₁) is not point identified so that F(y₁ –y₀) is consequently not point identified.¹³ When F(y₁ –y₀) is not point identified, that it can generally be partially identified is an important consideration in what follows.

Quantiles of the treatment effect distribution are denoted

$$\mathrm{q}\left(\alpha \right)\Delta =\text{min}\phantom{\{}\{{\mathrm{y}}_{{}_1}-{\mathrm{y}}_{{}_0}\mid \mathrm{F}\left({\mathrm{y}}_{{}_1}-{\mathrm{y}}_{{}_0}\right)\ge \alpha \}.$$ (6)

with the median-difference treatment effect given by¹⁴

$$\mathrm{m}\Delta =\mathrm{q}\left(.5\right)\Delta =\text{med}\left(\mathrm{F}\left({\mathrm{y}}_{{}_1}-{\mathrm{y}}_{{}_0}\right)\right)=\text{min}\phantom{\{}\{{\mathrm{y}}_{{}_1}-{\mathrm{y}}_{{}_0}\mid \mathrm{F}\left({\mathrm{y}}_{{}_1}-{\mathrm{y}}_{{}_0}\right)\ge .5\}.$$ (7)

This paper focuses mostly on Δm and mΔ rather than general quantile treatment effects, this owing to the prominence¹⁵ of medians in empirical clinical and health research; most results generalize readily to other quantiles. Manski (2007, chapter 7) notes the inequality

$$\mathrm{m}\Delta \ne \Delta \mathrm{m},$$ (8)

and proceeds to suggest that "a researcher reporting a median treatment effect must be careful to state whether the object of interest is the left- or right-hand side [of (8)]."

As will be seen in section 4 understanding and computing bounds on mΔ is facilitated by reference to the inequality probabilities (IPs) Pr(y₁ > y₀) . Like mΔ IPs are generally not point identified but are partially identified using information on the F_j. Define

$${\mathrm{D}}_{{}_{\text{jk}}}={\max }_{{}_{\mathrm{y}\in \mathrm{Y}}}\left\{{\mathrm{F}}_{{}_{\mathrm{j}}}\left(\mathrm{y}\right)-{\mathrm{F}}_{{}_{\mathrm{k}}}\left(\mathrm{y}\right),0\right\},\text{for}\text{j,k}\in \left\{0,1\right\}.$$ (9)

The D_jk are the largest vertical differences between F_j(y) and F_k(y) over the common support Y.¹⁶ These measures, known commonly as Kolmogorov's D-statistics, are depicted in figure 3. Note that at most one of the D_jk can exceed .5, a result that will prove useful later.

Figure 3 —

Defining D₀₁ and D₁₀

3. How Δm May Inform Decisions

Imagine that a decisionmaker selects treatment T_p ∈ {T₀,T₁} where p is determined by

$$\mathrm{p}=1\left(\mathrm{g}\left(\Delta \mathrm{m},\psi \right)>\mathrm{h}\right),$$ (10)

wherein ψ are features of Pr (y₀,y₁) beyond Δm that may matter to the decisionmaker. For instance with smaller values of y representing better health conventional null-hypothesis-test-based decisionmaking that gives deference to status quo (say T₀) over innovation (say T₁) might set g$\left(\Delta \mathrm{m},\psi \right)=-\Delta \mathrm{m}\text{and}\mathrm{h}=-{\mathrm{z}}_{{}_{\alpha ∕2}}\times {\widehat{\sigma }}_{{}_{\Delta \mathrm{m}}}$. Decisions that treat T₀ and T₁ symmetrically (see Manski and Tetenov, 2020) might set g (Δm, ψ) = −Δm and h=0. In either case ψ is ignored and Δm is the only feature of Pr(y₀,y₁) that influences choice.

Of course there are many standards by which two outcome distributions might be compared: moments, quantiles, dominance, etc. On what principles might Δm be advocated as the sole basis of decisions? While Δm may have reasonable sampling properties this is a different matter from it possessing well grounded conceptual properties. For instance comparison of the medians (or other quantiles) of observed marginal distributions are uninformative about the distribution of gains and losses—i.e. F(y₁ –y₀)—that owe to a change from T₀ to T₁.

Imbens and Wooldridge, 2009, raise two issues to support their claim that Δm may be of greater interest than mΔ or other features of F(y₁ –y₀). First, they assert that it is natural for decisionmakers to compare policies via differences between outcome distributions, which "can often be summarized by differences in the quantiles." Second, they note that F(y₁ –y₀) and associated parameters like mΔ typically cannot be point identified. Each claim has merit.

Yet two counterclaims might be advanced to support consideration of mΔ. The first is statistical: While mΔ cannot generally be point identified it can generally be partially identified informatively, and in a subset of such cases can be sign identified (section 4 and appendix C). The second is conceptual: While decisionmakers may assess treatments' effectiveness by comparison of resulting marginal outcome distributions, they may also care about policies' distributional consequences, e.g. the fractions of the population that benefit or suffer from policy change and how much they benefit or suffer. The latter concerns are not informed by examination of the marginal distributions but require consideration of the joint distribution Pr(y₀,y₁) or, specifically, F(y₁–y₀) . Heterogeneous response to policy in direction and magnitude may be important considerations: As articulated nicely by Koenker and Bilias, 2000, "treatment may make otherwise weak subjects especially robust, and turn the strong to jello."

Decisionmakers may or may not feel that the basis of their decisions is enhanced by information on F(y₁–y₀) or mΔ . Either way a revealed preference argument suggests that Δm measures of treatments' effectiveness provide them with at least some useful information about their choices (see footnote 15). Ease of computation, applicability with right-censored data, and parsimony in summarizing data are three plausible reasons for such popularity. The following two sections discuss decisionmaking that does (path 1; section 4) and does not (path 2; section 5) admit roles for partially identified parameters like mΔ working in conjunction with Δm . In both cases it is proposed that the basis of decisions may be enhanced by considering parameters beyond Δm . To implement such strategies analysts need appeal only to information contained in F₀ and F₁ ; how that information is used differs along the two paths.¹⁷

Finally, while the analytical results reported in sections 4 and 5 may be of interest per se, considering how they may be applicable in real-world decision and policy settings is appropriate. Rather than disrupting the flow of the paper here with lengthy discussion, appendix E describes several policy contexts wherein considerations of median or more general quantile TEs are, or at least arguably should be, prominent.¹⁸ To anticipate that discussion it is worth noting one example of regulatory language that naturally motivates such considerations. The regulatory language governing FDA's determination of biological products' effectiveness is:

Effectiveness means a reasonable expectation that, in a significant proportion of the target population, the pharmacological or other effect of the biological product, when used under adequate directions, for use and warnings against unsafe use, will serve a clinically significant function in the diagnosis, cure, mitigation, treatment, or prevention of disease in man. [21 CFR 601.25(d)(2)] (emphasis added)

Note that this standard entails considerations of both population quantiles ("significant proportion") and treatment effect magnitudes ("clinically significant function"). Appendix E suggests statistical formalization of this language and suggests further how the discussion in sections 4 and 5 may usefully address such regulatory questions.

4. Path 1: Assessing Treatments' Effectiveness via Δm and Partially Identified mΔ

The essence of decision problems in this context is captured by a simple picture. Figure 4 depicts some population's discrete Pr (y₀,y₁) that puts probability mass 1/3 on three (y₀,y₁) values, {(3,8), (15,11), (17,21)}. Imagine Pr(y₀,y₁) is known. There is nothing particularly peculiar about the depicted probability structure (Pearson correlation .77, rank correlation 1.0). Yet Δm = −4 < 0 and mΔ = 4 > 0. A decisionmaker knowing nothing more than the pictured information must select T₀ or T₁. Which should be chosen? Which would you choose?

Figure 4.

Defining Δm and mΔ with Known Pr(y₀,y₁)

In general one would appeal to the decisionmaker's utility or loss function to answer this question. But this is not the main issue here: Important here is whether the assumed knowledge of mΔ might influence even to some degree a decisionmaker's attitudes about the relative merits of T₀ and T₁ given that Δm is presumed known. Would such knowledge be even partially influential then the obstacle to decisionmaking based on mΔ in settings where Pr(y₀,y₁) is not known is mΔ's point identification not, as Imbens and Wooldridge hint, irrelevance of mΔ per se. In these cases partial identification may support such decisionmaking.

Partial Identification of and Bounds on mΔ

Since mΔ is generally partially but not point identified, a decisionmaker whose choice would depend to at least some degree on knowing it must accept that knowing a range of possible values is the best to be hoped for. Whether the knowable range of its possible values suffices for decisionmaking is context dependent.

A parameter θ is partially identified when it is known to reside in the closed, half-open, or open bounds interval b(θ) . Define the bounds set B(θ) = {L(θ),U(θ)} where L(θ) = inf (b(θ)) and U(θ) = sup(b(θ)).¹⁹ L(θ) and U(θ) are valid bounds since θ ∈ b(θ) . The bounds B(θ) are sharp when L(θ) and U(θ) are the largest and smallest values consistent with knowledge of θ that are revealed by the data. These tightest bounds and their corresponding b(θ) are denoted B*(θ) = {L*(θ),U*(θ)} and b*(θ) . θ is sign identified if b*(θ) excludes zero.^20,21 In passing it may be of interest to note that B*(mΔ) = {−6,6} for the data shown in figure 4, with B*(mΔ) computed here using the permutation method described in appendix C.

Bounds with Partially Observed Outcome Data

Because they are relevant in the case studies appearing in section 6, a few considerations involved in computing bounds on mΔ under two forms of partially observed outcomes—right-censored data and IQR data—are sketched here and discussed more fully in appendix C.

Right-censoring of outcome data is common in TTE studies. Given noninformative right-censoring at y_c (outcomes unobserved if they exceed y_c ) three scenarios can be considered:

1. both F₀(y_c) and F₁(y_c) exceed .5 (both m₀ and m₁ are point identified)

2. only one of F₀(y_c) and F₁(y_c) exceeds .5 (one of m₀ or m₁ is point identified,)

3. Neither F₀(y_c) nor F₁(y_c) exceeds .5 (neither m₀ nor m₁ is point identified)

These cases, depicted in the top panel of figure 5, have different implications for the identification of Δm and mΔ:

Figure 5 —

Identifying Δm and mΔ with Right-Censored Outcomes: Three right-censoring scenarios (top); Degree of right censoring affects mΔ bounds (bottom)

1. Δm is point identified; mΔ is partially identified with finite bounds

2. both Δm and mΔ are partially identified with one finite bound

3. neither Δm nor mΔ is informatively bounded (no finite bounds)

None of these results is surprising. Yet even under scenario (a) it is noteworthy that while the particular right-censoring threshold y_c is of no consequence for point identifying Δm —i.e. m₀ , m₁ , and Δm are invariant with respect to any y_c that exceeds both m₀ and m₁ —this is not so for computation of mΔ bounds. The extent of right-censoring affects the amount of information used to compute the mΔ bounds. Reducing right-censoring—i.e. increasing y_c — can never widen mΔ bounds but may narrow them. See appendix C for discussion.²²

The second form of partially observed data of interest here is interquartile-range data. IQRs are often reported alongside medians, sometimes in settings where data are also right censored. For instance IQRs are reported for the primary endpoints in the Cao study that reported entire data (their figure 2 and table 3) as well as in the Hung study that reported the two arms' quartiles only (their table 2; figure 6 here). Indeed it is not unusual to find the only outcome information reported to be that on each outcome's three sample quartiles.

Table 2 —

Empirical Results, Cao et al., 2020, Lopinavir–Ritonavir Trial (Output from medte Stata Program)

Median Treatment Effects and Related Parameters
Outcome variable: (integer-valued) tti
Group variable (g): tx
Uncensored Obs. for g=0:	70
Uncensored Obs. for g=1:	77
Right-censored Obs. for g=0:	30
Right-censored Obs. for g=1:	22
Median Treatment Effects
Sample median for g=0 (m0):	16
Sample median for g=1 (m1):	16
Diff. in medians m1-m0:	0
Lower bound on med(F(y1-y0)):	−9
Upper bound on med(F(y1-y0)):	7
Central Aperture Measures
a1(m*,.5):	0.0000
a2(.5):	0.0000
Bounds on Inequality Probabilities
Lower bound on Prob(y1>y0):	0.0000
Upper bound on Prob(y1>y0):	0.8259

Figure 6 —

Time-to-Negative-Test Outcomes in Hung et al., 2020, Combination-Treatment Study (Quartiles and IQR reported in Hung's table 2)

It turns out that mΔ can be bounded informatively using only the IQR_j (2). When the y_j are continuously distributed and the F_j everywhere-increasing the IQR-based bounds are

$${\mathrm{B}}_{{}_{\text{IQR}}}\left(\mathrm{m}\Delta \right)=\left\{\left({\mathrm{q}}_{{}_1}\left(.25\right)-{\mathrm{q}}_{{}_0}\left(.75\right)\right),\left({\mathrm{q}}_{{}_1}\left(.75\right)-{\mathrm{q}}_{{}_0}\left(.25\right)\right)\right\},$$ (11)

where the q_j(.25) and q_j(.75) are the observed data. When outcomes are measured as integers the corresponding IQR-based bounds are (see appendix C for details):

$${\mathrm{B}}_{{}_{\text{IQR}}}\left(\mathrm{m}\Delta \right)=\left\{\left({\mathrm{q}}_{{}_1}\left(.25\right)-{\mathrm{q}}_{{}_0}\left(.75\right)-1\right),\left({\mathrm{q}}_{{}_1}\left(.75\right)-{\mathrm{q}}_{{}_0}\left(.25\right)+1\right)\right\}.$$ (12)

Relationships between Δm and B* (mΔ)

While Δm and mΔ describe different quantities, understanding relationships between them may enhance understanding of treatment effectiveness when knowledge of Δm alone inadequately informs choice. If so, whether knowing B*(mΔ) instead of mΔ itself will satisfy a decisionmaker will depend on context.

This subsection presents results on relationships between Δm and B*(mΔ) . Knowledge of Δm turns out to be to some degree informative about B*(mΔ) and vice versa; since both Δm and mΔ derive from Pr(y₀,y₁) this is unsurprising. Appendix D explains these results.

Result 1 (R1): L*(mΔ) ≤ Δm ≤ U*(mΔ)

Result 2 (R2): Suppose Δm < 0 . Then:

1. 0 ≤ D₀₁ ≤ .5

2. 0 < D₁₀ ≤ 1

3. L*(mΔ) ≤ 0

R2(c) means that knowing sign (Δm) suffices to sign L*(mΔ) when Δm < 0 . An interpretation is that the data cannot reject sign(mΔ) = sign (Δm). Alternatively, if mΔ is sign identified—with Δm < 0 this means U*(mΔ) < 0 —then its sign must be the same as ΔM's , i.e. mΔ and Δm cannot be sign identified in opposite directions.

Result 3 (R3): Suppose D₁₀ > .5. Then:

1. Pr(y₁ ≤ y₀)> .5

2. Δm < 0

3. mΔ < 0

Finding D_jk > .5 is powerful, sufficing to sign identify mΔ and offer a strong statement about Pr(y_j < y_k).²³ Indeed, mΔ is sign identified if and only if one of the D_jk exceeds .5.

Figure 7 integrates these ideas. Shown are two different joint probabilities. Pr_(a)(y₀,y₁) , where y₀ and y₁ are proximate to each other, has Δm = 2 , mΔ = 2 , B*(mΔ) = {−1,4} , D₀₁ = .4 , and B*(Pr(y₁ > y₀)) = {.4,1}. In contrast Pr_(b)(y₀,y₁), where y₀ and y₁ are distant from each other, has Δm = 6 , mΔ = 6 , B*(mΔ) = {3,8} , D₀₁ = 1 (zero-order dominance; Castagnoli, 1984), and B*(Pr(y₁ > y₀)) = {1,1}. Joint distributions placing probability mass further northwest of the y₀ = y₁ locus yield stronger identifying information with larger median treatment effects; in such instances the marginals F₀ and F₁ are "farther apart" in an important sense explored further in section 5.

Figure 7 —

Comparing Implications of Two Different Joint Distributions

Informing Decisions with Δm and Partially Identified mΔ

Imagine a decisionmaker for whom both Δm and mΔ are decision-relevant parameters. Suppose ψ = mΔ in (10) so that treatment choice is T_p1 ∈ {T₀,T₁} , where

$$\begin{gathered} {\mathrm{p}}_{{}_1}=1\left(\mathrm{g}\left(\Delta \mathrm{m},\mathrm{m}\Delta \right)>0\right) \\ =1\left(\beta \Delta \mathrm{m}+\left(1-\beta \right)\mathrm{m}\Delta >0\right). \end{gathered}$$ (13)

β ∈ [0,1] is presumed known and reflects the relative importance to the decisionmaker of Δm and mΔ . Of course g(Δm, mΔ) = βΔm + (1 – β)mΔ is only partially identified, with

$${\mathrm{B}}^{{}^{\ast }}\left(\mathrm{g}\left(\Delta \mathrm{m},\mathrm{m}\Delta \right)\right)=\left\{\beta \Delta \mathrm{m}+\left(1-\beta \right){\mathrm{L}}^{{}^{\ast }}\left(\mathrm{m}\Delta \right),\beta \Delta \mathrm{m}+\left(1-\beta \right){\mathrm{U}}^{{}^{\ast }}\left(\mathrm{m}\Delta \right)\right\}.$$ (14)

Note from (13) that sign identification of g (Δm, mΔ) suffices for unambiguous decisionmaking. Consider three cases:

1. Suppose Δm = 0. Then for any β ∈ [0,1] g(Δm, mΔ) would be sign-identified only if mΔ is sign-identified. But from R1 mΔ is not sign identified when Δm = 0.

2. Suppose Δm ≠ 0 and mΔ is sign-identified. Then g(Δm, mΔ) is sign-identified for any β ∈ [0,1] since from R1 the signs of Δm and mΔ coincide when mΔ is sign-identified.

3. Suppose Δm ≠ 0 and mΔ is not sign-identified. Then g(Δm, mΔ) is sign-identified for some range of β values. E.g. suppose Δm < 0 and U*(mΔ) > 0 . Then g(Δm, mΔ) is sign identified if β∈(U*(mΔ)/(U*(mΔ)–Δm), 1].

Decisionmakers may be positioned to make defensible choices even when they know only (14) and not (13). If g (Δm, mΔ) is sign identified then the appropriate decision is unambiguous. If not then it is proper to admit reservations about whatever choice is made since its ex ante merits are uncertain; acting otherwise implies a certitude that lacks credibility (Manski, 2020a). Either way to the extent that mΔ is at least minimally important to a decisionmaker (e.g. as in (13)) its partial identification needn't hinder information on its bounds supporting or cautioning choices that must be made.^24,25

5. Path 2: Assessing Treatments' Effectiveness via Δm and Other Point-Identified Parameters

Recall from section 3 that Imbens and Wooldridge, 2009, argued that typical policy choices will appropriately consider quantiles of F₀ and F₁, writing: "choice should be governed by preferences of the policymaker over [ F₀ and F₁] (which can often be summarized by differences in the quantiles)." One might thus imagine a decision function wherein various quantiles hold different importance for a decisionmaker; with smaller y being better health,

$${\mathrm{p}}_{{}_{\omega ,\mathrm{J}}}=1\left({\sum }_{\alpha \in \mathrm{J}}-{\omega }_{{}_{\alpha }}\times \Delta \mathrm{q}\left(\alpha \right)>{\mathrm{h}}_{{}_{\omega ,\mathrm{J}}}\right),$$ (15)

where J is the set of quantiles of interest and the ω_α ≥ 0 are importance weights. Should all the Δq (α) in (15) be point identified then Imbens and Wooldridge's claim suggests that the value of P_ω,J will determine treatment choice T_pω,J ∈ {T₀,T₁} Basing choice on Δm alone is a specific version of (15).

Still in the spirit of the Imbens and Wooldridge suggestion of comparing marginal distributions, policy choices might alternatively depend on a set of probability treatment effects,

$${\mathrm{p}}_{{}_{\pi ,\mathrm{K}}}=1\left({\sum }_{\mathrm{k}\in \mathrm{K}}{\pi }_{{}_{\mathrm{k}}}\times \Delta \mathrm{F}\left({\mathrm{y}}_{{}_{\mathrm{k}}}\right)>{\mathrm{h}}_{{}_{\pi ,\mathrm{K}}}\right),$$ (16)

where K is the set of y-values of interest (e.g. 6-, 12-, and 24-month survival) and π_k ≥ 0 are importance weights. Like above p_π,K may determine treatment choice T_pπ,K ∈ {T₀,T₁}. Familiar decision criteria, e.g. differences in 12-month survival probabilities, are specific versions of (16).

Aperture and Weighted- Δm Measures of Treatment Effectiveness

Point-identified parameters beyond Δm may provide decisionmakers more-nuanced perspectives on treatment effectiveness than those offered by Δm alone. The search is for broadly applicable parameters that provide such perspectives while also being straightforward to implement in empirical studies. The perspective is that of a decisionmaker not prepared to entirely abandon Δm but willing to temper decisions by additional point-identified parameters. With reference to (10), consider

$$\begin{gathered} {\mathrm{p}}_{{}_2}=1\left(\mathrm{g}\left(\Delta \mathrm{m},\psi \right)>\mathrm{h}\right) \\ =1\left(\mathrm{g}\left(\Delta \mathrm{m},\delta \right)>\gamma \right) \\ =1\left(\Delta \mathrm{m}\times \delta >\gamma \right), \end{gathered}$$ (17)

where δ are various point-identified parameters to be described shortly and γ is some threshold of Δm×δ that is meaningful for a decisionmaker. Treatment choice is T_p2 ∈ {T₀,T₁}.²⁶

As used in the preceding paragraph "treatment effectiveness" is intended to describe a broad sense of the extent to which one treatment makes people better off than another. In the following discussion greater treatment effectiveness means some amalgam of larger α–quantile treatment effects Δq(α) over some α∈(0, 1) and larger y–probability treatment effects ΔF(y) over some y ∈ Y. In essence, greater treatment effectiveness means F₀ and F₁ are "farther apart" in some directions. The practical challenge is how to summarize this notion with a single parameter, i.e. how to define decision-informative yet simple and parsimonious representations of "farther apart" and "some direction."

Define the area between F₀ and F₁ over some interval y ∈ (r, s) as the local aperture of F₀ and F₁ and denote this area A(r, s) .²⁷ "Aperture" signifies that A(r, s) measures the "opening" between F₀ and F₁ over the interval (r,s). Intuitively, the larger is A(r,s) the more effective is T_j relative to T_k in the vicinity of (r,s) as F₀ and F₁ are locally "farther apart."²⁸

For any point (y,α) in the region bounded by F₀ and F₁ over y ∈(r,s) define two approximations to A(r, s) as

$$\begin{gathered} {\mathrm{a}}_{{}_1}\left(\mathrm{y},\alpha \right)=\left({\mathrm{F}}_{{}_0}\left(\mathrm{y}\right)-{\mathrm{F}}_{{}_1}\left(\mathrm{y}\right)\right)\times \left({\mathrm{F}}_{{}_1}^{{}^{-1}}\left(\alpha \right)-{\mathrm{F}}_{{}_0}^{{}^{-1}}\left(\alpha \right)\right), \\ =-\Delta \mathrm{F}\left(\mathrm{y}\right)\times \Delta \mathrm{q}\left(\alpha \right) \end{gathered}$$ (18)

and

$$\begin{gathered} {\mathrm{a}}_{{}_2}\left(\alpha \right)=.5\times \left({\mathrm{F}}_{{}_0}\left({\mathrm{F}}_{{}_1}^{{}^{-1}}\left(\alpha \right)\right)-{\mathrm{F}}_{{}_1}\left({\mathrm{F}}_{{}_0}^{{}^{-1}}\left(\alpha \right)\right)\right)\times \left({\mathrm{F}}_{{}_1}^{{}^{-1}}\left(\alpha \right)-{\mathrm{F}}_{{}_0}^{{}^{-1}}\left(\alpha \right)\right) \\ =.5\times \left({\mathrm{F}}_{{}_0}\left({\mathrm{F}}_{{}_1}^{{}^{-1}}\left(\alpha \right)\right)-{\mathrm{F}}_{{}_1}\left({\mathrm{F}}_{{}_0}^{{}^{-1}}\left(\alpha \right)\right)\right)\times \Delta \mathrm{q}\left(\alpha \right). \end{gathered}$$ (19)

Using the idea of aperture and its approximations to devise broadly applicable and point-identifiable measures of treatment effectiveness, it is natural to consider aperture at a "central" location in the data, or central aperture. Letting (r,s) = (min{m₀, m₁}, max{m₀, m₁}) = (m_min, m_max), define central aperture as A(m_min,m_max). From (18) and (19) and defining²⁹ m* = .5(m₀ + m₁) it follows that two easily computed approximations to central aperture are³⁰:

$$\begin{gathered} {\mathrm{a}}_{{}_1}\left({\mathrm{m}}^{\ast },.5\right)=-\Delta \mathrm{F}\left({\mathrm{m}}^{\ast }\right)\times \Delta \mathrm{m} \\ ={\delta }_{{}_1}\times \Delta \mathrm{m} \end{gathered}$$ (20)

and

$$\begin{gathered} {\mathrm{a}}_{{}_2}\left(.5\right)=.5\times \left({\mathrm{F}}_{{}_0}\left({\mathrm{F}}_{{}_1}^{{}^{-1}}\left(.5\right)\right)-{\mathrm{F}}_{{}_1}\left({\mathrm{F}}_{{}_0}^{{}^{-1}}\left(.5\right)\right)\right)\times \Delta \mathrm{m} \\ =.5\times {\delta }_{{}_2}\times \Delta \mathrm{m} \end{gathered}$$ (21)

a₁(m*,.5) and a₂(.5) are necessarily non-negative since for any y∈(m_min,m_max) has the same sign as the terms multiplying it in (20) and (21).³¹ For practical purposes it is useful to re-define a₁(m*,.5) and a₂(.5) so their signs indicate the Δm -direction of treatment effectiveness, i.e.

$${\mathrm{a}}_{{}_1}\left({\mathrm{m}}^{\ast },.5\right)=\text{sign}\left(\Delta \mathrm{m}\right)\times \left({\delta }_{{}_1}\times \Delta \mathrm{m}\right)$$ (22)

and

$${\mathrm{a}}_{{}_2}\left(.5\right)=\text{sign}\left(\Delta \mathrm{m}\right)\times \left(.5\times {\delta }_{{}_2}\times \Delta \mathrm{m}\right),$$ (23)

with A (m_min, m_max) analogously signed. These revised definitions are used henceforth.

Figure 8 depicts a₁(m*,.5) and a₂ (.5) when y₀ and y₁ are imagined to be gamma-distributed with m_Q = 4 and m₁ = 2 . In the top panel −a₁(m*,.5) = .70 is the area of the shaded rectangle while in the bottom panel −a₂(.5) = .64 is the area of the shaded trapezoid.³²

Figure 8 —

Approximate Central Aperture, m₀ = 4 , m₁ = 2 , y_j ~ gamma (γ_j, δ_j) with γ₀ = 4 and γ₁ = 1 : a₁(m*,.5) = −.70 (top); a₂(.5) = −.64 (bottom) (Note: (m_j, γ_j) fix δ_j)

Noteworthy from (22) and (23) is that a₁(m*,.5) and a₂(.5) respect but augment Δm. Both are intuitive, easily computed indicators of treatment effectiveness,³³ measures that provide broader characterizations of effectiveness than does Δm on its own. That is a₁(m*,.5) and a₂(.5) describe more comprehensively than does Δm the divergence of F₀ and F₁ in the "middle" of the data. In an important sense a₁(m*,.5)and a₂(.5) combine the motivations underlying (15) and (16); for instance

$${\mathrm{a}}_{{}_1}\left({\mathrm{m}}^{\ast },.5\right)=\text{sign}\left(\Delta \mathrm{m}\right)\times \left(-\Delta \mathrm{q}\left(.5\right)\times \Delta \mathrm{F}\left({\mathrm{m}}^{\ast }\right)\right).$$ (24)

The sense in which central aperture and its approximations are indicators of treatment effectiveness may be appreciated with reference to figure 9 where m₀ = 4 and m₁ = 2 in both panels wherein F₀ is the same but F₁ differs (each F_j is gamma distributed). In the top panel δ₁ = −.08 and (m*,.5) = −.16 while in the bottom panel δ₁ = −.18 and a₁(m*,.5) = −.35 . This difference in approximate central aperture conforms to a notion that the effectiveness of T₁ relative to T₀ is greater in the scenario depicted in the figure's bottom panel even though Δm = −2 in both cases. If a decisionmaker must choose among the treatment options whose outcomes are pictured in figure 9, which one do they select? Might knowledge of the respective a₁(m*,.5) influence or support their choice?³⁴

Figure 9 —

a₁(m*,.5), with m₀ = 4 , m₁ = 2 , y_j ~ gamma (γ_j,δ_j) (Note: (m_j,γ_j) fix δ_j): γ = .3 , γ₁ = .2 , a₁(m*,.5) = −.16 (top); γ₀ = .3 , γ₁ = .9 , a₁(m*,.5) = −.35) (bottom)

One might also consider tail aperture or quartile aperture to assess treatment effectiveness, akin to how one might consider comparisons of IQRs, e.g. A(q_j(α),q_k(α)) for α = .25 , α = .75 , etc., implemented using a_j(…) approximations. A vector of such measures might complement the Imbens and Wooldridge, 2009, approach described in (15) and (16), e.g. ${\mathrm{p}}_{{}_{\xi ,\mathrm{J}}}=1\left({\sum }_{\alpha \in \mathrm{J}}{\xi }_{{}_{\alpha }}\times \mathrm{A}\left({\mathrm{q}}_{{}_{\mathrm{j}}}\left(\alpha \right),{\mathrm{q}}_{{}_{\mathrm{k}}}\left(\alpha \right)\right)>{\mathrm{h}}_{{}_{\xi ,\mathrm{J}}}\right)$. Such indicators can be instructive regarding treatment effectiveness if m₀ or m₁ would not be identified due to right censoring (e.g. consider time to disease onset in COVID-19 vaccine trials; see for instance figure 3 in Polack et al., 2020) or when the information conveyed to a decisionmaker by a₁(m*,.5) or a₂(.5) is ambiguous (as with the Cao et al., 2020, data where A(m_j,m_t) = a₁(m*,.5) = a₂(.5) = Δm = 0 (see figure 2)).

Inspection of (22) and (23) suggests that a₁(m*,.5) and a₂(.5) can be interpreted as weighted Δm . One might thus consider from (17) a class of weighted Δm , Δm ×δ , where δ∈[0,1] are point-identifiable weights describing decision-relevant aspects of the divergence of F₀ and F₁ . Conventional analysis implicitly sets δ = 1 but there is no reason to believe δ = 1 best informs a decisionmakers' choices. Such weighted Δm parameters build on Δm but downweight it the smaller is δ . Reasonable candidates for δ include δ₁ and .5 × δ₂ as in (22) and (23) as well as other parameters characterizing vertical distances between F₀ and F₁ , e.g. the D_jk .³⁵

Tentative Conclusions

As noted above the signs of central aperture and its approximations are given once the sign of Δm is known. As such a decisionmaker whose choice relies only on the sign of Δm revealed in the data needn't bother with aperture measures. But decisionmakers whose choices hinge on treatment effectiveness magnitudes may well care about the specific nature of such magnitudes—in particular about whether more than merely Δm matters—and may thus appreciate that some weighted Δm measure usefully informs their choices. Three prominent examples where treatment effectiveness magnitudes ( γ in (17)) matter fundamentally in decisionmaking are in determining if there are clinically significant differences between treatments' outcomes, in noninferiority trials, and in defining incremental cost-effectiveness ratios Δc/Δe wherein the magnitude of the Δe denominator figures prominently. These are settings where the treatments typically do not stand ex ante in equipoise but rather where one is status quo or standard practice, the other an innovation (see Manski and Tetenov, 2020).

The preceding is not advocating a particular aperture measure or weighted Δm as a decision standard. It is instead an appeal for future research to assess the merits of easily implementable and point-identified criteria like a₁(m*,.5) and a₂(.5) for decisionmaking using point-identified parameters, the premise being that decisions should be informed more comprehensively than by appealing to Δm alone. Consideration of central aperture might be a useful starting point for such exploration but is unlikely to be its destination. Moving beyond the intuition that parameters akin to a₁(m*,.5) and a₂(.5) describe treatment effectiveness to a formal assessment of welfare differences they may describe might also be a valuable step.

Determining what values of a₁(m*,.5) or a₂(.5) —or indeed any other parameters advanced in such discussions—correspond to clinically significant or economically meaningful differences between F₀ and F₁ will also require novel consideration. (For instance if Δ_CS is considered a clinically significant Δm in a particular treatment context then might a magnitude of a₁(m*,.5) or a₂(.5) exceeding, say, .25×Δ_CS suggest clinical significance? Or .1×Δ_CS ? Or … ?) While ultimately essential, such pragmatic considerations ought not postpone exploring the merits of parameters that may better inform decisionmakers about treatment effectiveness than do those conventionally used, particularly when, like here, such novel parameters are straightforward to implement in practice.

6. COVID-19: Three Case Studies

Case Study 1: Remdesivir for COVID-19 Treatment (Beigel et al., 2020a)

The Beigel study reports results of a randomized, double-blind, placebo-controlled trial of intravenous remdesivir in adults hospitalized with COVID-19. Its primary outcome is time to recovery (TTR) measured in days, with recovery determined by a patient meeting a prespecified threshold on an ordinal scale. The intention-to-treat sample analyzed here³⁶ consists of 538 and 521 observations on treated and control patients, respectively, of which 147 and 181 observations are right-censored after 28 days. The bottom panel of figure 1 depicts these data.

Table 1 summarizes the results. As noted earlier Δm = −4, favoring treatment over control, while B*(mΔ) = {−15,10} . The bounds interval is fairly wide as can be expected from the wide IP bounds. While the Δm result might lead a decisionmaker to favor treatment over control, wide bounds on mΔ might encourage that same decisionmaker to be conservative in advancing such a recommendation should considerations beyond Δm be relevant. Using earlier arguments, however, note that p_β is sign identified (negative) for β∈[5/7,1] , so that a decisionmaker who appeals to (13) and who weighs mΔ no more than 2/7 is positioned to recommend treatment over control. Finally, for the Beigel data the central aperture measures are A(m₁,m₀) = −.394 , a₁(m*,.5) = −.386 , and a₂(.5) = −.375 .

Table 1 —

Empirical Results, Beigel et al., 2020a, Remdesivir Trial (Output from medte Stata Program)

Median Treatment Effects and Related Parameters
Outcome variable: (integer-valued) ttr
Group variable (g): tx
Uncensored Obs. for g=0:	340
Uncensored Obs. for g=1:	391
Right-censored Obs. for g=0:	181
Right-censored Obs. for g=1:	147
Median Treatment Effects
Sample median for g=0 (m0):	15
Sample median for g=1 (m1):	11
Diff. in medians m1-m0:	−4
Lower bound on med(F(y1-y0)):	−15
Upper bound on med(F(y1-y0)):	10
Central Aperture Measures
a1(m*,.5):	−0.3862
a2(.5):	−0.3747
Bounds on Inequality Probabilities
Lower bound on Prob(y1>y0):	0.0000
Upper bound on Prob(y1>y0):	0.8856

Case Study 2: Lopinavir-Ritonavir for COVID-19 Treatment (Cao et al., 2020)

The Cao study reports the results of an RCT involving hospitalized patients with confirmed SARS-CoV-2 infection. Treatment consisted of receipt of lopinavir-ritonavir plus standard care, while control consisted exclusively of standard care. The trial's primary endpoint is time to clinical improvement (TTI), measured in days. The trial was open label; moreover the authors report that placebo treatment in the control group was not possible due to the trial's emergency nature. The intention-to-treat sample consists of 199 patients with 99 and 100 randomized to treatment and control, respectively. By the administrative censoring time of 28 weeks clinical improvement was observed for 77 of the 99 treatment subjects and 70 of the 100 control subjects; thus 22 and 30 subjects' data, respectively, are treated as right-censored. Figure 2 reproduces the data depicted in Cao's figure 2.

Median TTI is 16 days in both arms of the trial so Δm = 0. A decisionmaker concerned only with marginal median outcomes would favor neither treatment. But B*(mΔ) = {−9,7} , so the magnitude of mΔ could be quite substantial in either direction. Regarding central aperture A(m_j, m_k) = a₁(m*,.5) = a₂(.5) = 0 in the Cao sample as Δm = 0 .^37,38

Case Study 3: Combination Therapy for COVID-19 Treatment (Hung et al., 2020)

The Hung study reports the results of an open-label randomized phase-2 trial comparing combination treatment for COVID-19 (interferon beta-1b plus lopinavir–ritonavir plus ribavirin) with lopinavir–ritonavir alone in a sample of hospitalized patients. The study's primary outcome is time to providing a nasopharyngeal swab negative for severe acute respiratory syndrome coronavirus 2 RT-PCR. 86 and 41 patients were randomly assigned to treatment and control. The time to negative test (TTNT) outcomes are reported as integers (days) and analyzed in an intention-to-treat context.³⁹

Figure 6 depicts the medians and IQRs reported in Hung's table 2; this is the only information the study provides on the outcomes' marginal distributions. As such, the reported TTNT data are right-censored and interval-measured (e.g. q₁(.25) = 5 implies F₁(5)∈[.25,.5), as depicted by the closed and open symbols in figure 6). For these data Δm = −5 . Applying (12) to these data yields B_IQR(mΔ) = {−11,4}.⁴⁰

7. Summary

Ease of computation, broad applicability, and parsimony in summarizing outcome data are three plausible statistical reasons for the prominence of Δm in health research, whether Δm informs decisionmakers about treatments' effects on patients' health, on healthcare resource demands, or on any other endpoints that may be of concern. By a revealed preference argument, Δm on its own must routinely provide decisionmakers with useful information about the choices they face. Yet this popularity could arise either because parameters other than Δm — ψ in (10)—are not of interest or because parameters beyond Δm are of interest but the cost of using them to inform choices is perceived to be too great.

This paper has suggested that it is straightforward to augment the signals about treatment effectiveness sent by Δm with information about other features of outcomes' distributions in such ways as should more comprehensively inform decisionmakers who must choose on the basis of the data at hand, whether or not those other features are point identified. In essence the strategies presented in this paper may serve to reduce the perceived costs of appealing to parameters beyond just Δm for those decisionmakers who do find them of interest.

While one might imagine a portfolio of alternative approaches, the specific strategies developed here are designed to be broadly roadworthy since they are easily implemented and require—whether a decisionmaker follows path 1 or path 2—assumptions hardly more stringent than those needed for identification of Δm itself. That the measures of treatment effectiveness proposed here are all anchored to Δm should also be of comfort to decisionmakers who have traditionally and broadly relied on Δm in making choices. How these approaches perform in practice, whether they provide useful information to decisionmakers, and how corresponding standards for clinical significance should be defined are among the issues still to be resolved.

Finally the paper has focused on issues involving identification, ignoring considerations of inference used to understand implications of sampling variation. Such considerations may be of interest in some decisionmaking contexts and, if so, would be useful to tackle in future study.