Win Ratio –An Intuitive and Easy-To-Interpret Composite Outcome in Medical Studies

Summary

In medical studies with multiple outcomes, researchers always need to make choices as to whether to use a composite outcome (after combining multiple outcomes) as their primary outcome. In this paper we review a new measurement of the treatment effect – win ratio, which can be easily used in studies with prioritized multiple outcomes. We also propose some research topics to be done in this area.

1. Introduction

Multiple outcomes are quite common in clinical and observational studies as it is difficult to use a single outcome to characterize treatment (exposure) effects. Another reason is that the complexity of a disease may be not adequately characterized by a single outcome. For example, in the study of depression in psychiatric studies, the outcomes may include (i) depression severity such as the Hamilton Depression Rating Scale^[1]; (ii) instrumental role functioning such as the Social Adjustment Rating Scale^[2]; (iii) social functioning such as the Short Form 36-Item Health Survey^[3] and (iv) depression remission status based on clinical assessment. In heart research^[4], outcomes of primary interest may include the time to the first heart failure (from study baseline), to death due to heart disease, and to death due to other reasons.

Most research studies attempt to collect as much information as possible in order to study the disease of interest from multiple aspects to find the optimal treatment. Thus, multiple outcomes are usually considered to study treatment effects. However, multiple outcomes impose challenges in data analysis as well as interpretations of study results:

(i) the multiple outcomes in the same study may have different scales. For example, in psychiatric studies, depression status is categorical, while depression severity and social functioning may be continuous (or treated as such if the scale has a wide range). It is difficult to use a multivariate distribution for this kind of mixed outcomes; and

(ii) directions of treatment effects may be different for different outcomes. For example, treatment for depression may reduce depression severity and improve daily living, but may also reduce sexual desires for depressed patients.

A composite outcome is a single outcome based on combinations of multiple study outcomes and is widely used to summarize information from the multiple outcomes. The Intelligence Quotient (IQ) test is an example of a composite outcome, as the test result is a weighted score from four subscales: verbal compression, perceptual reasoning, working memory, and processing speed.^[5-6] Composite outcome is also widely used in clinical and observational studies. Overweight or obese people are generally at higher risks for many diseases such as diabetes, hypertension, and heart disease. One widely used measure for obesity is the Body Mass Index (BMI)^[7] which is defined as

The validity of this index has been widely discussed in literature.^[8]

In heart research, patients may experience both heart failures and death, which are two different types of events.^[4,9] A treatment may decrease the risks of both types of events or may decrease the risk of one, and increase the risk of the other. How to measure treatment effect is a challenging problem for both physicians and statisticians. One popular practice in heart research is to define a composite outcome, for example,^[4] the time to heart failure or death, whichever occurs first. Some well-established methods, such as the Kaplan-Meir estimator^[10] and the Cox proportional hazards model^[11], can be readily used to analyze such composite outcomes.

The advantages and the disadvantages of composite outcomes have been discussed extensively in the medical literature.^[12-41] Heddle and Cook^[41] summarize the advantages and disadvantages of using composite outcomes in clinical studies. They also give some guidelines on how to select composite outcomes. The decision to use composite outcome may be debated forever.

In this report note, we review a relatively new method – win ratio- to analyze some types of multiple outcomes in clinical studies. In Section 2 we give a brief introduction to win ratio and statistical properties. Section 3 focuses on the interpretation of this parameter, the population win ratio. Section 4 reports simulation results of estimating win ratio if both the marginal survival functions of the fatal event and the conditional survival functions of the nonfatal event satisfies being in the Lehmman family,^[42] followed by conclusion and discussion in Section 5.

2. A brief introduction to win ratio method

Finkelstein and Schoenfeld^[43] first develop a nonparametric test, which will be referred to as the FS rank test below, to combine time-to-event and longitudinal outcomes in clinical studies. In this method, each pair chosen from the treatment and control group are compared based on these two types of outcomes. First, subjects are compared based on the time-to-event outcome. If the individual in the treatment group has a longer (shorter) time to event, the treatment group is said to win (lose) in this pair. If for some reason (for example, censoring), their times to event cannot be compared, the longitudinal outcomes will be compared and the win or loss will be determined similarly. The FS rank test is the difference between the numbers of wins and losses in the treatment group. The FR rank test is a generalization of the Wilcoxon rank sum test.

The idea of FS rank test was later expanded by Pocock and colleagues^[44] in a clinical trial with two types of events: fatal event (cardiovascular death) and non-fatal event (heart failure hospitalization). Fatal event is assumed to have a higher priority than the non-fatal event when evaluating treatment effects. For each pair of patients, we first determine which patient lives longer than the other. If this is unknown, we then determine who has a better non-fatal event. This is the same idea as the FS rank test. The ratio of the wins over losses in the treatment group is called the win ratio. The treatment is beneficial compared to the control if the win ratio is greater than 1.

Pocock and colleagues^[44] offered two approaches to calculating the sample win ratio:

(1) Match pair approach: This is a three-step method to calculating the win ratio statistic.

1. Use a risk score or risk stratification to select matched pairs of patients from treatment and control group.

2. For each matched pair, one first compares the priority event (e.g., fatal event). If the comparison cannot be made (e.g., which patient dies first), then check the second priority event (e.g., time to hospitalization). There are 5 possible results based on these two comparisons:

   1. patient in the treatment group dies first;

   2. patient in the control group dies first;

   3. if not (a) or (b), patient in the treatment group is hospitalized first;

   4. if not (a) or (b), patient in the control group is hospitalized first;

   5. none of the above.

3. The comparison results are summarized by N_a, N_b, N_c, N_d, and N_e, corresponding to the numbers of matched pairs in (a), (b), (c), (d), and (e).

The numbers of known wins and losses in the treatment group are N_w=N_b + N_d and N_l=N_a + N_c. The sample win ratio is

(1.1)

Pocock and colleagues^[44] constructed 95% confidence interval (CI) and p-value for the win ratio based on the matched pairs approach. Let p_w =N_w /(N_w +N₁) and

p_L=p_w-1.96[p_w(1-p_w)/(N_w+N_L)]^1/2, p_U=p_w+1.96[p_w(1-p_w)/(N_w+N_L)]^1/2.

the 95% CI of the win ratio is [p_L/(1-p_L), p_U/(1-p_U)]. For a significance test, (p_w-0.5)/ [p_w(1-p_w)/(N_w+N_l)]^1/2 has an asymptotic standardized normal distribution under the null hypothesis of no difference between the treatment and control group.

(2) Unmatched pair approach: Analogous to the matched pairs approach, the unmatched pairs approach classifies the comparison results into five categories. The difference is that we compare every subject in the treatment group with every subject in the control group. Let N_n and N_s be the number of patients in the treatment and control group, we need to make N_n×N_s comparisons.

In the unmatched pair approach, calculating the 95% CI and p-value for R_ware quite complex. Finkelstein & Schoenfeld^[43] described the general idea of significance test. Two-sample U- statistic theory provides expressions for the asymptotic variance of the win-ratio statistic for unmatched pairs. Luo and colleagues^[45] derive an alternative standard error estimate using counting process methods.

3. The mathematical meaning of win ratio

Recently, Oakes^[46] extended the win ratio statistic to define the probability win ratio, the ratio of the win and loss probabilities, for general survival models when follow-up of all patients is over a specified time interval [0, c]. This extension not only avoids the complex subjecting matching process (or comparisons of all subjects between the treatment and control group), but also gives rise to nice statistical properties.

Let (T, X) denote the pair of time to death (T) and to first non-fatal (X) event. For j=0, 1 with 0 (1) denoting the control (treatment) group, let

S_j(t)=Pr[T_j>t], G_](x|c)=Pr[X_]>x|T_]>c], t>0, x>0,

where S_j(t)=Pr[T_j>t] (G_j(x|c)=Pr[X_j>x|T_j>c]) is the marginal survival function of T_j (conditional survival function of X_jgiven T_j>c). The win and loss probabilities are

(1.2)

(1.3)

Thus the probability win ratio is the ratio of W(c) over L(c), Note that unlike the original sample win ratio in (1.1), the probability win ratio is an unknown constant or a parameter, rather than a statistic as in (1.1).

The probability win ratio of greater than 1 indicates evidence of treatment effect in favor of the treatment group. For example, PR_w = 1.5 means that there is a 50% increase in number of deaths (if observed) or non-fatal events (if death is censored) for a patient in a treatment group than in a control group.

It is clear that the win ratio defined above depends on the specified observation window [0, c]. This means that for the same study, if we stop the study at different follow-up times, we will be estimating different win ratio parameters, PR_W (C). In the extreme case, if c is sufficiently large, deaths censored by the observation window will be negligibly small and the second priority event may become irrelevant. In general, PR_W (C) is a function of c. However, as shown in Oakes,^[46] the win ratio will be independent of the observation time c if the marginal survival function and conditional survival function are both in the Lehmann family sharing the same parameters. Moreover, this shared parameter has the interpretation of both the hazard ratio and loss ratio (the reciprocal of the win ratio). This is quite a nice statistical property as the hazard ratio is widely used to measure treatment effects under the popular Cox proportional hazards model.

4. A small simulation study

We now discuss how to estimate the probability win ratio under the assumption that both the marginal and conditional survival functions are in the Lehmann family with the same parameters. We also show how to use the Wei-Lin-Weissfeld^[47] method to obtain a more efficient estimator by combining the estimators obtained from fatal and non-fatal events separately.

In the control group, we assume the joint survival function of (T, X) is

For the treatment group we assume the joint survival function of (T, X) is

The correlation of T and X is described by α; T and X become highly if α is close to 0 and independent when α=1. Under the above specifications, the win ratio parameter in the absence of censoring is PR_W = 1/θ^α. We also assume that both the fatal and non-fatal events are subject to independent censoring with the censoring time C exponentially distributed with a rate q.

We first used the Cox proportional hazards model to estimate the win ratio based on (1) the non-fatal event, where the non-fatal event time may be censored by either the censoring time C or the fatal event time; (2) fatal event, where the time may be censored by the censoring time C. The Wei-Lin-Weissfeld^[47] method was used to combine these two estimators.

Table 1 reports the results after 1,000 Monte Carlo simulations based on sample size n=2,000(λ₁=0.1, λ₂=0.6, θ=0.5, and q=0.5). With such a large sample size, both estimates from fatal events and non-fatal events should be close to the true win ratio parameter. The estimates obtained from the Wei-Lin-Weissfeld^[48] combination method is always more efficient than the other two methods, especially when the correlation between the fatal and non-fatal event is weak (large α).

Table 1.

Estimate of win ratio

α	Win ratio	Fatal event		Non-fatal event		Combined
		Estimate	SD	Estimate	SD	Estimate	SD
0.1	1.072	1.073	0.048	1.073	0.047	1.073	0.045
0.2	1.149	1.149	0.054	1.149	0.051	1.149	0.049
0.3	1.231	1.232	0.065	1.233	0.057	1.232	0.055
0.4	1.320	1.321	0.076	1.319	0.063	1.319	0.059
0.5	1.414	1.417	0.084	1.416	0.066	1.416	0.062
0.6	1.516	1.520	0.104	1.519	0.072	1.518	0.069
0.7	1.625	1.627	0.117	1.621	0.078	1.622	0.071
0.8	1.741	1.741	01.34	1.743	0.083	1.741	0.075
0.9	1.866	1.874	0.161	1.868	0.092	1.867	0.083

5. Conclusion and Future work

In medical studies with multiple outcomes, it is often difficult to construct appropriate composite outcomes to evaluate treatment effects that reflect the multi-faceted nature of interventions. In this paper we review the concept of win ratio and associated methods to facilitate such a difficult process. The win ratio is an intuitive and easy-to-interpret composite outcome and is readily implemented within the context of survival analysis, especially with the probability win ratio.

Since win ratio is new, more work is needed to study its properties and applications. We propose two future research areas in win ratio analysis:

1. In most clinical or observational studies, we want to estimate the treatment effects after adjusting the confounding effects of other covariates. How to incorporate the estimation of win ratio in semiparametric regression analysis is a new area to be explored.

2. Power analysis based on win ratio. In the clinical study design, the sample size calculation is usually based on the proposed treatment effects. How to calculate the sample size given the proposed win ratio is another research topic.

Biography

Dr. Hongyue Wang obtained her BS in Scientific English from the University of Science and Technology of China (USTC) in 1995, and PhD in Statistics from the University of Rochester in 2007. She is a Research Associate Professor in the Department of Biostatistics and Computational Biology at the University of Rochester Medical Center. Her research interests include longitudinal data analysis, missing data, survival data analysis, and design and analysis of clinical trials. She has extensive and successful collaboration with investigators from various areas, including Infectious Disease, Nephrology, Neonatology, Cardiology, Neurodevelopmental and Behavioral Science, Radiation Oncology, Pediatric Surgery, and Dentistry. She has published more than 70 statistical methodology and collaborative research papers in peer-reviewed journals.