Clinically informative measures of the effect of drugs or other interventions

Abstract

AIMS

Measures to compare two drugs are often affected by the background risk in the reference group; a ceiling effect results when the background risk is small. We review measures of the effect of drugs, including a special formulation of the relative risk difference, the attained effect or clinical result ratio, that addresses background risk and ceiling effect.

METHODS

Existing measures are the risk and odds ratios, the absolute and relative risk differences, and the number needed to treat. The attained effect is defined as the observed gain in success (the difference of proportion of success between the two interventions), divided by the maximum attainable gain, the maximum proportion of success one can expect. We illustrate the relationship between these measures with published results of two meta-analyses.

RESULTS

In studies of the effectiveness of cell salvage, the baseline risk ranged between 8 and 95%. This variability affected the risk difference and number needed to treat, while the attained effect, with a ceiling residual risk of 2%, showed that the gain in success was half the maximum attainable gain. In studies of the effectiveness of therapy in patients infected by the human immunodeficiency virus, where the baseline risk was less variable, and there was no ceiling effect, the maximum attained effect indicated that the gain could be much smaller.

CONCLUSION

The attained effect, interpreted as the proportion of effectiveness that remains to gain for future interventions, can usefully complete the number needed to treat as a clinically informative effect measure.

WHAT IS ALREADY KNOWN ABOUT THIS SUBJECT

• Several comparative measures (ratios, differences, or the number needed to treat) are used to express the effect of a drug or another intervention.

• These measures can vary in the way they are affected by the background risk measured from the reference group.

WHAT THIS STUDY ADDS?

• This paper reviews the formulation, interpretation and limitations of measures of effect.

• We describe a little-known parameter, the attained effect or clinical result ratio, a positive reformulation of the relative risk difference, and suggest how available parameters can be best used to summarize results of studies of the effect of drugs.

Introduction

Several comparative measures are used when assessing the effect of a drug or another form of intervention [1]. Whether the objective of the study is to assess the efficacy, the individual effectiveness or the population effectiveness [2], these measures compare the rate or risk of an outcome of the disease in a group exposed to the new intervention with that in a comparison or reference group. The latter can be untreated, or receive a placebo or a reference intervention. These comparisons are usually based on the calculation of a ratio, a difference, or a more complex parameter, such as the number needed to treat [1, 3]. For example, in a paper on the efficacy of antiretroviral combination therapy [4], results were reported as odds ratios; in another paper on the effectiveness of cell salvage [5], results were reported as risk ratios. In both studies, several other measures of effect could easily be computed from the results provided.

Often, existing measures are based on the analysis of failures. These measures, however, vary in the way they transform the same data on risk to reflect the usefulness of the new intervention. The resulting measure can convey information on the association between intervention and outcome (measure of association), or on its potential impact on the population (measure of impact). These measures can also vary in the way they are affected by the background risk measured from the reference group. A related issue is that of the ceiling effect, by which the potential gain in effect cannot be as large in absolute terms for a treatment when the background risk is small as when it is large.

In this paper, we review the formulation, interpretation and limitations of measures of effect based on risk reduction. Following the work of Babbs [6], we propose the attained effect or clinical result ratio, a more positive perspective in expressing the effect. This parameter is based on success, and addresses the issues of background risk and ceiling effect. We submit that the attained effect, a reformulation of the relative risk difference, may provide a useful complement to existing measures.

Methods

Background and notations

Consider a randomized controlled trial or cohort study designed to compare the risk of a given unwanted outcome under a new intervention A with that of a reference intervention B (Table 1). For simplicity, we assume that the time of follow-up is equal and complete for all patients in both groups, so that the risk is simply a proportion. Let r_B be the proportion of patients in group B who progress, during the time of follow-up, to the outcome, hereafter referred to as failures, and r_A the observed proportion of failures in group A; r_B is usually referred to as the background risk. The complementary proportions of patients who did not progress to the outcome, hereafter referred to as successes, are s_B = 1 − r_B in group B, and s_A = 1 − r_A in group A. We assume further that A is truly better than or equal to B. Under this assumption, and because r_B, r_A, s_B, and s_A are proportions, 0 ≤ s_B ≤ s_A ≤ 1, and 1 ≥ r_B ≥ r_A ≥ 0.

Table 1.

Typical display and notation for a randomized controlled trial with two interventions

Intervention	Failures	Successes	Denominator	Risk
A (new)	FA	NA − FA	NA	rA = FA/NA
B (control)	FB	NB − FB	NB	rB = FB/NB

Commonly used measures of effect

The existing measures of effect are either measures of association between the intervention and the risk of failure, or measures of impact of the intervention on risk. All measures are based on comparisons of r_A and r_B. For each measure, we provide in Table 2 the formula of the parameter, its interpretation, and its possible values under the above-mentioned assumptions. The variances of the estimators are provided in the Appendix.

Table 2.

Formulas and range of commonly used measures of effect, and the attained effects

Measure	Abbreviation	Estimator*	Interpretation	Estimate when effectiveness of A is
				Nil	Maximum
Of association
Risk ratio	RR	rA/rB	Reflects the strength of the association between intervention and outcome, might be useful for causality assessment, when risks can be directly estimated (in cohorts and trials); usually not useful for expressing the positive effect of drugs	1	0
Odds ratio	OR	(rAsB)/(sArB)	Reflects the strength of the association between intervention and outcome; considered inadequate when risks can be directly estimated; not useful for expressing the positive effect of drugs	1	0
Of impact
Absolute risk difference	ARD	rA − rB	Reflects the absolute impact of the new intervention; more informative when expressed as the number needed to treat (see below)	0	– rB
Relative risk difference	RRD	(rA − rB)/rB	Reflects the relative impact of the new intervention; preferred parameter in the absence of a ceiling effect, especially when expressed as a percent relative risk reduction	0	− 1
Number needed to treat	NNT	1/|rA − rB|	Reflects the absolute impact of the new intervention, expressed as a number of patients to treat with the new intervention to avoid one failure above what the reference intervention does; preferred expression of the clinical usefulness of a drug	+∞	1/rB
Attained effect	AE	(sA − sB)/(st − sB)	Proportion of possible gain in success attained by the new intervention; useful in the presence of a ceiling effect	0	1
Maximum attained effect	MAE	(sA − sB)/(1 − sB)	Proportion of possible gain in success attained by the new intervention; equivalent to the RRD in the absence of ceiling effect	0	1

^*See Table 1 and text for notation and definitions.

Two existing measures of association are the risk ratio (RR) and the odds ratio (OR). These measures of effect reflect the strength of the association between type of treatment and risk of failure (RR) or odds of failure (OR). Measuring the strength of association might be useful in assessing the causal relationship between the intervention and the observed effect [7]. The OR is often abusively considered as an estimator of the population RR, but will be further away from the null hypothesis value of 1 whenever the background risk is >10%.

The absolute risk difference (ARD) is a measure of the absolute impact of the new treatment on the risk of failure during the time of follow-up. The ARD will vary with the baseline risk, depending on the choice of the reference intervention and the targeted population. To take into account the background risk, the impact can be measured as a relative risk difference (RRD). It is a measure of the relative impact of the new intervention on the risk of failure. For a given effect, the RRD is usually constant across populations with different background risks. Although formally negative, both the ARD and RRD are often expressed positively; they are then referred to as absolute and relative risk reductions [8].

The number needed to treat (NNT) was proposed as a more direct way to convey to clinicians the magnitude of impact of a treatment [3]. The NNT, which is the inverse of the absolute value of the ARD, is a measure of the average number of patients who would have to receive treatment A to prevent one failure in that group, compared with the number of failures prevented in group B during the time of follow-up. For example, if the ARD is −0.25, then the NNT is 1/0.25 = 4. The NNT varies with the baseline risk, and its interpretation is correct only if A is actually better than B.

Measures of attained effect

Our initial objectives in seeking a new measure of effect were to focus on successes rather than failures, and to take into account the ceiling effect. This effect is related to the fact that, in some circumstances, proportions of success cannot reach 1, and will be limited by a ceiling, s_t, defined as the maximum proportion of successes that one can expect, given the characteristics of the population targeted by the intervention or the nature of the outcome. This ceiling, s_t, defined by Babbs [6] as the ‘clinically optimal or ideal outcome’, can vary from s_B (no expected gain in effect) to 1 (all failures are expected, at least theoretically, to be avoidable). The gain in success of a treatment, consequently, is limited by the maximum attainable gain in success, i.e. the distance between s_B and s_t. Thus, an ideal measure of effect should take the ceiling effect into consideration and compare the observed gain in success with this maximum attainable gain.

Consequently, we suggest the term ‘attained effect’ (AE) to describe the parameter first defined as the ‘clinical result ratio’ by Babbs [6] as (s_A − s_B)/(s_t − s_B). It is equal to the (observed) absolute gain in success, or absolute success difference, divided by the maximum attainable gain in success. In the best expected circumstances, where s_t = 1, the maximum AE (MAE) is (s_A − s_B)/(1 − s_B); its numerator is equal to 1 − ARD, and its denominator to r_B. In that special case, the MAE is simply the positive formulation of the RRD; it is also equal to the complement to 1 of the RR. Whatever is the value set for s_t, the range of the AE, 0–1, is a direct reflection of the direction and size of the effect. The AE can be expressed as a percentage, and is interpreted as the proportion of possible gain in success attained by the new intervention.

Illustration

To illustrate the interpretation and relationship between measures of effectiveness, we used data from studies included in the above-mentioned meta-analyses [4, 5]. These data were extracted from published randomized controlled trials of the effectiveness of cell salvage to minimize perioperative blood transfusion in orthopaedic patients (Table 3) [5], or comparing the effect of dual and triple therapy on progression of acquired immunodeficiency syndrome or death in human immunodeficiency virus-infected patients (Table 4) [4].

Table 3.

Data from randomized controlled trial of the effectiveness of cell salvage to minimize perioperative allogeneic transfusion in orthopedic surgery (data extracted from Figure 1 in [5]), and corresponding measures of effectiveness

Study*	Intervention group			Control group			Measure of effectiveness†
	Failure (FA)	Total (NA)	% (rA × 100)	Failure (FB)	Total (NB)	% (rB× 100)	RR	ARD	RRD	NNT	AE	MAE
Majkowski et al.[9]	7	20	35.0	19	20	95.0	0.37	−0.60	−0.63	1.7	0.65	0.63
Heddle et al.[10]	10	40	25.0	27	41	65.9	0.38	−0.41	−0.62	2.4	0.64	0.62
Healy et al.[11]	14	84	16.7	15	44	34.1	0.49	−0.17	−0.51	5.7	0.54	0.51
Riou et al.[12]	1	25	4.0	2	25	8.0	0.50	−0.04	−0.50	25.0	0.67	0.50
Rosencher et al.[13]	6	20	30.0	6	10	60.0	0.50	−0.30	−0.50	3.3	0.52	0.50
Ayers et al.[14]	5	103	4.9	29	129	22.5	0.22	−0.18	−0.78	5.7	0.86	0.78
Newman et al.[15]	3	35	8.6	28	35	80.0	0.11	−0.71	−0.89	1.4	0.92	0.89

^*See [5] for original references, definition of abbreviations, and final results of the meta-analysis.

^†Attained effect calculated assuming that the maximum attainable proportion of successes (s_t) would be 0.98. RR, risk ratio; ARD, absolute risk difference; RRD, relative risk difference; NNT, number needed to treat; AE, attained effect; MAE, maximum attained effect.

Table 4.

Data from randomized controlled trial of the effectiveness of triple therapy to decrease progression to acquired immunodeficiency syndrome and death in human immunodeficiency virus-infected patients (data extracted from Figure 2 in [4]), and corresponding measures of effectiveness

Study*	Intervention group			Control group			Measure of effectiveness†
	Failure (FA)	Total (NA)	% (rA × 100)	Failure (FB)	Total (NB)	% (rB × 100)	RR	ARD	RRD	NNT	MAE (= AE)
AIDS clinical trial group 229 [16]	3	98	3.1	6	100	6.0	0.51	−0.03	−0.49	34.0	0.49
AIDS clinical trial group 320 [17]	33	577	5.7	63	579	10.9	0.53	−0.05	−0.47	19.4	0.47
Merck 035 [18]	0	33	0.0	1	33	3.0	0.00	−0.03	−1.00	33.0	1.00
RTV Study Group. [19]	119	543	21.9	205	547	37.5	0.58	−0.16	−0.42	6.4	0.42
Merck 039 [20]	11	108	8.3	16	105	15.2	0.55	−0.07	−0.45	14.5	0.45
Spanish Earth-1 [21]	1	33	3.0	1	33	3.0	1.00	−0.00	−0.00	+∞	0.00
PISCES [22]	76	955	8.0	142	942	15.1	0.53	−0.07	−0.47	14.1	0.47

^*See [4] for definition of study abbreviations and final results of the meta-analysis.

^†Attained effect (AE) calculated assuming that the maximum attainable proportion of successes (s_t) could be 1, i.e. AE = MAE. RR, risk ratio; ARD, absolute risk difference; RRD, relative risk difference; NNT, number needed to treat; MAE, maximum attained effect.

Results

In Table 3, we assumed s_t could not be 1, because a small proportion of patients (2%) would always have abundant bleeding so that blood transfusion could not be avoided. With variations in the baseline risk of transfusion ranging between 8 and 95%, the possible gain by the evaluated treatment differed markedly from one study to the other. Indeed, there is little room for improvement with an 8% transfusion rate, whereas the possibilities of improvement are huge with the 95% transfusion rate. This variability is particularly reflected in the ARD and the NTT. On the other hand, the RR, the only measure reported in the meta-analysis [5] despite its lack of relevance, RRD and AE were less affected by this variability. Moreover, the AE expresses in the simplest form the strong effectiveness of the assessed treatment. It indicates that the gain in successes, ranging from 52 to 92%, was at least half of the maximum attainable gain. Furthermore, the MAE, which should not be reported given the existence of a ceiling, was smaller than the AE in all studies, and could even be quite different in studies where the background risk was small.

In Table 4, we assumed that s_t could be 1, as suggested by the results of one study. Consequently, only the MAE needed to be reported. The variation in the baseline risk was less important in this group of trials (3–37.5%), and the possible gain by the evaluated treatment was consequently more stable. The MAE indicates that the gain in successes was, except for one study, less than half of the maximum attainable gain, i.e. that there is still much room for improvement.

Discussion

The formal definition of parameters and their variable sensitivity to ceiling and background risk illustrates that the choice to report one parameter rather than another should always be justified. For instance, although both the RR and OR can be easily calculated from the same two-by-two table, they are usually not considered informative when the focus is on positive effects of drugs, discussed in this paper. Their contribution to the assessment of causality is useful only when assessing negative effects of drugs in pharmacoepidemiology studies; the choice of the parameter, in such circumstances, depends on the study design (RR for cohort studies and randomized controlled trials, OR for case–control studies).

The choice between an absolute (ARD and NNT) or a relative measure of impact (RRD and AE) is more difficult, as both report important information. Both the ARD and the NNT are useful as they provide an estimation of how many events can be avoided by the application of the new treatment, but have different advantages. The NNT has been proposed as a more clinically oriented expression of the absolute impact, but the ARD is simpler to compute. Moreover, the interpretation of the NNT as a measure of effectiveness might be counterintuitive to readers still unfamiliar with the concept, as its value is the highest when the effectiveness is the smallest. We believe, however, that increased use of the NNT will eventually result in the dereliction of the ARD.

The RRD, AE and MAE can be adequately used to convey the relative impact of the new treatment. Thus, they can be used to compare the impact of treatments used for different diseases with different background risks. The MAE can be seen merely as a reformulation of the RRD. We believe, however, that the MAE, and the AE, carries some advantages over existing parameters. First, the expression of results, as positive numbers, is the direct result of the definition of the estimator. Indeed, although the ARD and RRD are usually expressed positively, as absolute and relative risk reductions, they are really negative values. For example, the RRD = −0.78 in the study by Ayers et al.[14] (Table 3) would be reported as a 78% reduction of the risk. From a mathematical point of view, this is worrisome, as both the ARD and RRD can be positive, e.g. in randomized clinical trials or cohort studies with a two-sided null hypothesis, and results in favour of the reference intervention. This misconception of the mathematical meaning of the ARD and RRD is reflected, for example, in some definitions of ARD, RRD and NNT – see, for example, Appendix 1 in [8], where the absolute risk reduction and the NNT would be negative, and the relative risk reduction is defined by two formulae, one positive and one negative.

Formally, a mathematical and statistical approach virtually similar to the AE has first been published by Babbs [6]. In his paper, Babbs proposed the use of the ‘clinical result ratio’, which is identical to the AE, and demonstrated how this parameter can be applied both to continuous and categorical outcomes, and integrated in a meta-analysis of randomized controlled trials. However, our paper goes beyond the original approach by Babbs by comparing the different risk measures and the relative strength and weaknesses of these alternative measures of treatment effects. Furthermore, we believe our proposed terminology of attained effect is a better reflection of what the AE (or clinical result ratio) does reveal, i.e. the proportion of possible gain in success attained by the new intervention.

From the user's perspective, we believe the AE shares the advantage of the NNT of directly reflecting the potential benefit of the intervention for targeted individuals. It also shares the advantage of the RRD, when the latter is presented as a relative risk reduction, of having the range of a simple proportion, which can be less misleading than the ranges of the RR (a logarithmic scale), or of the ARD or NNT (ranges affected by the baseline risk). Compared with the relative risk reduction, however, the AE is based on success rather than risk, which can be a more positive way to present the potential benefit, at least for some patients and health professionals.

The possibility of calculating a measure of potential benefit in circumstances where the maximum expected proportion of successes is limited provides a more generic measure of the effect of an intervention. Indeed, a ceiling of 1 can seem reasonable, at least theoretically, in many studies, but becomes questionable when the individuals included have characteristics incompatible with a 100% success or when the follow-up is long and the failures of interest include outcomes such as death. For example, studies of the effectiveness of trauma care will always be limited by the fact that up to two-thirds of prehospital deaths are not preventable [23]; in long-term follow-up of patients treated for acquired immunodeficiency syndrome, the proportion of successes cannot be 100%, given the high frequency of severe side-effects [24]. In such a context, one could compare two interventions and choose that with the highest MAE, or, if MAEs are the same, that with the least side-effects. Finally, for researchers and decision makers, we believe that providing an effectiveness result as a proportion, or percentage, of the maximum attainable effectiveness is a good reflection, not only of the gain in effectiveness, but also of what remains to gain for future interventions.

One major difficulty in using the AE, however, is to justify and estimate the ceiling. In his paper [6], Babbs proposed that, for binary outcomes, the ceiling could be set to 1, or 100% successes, in most circumstances. Whenever success is measured by survival, but a 100% survival is not realistic, the ceiling could be the predicted survival of healthy persons over the same time period [6]. Estimations of the ceiling can also be provided, whenever possible, by large cohort studies of the natural history of the disease, or by pooling estimates of background risks in previously published meta-analyses. However, Babbs suggested in his paper's discussion [6] that there could be abuse in the choice of the ceiling value, as there is some latitude in deciding what the ideal response would be. Changes in dichotomous variables to 100% survival are often obvious choices, and any other value should always be clearly justified, so that readers would be able to judge and dispute the rationale of the chosen ceiling [6]. Other points are that the ceiling is always estimated with a degree of uncertainty, and that the difference between the AE and the MAE, and therefore the RRD, will be small whenever the background risk is large, as illustrated in our first example. Consequently, estimations of the AE and related parameters should always be provided with confidence intervals, and the resulting conclusions submitted to robustness analyses.

Although the focus was on simple proportions, this review of effectiveness measures applies to all comparisons of parameters with a ceiling effect. The AE therefore might be an appropriate measure in cohort studies of protective factors [25], where cumulative incidences are compared, and in randomized trials with product-limit estimations of survival or event-free rates [1]. In conclusion, the AE is not meant to replace existing measures of effectiveness, nor is it formally different from the clinical result ratio proposed by Babbs [6]. It should be seen merely as a new formulation of existing ways to report results of randomized clinical trials and cohort studies. Nevertheless, we believe that all studies reporting drug effects could be limited to the NNT and either the RRD or AE, depending on the existence of a ceiling effect. An appropriate use of these parameters may improve our ability to take wise decisions regarding new drugs or other interventions.

Competing interests

S.S. is the recipient of a Distinguished Investigator award from the Canadian Institutes of Health Research (CIHR).

Appendix: Variances of effect parameters

Let X and Y be two independent random variables with μ_X = E(X), , μ_Y = E(Y) and ; for a given function f(X,Y), .

In our case, where , , , and , the functions and resulting variances are:

(1) for the RRf(X,Y) X/Y; ;

(2) for the OR; ;

(3) for the ARDf(X,Y) X −Y; ;

(4) for the RRD; ;

(5) for the NNT; ;

and

(6) for the AE; .