Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects

Abstract

The use of evidence from clinical trials to support decisions for individual patients is a form of “reference class forecasting”: implicit predictions for an individual are made on the basis of outcomes in a reference class of “similar” patients treated with alternative therapies. Evidence based medicine has generally emphasized the broad reference class of patients qualifying for a trial. Yet patients in a trial (and in clinical practice) differ from one another in many ways that can affect the outcome of interest and the potential for benefit. The central goal of personalized medicine, in its various forms, is to narrow the reference class to yield more patient specific effect estimates to support more individualized clinical decision making. This article will review fundamental conceptual problems with the prediction of outcome risk and heterogeneity of treatment effect (HTE), as well as the limitations of conventional (one-variable-at-a-time) subgroup analysis. It will also discuss several regression based approaches to “predictive” heterogeneity of treatment effect analysis, including analyses based on “risk modeling” (such as stratifying trial populations by their risk of the primary outcome or their risk of serious treatment-related harms) and analysis based on “effect modeling” (which incorporates modifiers of relative effect). It will illustrate these approaches with clinical examples and discuss their respective strengths and vulnerabilities.

Introduction

Austin Bradford Hill, the epidemiologist who formalized randomized clinical trial (RCT) methods, noted in the 1960s that although RCTs can determine the better treatment on average, they “do not answer the practicing doctor’s question: what is the most likely outcome when this particular drug is given to a particular patient?”¹ But, if not with an RCT, how can we forecast outcomes in individuals under alternative treatments?

Kahneman and others have described two distinct approaches to single case prediction, the “inside view” and the “outside view.”^{2 3} The inside view considers a problem by focusing on the specifics of each case and understanding the many characteristics that make it unique. It is the view prioritized by “traditional” physicians who emphasize clinical experience and expert judgment and the view we spontaneously adopt for making decisions in virtually all aspects of life. By contrast, the outside view predicts by explicitly identifying a group of similar cases (a “reference class”) and ignoring some potentially important particulars; the reference class provides a statistical basis for prediction. This is referred to as “reference class forecasting.”

The central premise of evidence based medicine (EBM) is the recognition that Hill’s assertion was (at least partially) wrong: RCTs can be used to guide clinical decision making for individuals. In emphasizing this, RCTs were repurposed from tools to establish causality into tools for prediction, through reference class forecasting, in individual patients. There is now a wealth of evidence—in medicine and other fields—that predictions based on the inside view (even by “experts”) are vulnerable to all manner of cognitive biases, and that prioritizing impersonal data generally improves decision making.^{2 4} EBM has become the dominant paradigm both for medical decision making and for clinical practice guidelines.

Nevertheless, it is easy to recognize that Hill’s view was, in part, right. The result of a positive RCT only provides evidence that at least some of the enrolled patients benefited from the intervention. Logically, the impact this knowledge has on decision making in an individual (even one qualifying for the trial) is unclear when treatments can have very different effects in different patients. For example, thrombolysis in acute ischemic stroke can improve functional outcomes (through recanalization) but also worsen functional outcomes (through intracerebral hemorrhage); angiotensin converting enzyme inhibitors can prevent progression of renal insufficiency but can also cause it in some patients; antihypertensives prevent serious cardiac events but can also cause them; bisphosphonates can prevent fracture from osteoporosis but can also cause them⁵; carotid endarterectomy for symptomatic carotid stenosis can prevent strokes but can also cause them.⁶ Moreover, individual patients have many characteristics that might affect the likelihood of an outcome and the benefits or harms of treatment. Determining the best treatment for a given patient, the task of a clinician, is thus very different from determining the best treatment on average.

Thus, interest in understanding how a treatment’s effect varies across patients—a concept described as heterogeneity of treatment effects (HTE)—has been growing. This concept is central to the agenda for both personalized (or precision) medicine and comparative effectiveness research. HTE has been defined as non-random variability in the direction or magnitude of a treatment effect, in which the effect is measured using clinical outcomes.⁷ Despite this definition, the broad concept of HTE accommodates different perspectives⁸ and different goals,⁹ which have at times confused discussions.¹⁰

In this article, we focus on what we consider the most essential goal of HTE analysis for clinical decision making: prediction in the individual patient of outcomes under alternative treatments. Although we discuss fundamental difficulties in the prediction of treatment effects for individuals, we emphasize this goal because HTE analysis is of little value if it does not improve our ability to make predictions and decisions one patient at a time. Below, we discuss: fundamental difficulties with the prediction of “individual” risk and treatment effect common to all approaches; limitations of conventional (one-variable-at-a-time) subgroup analysis; and several different regression based approaches to “predictive” HTE analysis.

Sources and selection criteria

This narrative review provided background for a larger project supported by both a 14 member technical expert panel and an evidence review committee. We used our extensive libraries for the review of basic epidemiological and statistical concepts relevant to HTE. For emerging methods related to predictive approaches to HTE, articles recommended by the technical expert panel and two targeted systematic searches by the evidence review committee were also used. The aims were to discover consensus based methodological recommendations for predictive HTE analysis in RCTs and to identify methodological papers evaluating regression based approaches to predictive HTE analysis. Key search terms included “heterogeneity of treatment effect”, “treatment effect”, “regression”, “statistical models”, “randomized controlled trials” (as topic), and “precision medicine”. These search terms were combined using appropriate Boolean operators to yield 2851 abstracts, which were hand searched. The evidence review committee prepared an annotated bibliography (see supplemental table 1).

Conceptual background

Although the goal of predictive HTE analysis is to improve the prediction of the treatment effect and decision making in each patient,^{9 11} we acknowledge that this enterprise has fundamental limitations. Both risks and treatment effects can be determined only at the group level.^{12 13 14 15} Indeed, under a deterministic framework (that is, when outcomes in patients are viewed as being fully determined by prior causes and conditions), given complete knowledge, the only “true risk” for an individual would be either 0 or 1 for a binary outcome (such as death), and risk prediction should be regarded as a quantification of the limits of our knowledge, rather than an intrinsic property of the patient. Even if we accept the existence of a “true” risk for an individual (that is, a fundamentally stochastic universe), this true risk cannot be directly measured. Instead, a person’s risk is estimated by examining the frequency of outcomes in a group of other “similar” patients. But because similarity can in practice always be defined in many different ways (as we will discuss), a person’s risk cannot typically be uniquely determined; rather, it is a “model dependent” property.^{14 15}

The prediction of treatment effect in individual patients is even more challenging than prediction of outcomes. This is because treatment effects at the person level are inherently unobservable even in retrospect; outcomes under two counterfactual treatment conditions cannot be ascertained in the same person simultaneously. Thus, predicting treatment effect, and evaluating models that predict treatment effect, is fundamentally different from (and more difficult than) predicting outcome risk, because we are attempting to predict an “outcome” (that is, the difference in potential outcomes, with and without treatment) that is only partially observable in any patient.

Thus, both risk and the prediction of treatment effect must rely on assigning patients to groups (reference classes) to which the individual of interest is similar. But how can similarity be defined? Mathematician John Venn pointed out in 1876 that “every single thing or event has an indefinite number of properties or attributes observable in it, and might therefore be considered as belonging to an indefinite number of different classes of things.”¹⁶ Alternative methods of classifying patients will lead to different inferences for any given patient. This “reference class problem” has been subject to much discussion in other fields but has received surprisingly scant attention in the EBM literature.

The approach of EBM to the reference class problem has generally been to emphasize the broad reference class of the RCT population. Guyatt and colleagues’ classic User’s Guide to the Medical Literature II stated: “if the patient meets all the inclusion criteria, and doesn’t violate any of the exclusion criteria—there is little question that the results are applicable.”¹⁷ The enthusiasm for pragmatic trials, enrolling ever broader populations, represents an extrapolation of the view that broad based populations provide the most useful reference class for clinical decisions.¹⁸

Another approach to the reference class problem was suggested by Reichenbach, the theorist who first coined the term. He recommended calibration to “the narrowest reference class for which reliable statistics can be compiled,”¹⁹ but matching on just 10 binary characteristics gives rise to more than 1000 distinct subgroups (and 20 binary characteristics give rise to more than a million). Thus, this approach is limited by the problem of small samples, leaving the reference class problem unresolved. The narrowest possible class is the patient himself or herself, who is unique; the uniqueness of each case is why medicine at times becomes an improvisational, “inside view” enterprise so dependent on “clinical intuition.” What is needed is a principled way of prioritizing relevant patient characteristics.

The selection of an appropriate reference class is the central problem when using group evidence to forecast outcomes (or treatment effects) in individuals.²⁰ The mapping of an individual to a group of similar (but non-identical) patients always requires (implicitly or explicitly) a model or scheme, whether that be the inclusion criteria of the overall trial or some narrower classification scheme. In this article we will review three broad analytic approaches used to derive more personalized treatment effect estimates: conventional (one-variable-at-a-time”) subgroup analysis, risk based subgroup analysis (or risk modeling), and treatment effect modeling.

Conventional subgroup analysis

The most common approach to HTE analysis is to divide patients serially on the basis of single characteristics defined at baseline (such as male v female; old v young) and to serially test whether the treatment effect varies across the levels of each attribute. The literature and guidance on the conduct of subgroup analyses is extensive (and largely pejorative).^{21 22 23 24 25 26 27 28 29 30 31 32 33 34} Nevertheless, subgroups remain routinely reported, often in the form of forest plots (fig 1). Understanding these analyses and their limitations is central to the understanding of predictive HTE analysis.

Fig 1.

³⁵ HRs (black squares) and 95% confidence intervals (horizontal lines) for the primary outcome for PCI versus medical therapy for subgroups are shown. Despite what seems to be clinically significant differences in treatment effects across several variables (eg, qualitative interactions for both age and sex), no statistically significant interaction was found between treatment and any of the subgrouping variables, indicating “consistency of effects across clinical significant subgroups.” The discrepancy between the apparent clinical importance of the observed effect heterogeneity and the lack of statistical significance reflects the very low statistical power for interaction effects, which is typical of most trials. (B) The DANAMI-2 trial also showed “consistency of effects” across all subgroups for the primary composite endpoint of death, reinfarction, or disabling stroke in 1572 patients randomly assigned to primary angioplasty versus fibrinolysis.³⁶ Despite the similarity of effects in these one-variable-at-a-time subgroup analyses, a subsequent risk stratified analysis,³⁷ using the TIMI (mortality) risk score, showed that patients who are at low risk of mortality are less likely to benefit than those at high risk, particularly on the clinically important absolute risk difference scale. Indeed, for the outcome of mortality, there was a slight trend for harm among the three quarters of patients at lowest risk and a very large benefit for the quarter of patients classified as high mortality risk (see fig 5). Conventional subgroup analyses, such as those described in this forest plot, can miss these clinically important differences because, when patients are serially divided into groups defined one-variable-at-a-time, each analysis grossly under-represents the heterogeneity across individual patients who differ from one another in many variables simultaneously. These analyses also obscure variation in treatment effect on the risk difference scale, which is the most important scale to assess clinically. Abbreviations: ACE: angiotensin converting enzyme; DANAMI-2: Danish Multicenter Randomized Study on Fibrinolytic Therapy Versus Acute Coronary Angioplasty in Acute Myocardial Infarction; LAD: left anterior descending; MI: myocardial infarction; OAT: Occluded Artery Trial; PCI: percutaneous coronary intervention.

Why most positive subgroup analyses are false

It is often emphasized that the appropriate statistical method for assessing HTE is to test for the contrastin effects among the levels of a baseline variable with a statistical test for interaction.^{38 39 40 41} This typically compares the relative risk (or the odds ratio or hazard ratio) across the levels of the subgrouping variable and corresponds to the epidemiologic concept of effect modification. A common mistake is to claim heterogeneity on the basis of separate tests of treatment effects within each subgroup^{22 23}—for example, when a P value reaches statistical significance in one group (say, men) but not in another (say, women).

However, even when adhering to the recommended practice of performing interaction tests, the credibility of “statistically significant” subgroup effects should be regarded cautiously. Several recent meta-epidemiological studies have shown that very few are corroborated in subsequent studies.^{24 42 43} A recent empirical evaluation of sex-by-treatment interactions in 109 topics found only eight (7%) with statistically significant sex-by-treatment interactions⁴²—a result that was not much greater than what would be expected by chance if relative effects between the sexes were always identical. These results suggest that most statistically significant subgroup effects represent false discoveries.²⁴ Well known examples of misleading positive subgroup analyses include not just the influence of astrological signs on the effects of aspirin for patients with myocardial infarction,⁴⁴ but far more plausible and therefore more harmful results (eg, aspirin is ineffective in secondary stroke prevention in women,⁴⁵ beta blockers are ineffective in inferior wall myocardial infarction).^{22 46}

The low credibility of positive subgroup results is understandable because RCTs are powered for the main effect of treatment; at least four times the sample size would be needed to provide similar power for an interaction effect of similar magnitude (eg, for a relative odds ratio equal to the odds ratio of the main effect), even for a perfectly balanced subgroup. Alternatively phrased, these interaction effects are anticipated to be powered at about 30% for perfectly balanced subgroups (eg, males v females) in trials powered at 80% for the main treatment effect,^{38 47} and less for unbalanced subgroups or for smaller effects. Moreover, because subgroup analyses are typically viewed as being without cost, they are often performed promiscuously across variables, with far less previous evidence than for the main effect in a RCT (which is typically not undertaken without a reasonable probability of success). The combination of a low proportion of anticipated true effects and low power explains the high proportion of false discoveries among “statistically significant” effects (fig 2). Thus, subgroup analyses generally provide the essential conditions for the reliable generation of false discoveries: weak theory and noisy data—that is, exploratory analyses testing multiple hypotheses performed in databases with low power.^{48 50} In addition to false discovery, effect exaggeration—that is, “testimation bias” (also known as the “winner’s curse”)^{49 51}—can be anticipated because overestimated effects are preferentially selected through the use of a statistical criterion (such as a P value threshold). These two concerns are important not only in conventional subgroup analysis, but also when considering how best to develop multivariable prediction models to estimate effects for individual patients, which is the focus of this article.

Fig 2.

Why most positive subgroup effects are false or overestimated. The well known unreliability of subgroup analysis arises from the fact that interaction tests typically have weak power when performed in randomized clinical trials designed to have 80% or 90% power to detect main treatment effects, and also by the fact that multiple poorly motivated subgroups are typically evaluated.⁴⁸ “Exploratory” analyses are depicted by the distributions on the left, in which subgroup analyses are undertaken across multiple variables to detect the 5% that represent true effect modification (shown in red). This prevalence of “true effects” was chosen to emulate previous meta-epidemiologic studies.⁴² Assuming 30% power to detect interaction effects,^{38 47} only a minority of these true effects (1.5/5=30%) are anticipated to show statistically significant effects. Meanwhile, with an α of 0.05 (P value threshold), 5% of the null variables (shown in black) are also anticipated to be statistically significant (5/95=4.8%). Thus, only a minority of results with a P value <0.05 (1.5/6.3 of the effect estimates falling to the right of the blue threshold) represent true subgroup effects. The false discovery rate is much lower when only variables with a higher prior probability are tested. The distribution on the right depicts “confirmatory” analyses with a prior probability of 25%. Here, about two thirds of subgroups with a P value <0.05 (7.5/11.3) are anticipated to represent true effects. Even then, subgroup effects will generally be overestimated because exaggerated effects are preferentially identified. This exaggeration of effects has been referred to as “testimation bias” because it arises when hypothesis testing statistical approaches (eg, for biomarker discovery) are combined with effect estimation.⁴⁹

Why claims of “consistency of effect” are often misleading

Results similar to those shown in fig 1 (in which none of the tested subgroup interaction effects reach statistical significance) are often the basis for claims of “consistency of effects.” However, because trials are usually underpowered for subgroup analyses, the inability to find significant interactions should be anticipated. For example, fig 1A (the Occluded Artery Trial³⁵) shows how clinically significant differences in effects between men and women and between young and old patients may not be statistically significant, even in large trials, and even when the point estimate of these effects is qualitatively different (harm in one stratum and benefit in another). Additionally, even when results seem to be highly consistent across “clinically important subgroups” (as in the Danish Multicenter Randomized Study on Fibrinolytic Therapy Versus Acute Coronary Angioplasty in Acute Myocardial Infarction (DANAMI-2) trial; fig 1B), null subgroup analyses do not imply that benefit-harm trade-offs are likely to be similar across all trial enrollees or that the overall treatment effect applies similarly across trial subjects. Indeed, a core assumption of personalized medicine is that, at the person level, HTE is ubiquitous (some patients benefit and others don’t, and this is not totally random).^{13 52} Because one-variable-at-a-time subgroup analyses compare groups of patients who differ systematically on only a single variable, whereas individual patients differ from one another across many variables simultaneously, the conventional approach greatly under-represents the heterogeneity clinicians observe clinically (that is, at the person level). Subgrouping schemes, defined more comprehensively across many clinically salient variables simultaneously, may detect important differences in treatment effects that are obscured in conventional subgroup analysis.⁵³ Indeed, clinically important HTE was subsequently identified in the DANAMI-2 trial when a risk modeling approach was applied.³⁷

Why conventional subgroup analyses are incongruent with the goals of predictive HTE analysis

Conventional subgroup analysis may detect “relative effect modification.” This can help inform theories about conditions under which treatments are especially effective or ineffective. However, this approach does not directly address the reference class problem—that all patients belong to multiple different subgroups, each of which may yield different inferences. For example, even assuming the that subgroup effects shown for both age and sex in the Occluded Artery Trial (fig 1A) are wholly credible, the optimal treatment for a young woman (or an old man) would be unclear. Because a patient has an indefinite number of attributes and can thus belong to an indefinite number of different reference classes, there are as many probabilities for a given patient (and by extension estimable treatment effects) as there are specifiable classes.

The application of conventional subgroup analysis to clinical decision making is further complicated because HTE is typically tested (and presented) on a relative scale (eg, odds ratio or relative risk), whereas the absolute risk difference (RD) scale (or its inverse, number needed to treat (NNT)) is the most important scale for clinical decision making.^{13 54 55 56} Although the literature sometimes emphasizes the distinction between “predictive factors” (relative effect modifiers) and “prognostic factors,” this distinction is somewhat artificial and can be as confusing as it is clarifying. This is because prognostic factors are “predictive” (that is, effect modifying) when effect is considered on the clinically important absolute scale, and predictive factors typically have “prognostic” effects that complicate clinical interpretation. For clinical decision making, prognostic and predictive effects should be considered simultaneously, because the ARD is a product of both the outcome risk and the relative treatment effect (fig 3). Thus, the presence of statistically significant heterogeneity on the relative scale does not necessarily imply clinically important HTE, which should always be assessed on the ARD scale (fig 3). Indeed, prognostic modeling can often reveal clinically important HTE, because differences in outcome risk are just as important as similar changes in relative risk when determining the ARD. Moreover, prognostic factors are much easier to model than relative effect modifiers, given abundant prior knowledge and much greater statistical power for main effect analyses rather than tests for interaction.

Fig 3.

The value of a marker for targeting of treatment depends on its influence both on outcome risk and on relative treatment effect. The domain along the x axis quantifies prognostic effects; the range along the y axis quantifies relative effect modification (sometimes called “predictive” effects). The clinically significant effect measure (absolute risk difference or number needed to treat (NNT)) is depicted by the contour plot. The average effect in the overall trial is shown by the large red dot, which can be disaggregated into subgroups (shown by the smaller black and white dots) in different ways. Both pure prognostic markers (which scatter patient subgroups horizontally) and pure relative effect modifying (“predictive”) markers (which scatter patient subgroups vertically) help discriminate patient groups with different degrees of absolute benefit. Asymmetry of the scatter represents the usual non-normal distribution of risk (here shown as log normal, with a greater number of low risk and low benefit patients). Generally, “predictive” markers are more difficult to identify than prognostic markers, both because reliable information about effect modifiers is usually scant and because power to examine treatment effect interactions is substantially lower than prognostic effects. However, factors are often both prognostic and relative effect modifying, and these effects may be “synergistic” (relative risk reduction and outcome risk positively correlated) or “antagonistic” (relative risk reduction and outcome risk negatively correlated). The most useful factor for treatment selection is that for which the absolute risk difference most varies as a function of that factor’s value (here, the “synergistic” example). This corresponds to improved discrimination for treatment benefit on the risk difference scale. Note that for the factor with antagonistic effects, patients with the largest relative treatment effect paradoxically benefit the least on the absolute scale. From a decision analytic perspective, the clinical value of the marker is determined by its ability to distribute patients across a decisionally important threshold, which depends on the treatment burden (accounting for patient preferences, adverse effects, and costs). These decision thresholds are represented by the contours

Limitations of guidance for subgroup analysis

Guidance for analyzing, reporting, and interpreting subgroup analysis typically includes key recommendations¹³: subgroups should be fully defined a priori (to prevent data dredging); be limited in number (or corrected for multiplicity, or both); be well motivated by clinical reasoning or previous empirical studies; be in the expected (and pre-specified) direction^{9 22}; be supported by formal tests for interaction; and be fully reported and cautiously interpretted.^{21 22 30 57 58 59} It has also been recommended that the type of subgroup analysis (eg exploratory (fun to look at) or confirmatory (potentially actionable)) should be specified.^{9 56 60} A further refinement is the development of an instrument to help evaluate the credibility of any positive subgroup effects.^{21 30 61}

Although this guidance thoughtfully deals with one aspect of the central dilemma of subgroups—the risk of a falsely positive subgroup—it mostly ignores the other term: the risk of overgeneralizing summary results to all patients who meet the enrollment criteria. Although the potential importance of HTE is increasingly recognized,^{34 62 63 64 65 66} trialists, peer reviewers, and regulators have very little guidance on which subgroup analyses should be routine, expected, and necessary for the results to be considered fully and transparently reported.

Predictive approaches to heterogeneous treatment effects

Predictive approaches to HTE are intended to ameliorate many of the above limitations of one-variable-at-a-time subgroup analysis. The goal of predictive HTE analysis is to develop models that can be used to predict which of two or more treatments will be best for individual patients when multiple variables that influence the benefits or harms of treatment are taken into account. We divide this type of analysis into two subcategories:

Firstly, risk modeling: an approach to predictive HTE analysis whereby a multivariable model (either externally or internally developed) that predicts the risk of an outcome (usually the primary study outcome) is applied to disaggregate patients in trials so that treatment effects can be examined across risk groups

Secondly, treatment effect modeling (or “effect modeling”): an approach to predictive HTE analysis that develops a model directly on trial data to predict treatment effects (that is, the difference in outcome risks under two alternative treatment conditions). Unlike risk modeling, such a model incorporates a term for treatment assignment and permits the inclusion of treatment by covariate interaction terms.

Risk modeling

We have previously proposed a framework for risk modeling that prioritizes the reporting of relative and absolute treatment effects across risk strata for the primary trial outcome and suggests that these should be routinely reported.⁵⁶ Why should outcome risk be prioritized as a subgrouping variable over other variables, such as age, sex, or comorbidities? Unlike other variables that may or may not modify treatment effect, outcome risk is a mathematical determinant of treatment effect. Table 1 shows the definition of several different measures of treatment effect. All of these measures depend on the outcome rate in the control group (the control event rate; CER), which is itself an observable proxy for outcome risk. Because outcome risk typically varies substantially in a trial population when risk is described through a combination of factors,⁶⁷ the CER will also vary across the trial population when it is disaggregated with a prediction model. Except when trials have null effects, the ARD will generally vary when CER varies across the population (fig 3). Mathematically, only one measure of treatment effect (at most) can remain consistent when risk varies across the population.

Table 1.

Treatment effect is mathematically dependent on the control event rate^*

Measure	Definition
Absolute risk difference	CER-EER
Relative risk reduction	1-(EER/CER)
Odds ratio	EER/(1-EER) ÷ CER/(1-CER)

^*CER: control event rate; EER: experimental event rate.

Figure 4 shows the 30 day mortality risk estimates for 1058 patients with ST elevation myocardial infarction based on pretreatment clinical and electrocardiographic variables.⁶⁹ The risk of mortality in the quarter of patients at highest risk is about 16 times higher than it is in the quarter at lowest risk. Doctors know (and simple algebra confirms) that for interventions that carry some risk of serious treatment related harm, benefit-harm trade-offs differ in patients at such different risks of mortality. However, it is common practice in research to aggregate these patients together in a trial and emphasize the overall summary results, thereby obscuring whether the differences in treatment effect across risk categories are clinically important. Thus, our view is that trial results are incompletely disclosed unless both outcome rates and treatment effects across risk groups are described.^{56 66 70 71}

Fig 4.

Distribution of mortality risk. This distribution displays the predicted mortality risk in 1058 patients who received reperfusion therapy for ST elevation myocardial infarction at 28 US hospitals from the lowest risk (0th centile) to the highest risk (100th centile). Mortality risk is calculated using the individual patients’ clinical and electrocardiographic variables and a validated logistic regression equation.⁶⁸ The dotted red line indicates that the average mortality risk is about 6%. However, about three quarters of patients have a risk lower than the average risk, and the typical (median) risk patient has a risk that is around half the average risk. The quarter of patients at lowest risk have only a 1% probability of 30 day mortality, so an invasive procedure such as percutaneous coronary intervention, is unlikely to reduce the risk of mortality any further in these patients. However, the quarter of patients at highest risk have substantial potential for benefit. In a conventional clinical trial, these patients with highly different risks are collapsed into a single overall population, even though benefit-harm trade-offs may differ greatly. This risk distribution is typical of trials with a low outcome rate, when a reasonably good multivariable predictive model is available to describe risk.⁶⁷

Figure 4 illustrates another commonly observed property^{67 72}—that the distribution of the predicted risk is skewed, such that the risk of mortality is lower than the average risk for about 75% of patients; the risk of mortality in the “typical” (median risk) patient is about 3%, about half the average risk that would be reflected in the summary result. The higher mortality risk is driven by the influential quarter of patients at highest risk. When the risk distribution is skewed, the overall benefit for a treatment seen in the trial’s summary results may not reflect the benefits or the benefit-to-harm trade-offs even in patients who are at typical risk (especially when there is some treatment related harm).^{66 72}

An understanding of the underlying distribution of risk for patients in RCTs can help inform anticipated subgroup effects, which by their nature are more credible than unanticipated subgroup effects (in the same way that confirmatory subgroup analysis is more credible than exploratory subgroup analysis (fig 2)). For example, when considering the use of a potentially effective invasive procedure (such as percutaneous coronary intervention; PCI) with a small risk of serious treatment related harm, it is anticipated that the benefit-harm trade-offs would be very different across the risk distribution shown in fig 4. Thus, despite “consistency of effects” in conventional subgroup analysis of the DANAMI-2 trial (fig 1B) (which compared PCI versus medical therapy in patients with ST-elevation myocardial infarction (STEMI)), clinically important HTE emerged when the population was subsequently stratified by mortality risk using the TIMI (thrombolysis in myocardial infarction) score (fig 5A). A risk stratified analysis based on an internally derived model using the data from the RITA-3 trial, which compared an invasive to a non-invasive approach for patients with non-STEMI/unstable angina, showed similar results (fig 5B).

Fig 5.

Analyses showing that invasive coronary procedures improve mortality in patients with ST elevation MI (DANAMI-2) in high risk but not low risk groups; this pattern holds true for mortality or reinfarction in non-ST elevation MI (RITA-3). (A) The DANAMI-2 trial tested an invasive procedure (PCI) against medical treatment in patients with ST elevation MI. (B) The RITA-3 trial compared an invasive strategy against medical treatment in patients with non-ST elevation MI/unstable angina. Event rates (upper plot), hazard ratios (middle plot) and absolute risk reductions (lower plot) are shown for each trial, with the average effect depicted by a dotted line. In DANAMI-2 (N=1527), a post hoc subgroup analysis stratified by risk showed that the approximately 75% of patients at low risk (TIMI score 0-4) received no mortality benefit—indeed, they had a non-significant trend towards harm. High risk patients (TIMI score ≥5) benefitted greatly from the invasive procedure (∼10% absolute reduction in mortality). The interaction (on the hazard ratio scale) between TIMI risk score and treatment effect was statistically significant (P<0.008). These effects were seen despite “consistency of effects” across all subgroups in conventional (one-variable-at-a-time) subgroup analyses. The RITA-3 trial (N=1810) showed a similar risk by treatment interaction for the outcome of death or non-fatal MI at four months when analyzed with an internally derived risk model. Absolute risk reduction in the primary outcome was very pronounced in the eighth of patients at highest risk, whereas the half at lowest risk received no benefit. DANAMI-2: Danish Multicenter Randomized Study on Fibrinolytic Therapy Versus Acute Coronary Angioplasty in Acute Myocardial Infarction; MI: myocardial infarction; OAT: Occluded Artery Trial; PCI: percutaneous coronary intervention; RITA-3: Randomized Intervention Trial of unstable Angina 3.

The pattern observed in these trials is not rare. Rather, risk distributions seem to conform to predictable patterns, based on the prevalence of the outcome and the discriminatory performance of the prediction model.⁶⁷ Other examples in which effects in high risk subpopulations obscure the lack of benefit (and even harm) in many typical or low risk patients include more intensive versus less intensive thrombolytic therapy in STEMI,⁷³ activated protein C for sepsis (https://s3-us-west-2.amazonaws.com/drugbank/fda_labels/DB00055.pdf?1265922807),⁷⁴ enoxaparin or tirofiban in acute coronary syndrome,^{75 76 77} anticoagulation for stroke prevention in non-valvular atrial fibrillation,^{78 79} fidaxomicin versus vancomycin to prevent recurrence of Clostridium difficile infection, and many others.^{6 73 80 81 82 83 84}

The examples in fig 5 show how risk modeling can lead not only to important changes on the ARD scale but to statistically significant HTE on the relative scale. This interaction can emerge for many reasons but should be expected when there are known treatment related harms that are reflected in the primary outcome, because similar degrees of treatment related harm will outweigh (or substantially reduce) the benefits in low risk patients but not high risk patients.^{53 66} At the same time, the importance of a significant “P value for interaction” should not be overemphasized when subgroups have very different outcome rates because the clinical importance of HTE needs to be determined on the absolute scale. For example, the Diabetes Prevention Program (DPP) trial tested both a lifestyle modification program and metformin pharmacotherapy against usual care in patients with pre-diabetes. It provides an interesting case where statistically significant relative effect modification was shown for one intervention (lifestyle modification) but not the other (metformin), even though clinically important HTE was shown for both interventions when effects were examined on the absolute scale (fig 6).

Fig 6.

High risk patients with pre-diabetes benefit more than low risk patients from interventions with both homogeneous relative treatment effects (lifestyle) and heterogeneous relative treatment effects (metformin). The Diabetes Prevention Program trial compared three approaches to diabetes prevention among patients with pre-diabetes: (1) a rigorous lifestyle modification program; (2) metformin treatment; (3) and usual care. (A) The graphs show event rates, hazard ratios, and risk differences for (A) lifestyle modification versus usual care and (B) metformin versus usual care for the outcome of development of diabetes. Overall results are depicted by the horizontal dotted line; both lifestyle modification and metformin showed substantial effectiveness in preventing diabetes.⁸⁵ When patients were stratified by their risk of diabetes according to a simple internally developed risk model,⁸⁶ the treatment effect was homogeneous on the hazard ratio scale for lifestyle modification, but strongly heterogeneous for metformin (P_intervention <0.001). Nevertheless, similar HTE across risk strata was seen when the treatment effect was expressed on the risk difference scale. This analysis demonstrates the limited clinical value of null hypothesis testing for HTE on the proportional scale when the outcome rate differs so dramatically across risk groups. The clinical significance of HTE needs to be evaluated on the absolute scale, where the benefits of the strategies for preventing diabetes can be weighed against the treatment burdens. Stratification with an externally derived model yielded similar results, with strata specific point estimates of effects indicated by asterisks (*).⁸⁷ HTE: heterogeneity of treatment effect.

The importance of risk as a determinant of absolute benefit is widely accepted. The concept has entered guidelines, notably in the recommended approach to lipid lowering treatment for the prevention of coronary artery disease.⁸⁸ The concept also underpins several algebraic approaches to “individualizing” evidence that are based on risk predictions and an assumption of consistent relative effects.^{89 90 91 92} Risk based analyses of RCTs permit this assumption to be examined.

External versus internal models

Although an applicable externally derived model would enable translation into practice, especially if well validated and clinically accepted, many of the above examples used internally developed risk models. These were derived on trial data “blinded” to treatment assignment. As long as good modeling practice (such as a large number of events per independent variable and a priori selection of risk variables based on previous literature) has been adhered to, models derived directly from RCT data provide “honest” (internally valid) treatment effect estimates within risk strata.^{51 93} Although some researchers recommend that the control arm be used to model risk only,^{94 95 96} this approach can potentially induce differential model fit on the two trial arms, biasing treatment effect estimates across risk strata, and exaggerating HTE.⁹⁷ Indeed, with this approach, overfitting on the control arm can make completely innocuous and ineffective treatments appear to be beneficial in high risk patients and harmful in low risk patients. Various cross validation techniques have been proposed to mitigate this bias.⁹⁸ However, given the small scale of the ARD compared with the predicted outcome risk, even very modest overfitting on the control arm can substantially bias estimates of the treatment effect.

Although internally derived (or endogenous) prognostic models can provide reliable estimates of treatment effects within trial risk strata,⁹⁸ the implementation of an externally valid prognostic model is necessary for translation into practice. The finding of clinically important HTE across risk strata within a trial provides an important impetus for implementing an externally valid model. It should be noted that external validity is a general concern for RCT results and is not confined to results subgrouped using risk models.

Other dimensions of risk: heterogeneity of treatment related harm

It is also important to examine whether treatment related harms vary across risk strata because the treatment burden might not be constant across strata defined by outcome risk. When the two dimensions of risk are highly correlated (when high risk patients are also at greatest risk of treatment related harms), it becomes more difficult to segregate treatment favorable patients from treatment unfavorable ones.^{99 100} Thus, to facilitate the interpretation of benefit-harm trade-offs, important treatment related harms should be reported at the same level of disaggregation (that is, in each of the risk strata) as the primary outcome.

For treatments with serious treatment related harm, a better understanding of the variation in the risk of these adverse events may help to “deselect” patients with unfavorable benefit-harm trade-offs.¹⁰¹ Figure 7 illustrates two recent analyses that showed clinically important variation in the benefit-harm trade-offs in patients who were stratified by internal risk models for the treatment related harm (fracture in the case of pioglitazone; bleeding in the case of long course versus short course dual antiplatelet therapy). Although these analyses can be highly informative, differential overfitting may occur when the adverse outcome is rare in the control group, underscoring the importance of model validation.

Fig 7.

Benefit-harm trade-offs change substantially when subgroups are stratified by their risk of treatment related harms. (A) In the IRIS study, pioglitazone was shown to reduce the risk of recurrent events (stroke or MI) (RR=0.76) in patients with ischemic stroke and insulin resistance, but with an increase in the risk of fracture. At five years, the incremental risk of fracture was 4.9% (13.6% v 8.8%; HR 1.53). When patients were stratified by their risk of fracture using a simple risk score with eight variables, for each 100 patients at low risk of fracture treated with pioglitazone for five years, two to three had a pioglitazone related fracture, compared with six to seven in each 100 patients at high risk.¹⁰² During this same interval, in both risk groups three to four fewer patients treated with pioglitazone had a recurrent stroke or MI. Thus, the number of ischemic events prevented per fracture caused was two in the group at low risk of fracture and 0.5 in the high risk group. When only serious fractures were considered (those requiring hospital admission or surgery), pioglitazone prevented six ischemic events per serious fracture caused in those at low risk of fracture, but only about one event in those at high risk. These clinically important differences in benefit-harm trade-offs across strata emerged despite consistency of effects on the proportional scale for both the harm and benefit of treatment. (B) Similarly, when patients were stratified by their bleeding risk using a simple five variable risk score, prolonged DAPT (aspirin plus clopidogrel or ticagrelor) after percutaneous coronary intervention had a very favorable harm-benefit trade-off in patients at low risk of bleeding but not in those at high risk.¹⁰³ DAPT: dual antiplatelet therapy; HR: hazard ratio; IRIS: Insulin Resistance In Stroke; MI: myocardial infarction; RR: relative risk.

Several trials have been stratified by combining models for outcome risk and for treatment related harm to make more comprehensive benefit-harm models.^{6 73 104} Although this is ultimately the goal of evidence personalization, the arithmetic combination of predictions from different models poses serious challenges related to the calibration of predictions that are beyond the scope of this discussion. Finally, because the primary outcome is sometimes a composite of outcomes with treatment responsive causes and those with treatment unresponsive (or competing) causes, it may also be useful to stratify the trial population by an index that predicts the fraction of outcomes attributable to the treatment responsive cause.^{105 106 107} For example, implantable cardiac defibrillators may be of greater benefit in those who have a higher risk of sudden cardiac death compared with their risk of pump failure death¹⁰⁸; PFO closure may be more beneficial in a subset of patients with stroke and PFO who are more likely to have a stroke that is caused by PFO rather than another occult mechanism^{109 110}; an anti-endotoxin specific therapy may be of greater benefit in patients with sepsis who are at higher risk of Gram negative rather than Gram positive causes of sepsis. Stratification of patients by prediction models that estimate risk of important competing events might also be informative in some circumstances.^{109 110}

Treatment effect modeling

Although subgrouping on the basis of prognostic modeling has advantages over conventional subgroup analyses, outcome risk may not represent the optimal classification scheme. Prediction models developed on RCT data “unblinded” to treatment assignment have the potential to capture relative effect modification through the inclusion of treatment-by-covariate interaction terms. This may be important for determining (both relative and absolute) treatment effects and highly important for optimizing treatment selection.¹¹¹ Indeed, approaches to stratified and personalized medicine have often focused exclusively on the discovery of effect modifiers on the relative scale,¹¹² and some researchers reserve the term HTE to refer only to heterogeneity on the relative scale.¹¹³ When strong and well established effect modifiers exist—such as time from onset of symptoms to treatment for reperfusion therapies in myocardial infarction—treatment interaction effects can be included in the model, regardless of statistical significance. For example, stratification by predicted benefit (predicted outcome risk with treatment minus predicted outcome risk without treatment) could then stratify some lower risk patients with acute myocardial infarction who present very early as being more treatment favorable than some higher risk patients who present later.

However, the incorporation of relative effect modifiers (treatment interaction terms) that were selected on the basis of modeling on the trial itself into prediction models has special challenges. The selection of “statistically significant” relative effect modifiers for inclusion in a prediction model is identical in many respects to one-variable-at-a-time subgroup analysis and has many of the same vulnerabilities—weak theory and noisy data—that can lead to “false positives” and exaggerated effects (from testimation bias⁴⁹ and other forms of overfitting). The number of events per interaction term needed for more accurate modeling of effect modification is many times greater than the number needed for main prognostic effects and has not been well studied. “Treatment benefit” prediction models using naive regression to select “statistically significant” interactions should be expected to provide misleading estimates of within strata effects because of unreliable, exaggerated, and highly influential interaction terms.^{114 115} The vulnerability to overfitting leaves this approach prone to discovering false subgroup effects, even for treatments that are completely ineffective.

Nevertheless, the further individualization of treatment selection often depends on the discovery of treatment effect modifiers that are not well established. One promising approach is to select a set of variables anticipated to be relative effect modifiers on the basis of a priori clinical reasoning, and to use an omnibus test for significance (with the appropriate degrees of freedom) across all the included putative interaction terms. If the result of this overall test is statistically significant, all interactions are included in the model; otherwise, none are. Because interaction terms are still prone to overfitting, this process should be combined with penalized regression methods (such as lasso regression,^{116 117} ridge regression,^{118 119} or elastic net regularization regression),^{120 121} which shrink model coefficients on the basis of model complexity to yield better predictions of the absolute treatment effect within new populations. Alternatively, when developing models “unblinded” to treatment assignment, a different set of data should be used for variable and model selection (that is, to define the reference class or subgrouping scheme) and for estimation of the treatment effect across strata. There is intense research interest in methods that combine effect modifier (biomarker) discovery with treatment effect estimation, including both machine learning approaches and regression based methods^{122 123 124 125 126 127 128 129 130 131} (see supplemental table 1 for additional examples), although clinical application remains limited.¹²¹ These more complex and aggressive prediction approaches require more rigorous validation.

The SYNTAX score II (fig 8) is an example of a model for predicting benefit; eight variables were used as both prognostic variables and effect modifiers (in treatment interaction terms), in a score that predicts outcomes for patients with non-acute coronary artery disease under two revascularization strategies—coronary artery bypass graft surgery (CABG) versus PCI.¹³³ Although the overall trial showed substantial benefit for CABG (the primary outcome was reduced from 17.8% with PCI to 12.4% with CABG; P=0.002),¹³² stratification by predicted benefit according to the SYNTAX score II indicated that the benefits of population-wide CABG may largely be achieved by targeting to the most treatment favorable quarter of patients, potentially avoiding the substantial trauma and morbidity associated with an open chest procedure in most patients.

Fig 8.

The SYNTAX score II stratifies patients with non-acute coronary artery disease on the basis of their risk of mortality with CABG versus PCI and is a useful guide to decision making. In the SYNTAX trial, rates of major adverse cardiac or cerebrovascular events at 12 months were significantly higher in the PCI group (17.8%) than in the CABG group (12.4%; P=0.002), confirming that CABG should be the preferred approach for patients with untreated three vessel or left main coronary artery disease.¹³² The SYNTAX score II was developed by applying a Cox proportional hazards model to the SYNTAX (Synergy Between Percutaneous Coronary Intervention With Taxus and Cardiac Surgery) trial (N=1800). It contains eight predictors: a previously developed anatomical SYNTAX score, age, creatinine clearance, left ventricular ejection fraction, presence of unprotected left main coronary artery disease, peripheral vascular disease, female sex, and COPD, plus treatment interaction terms with each of these variables. The graphs show (A) event rates, (B) hazard ratios, and (C) absolute risk reductions for CABG versus PCI. Unlike the examples shown in other figures, event rates do not increase monotonically across quarters because patients are stratified not by predicted risk but by predicted benefit (outcome risk with PCI minus outcome risk with CABG). Overall results, depicted by the horizontal dashed line, show a trend that favors CABG. However, when patients are stratified by their expected benefit, a quarter of patients who are treatment unfavorable is identified (P_interaction=0.0037 for eight interaction terms), and benefit is largely confined to the quarter of patients at highest benefit. Although the SYNTAX score II has been validated for prediction of outcomes, it has not yet been validated for the prediction of benefit. CABG: coronary artery bypass graft surgery; COPD: chronic obstructive pulmonary disease; PCI: percutaneous coronary intervention.

Evaluating models that predict treatment benefit

The evaluation of a prediction model intended to estimate benefits using the usual metrics for outcome discrimination (eg, c-statistic) and calibration does not provide information on how well a model performs for predicting benefit—that is, the difference between outcome risk with two alternative strategies. Efforts to develop measures to assess model accuracy for predicting benefit are hampered by the fundamental problem of causal inference.¹³⁴ Unlike individual patient outcomes, individual patient treatment effects (that is, who benefits and who does not) are inherently unobservable because patients do not simultaneously receive both counterfactual treatments to which they are randomized.¹³⁵

Recently, the c-statistic, commonly used to measure discrimination in outcome risk models, has been adapted to evaluate the prediction of treatment effect.¹³⁶ To do this, two patients who are discordant on treatment assignment are matched according to their predicted benefit (the absolute difference in their outcome risk with and without treatment). These matched pairs of patients with a similar “propensity for benefit” can then be classified into three categories according to their “observed benefit” by comparing outcomes in the control and experimental patient—benefit (1, 0); no effect (1, 1 or 0, 0); or harm (0, 1)—where 1 represents a bad outcome and 0 represents a good outcome in each of the two study arms; the c-statistic assesses how well the model discriminates pairs of patients on the basis of this trinary “outcome.”¹³⁶ This approach assumes no correlation in the distribution of outcomes under the two treatments, conditional on the variables in the prediction model; this strong assumption leads to generally low values of the “c-for-benefit” statistic. Similarly, a model based ROC (receiver operating characteristic) measure has been proposed for treatment selection markers using a potential outcomes framework, but this approach relies on the assumption that model predictions are correct.¹³⁷

Ultimately, the usefulness of a model depends not just on its ability to predict accurately and provide honest estimates of within strata treatment effects, but on its ability to improve decisions. This depends on model performance relative to a specific decision threshold—that is, a risk distribution that perfectly balances the burdens, harms, and costs of treatment. Decision curve analysis¹³⁸ has been proposed to evaluate the clinical usefulness of prediction models and has been adapted to evaluate models that predict HTE in trials.¹³⁹ These methods evaluate whether a particular prediction-decision strategy optimizes net benefit for a population at a particular decision threshold, compared with the best overall strategy (that is, treat all or treat none).¹⁴⁰ The ultimate test of a predictive approach is to compare decisions (or outcomes) in settings that use such predictions with usual care in an experiment,¹⁴¹ such as a cluster randomized trial.

Use of observational data for predictive HTE analysis

Observational data have tremendous appeal for predictive HTE analyses. In particular, the growing availability of large databases that capture electronic health records and claims on millions of patients can provide statistical power far beyond that typically achieved by single or pooled RCTs.^{142 143} In addition, because these databases capture a broader, more heterogeneous population, representing the full spectrum of patients seen in routine practice, they may be an excellent substrate for risk prediction. Nevertheless, because randomization remains the gold standard for unbiased estimation of causal treatment effects, RCTs are also the preferred substrate for HTE analysis. Although modern methods for de-confounding may produce unbiased average treatment effect estimates in observational data, it is not possible to know whether all model assumptions are met in any given analysis.¹⁴⁴ In addition, for HTE analyses, the assumptions necessary for deconfounding need to be met within each stratum, a more stringent requirement than for the estimation of an overall average treatment effect. Apart from confounding by indication, large observational data sources collected from routine care are often plagued by missing data and misclassification. A growing body of research is focused on improving the understanding of the necessary conditions for trustworthy, unbiased observational results, including research on methods to achieve balance in covariates across subgroups.^{145 146 147} Nevertheless, the use of observational data potentially compounds and complicates the well known problems with credibility that already undermine subgroup analyses even in RCTs.

Conclusion

Although a positive RCT result provides strong evidence that an intervention works for at least some patients included in the trial, clinicians still need to understand how a patient’s multiple characteristics combine to influence his or her potential treatment benefit—that is, the difference between outcome risk with and without the treatment. Disaggregation of the overall results according to absolute risk can yield more informative, narrower reference classes for more patient specific effect estimates of benefit and support more patient specific decision making. Routine use of absolute risk modeling is usually feasible for large phase III trials; journal editors, funders, and the research community should insist on these analyses. New statistical approaches, devised to model treatment effect directly, may offer additional advantages (increasing “benefit discrimination”), although with greater potential for statistical overfitting, false discovery, and biased predictions in new patient populations. These approaches merit more research.

Nonetheless, substantial barriers still need to be to overcome.¹⁴⁸ We list a few of the outstanding research questions related to the problems covered in this article in the Questions for future research box. In addition, we need research aimed at:

• Improving the integration of clinical prediction into practice¹⁴⁹

• Improving our understanding of how to individualize clinical practice guidelines

• Establishing or extending reporting guidelines¹⁵⁰

• Establishing new models of data ownership to facilitate data pooling¹⁵¹

• Re-engineering the clinical research infrastructure to support substantially larger clinically integrated trials sufficiently powered to determine thee HTE, or to develop our ability to predict when observational data will probably be sufficiently de-biased for reliable HTE determination, or both.^{146 152}

Many recent and ongoing organizational and technical advances should enable this evolution.

As Hill pointed out, at the level of the individual the right decision is fundamentally under-determined by the results of a trial. Even in retrospect, it is usually impossible to tell whether the right decision was made for any individual patient. Thus, although the goal of prediction is to improve decisions in each patient, paradoxically, like any other intervention, this can be assessed only by examining whether more precise prediction improves outcomes at the population level. As experience with these approaches grows, in addition to stronger methodological and evidentiary standards, we will need empirical studies to ensure that these more flexible (and manipulable) methods realize in practice their potential to improve population outcomes.

Glossary.

• Effect modification: This occurs when the size of the effect of a treatment or exposure on an outcome depends on the level of a third variable (eg, patient characteristics). In the presence of effect modification, the use of an overall effect estimate is inappropriate.

• Heterogeneity of treatment effect (HTE): Non-random variability in the direction or size of a treatment effect, measured using clinical outcomes. HTE is fundamentally a scale dependent concept and therefore, for clarity, the scale should generally be specified. (It should be noted that some people reserve the term to describe variability on a relative scale only, such as changes in the odds ratio or relative risk.)

• Clinically important HTE: This occurs when variation in the risk difference across patient subgroups spans a decisionally important threshold, which depends on treatment burden (including treatment related harms and costs). It is generally assessed on the absolute scale.

• Predictive HTE analysis: The main goal of predictive HTE analysis is to develop models that can be used to predict which of two or more treatments will be better for an individual by taking into account multiple relevant variables.

• Risk modeling approach: An approach to predictive HTE analysis in which a multivariable model that predicts the risk of an outcome (usually the primary study outcome) is applied to disaggregate patients in trials and examine risk based variation in treatment effects.

• External risk models versus endogenous/internal risk models: External risk models have been developed from an external trial or cohort but can be used for HTE analysis of other trials. Internal risk models are developed directly from the trial population.

• Treatment effect modeling approach: An approach to predictive HTE analysis that develops a model directly on randomized trial data to predict treatment effects (the difference in outcome risks under two alternative treatment conditions). Unlike risk modeling, the model incorporates a term for treatment assignment and permits the inclusion of treatment by covariate interaction terms.

• Net benefit: A decision analytic measure that puts benefits and harms on the same scale. This is achieved by specifying an exchange rate on the basis of the relative value of benefits and harms associated with interventions. The exchange rate is related to the probability threshold determining whether a patient is classified as being positive or negative for a model outcome, or (when applied to trial analysis) as being treatment favorable versus treatment unfavorable.

• Overfitting: A situation where predictions do not generalize to new subjects outside the sample under study. Overfitting occurs when a model conforms too closely to the idiosyncrasies or “noise” of the limited data sample from which it is derived and is a threat to the validity of a model.

• Penalized regression: A set of regression methods, developed to prevent overfitting, in which the coefficients assigned to covariates are penalized for model complexity. Penalized regression is sometimes referred to as shrinkage or regularization. Examples of penalized regression include lasso, ridge, and elastic net regularization.

• Predictive factors: Patient characteristics that result in modification of the treatment effect and are often assessed using statistical interaction terms on the relative scale. Generally, predictive factors are substantially harder to identify than prognostic factors, given the more limited a priori information on their effects and the greater statistical power needed to test interactions.

• Prognostic factors: Patient characteristics that influence the risk of the outcome of interest. These factors may also help discriminate patient groups with different degrees of absolute benefit. A single characteristic may be both predictive and prognostic.

• Reference class: A group of similar cases that is used to make predictions for an individual patient of interest. The “reference class problem” refers to the fact that similarity can be defined in an indefinite number of different ways because individuals have many different potentially relevant attributes.

• Testimation bias: Refers to the fact that, on average, the effect sizes of newly discovered true (non-null) associations are inherently inflated. Testimation bias arises from the use of statistical thresholds in the process of discovering associations or of selecting variables for a model. Inflation is expected when an association has to pass a certain threshold of statistical significance to be deemed positive (or included in a model) and the study has suboptimal power. The problem is also referred to as the “winner’s curse.”

How patients were involved in the creation of this article.

To gain insight into the importance of heterogeneity of treatment effects from the patient’s perspective, we held three 90 minute webinar enabled group discussions with patient stakeholder representatives of three patient powered research networks (PPRNs): ARthritis Partnership with Comparative Effectiveness Researchers (AR-PoWER), the Health eHeart Alliance, and iConquerMS.

Questions for future research .

• How can we jointly predict multiple important outcomes or risk dimensions (eg, risk of the primary outcome versus risk of treatment related harm)?

• How can we determine when relative effect modifiers are sufficiently reliable for inclusion in treatment effect models?

• Do machine learning techniques have distinct advantages over traditional statistical approaches for predicting treatment effect? If so, under what conditions?

• How can models be updated and recalibrated in the absence of new randomized trials?

• Under what conditions can observational big data sources provide a substrate for reliable predictive heterogeneity of treatment effect analysis?