Tipping Point Analysis

During a randomized clinical trial (RCT), despite clinical trialists’ best efforts to follow up with all participants for study outcomes, enrolled participants may be lost to follow-up, making it impossible to determine their final outcomes. Participants with missing outcome data may be systematically different from those with complete outcome data, eg, in their prognosis, stage of disease, responsiveness to therapy, or ability to tolerate the assigned treatment regimen. Therefore, estimates of treatment effect based on the available outcome data may not reflect the treatment effect in the entire enrolled population. The direction of bias is often difficult to predict.

Methods such as multiple imputation have been developed to approximate the results that would have been achieved had all outcome data been available. However, such methods often require strong assumptions, eg, the “missing at random” assumption that the pattern of missingness can be explained by the observed data.^1,2 The validity of the missing at random assumption is generally impossible to verify.

Alternatively, rather than trying to replicate the results that would be achieved without missing data, researchers can assess how systematically different the missing and observed study outcomes would have to be to alter the overall conclusion drawn from the observed data. By evaluating the effects of a range of patterns in missing outcome data, tipping point analyses can identify the conditions under which the study’s conclusions would be different had no participants been lost to follow-up.

In an article published in JAMA,³ Ojji and colleagues reported the results of a multicenter RCT in Nigeria comparing a triple-pill protocol with standard care for hypertension, with a primary outcome of mean systolic blood pressure (SBP), quantified as a change from baseline over 6 months. In the primary analysis, the triple pill was more effective than standard treatment. The 6-month SBP measurement was missing for 28 of 300, or 9%, of study participants. A tipping point analysis was conducted to explore how assumptions about the missing blood pressure values would influence the primary conclusion.

Use of Tipping Point Analysis

What Is Tipping Point Analysis?

Tipping point analysis assesses the robustness of study results by varying assumptions about the unknown values of the missing outcome data in order to identify the “tipping point” at which the conclusion about the treatment effect would change.⁴ The observed data, which are not altered, are combined with the hypothetical data for the outcomes that were missing, and the primary analysis is repeated under each scenario.

For a continuous outcome like blood pressure, the tipping point analysis may be completed by incorporating a range of potential differences of treatment effects among participants with missing outcomes directly into the calculation of the statistical test.^4,5 The consistency of study conclusions can then be evaluated across plausible scenarios in which outcomes differed among those who completed treatment vs dropped out of each treatment group. Alternatively, the analysis may begin with an imputation method such as multiple imputation before applying a range of adjustments or “shift parameters” to the now-replaced missing values in 1 or both treatment groups.⁶ These shift parameters represent assumed, hypothetical differences between the unknown values for the missing outcome data and the imputed outcome data.

For a dichotomous outcome, the process generally involves assigning varying proportions of positive or negative outcomes to the missing data in each treatment group, allowing for exploration of all possible combinations of outcomes across treatment groups.⁷

The Figure shows the process of tipping point analysis for a hypothetical study of treatment for hypertension with a continuous blood pressure outcome. Initially, the missing data are replaced with values using multiple imputation, and the completed datasets form the basis for the primary analysis. Then, the imputed values are progressively shifted, eg, adding a fixed value to the treatment group and subtracting a fixed value from the control group. The tipping point is identified when the upper boundary of the confidence interval for the treatment effect crosses zero.

Figure.

The Process of Tipping Point Analysis for a Hypothetical Study of Treatment for Hypertension With a Continuous Blood Pressure Outcome

Why Is Tipping Point Analysis Important?

Understanding the potential impact of missing data on the validity of conclusions drawn from clinical trials is important to avoid being overly confident in a potentially incorrect interpretation of the trial results. Tipping point analysis offers asystematic framework forexploring the robustness of results by identifying hypothetical scenarios under which the study’s conclusions, if all data had been observed, would change. For instance, in a trial with a positive conclusion but significant dropout, tipping point analysis can tell us how much worse the outcomes for the missing patients in the treatment group, compared with those in the control group, would need to be for the resulting treatment effect to lose statistical significance.

Depending on the results of the tipping point analysis, confidence in the robustness of the findings may increase if the study’s conclusions remain stable across a wide range of assumptions. Conversely, confidence may decrease if the results are sensitive to small changes in the assumptions about outcomes in participants with missing data. Regulatory agencies such as the US Food and Drug Administration and the European Medicines Agency often require tipping point analyses to ensure that study results are not overly dependent on untestable assumptions about missing data.^5,8

Limitations of Tipping Point Analysis

Tipping point analysis is not an adjustment method for missing data or an attempt to estimate the treatment effect that would have been observed in the absence of missing data. Furthermore, tipping point analysis does not predict the direction or magnitude of the bias due to missing data, nor does it quantify the likelihood of different missing data patterns across the potential scenarios that are tested. Tipping point analyses may rely on subjective decisions regarding adjustment parameters forhypothetical scenarios.⁶ Forexample, deciding how much worse the outcomes for missing data with a continuous outcome could be in one group compared with another may require clinical judgement.⁹

How Was Tipping Point Analysis Used?

Ojji et al³ focused on missing SBP data at the month 6 clinic visit, for which a total of 9% of data were missing. To address the missing data, they used multiple imputation to create complete datasets in which the outcome patterns were similar for participants with observed and missing data, followed by the tipping point analysis. In the tipping point analysis, the authors explored hypothetical scenarios by applying a “shift parameter” to the missing data inthe 2 treatment groups. In each scenario, the shift parameter was applied to the missing data for both groups, with separate shifts used for the triple-pill and standard care groups. For example, the authors defined scenarios in which participants with missing outcomes in the triple-pill group could have a shift ranging from 0 mm Hg to 31 mm Hg, while participants with missing outcomes in the standard care group had shifts from −6 mm Hg to 0 mm Hg. In each scenario, the multiply imputed values were adjusted by adding one shift parameter to the missing data for the triple-pill group and a second shift to the missing data for the standard care group. The primary outcome analysis was then repeated for each scenario to identify those for which the effect of the treatment on SBP lost statistical significance.

How Should the Tipping Point Analyses Be Interpreted?

The tipping point analysis demonstrated that for the observed benefit to lose statistical significance and the study’s conclusions to be overturned, the multiply imputed missing outcome data would need to be shifted to an implausibly large extent. For example, the results would lose statistical significance if the missing SBP values in the triple-pill group were shifted 26 mm Hg higher than in that group’s multiply imputed data while, simultaneously, those in the standard care group were decreased by 6 mm Hg. These assumed differences between observed and unobserved SBP data represent extreme scenarios, and the authors concluded that they were implausible, supporting the robustness of the study’s conclusions.