Refinement of Operating Characteristics for Model‐Based TGI Metrics Decision Support to Ungate a Pivotal Trial in Oncology

ABSTRACT

The oncology market is evolving, with an increasing number of therapeutic options making the inference of treatment benefit from early development critical. The “Go” versus “No Go” decisions from such trials usually rely on probabilities of achieving meaningful improvements (effect sizes) in endpoints like objective response rate, tailored to products' profile, supporting faster transitions to confirmatory trials. Operating characteristics are sensitive to methodological details, important for establishing an early decision framework for ungating a pivotal clinical trial.

1. Framework Previously Developed

A recently conducted retrospective analysis [1] evaluated the operating characteristics (OCs) of model‐based tumor growth inhibition (TGI) metric estimates of effect size to support Go/No‐Go decisions in early randomized clinical studies of immunotherapy combinations. The TGI metrics evaluated were the tumor growth rate constant (KG), the ratio of model‐predicted tumor size at Week 24 (TR24) to baseline, and the time to tumor regrowth (TTG). The effect size was estimated using median differences (control versus experimental). Classical endpoints like objective response rate (ORR) difference and progression‐free survival (PFS) hazard ratio (HR) were also evaluated for comparison.

This analysis used data from the IMpower150 Phase III trial in first‐line non‐small cell lung cancer (NSCLC) [2, 3], comparing atezolizumab plus bevacizumab plus carboplatin plus paclitaxel (ABCP), atezolizumab plus carboplatin plus paclitaxel (ACP), and bevacizumab plus carboplatin plus paclitaxel (BCP) regimens. ABCP demonstrated significant PFS and overall survival (OS) improvement over BCP in intent‐to‐treat patients without EGFR or ALK mutations. To mimic early‐phase trials, 500 replicates of 40‐patient‐per‐arm datasets were generated by resampling the longitudinal sum of longest diameters (SLD), ORR, PFS, and baseline characteristics from patients in the ABCP and BCP arms, the data were restricted to a 24‐week follow‐up after the last patient was included [1], as is done in a real clinical trial. An incorrect “Go” decision scenario was also mimicked by resampling twice the control arm (BCP).

The main results compared decision performance based on effect size (experimental versus control) across 500 replicates for both ABCP/BCP (true treatment effect) and BCP/BCP (null scenario). Effect sizes for each of the investigated endpoints (TGI metrics, ORR, and PFS) comparing ABCP versus BCP or BCP versus BCP were calculated as the ratio of medians (MedR) for estimated TGI metrics, the observed difference in ORR, and observed HR for PFS.

The study demonstrated that model‐based TGI metrics, particularly KG and TR24, showed strong predictive capabilities and outperformed Response Evaluation Criteria in Solid Tumors (RECIST) endpoints, ORR, and PFS for identifying effective treatments early in this treatment setting.

In this paper, we refine and streamline the proposed OCs assessment process [1] due to its broad relevance and potential utility in early oncology drug development, based on recent methodological learnings, on the following aspects:

• TGI model estimation

• TGI metrics

• Results processing

  • ○
    Effect size assessment
  • ○
    Summarization and visualization of results.

2. Refinement of the Methodology

2.1. TGI Model Estimation

Longitudinal SLD data from the resampled datasets were analyzed using a biexponential TGI model, implemented as a nonlinear mixed‐effects model in NONMEM (version 7.3.0) [4], as previously described [1]. The model was applied to each of the 500 replicated datasets, with treatment arm included as a fixed effect on both KG and shrinkage rate (KS) parameters to estimate population‐level effects. During model refinement, we introduced treatment‐specific TGI parameters (fixed‐effect thetas) under both the alternative hypothesis (experimental versus control) and the null hypothesis (control versus control). This approach minimizes shrinkage in parameter estimates and more accurately reflects the distinct treatment effects and underlying assumptions associated with each hypothesis scenario, ultimately providing a more robust framework for evaluating TGI metrics.

2.2. TGI Metrics

Tumor size ratios relative to baseline were derived from model predictions at specific early time points: Week 24 (TR24), Week 12 (TR12), Week 6 (TR6), and at the time of maximum tumor shrinkage (TRMAX). These metrics were evaluated and compared with the model‐estimated KG. In contrast to KG, which is directly estimated as a structural parameter in the model, TR metrics are derived quantities that integrate both KG and KS. The analysis suggests that TR24 and potentially earlier ratios such as TR12 may offer practical advantages in early decision‐making settings. In this context, TR metrics, notably TR24, demonstrated robust OCs and favorable predictive performance, with effect sizes supporting their use as early indicators of treatment efficacy. Finally, TR24 is more easily interpretable than KG. Overall, while both TR24 and KG are useful metrics for evaluating treatment efficacy, TR24 offers several advantages over KG in terms of robustness, reliability, and interpretability. TTG, the time point at which tumor size begins to increase following initial shrinkage (nadir), was also evaluated. TRMAX is derived at the time of the TTG, TTG estimate might be shrunk in some patients.

3. Results Processing

3.1. Effect Size Assessment

In our previous work [1], we used the ratio of the median of individual estimated metrics (MedR, experimental/control) to assess effect size. However, TTG is inherently a time‐to‐event variable, and analyzing it using survival methods is more appropriate than summarizing it with medians and computing the median ratio between treatment arms. Deriving a HR from individual TTG values in respective arms (experimental and control) enables a more accurate comparison of treatment effects with other parameters, such as KG and TR24. HR for time‐to‐event endpoints, such as PFS, and now TTG, were estimated using a Cox proportional hazards model implemented in R with the survival package [5]. For each replicate, the coxph() function was used to model the survival object, defined by event time and censoring status, as a function of the treatment group. However, in that particular case, all patients have a TTG estimate, and there is no censoring. The TTG HR was then derived as the exponentiated regression coefficient representing the relative risk of an event (regrowth) in the experimental arm compared to the control. This approach allows for a consistent comparison of treatment effects using time‐to‐event endpoints and proper evaluation of the predictive performance of TTG.

3.2. Summarization and Visualization of Results

In the previous work [1], we used a Receiver Operating Characteristic (ROC) curve of correct Go (power) versus incorrect Go (type I error) for graphical representation of the performances. The ROC curves were used to evaluate and compare the performance of model‐derived metrics or TGI parameters, graphically representing correct Go decisions and incorrect Go decisions. Initially, discrete points were plotted based on the distributions of metric‐specific effect sizes under the null and alternative hypotheses [1]. To refine this approach, smoothed ROC curves were generated using the pROC package [6] in R, enabling a more interpretable comparison of the metrics' abilities across scenarios, with the ROC curves, associated confidence intervals and area under the curve (AUC) calculation.

4. Key Messages

The comparison of OCs (correct Go rate versus incorrect Go rate) for the different TGI metrics is presented in Figure 1, which displays the probabilities of effect sizes being less than or greater than target values for the ABCP/BCP comparison versus the BCP/BCP comparison. The gray region (top left) represents a correct Go decision rate > 70% for an incorrect Go rate < 20%. These probabilities are derived from the distributions of point estimates across 500 samples. In this analysis, the comparison of TGI metrics revealed that TR24 MedR exhibited the most favorable operating characteristics, closely followed by TRMAX, KG, TR12, TR6 MedR, and TTG HR (Figure 1).

FIGURE 1.

Probabilities to achieve desired effect sizes when comparing ABCP versus BCP (correct Go rate) and BCP versus BCP (incorrect Go rate) for the primary scenario (40 patients/arm, 24 weeks follow‐up) for TGI parameters or metrics. Horizontal and vertical dotted lines: Correct Go rate 70% and incorrect Go rate 20%. Gray zone: correct Go > 70%, incorrect Go < 20%. For each metric, ROC curves, AUC and their respective 95% confidence intervals are represented and reported.

When compared with RECIST endpoints, TR24 MedR demonstrated superior performance compared to ORR difference and PFS HR (Figure 2), with thresholds of TR24 MedR < 0.90 or < 0.85 emerging as promising targets to support decision‐making, contingent upon the acceptable risk of incorrect Go (< 20% or < 10%, respectively). TR24 showed significantly higher AUC than both PFS (0.934 [0.918; 0.949] versus 0.828 [0.802; 0.853]) and ORR difference (0.934 [0.918; 0.949] versus 0.865 [0.843; 0.887]).

FIGURE 2.

Probabilities to achieve desired effect sizes when comparing ABCP versus BCP (correct Go rate) and BCP versus BCP (incorrect Go rate) for the primary scenario (40 patients/arm, 24 weeks follow‐up) for the selected TGI metrics or RECIST endpoints. Horizontal and vertical dotted lines: Correct Go rate 70% and incorrect Go rate 20%. Gray zone: correct Go > 70%, incorrect Go < 20%. For each metric/endpoint, ROC curves, AUC and their respective 95% confidence intervals are represented and reported.

In contrast to the findings previously reported [1], the ROC curve for KG MedR exhibited a downward and rightward shift, indicating reduced power and an increased incorrect Go rate (type I error), similar to the performance observed for the ORR difference, still superior to PFS. This shift in KG MedR performance is likely due to excessive shrinkage in KG estimation on short follow‐up studies.

This refinement highlighted that OCs may be sensitive to methodological details; therefore, it is strongly advised that the study design of the subsamples mimics the real drug development decisions that are evaluated: sample size, choice of the control (treatment, leveraging a historical control versus having randomized data), and duration of follow‐up. Assessing OCs of TGI metrics seems to be a great way to support decision‐making in early drug development in oncology; further work is ongoing to evaluate different resampling approaches and the robustness of the model used to assess OCs. Finally, the interpretation of the results should account for the association of the selected metric with the primary endpoint of the pivotal trial we aim to ungate.

5. Conclusions

OCs' assessment provides a quantitative framework for characterizing alternative decision endpoints, thereby enhancing the efficiency of early drug development in oncology. The utilization of TGI metrics, in particular, holds promise as a supportive tool for informed decision‐making. Nevertheless, careful consideration must be given to methodological nuances that can significantly influence outcomes, including the choice of modeling strategy, the estimation of the parameters, the processing of TGI metrics values, and the size of the control arm. A thorough understanding and meticulous handling of these factors are crucial to ensure the reliability and generalizability of results derived from OCs assessment. Further evaluations are ongoing in different settings, both prospective and retrospective in multiple myeloma and biliary tract cancer [7, 8] to confirm the approach and gain confidence within multidisciplinary drug development project teams.

Funding

The authors have nothing to report.

Conflicts of Interest

M.M., A.G. are or were employees of Certara and were under contract with Genentech Inc/F. Hoffmann‐La Roche Ltd. for this project; R.B. is under contract with Roche for this project. K.Y., R.B., C.L., P.C., J.J. are or were employees at Roche or Genentech and hold Roche stocks.