Update to improve reproducibility and interpretability: A response to “Machine Learning for Tumor Growth Inhibition”

Reproducibility is an important aspect of pharmacometric research, but complications can occur when complex data management and an understanding of modeling approaches from multiple disciplines are required, such as the case when machine learning (ML), a tumor growth inhibition (TGI) model, and baseline covariates were used to predict overall survival (OS) outcome. In addition, continuous updates of our original work, ¹ based on more sophisticated ML methodologies, are expected to increase the interpretability of the modeling results.

In the commentary by Meid et al., ² the authors originally set out to reproduce our previously published work that explored four ML methods to support the validity of TGI metrics in predicting OS using data from a phase III clinical trial. ¹ After eventually obtaining access to the clinical trial data through a data‐sharing platform (www.vivli.org), Meid et al. ² developed an alternative model to predict OS based on the conditional average treatment effect of each patient. To increase the interpretability of the contribution of the covariates in predicting the outcome, Meid et al. also investigated the following two additional methods: variable importance and partial dependence plots. ²

During our various prior communications with the primary author of Meid et al. ² to facilitate data access and analysis data set construction, we regret the hurdles that the authors experienced in submitting a research proposal to Vivli (as a simple publication reproduction was not a valid objective in the data request) and constructing an analysis data set from Study Data Tabulation Model (STDM) databases. Dataset building for pharmacometric analyses is not a straightforward process, ³ and the attempt by Meid et al. ² was also complicated by the lack of readily available TGI metrics in the STDM database, as the individual TGI metrics were model‐derived parameters based on modeling of longitudinal tumor size data. In addition, an understanding of TGI models is essential to reproduce the TGI metrics. As alluded to in our publication, a biexponential TGI model ⁴ was used to fit tumor size data using a nonlinear mixed effect modeling approach. ⁵ , ⁶ An example model file as implemented in NONMEM is available as supporting information in a separate publication on applying the TGI‐OS framework to several tumor types ⁷ and could be helpful for deriving TGI metrics from longitudinal tumor size data in general.

The commentary by Meid et al. also underscored the importance of ensuring ML model transparency and interpretability to clinicians. ² Our previously published work used Brier scores to compare the predictive performance among models, and the marginal effect of TGI metrics on OS were shown using a simple two‐dimensional plot. ¹ As mentioned in our publication, to our knowledge, this work that was originally conducted in 2018 was the first work that leveraged ML to characterize the relationship between TGI metrics and OS. As such, the analysis was intended to be exploratory in nature, based on four well‐established ML methodologies (i.e., lasso, boosting, random forest, and kernel machine), and TGI metrics were shown to be the top predictors of OS, regardless of the method used.

We agree with Meid et al. ² regarding the importance of ML model interpretability and also considered alternative, more sophisticated model explanatory methodologies to show the impact of covariates on the outcome. As a result, an extended analysis has been conducted, and the results were presented at the annual meeting of American Society for Clinical Pharmacology and Therapeutics in March 2021. ⁸ In this subsequent work, an XGBoost model identified a parsimonious feature set, which also included TGI metrics among the top predictors, and SHapley Additive exPlanation (SHAP) values ⁹ enabled visualization of the marginal effects from the covariates on OS in a more comprehensive fashion. As shown in Sundrani and Lu, ¹⁰ SHAP analysis delivers not only a ranked order of variable importance but also provides a detailed attribution of variable contribution to the predicted hazard rates at the individual patient level. As a result, the SHAP methodology has become widely adopted for biomedical applications to enable better interpretability of ML models. A full manuscript is planned to follow with details on the methods of the updated analysis and additional results such as predicted OS hazard ratios to reflect the relative treatment benefit as well as their confidence intervals that can be computed from ML models, ¹⁰ which would extend further the interpretability and practical applicability of our work.

The increasing popularity and accessibility of ML approaches in recent years has prompted us to investigate their applicability on leveraging TGI metrics and baseline covariates to predict OS outcome. With the continuous improvement and development of new ML algorithms and the availability of large data sets, additional iterations and updates of our original work are expected in the future.

CONFLICT OF INTEREST

The authors are employees and stockholders of Genentech, Inc.