Developing and Validating Risk Assessment Models of Clinical Outcomes in Modern Oncology

Abstract

The identification of prognostic factors and building of risk assessment prognostic models will continue to play a major role in 21^st century medicine in patient management and decision making. Investigators are often interested in examining the relationship between host, tumor-related, and environmental variables in predicting clinical outcomes. We make a distinction between static and dynamic prediction models. In static prediction modelling, typically variables collected at baseline are utilized in building models. On the other hand, dynamic predictive models leverage the longitudinal data of covariates collected during treatment or follow-up, and hence provide accurate predictions of patients prognoses. To date, most risk assessment models in oncology have been based on static models. In this article, we cover topics that are related to the analysis of prognostic factors, centering on factors that are both relevant at the time of diagnosis or initial treatment and during treatment. We describe the types of risk prediction and then provide a brief description of the penalized regression methods. We then review the state-of-the art methods for dynamic prediction and compare the strengths and the limitations of these methods. While static models will continue to play an important role in oncology, developing and validating dynamic models of clinical outcomes need to take a higher priority. It is apparent that a framework for developing and validating dynamic tools in oncology is still needed. One of the limitations in oncology that modelers may be constrained by the lack of access to the longitudinal biomarker data. It is highly recommended that the next generation of risk assessments consider the longitudinal biomarker data and outcomes so that prediction can be continually updated.

Introduction

Identifying prognostic factors and building risk assessment prognostic models will continue to play a major role in 21^st century medicine in patient management and decision making [1]. Prognostic factors in oncology associate host and tumor variables to clinical outcomes independent of treatment [2]. Gospodarowicz et al., classify factors as either tumor-related, host or environmental factors [3]. Tumor-related factors are variables that are related to the tumor and reveal the tumor biology and pathology (such as size of tumor, lymph node involvement, presence of metastasis and molecular markers (over expression of PTEN gene, presence of androgen receptor variant AR-V7). Host-related factors are associated to the patients’ characteristics, such as age and comorbities. Lastly, environmental factors are external to the patient such as access to healthcare [3]. Prognostic models are increasingly used in the design, conduct, and analysis of clinical trials. For example, in several trials of prostate cancer, randomization was stratified by the predicted survival probability determined by prognostic model of overall survival [4–7]. In the TAILORx trial, OncotypeDx, a 21-gene score that predicts likelihood of recurrence, is used to classify women with breast cancer by their risk group of recurrence [8]. Prognostic factors have also been used for enriching the patient population in trials with targeted therapies. For example, in the TOGA trial, gastric cancer patients with HER-2 positive were randomized to either trastuzumab plus chemotherapy or chemotherapy alone [9].

In this article, we cover topics that are related to the analysis of prognostic factors, centering on factors that are both relevant at the time of diagnosis or initial treatment and during treatment or follow-up. We use the terms prognostic models, risk models, and risk assessments interchangeably. This article is organized in the following way. We first describe the type of risk prediction and then provide a brief description of the penalized regression methods. We then review the state of the art methods for dynamic prediction and compare the strengths and the limitations of these methods. We next offer a concise discussion of validation and metrics for assessing models. Lastly, we present recommendation for the next generation of risk assessments methods to be built in modern oncology.

I. Types of Risk Predictions

Investigators are interested in examining the relationship between host and tumor-related in predicting clinical outcomes (Figure 1A). We make a distinction between static and dynamic prediction. In static prediction modelling, typically variables collected at baseline are utilized in building models. For example, prostate specific antigen (PSA) measurements at baseline are utilized for prediction of recurrence. On the other hand, dynamic predictive models explicitly leverage the longitudinal data of covariates that are collected during treatment or follow-up. In advanced patients with cancer, the disease has substantially evolved and has a heterogeneous presentation within the patient [10]. The inter and intra-patient variability should be taking into account in statistical modeling [11, 12]. Dynamic prediction incorporates time-dependent covariates so that risk prediction would be continually updated with new observations to reflect the patient’s prognosis.

Figure 1A.

Relationship between host and tumor related factors and clinical outcomes. Modified from Seminars in Oncology[2].

We define terminology that is typically used in dynamic risk prediction. The term landmark time is defined as a current time point at which we have data (host, tumor-related variables and outcomes). The term horizon time is defined as a future time point at which we want to predict a time-to-event outcome, such as overall survival. Dynamic predictive models essentially capture the historical information of the longitudinal measurements from the study baseline (t) to the landmark time (u), such that the risk at horizon time can be accurately predicted (Figure 1B).

Figure 1B.

Dynamic risk prediction framework

II. Identification of Prognostic Factors

Several popular strategies exist for identifying prognostic factors in static risk assessment. Standard variable selection approaches such as forward selection, backward selection, etc. with logistic regression for binary endpoints [13], proportional hazards regression for time-to-event endpoints have been applied [14]. Criticism for the stepwise variable selection has been well documented [15, 16]. It is also worth noting that classification trees, such as recursive partitioning for both binary and time to event endpoints [17–20]have been frequently utilized [2, 21–26].

We concentrate on penalized methods that fit and shrink p predictors and in doing so reduce the variance of the coefficient estimates [27]. Thus, these method would improve the accuracy of the model [28] The least absolute shrinkage and selection operator (LASSO) and adaptive LASSO (ALASSO) have been widely employed to develop prognostic models of clinical outcomes [29, 30]. We will briefly describe ridge and penalized methods. Ridge regression minimizes the residual sum of squares function, but it has a caveat as it does not reduce all the coefficients exactly to zero [28]. Let y_i be the response, x_ij is the j^th covariate value (j = 1,2,…,p) corresponding to the i^th individual , β_j is the regression coefficient j^th covariate and λ is a tuning parameter. Similar to ridge regression, the first term in LASSO is the residual sum of squares and LASSO minimizes this function subject to the l₁ penalty (Eq 1):

$$\sum _{i=1}^n{\left(y_i-{\beta }_0-\sum _{j=1}^p{\beta }_j{x}_{ij}\right)}^2+\lambda \sum _{j=1}^p\left|{\beta }_j\right|$$ (1)

A large tuning parameter causes the coefficients estimates to be equal to zero, then the LASSO will have the sparsity property [28]. LASSO is an improvement over ridge regression, although it has the main limitation of tending to select too many unimportant variables and it also performs poorly in situations when p>n [20, 31–33]. Adaptive LASSO has been proposed as an improvement over LASSO in order to overcome the limitation of LASSO [34]. ALASSO minimizes this function (Eq 2):

$$\sum _{i=1}^n{\left(y_i-{\beta }_0-\sum _{j=1}^p{\beta }_j{x}_{ij}\right)}^2+\lambda \sum _{j=1}^p{w}_j\left|{\beta }_j\right|$$ (2)

ALASSO utilizes a weighted penalty term in the L₁ penalty where w = (w₁,w₂,…,w_p) is the weight vector. If $\widehat{\beta }$ is a $\sqrt{n}$-consistent estimator, e.g. $\widehat{\beta }$(OLS), of β (= β₁, β₂,…, β_p), then an appropriate choice of the weight w is $1/\left|\widehat{\beta }\right|$. ALASSO is considered to be an improvement over LASSO as it has consistent variable selection as well as lower prediction error. Consequently, ALASSO tends to select fewer non-zero coefficients than the LASSO despite having smaller prediction error. The ALASSO enjoys the oracle property [30, 34].

Elastic net regression utilizes a combination of l₁ penalty and ridge l₂ penalty and is a compromise between LASSO and ridge regression, and one of its main advantages when p > n is that it retains more than n variables in the model [35]. Hastie et al. provide a thorough comparison of these shrinkage techniques [28].

We applied LASSO and ALASSO from CALGB 90401, a phase III clinical trial in advanced prostate cancer with the overall goal of building a model of overall survival [5]. We had 22 variables that were common between CALGB 90401 and the Enthuse trial (external data set) [36]. Because of missing data in the covariates, we utilized methods to impute them as described by [37]. The regression’s estimates from the proportional hazards model, LASSO and ALASSO are presented in Table 1. We considered LASSO and ALASSO and applied both the Akaike information criteria and the Bayesian information criterion to choose the optimal model of overall survival. LASSO and ALASSO selected eight and nine variables, respectively (Table 1). We determined the ALASSO model as the final optimal model since it included the site of metastases for bone disease. Figure 2 presents the solution path for ALASSO and we observe that LDH greater than upper limit of normal and ECOG performance status were selected early in the l₁ path compared to the other variables. This is followed by visceral disease, alkaline phosphatase, albumin, hemoglobin, pain, bone metastases and then PSA (the BIC stopped at PSA). The final model selected the following prognostic factors: LDH greater than upper limit of normal, ECOG performance status, metastatic site (presence of visceral disease, presence of bone metastases), PSA, alkaline phosphatase, albumin, hemoglobin and analgesic opioid use.

Table 1.

Identified Prognostic Factors by LASSO and ALASSO in the Training Set

Description of variable (variable name)	$\widehat{\beta }$ Cox model	$\widehat{\beta }$ LASSO	$\widehat{\beta }$ ALASSO
Site of Metastases: Bone (DS 2)	0.234	-	0.058
Visceral (DS 3)	0.400	0.056	0.293
Liver (LIVER)	0.013	-	-
Lung (LUNG)	0.199	-	-
Opioid analgesic use (PAIN)	0.136	0.077	0.088
Age in years (AGE)	−0.003	-	-
Body Mass Index (BMI)	−0.021	-	-
Race (Caucasian)	0.034	-	-
ECOG Performance Status (ECOG)	0.278	0.190	0.305
Comorbidity (Comorb)	0.070	-	-
Gleason Score (GLEAST)	0.052	-	-
Prior treatment with radiotherapy (Radio)	0.105	-	-
LDH ≥ ULN (LDH.High)	0.325	0.203	0.335
Albumin (ALB)	−0.133	−0.080	−0.122
Bilirubin (BILI)	−0.017	-	-
Hemoglobin (HGB)	−0.094	−0.085	−0.065
Platelets (PLT)	0.000	-	-
White blood cells (WBC)	0.055	-	-
Alkaline Phosphatase (ALKPHOS)	0.145	0.138	0.145
Aspartate Aminotransferase (AST)	0.025	-	-
Prostate Specific Antigen(PSA)	0.063	0.026	0.015
Testosterone (TESTO)	−0.088	-	-
Training C-index		0.662	0.660
Integrated Time AUC		0.742	0.740

Figure 2.

Solution Path for ALASSO. Printed with permission from Chapman & Hall/CRC [96].

We have focused on variable selection when the number of predictors is small relative to the sample size. There are two main challenges in identifying potential prognostic factors in high dimensional space: computational intensity and a high false discover rate [38, 39]. It is important to point that several pre-screening methods are useful in identifying prognostic features in both the large p, small n problem and in ultra-high dimensional space [31, 32, 39–41].

The concept of variable selection is more challenging in building dynamic models as the main goal is to identify important factors that are related to the longitudinal process and the outcomes. In recent years, a few statistical studies extended the penalized method for the joint modeling of longitudinal data and survival outcomes [42, 43]. The general idea is to postulate the joint likelihood linking the two submodels via latent random variables and to add shrinkage operators to select fixed and random effects. He et al. [42] proposed a variable selection method for joint modeling with a univariate longitudinal outcome, Chen and Wang [43] extended the framework to incorporate multiple longitudinal outcomes. While these methods have not been implemented in oncology, the statistical development paves the way for dynamic risk prediction.

Heterogeneity of treatment effect (HTE) is another important area to consider when building prognostic models. It is the nonrandom, explainable variability in the direction and magnitude of treatment effects for individuals within a population [44]. There are different sources for HTE and it may arise from an underlying causal mechanism, artifacts, measurements or methods [45, 46]. One main goal of the HTE analyses is to predict whether a patient might benefit from a treatment. Traditionally, the Cox regression method has been employed to identify subgroups of patients who may benefit from treatment [14]. Recursive partitioning has also been utilized to identify a subgroup of patients [47]. While these classification tress methods have several advantages, they can create complicated structures and produce overfitting [19, 41, 47]. Other methods, such as permutation methods, SIDES, doubly robust augmented inverse probability weighted estimator, and virtual twins have been developed to take HTE into consideration [45, 48–52]. The personalized prediction can be adequately addressed by the dynamic prediction framework, although it is an area for future research.

III. Estimating Patient-Specific Outcome Prediction and Constructing Risk Groups

Once the final model is chosen, the next step is estimating patient-specific outcome prediction. The estimated survival function at time t is (Eq 3):

$$\widehat{S}\left(t\right)={\left[\widehat{S_0}\left(t\right)\right]}^{\mathit{\text{expR}}}$$ (3)

where R is the estimated linear predictor or risk score for the i^th individual ${\mathit{x}}_{\mathit{i}}{}^{\text{T}}\widehat{\mathit{\beta }}={\sum }_{j=1}^p{\widehat{\beta }}_j{x}_{ij}$ , $\widehat{{\text{S}}_0}\left(t\right)=\exp \left(-{\widehat{\text{H}}}_0\left(t\right)\right)$ is the baseline survival function, and ${\widehat{\text{H}}}_0\left(t\right)$ is the baseline cumulative hazard function. Turning to our prognostic model of overall survival in prostate cancer, we computed a risk score from the estimated regression coefficients and the predicted survival at 24-months using the baseline cumulative hazard. We present the profiles of two patients with different baseline prognostic factors and their predicted overall survival at 24 months (Table 2) [5]. We observe that Patient 2 has a worse predicted survival probability at 24-months than Patient 1.

Table 2.

Profiles of Patient-Specific Predicted Probabilities [5].

Patient Number	Disease Site	Opiate Use	PS ECOG	LDH>ULN	ALB (g/dL)	HGB (g/dL)	ALK (U/L)	PSA (ng/ml)	Predicted Probability at 24 months (95% CI)*
1	Bone	Yes	1	No	4	14	130	90	0.47 (0.42–0.52)
2	Visceral	Yes	1	Yes	3	10	90	75	0.28 (0.19–0.38)

^*Computed by https://www.cancer.duke.edu/Nomogram/firstlinechemotherapy.html [97]

Another important task in static predictive modelling is to construct prognostic risk groups, which can be formed based on their quantiles from the estimated linear predictor. In our overall survival model, we constructed two- and three- prognostic risk groups and determined the cut-points from the training set based on quantiles (33th, 50th and 67th percentiles) [5]. While demonstrating that the overall survival curves differ across the three risk groups are appropriate, it is not a sufficient approach [53]. The optimal strategy would be to compute a measure of discriminative ability of the model.

IV. Methods for Dynamic Modelling

Let T_i denote the true failure time, let f_i (t) denote a set of longitudinal measurements at some time points up to landmark time t. We are interested in predicting the probability that a new patient i* is event-free at least up to horizon time u > t given survival up to t. The conditional probability is defined as (Eq 4)

$${\pi }_{i*}\left(u|t\right)=Pr\left(T_{i*}\ge u|T_{i*}>t,f_{i*}(t),D_n\right),$$ (4)

where D_n denotes a sample from the target population and on which the prediction is based. This formulation enables a dynamic updating scheme. Indeed, if a new measurement for patient i* is observed at time t′ > t, we can update the risk prediction by calculating π_i*(u|t′).

There are two general dynamic risk prediction frameworks: joint modelling and landmark analysis. Joint modeling is comprised of two linked submodels, one for the longitudinal process, one for the time-to-event data, and both are dependent on a common set of latent random variables [54, 55]. In particular, the longitudinal data are usually modeled by a linear mixed effects model. The time-to-event data are modeled by the proportional hazards model with true longitudinal process as time-varying covariates. The Cox regression coefficient quantifies the association between the latent longitudinal process and the hazard rate at time t. The longitudinal process and the event time process are assumed to be independent given the latent random effects, and the joint likelihood can be derived. The model can be estimated either using a frequentist approach that attains maximum likelihood through EM algorithm [54, 56–58] or a Bayesian approach that uses Markov Chain Monte Carlo (MCMC) to obtain posterior means [59–61]. Assuming that the parameters are readily estimated from the observed data, then the conditional probability π_i*(u|t) can be computed. A Monte Carlo estimate of π_i*(u|t) can be obtained by sampling the random effects and the parameters from the corresponding distributions [62].

On the other hand, landmarking [63–66] consists of a series of related Cox regression models, each one is defined at a distinct landmark time t [63–66]. For each pair of {u, t}a separate model is fitted to the individuals who remain in the study and have not yet experienced the event of interest. The baseline hazard can be estimated using the Breslow’s estimator [67]. Then π_i*(u|t) is computed as the survival probability treating the longitudinal observation at time t as a baseline covariate.

V. Comparison between Joint Modeling and Landmarking

Joint modeling and landmarking approaches differ in three aspects: model assumptions, information utilized, and computational complexity. Joint modeling models the dual distribution of the longitudinal process and the failure times, and hence satisfies the consistency conditions for dynamic prediction [68]. Moreover, it exploits the full information of collected data and takes into account of the measurement error of the longitudinal data. This latter point is critical as this implies that joint modelling is a more efficient than landmarking. However, joint modeling needs to specify a correct model for the longitudinal process, and requires stronger assumptions than landmarking. It also takes a considerable effort to estimate the parameters and the computational cost is high as it involves complicated joint distribution and numerical integration. In contrast, landmarking circumvents the aforementioned model assumptions and computational burden, but it is not a comprehensive probability model of the longitudinal process and the failure times, and as such does not satisfy the consistency conditions. Another major shortcoming is that landmarking only considers the patients at risk at the landmark time and does not fully explore the information.

Recent articles have focused on the comparison of joint modeling and landmarking. Rizopoulos et al. [69] compared the two prediction frameworks and proposed a compromise between joint modeling and landmarking. Suresh et al. [70] contrasted joint modeling and landmarking for dynamic risk prediction in the context of a binary longitudinal marker, and also applied these approaches to a prostate cancer study [70]. Ferrer et al. [71] compared the two approaches in case of model misspecification, and they aimed for predicting competing risks of prostate cancer from the PSA history.

Dynamic risk prediction (joint modelling) relies on model assumptions, and hence its performance suffers from model misspecification. Functional data analysis, which is a nonparametric framework, has received considerable attention recently in medical studies as it is a flexible tool for modeling nonlinear longitudinal process [72–74]. In particular, these methods have been incorporated with joint modeling [75] and are exploited to construct dynamic prediction models [61, 76–78].

VI. Examples

There are a few examples in cancer where longitudinal data were modelled with clinical outcomes. We have previously explored whether PSA decline at different landmark times is prognostic for overall survival in patients with advanced prostate cancer [79]. Fontein et al. developed a dynamic model for predicting overall survival in patients with breast cancer [80]. The authors validated the overall survival model using dynamic cross-validated c-index and reported c-index of 0.72, 0.76 and 0.79 at 1-, 2- and 3- years, respectively. Suresh et al. [70] and Ferrer et al. [71] extended the landmarking approach and used prostate cancer studies as the testing bed. In addition, Proust-Lima and Taylor [81] developed a dynamic prognostic tool based on joint modeling using PSA as a biomarker for prostate cancer recurrence. There are other applications of dynamic models in prostate cancer [82–85] and colorectal cancer [86].

We demonstrate the application of dynamic risk prediction utilizing the DATATOP study [87], a clinical trial that was designed to examine the benefits of deprenyl and alpha-tocopherol in slowing the progression of Parkinson’s disease (PD) [88]. Multiple longitudinal biomarkers were collected in the DATATOP study, including Unified PD Rating Scale (UPDRS) total score, modified Hoehn and Yahr (HY) scale, and Schwab and England activities of daily living (SEADL). The biomarkers measured at baseline, 1-month, and every 3-months show strong correlation between the PD symptoms and the terminal event. We applied a joint modeling framework to account for the informative event times. We developed a Bayesian approach for parameter estimation and predicted patients’ future outcome trajectories (Figure 3A) and risk of functional disability (Figure 3B). A more severe Patient 169 with earlier development of functional disability and a less severe Patient 718 were selected to illustrate the patient-specific predictions at a clinically relevant future time points, conditional on their available longitudinal measurements. The predicted UPDRS trajectories were biased with wide uncertainty bands when only baseline measurements were used. Although dynamic prediction for longitudinal trajectories is an advantage of the joint modeling, our major interest was to predict the probability of functional disability after visits at time t given the patient’s longitudinal profiles and the event-free status up to time t. We found a similar pattern in that the risk predictions with higher accuracy were achieved based on more longitudinal observations. For example, based on the longitudinal profiles of the first 12 months, the predicted probabilities in the next 3, 6, 9, and 12 months were 0.21, 0.46, 0.78, 0.97 and 0.02, 0.06, 0.13, 0.30 for Patient 169 (Figure 3B, last plot first row) and Patient 718 (Figure 3B, last plot second row), respectively. Therefore, Patient 169 with a higher risk of functional disability in the next few months might need attentive medical intervention to control disease progression. Meanwhile, the Brier’s scores were 0.216 and 0.108, respectively, which implied a better prediction in terms of calibration given more follow-up data.

Figure 3A.

Predicted UPDRS trajectories for Patient 169 (first row) and Patient 718 (second row). Solid lines are the means of 2000 MCMC samples. Dashed lines are the 2.5% and 97.5% percentiles of the 2000 MCMC samples. The dotted vertical lines represent the landmark time. Printed with permission from Annals of Applied Statistics [88].

Figure 3B.

Predicted conditional failure probabilities for Patient 169 (first row) and Patient 718 (second row). Solid lines are the means of 2000 MCMC samples. Dashed lines are the 2.5% and 97.5% percentiles of the 2000 MCMC samples. Printed with permission from Annals of Applied Statistics [88].

VII. Validation and Assessment of Prognostic Models

The primary goal of a risk assessment is to provide accurate outcome prediction in new patients [16, 89]. Overfitting remains one of the main challenges in model building. Overfitting occurs when a high predictive accuracy is estimated from a model that has been applied to the training set, but has low accuracy when assessed in an independent dataset [90]. A good example of overfitting is provided in Halabi and Owzar [2]

Validating a prognostic model is considered as a critical step after a risk assessment model has been built. There are two types of validation: external and internal [16, 89, 90]. External validation where the frozen model from the training data is applied to an independent data set, is the most rigorous approach. However, often investigators may not have access to external data set. It is important to emphasize that other types of resampling methods, such as cross-validation, bootstrapping and bootstrapping using 0.632+ are considered appropriate approaches of model validations [32, 53, 91, 92].

Assessing the performance of the model is usually conducted by examining the calibration and the discriminative ability of the model. Calibration signifies the extent of the match between the predicted and observed outcome [16]. Often investigators plot the predicted versus the observed outcome. The model would be calibrated if the data fall on a 45-degree line. Using data from two phase III clinical trials, we evaluated the overall survival model for calibration at different landmarks [36, 93]. Figure 4A shows the predicted survival probabilities at 18-months were close to the proportion of patients surviving 18-months using the Enthuse 33 trial. On the other hand, Figure 4B demonstrates that the model was not well-gauged as the observed-predicted data points did not fall on the 45-degree line. The first two points (circles) show that the model over-predicted the proportion of patients surviving 12-months, whereas the third and fifth data points demonstrate that the model under-predicted the proportion of patients surviving 12-months.

Figures 4A-4B.

Calibration of the Overall Survival Model on Two Datasets.

Discrimination describes the ability of a prognostic model to distinguish between patients with and without the outcome of interest [16]. Several metrics are employed to report the performance of a model. A widely used measure is the concordance (c-index), which is the agreement between observed outcomes and prediction. Another widely used measure of predictive accuracy is the time-dependent area under the receiver operating characteristic curve (tAUROC) [94], which can be combined to form an integrated area under the receiver operating characteristic curve for the whole range of the study (iAUROC) [95].

Circling back to our prognostic model in prostate cancer, we evaluated the performance of the model by implementing Uno’s integrated measure for the time dependent area under the receiver operating characteristic curve [95], which were 0.73 (95% CI=0.70–0.73) and 0.76 (95% CI=0.72–0.76) in the testing and validation sets, respectively [5].

It is important to emphasize that dynamic models need also to be assessed for their discriminative ability and calibration. These measures can evaluate the performance of the model at various time points of the prediction. The tAUROC and Brier’s score are widely used for dynamic prediction validation [69, 70, 82, 87]. In the DATATOP study [87], we applied 5-fold cross-validation to evaluate the predictive performance of our framework. Conditional on the longitudinal history up to month 3 and month 12, our model yielded 0.744 and 0.766 tAUROC, respectively, for correctly assigning higher risk of functional disability by month 15 to more severe patients.

Criteria for evaluating risk assessments have been published by the Precision Medicine Core of the American Joint Commission on Cancer [53]), the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis [92] and the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies [91]. Investigators are encouraged to follow these guidelines as more rigorous tools of clinical outcomes would be developed in oncology. It is anticipated that these models will be implemented in both the design and conduct of future trials.

While static models will continue to play an important role in oncology, developing and validating dynamic models of clinical outcomes need to take a higher priority. It is apparent that a framework for developing and validating dynamic tools in oncology is needed. One of the limitations is that modelers may be constrained by the lack of access to the longitudinal biomarker data. It is highly recommended that the next generation of risk assessments take into consideration the longitudinal biomarker data and outcomes so that prediction are updated.

In summary, risk assessment will remain an important research task in precision oncology. We advocate for good clinical practice in risk assessment studies and recommend that investigators design these studies prospectively, in order to obtain accurate individual outcome prediction and prognostic risk group classification. Prognostic studies should begin by asking fundamental questions that are pertinent to patient outcomes, define the primary endpoint a priori, justify the sample size, and describe the appropriate methods for variable selection and model assessment. Lastly, they should be validated using external datasets if available.

Understanding the longitudinal relationship between host and tumor related factors and their impact on clinical outcomes is critical. We expect to see an upsurge in dynamic risk assessments in oncology, and as such, the AJCC and the TRIPOD guidelines should be extended to dynamic predictive modelling. Regardless whether static or dynamic modelling is the primary objective, we envision that this review will serve as a catalyst to bridge gaps in knowledge and motivate investigators to take risk assessment as a discipline by itself. Funding opportunities with the primary goal of building and validating high quality prognostic models will be critical for personalized predictions.

Context Summary.

Key objective:

Building prognostic models will continue to play a role in 21^st century medicine in patient management. We make a distinction between static and dynamic predictive models and provide a review of the state-of-the art methods for dynamic predictive models to promote them for future use.

Knowledge generated:

To date, most risk assessment models in oncology have been based on static prognostic models. Understanding the longitudinal relationship between host and tumor related factors and their impact on clinical outcomes is critical. Regardless whether static or dynamic modelling is the primary objective, we envision that this review will serve as a catalyst to encourage investigators to take risk assessment as a discipline by itself.

Relevance:

We expect to see an upsurge in dynamic risk assessments and as such it is highly recommended that the next generation of models consider the longitudinal data and outcomes so that prediction are updated.