The Visual Predictive Check and Real‐World Data

Abstract

The visual predictive check (VPC) is a standard tool for assessing pharmacometric model suitability, producing visualizations that compare observed data with simulated data such that both model structure and model variability terms can be assessed. However, real‐world data commonly reflect clinical decision‐making that adapts therapy in response to patient data. As a result, VPCs constructed from such data may display apparent model misspecification, even when models are well‐specified. To illustrate possible confounders, four common real‐world therapy adaptation scenarios were simulated: (1) varying dose amount with constant interval, (2) varying dosing interval with constant dose amount, (3) patient dropout after identification of a suitable maintenance dose, and (4) variability in sample timing based on individual pharmacokinetic parameter estimates. For all scenarios, simulated observations were generated using the same model used to produce the VPC simulations, ensuring that the model was unbiased. When only the dose amount varied, the prediction‐corrected VPC (pcVPC) appropriately indicated a well‐specified model. In all other cases, both VPC and pcVPC erroneously suggested a misspecified model. These simulated results, together with our broader experience with complex real‐world data, demonstrate that VPCs can be misleading when applied to real‐world data if dosing interval, sample timing, or sample frequency varies in response to measured drug concentrations. We propose these four scenarios as benchmarks for evaluating the robustness of current and future model diagnostics to real‐world data characteristics.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

The visual predictive check (VPC) is a core diagnostic for evaluating pharmacometric model suitability. Prediction‐corrected VPCs (pcVPCs) were introduced to account for variability associated with dose adaptation driven by covariates or Bayesian estimates.

• WHAT QUESTION DID THIS STUDY ADDRESS?

Real‐world pharmacokinetic data are often shaped by clinicians adapting dosing and sampling schedules in response to patient condition and biomarkers. This study examined how such adaptive data‐generation processes affect interpretation of the VPC and pcVPC.

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

This study proposes four simulated cases grounded in realistic clinical scenarios that demonstrate that even a perfectly specified model can appear misspecified when evaluated using a VPC. These biases persist with the pcVPC and arise from adaptive dosing intervals, sampling schedule modifications, and patient dropout.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

These findings show that VPCs and pcVPCs can produce misleading conclusions when applied to real‐world data shaped by adaptive clinical decision‐making. This work provides context for model developers interpreting apparent bias in VPCs constructed from real‐world data, and proposes four relevant case studies for developers of new model diagnostics to consider.

During routine clinical care, a patient's care team collects information about the patient over time and iteratively adapts treatment in response. For example, in response to a serum drug concentration, a clinician may alter a dosing regimen or request additional drug monitoring. The data generated over the course of routine clinical care therefore depend on preceding observations. This characteristic is typically not the case in clinical trial data, where sample collection, dose size, and dosing frequency are defined prior to the start of therapy. Although generally more complex, real‐world data reflects broader patient populations and typical therapeutic applications, and is therefore a rich source of information for improving patient care. ¹ , ²

One important application of real‐world data is for development and validation of population pharmacokinetic (popPK) models, particularly as these models play a growing role in clinical decision‐making. ³ , ⁴ , ⁵ The visual predictive check (VPC) has been adopted as one of a small set of core tools for diagnosing and evaluating model fit. ⁶ The VPC graphically compares empirically observed percentiles and simulated data distributions, typically stratified over time, providing visual indicators of the suitability of both model structure and variability terms. However, when predictions vary according to some other independent variable, such as dose amount or covariate effects, the VPC can incorrectly display model misspecification. The prediction‐corrected VPC (pcVPC) was developed to address this source of variability and has been broadly adopted for such scenarios. ⁷ , ⁸

While the pcVPC can correct for variability due to dose amount adaptation, clinical practice often results in data sets that contain other types of adaptation. In our experience, these sources of variability produce VPCs and pcVPCs that appear misspecified, even when the model itself is unbiased. In this work, we use simulations to explore four therapy adaptation scenarios, illustrating how a perfectly specified model may nevertheless appear inadequate when evaluated with VPCs or pcVPCs. These scenarios are intentionally simplified to isolate the effect of each type of adaptation on model diagnostics, rather than replicate the full complexity of clinical practice. We propose them as test cases for evaluating new and existing model diagnostics.

METHODS

Cases were designed to minimally extend the dose adaptation trial proposed by Bergstrand et al. ⁷ (“Example 2”). The same popPK model was used: a one‐compartment intravenous model of an arbitrary drug with a clearance of 8.66 L/h (inter‐individual variability (IIV) of 50%), a volume of distribution of 100 L (IIV of 30%, no covariance), and a proportional error of 20%. Each case included 200 patients, and regimens were adapted to achieve a trough concentration of 12.5 U/L with doses rounded to the nearest whole number. Simulations were performed using the PKPDsim R package, v1.5.0. ⁹ Individual PK parameters were estimated with maximum a posteriori (MAP) Bayesian estimation using the PKPDmap R package, v1.1.6. ¹⁰ Adaptive dosing scenarios (Cases 0–2) were simulated using a PKPDsim‐based package designed for adaptive dosing simulations, mipdtrial v0.0.0.02. ¹¹ VPCs, with or without prediction correction, were created using the R package vpc, ¹² v1.2.4, using 1,000 simulations per individual. All code is available at github.com/InsightRX/vpc_and_rwd.

Case 0: Adaptive dosing with variation in dose size

This scenario replicates the adaptive design originally proposed by Bergstrand et al. ⁷ Briefly, simulated patients were given once‐daily bolus doses, beginning with 10,000 U and subsequently adjusted each day based on measured trough concentrations. Troughs were sampled 23.5 hours after each dose.

Case 1: Adaptive dosing with variation in dosing intervals

Case 0 was repeated with the modification that now the dosing interval was adapted while keeping dose quantity fixed. Patients were started on a regimen of 4,000 U every 12 hours. Drug concentrations were measured at 23.5 hours and then every subsequent 48 hours, and the interval was adjusted to achieve the target trough concentration. Allowed intervals were 4, 6, 8, 12, 24, and 48 hours. This design ensured uniform sampling time across all patients and equal numbers of samples per patient, thereby isolating the dosing interval as the only source of variation. The trial was repeated for 10 different random seeds (Electronic Supplemental Material, Appendix S1 ).

Case 2: Adaptive dosing with dropout based on target attainment

Case 0 was repeated with the modification that once the patient achieved a trough within 20% of the target concentration (i.e., 10–15 U/L), sample collection ceased. An extension of this case, inspired by concentration‐dependent cessation of sampling in high dose methotrexate, is supplied in the Electronic Supplementary Material (Appendix S2 ; Case 2b).

Case 3: Sampling scheme based on individual PK estimates

Patients with unexpectedly low clearance (or other unusual observed PK) may be subject to closer monitoring. Here, we repeat Case 0 except without dose adaptation (i.e., all patients received a flat daily dose of 10,000 U). Instead, prior to each dose, all previously collected samples were used to estimate patient clearance using MAP Bayesian estimation. For patients where the estimated clearance was less than 50% of the population value, a peak sample (0.5 hours post‐dose) was collected in addition to the trough sample for the following dosing interval. This design isolates the impact of changes in sampling based on individual PK by removing variation in dosing.

RESULTS

VPCs and pcVPCs for all scenarios are shown in Figure 1 . Because the same model was used to generate observations and perform simulations, these visualizations should indicate no model misspecification. As previously reported, when dose size was adapted in response to drug concentrations (Case 0), the VPC indicated overestimation of the variability in drug concentration (Figure 1a ). This occurred because the observations for each individual converged to the target concentration of 12.5 U/L while expected variability in exposure given these adapted regimens was considerably larger. Prediction correction mostly resolved this issue, showing good alignment between model and data (Figure 1b ).

Figure 1.

Comparison of visual predictive check (VPC) and prediction‐corrected VPC (pcVPC) for each considered case. Case 0: adaptive dosing with dose quantity adjustment, (a) VPC and (b) pcVPC. Case 1: adaptive dosing with dose interval adjustment, (c) VPC and (d) pcVPC. Case 2: adaptive dosing with dropout based on identification of a suitable maintenance dose, (e) VPC and (f) pcVPC. Case 3: Sampling strategy varying based on measured values, (g) VPC, (h) pcVPC. Solid black lines (dashed black lines) indicate the median (5^th and 95^th percentile) of the observed data. Dark blue (light blue) shaded regions indicate the 95% confidence interval for the predicted median (5^th and 95^th percentile) based on simulation.

When dose adjustments were made by varying dosing intervals while holding dose quantity constant (Case 1), the VPC again suggested inflated variability (Figure 1c ), similar to Case 0. However, in this scenario, prediction correction did not resolve the issue (Figure 1d ). The substantial separation between the observed and predicted 5^th and 95^th percentiles would falsely indicate misspecification in model variability terms. Repetition of the trial under varied random seeds did not produce qualitatively different results (Electronic Supplementary Material, Appendix S1 ).

When dose size was adapted until an appropriate maintenance regimen was identified (Case 2), the VPC indicated both variability‐related misspecification (as in Case 0) and poor alignment in central tendency (Figure 1e ). As before, prediction correction showed better alignment in model variability, but with consistent misalignment between the observed and simulated median (Figure 1f ), misleadingly suggesting unexpected accumulation of drug over time.

When dosing was held constant but instead sample collection timing varied based on individual PK (Case 3), the VPC indicated structural misspecification; peak samples appeared to be significantly underestimated by the model (Figure 1g ). Prediction correction did not resolve the issue (Figure 1h ).

DISCUSSION

Real‐world data provides valuable opportunities to develop new pharmacometric models, improve the predictive performance of existing ones, or create specialized models for underrepresented patient groups. ¹³ However, unlike data sets created under the controlled dosing and sampling schemes of clinical trials or through opportunistic sampling from patients with well‐defined dosing and sampling schemes, real‐world data reflect clinician‐driven adaptations that introduce biases, requiring special consideration during model validation and evaluation. Here, we investigated several common scenarios in real‐world clinical practice that produce misleading VPCs, even when the underlying model is perfectly specified. While each of the considered cases examines a single source of variability, multiple such factors often coincide in clinical practice.

Case 1 is commonly encountered for biological therapies, where varying the dosing interval is often preferable to varying the amount. ¹⁴ , ¹⁵ It is also relevant for many antibiotics, including beta‐lactams, aminoglycosides, and vancomycin, for which both interval and dose amounts may vary over the course of a single treatment course.

Case 2, where clinicians cease drug concentration monitoring once a suitable maintenance dose has been identified, and Case 3, where the clinician responds to unexpected individual PK by ordering additional samples, are common for the above drugs as well as other individualized medications. When analyzing data collected during the course of routine model‐informed dose adjustment, we sometimes encounter the paradoxical finding where the more data available for a patient, the worse the performance of the model; the explanation is that a relatively “typical” patient may only provide one or two samples, while clinicians will monitor more challenging or treatment‐resistant patients much more closely.

For densely‐sampled treatment courses, excluding the regions of biased sampling may produce a VPC that more accurately represents model suitability (e.g., Case 2b), however for sparse sampling (e.g., Case 2), this may lead to excessive data loss. If the dropout or censoring mechanism is known exactly, it can be incorporated into the simulations underlying the VPC (Figure S2e,f ). ¹⁶ In clinical practice, however, the dropout mechanism is often unknown, poorly‐defined, or dependent on unobserved factors. For example, after identification of a suitable maintenance dose, sampling may decrease in frequency or may vary according to comorbidities, comedications, and other risk‐factors that are not captured in the available data. Clinical protocols governing these decisions may differ substantially between institutions and may not always be followed in practice. In these cases, modeling the sampling bias may be as complex as building the popPK model itself, outweighing the VPC's utility as a diagnostic tool. ¹⁷

A related example is when additional samples are collected based on some patient characteristic; for example, patients with elevated liver enzymes may be monitored more closely. A practical solution may be to include these additional samples in model‐building but to exclude them from the VPC, due to the bias they introduce. However, again this exclusion criteria require knowledge of which patients are subject to closer monitoring and for what reason, which is not always evident from the health record.

Based on these results, we recommend against the use of the VPC and the pcVPC when sampling strategies or dosing intervals vary based on the individual response variable of interest, as summarized in Figure 2 . The pcVPC remains a powerful diagnostic when the sampling scheme and dosing interval predominantly remain unrelated to individual PK, or when sampling, dropout, and dosing adaptation decisions can be modeled with a high degree of confidence.

Figure 2.

Flowchart describing proposed decision‐making process for the use of visual predictive checks (VPCs) and prediction‐corrected VPCs (pcVPCs) when using real‐world data for pharmacometric model development and evaluation. For cases where the VPC and pcVPC may not serve as meaningful diagnostics, recommendations are further divided by intended use: model‐informed precision dosing (MIPD) or more general applications.

When interpreting model diagnostics, it is important to consider both the context in which the data were collected and the model's intended use. For applications like model‐informed precision dosing, the objective is to accurately predict future drug exposure for an individual patient to ultimately improve treatment success. Tools such as perl‐speaks‐NONMEM proseval and mipdeval, ¹⁸ , ¹⁹ which iteratively perform MAP Bayesian predictions to mimic model performance in clinical settings, may provide a clearer picture of how a model will perform in practice than a VPC, which does not assess the model's ability to predict the future.

Although we did not investigate the impact of adaptive intervals and sampling designs on other model diagnostics, many simulation‐based tools are also likely influenced by the biases inherent to real‐world data. Like the VPC and pcVPC, normalized prediction distribution error (NPDE) analyses simulate data without incorporating the relationship between the dependent variable and dosing or sampling decisions, and therefore are susceptible to the same biases described here. The variability‐aligned covariate harmonized‐effects and time‐transformation equivalent (VACHETTE) is a visualization method that applies covariate‐driven transformations to data, ²⁰ but since it does not control for sampling and dosing designs adapted to individual PK, it is therefore unlikely to resolve the misspecification artifacts presented here. Reference‐corrected VPCs (rcVPCs) have been introduced as a more interpretable alternative to pcVPCs ²¹ and may help address challenges arising from adaptive changes in dosing intervals. However, extending rcVPCs to this use case is not straightforward and warrants further investigation. The four scenarios described here may serve as a useful benchmark for evaluating the robustness of these and other diagnostics (e.g., basic goodness‐of‐fit plots) to real‐world data characteristics.

Real‐world model‐informed precision dosing may involve additional alterations in sample timing and dose adjustment not considered here. For example, optimal sampling algorithms could further complicate real‐world data sets; based on individual PK estimates, clinical decision support platforms could recommend maximally informative timings for subsequent measurements, introducing yet another individual PK‐driven bias into the dataset. We also did not directly address adaptation based on pharmacodynamic endpoints, such as toxicity‐driven dose reduction in oncology or dose titration to achieve a target clinical response. We believe these practices would result in issues analogous to those described here, as the same feedback between observed response and clinical decision‐making applies. The four scenarios presented here may therefore serve as useful archetypes for evaluating model diagnostics in pharmacometric contexts beyond PK.

In routine clinical care, data and clinical decision‐making form a tightly‐coupled system: the data collected inform clinical decisions, the decisions made impact the data collected, and this feedback loop may be mediated by predictive models. The clinician's patient‐centered role in data generation complicates the interpretation of traditional diagnostics such as the VPC. Recognizing this interdependence is essential for appropriately evaluating model performance in real‐world MIPD applications.

FUNDING

No funding was received for this work.

CONFLICT OF INTEREST

The authors declared no competing interests for this work.

AUTHOR CONTRIBUTIONS

J.H.H., M.B., and R.J.K. designed the research. J.H.H. performed the research. J.H.H., M.B., and R.J.K. analyzed the data. J.H.H. wrote the manuscript.

Supporting information

Appendix S1.