Using forest plots to interpret covariate effects in pharmacometric models

Abstract

The inclusion of covariates in pharmacometric models is important due to their ability to explain variability in drug exposure and response. Clear communication of the impact of covariates is needed to support informed decision making in clinical practice and in drug development. However, effectively conveying these effects to key stakeholders and decision makers can be challenging. Forest plots have been proposed to meet these communication needs. However, forest plots for the illustration of covariate effects in pharmacometrics are complex combinations of model predictions, uncertainty estimates, tabulated results, and reference lines and intervals. The purpose of this tutorial is to outline the aspects that influence the interpretation of forest plots, recommend best practices, and offer specific guidance for a clear and transparent communication of covariate effects.

INTRODUCTION

The inclusion of covariates in pharmacometric models is important due to their ability to explain variability in drug exposure and response. Examples of covariates include patient‐specific characteristics, such as demographic factors, concomitant medications, disease attributes, and genetic variations. This facilitates a more precise and individualized estimation of pharmacokinetic (PK) and pharmacodynamic parameters, thereby enhancing the accuracy and applicability of the models.

Clear communication of the impact of covariates is needed to support informed decision making in clinical practice and in drug development. However, effectively conveying these effects to key stakeholders and decision makers can be challenging due to the statistical relationships involved, and the need to strike a balance between simplicity and comprehensiveness creates a risk of misinterpretation or oversimplification of the results.

Forest plots have been proposed as a way to meet these communication needs. ¹ Such plots have been used extensively in meta analyses and the name relates to the fact that they often include a “forest” of lines. ² Forest plots offer an efficient means of conveying the impact of covariates in pharmacometric models, presenting visually appealing and intuitively understandable representations of the expected changes in relevant metrics (such as drug exposure) at specific values of the covariates. They also provide measures of uncertainty around these expectations and can incorporate reference lines and intervals of clinical relevance, inviting the viewers to make inferences about statistical significance and clinical relevance.

However, forest plots for the illustration of covariate effects in pharmacometrics are complex combinations of model predictions, uncertainty estimates, tabulated results, and reference lines and intervals (Figure 1). Such plots are not trivial to assemble but fortunately there are currently several R packages available to support the analyst with this. For example, in a recent tutorial, Marier et al. ³ describe how simulation‐based methodologies can be used to evaluate covariate effects along with the introduction of the coveffectsplots R package ⁴ to create forest plots. Another example is the pmforest R package. ⁵ The forest plots in the current paper are created using the R package PMXForest ⁶ and an example of how it can be used is provided in Appendix S1.

FIGURE 1.

An example Forest plot for the covariate impact on AUC in a hypothetical analysis. The different components of the plot are indicated by the squared letters and the corresponding explanations are provided to the right. AUC, area under the curve; CI, confidence interval; CRCL, creatinine clearance; EM, extensive metabolizer; IM, intermediate metabolizer; NCI, National Cancer Institute; PM, poor metabolizer; UM, ultra‐fast metabolizer.

However, despite the technical challenges involved in constructing forest plots for covariate effects, the true difficulty lies in their interpretation. Although forest plots may be regarded as standardized and effective tools for communicating covariate effects, they do not eliminate underlying complexities. It is essential to acknowledge the impact of choices and assumptions made by the analyst during the establishment of the covariate model, derivation of uncertainty, and selection of input parameters (e.g., covariate values, reference subject, and reference area width) for the forest plot. It is also important to recognize the inherent limitations imposed by the design of the underlying data, which affects the relevance of the inferences that can be drawn from these plots.

These above types of challenges and how they should be managed in practice were beyond the scope of the tutorial by Marier et al. ³ and, to our knowledge, have not been comprehensively addressed elsewhere either. Thus, the purpose of this tutorial is to outline the aspects that influence the interpretation of forest plots. As will be obvious, there is a need for a dual responsibility between the creator and viewers of forest plots to avoid misuse and misinterpretation. The tutorial will also provide best practice advice, checklists for both the creator and viewer of forest plots, and will provide a framework for structured captions for forest plots.

The tutorial will first go through the components of a forest plot and explain under what circumstances unambiguous conclusions can be drawn and how the learnings from a forest plot should be formulated. This will be followed by a detailed discussion of the various choices and assumptions that influence the creation of forest plots, from study design and conduct to choices and assumptions made during the analysis, to the choices made during the creation of the plot. The last part of the tutorial will provide concrete advice and best practices for creating, communicating, and evaluating forest plots.

COMPONENTS OF A FOREST PLOT AND BASIC INTERPRETATION

An example forest plot for the covariate impact on the area under the curve (AUC) from a hypothetical analysis is shown in Figure 1 together with explanations of the different components in the plot.

Most of these components can be understood, at least superficially, without further details, but for the sake of interpretation it is worth pointing out two important features: the reference line (I in Figure 1), which represents the value to which we look for a difference from and the reference area (H in Figure 1), which indicates the size of the covariate effect that is considered clinically irrelevant.

The interpretation of a forest plot depends on the location of the point estimates (D in Figure 1) in relation to the reference area and on the (potential) overlap of the confidence interval (E in Figure 1) with the reference line and the reference area. There are in total seven possible combinations and these are indicated in Figure 2. Ultimately, the desire is to learn if the effect of the covariate is different from the reference value or not, and if any difference is clinically relevant or not. There are only two out of the seven combinations that support a clear and unambiguous conclusion (4 and 7 in Figure 2). The conclusion that can be drawn for each of the seven possible outcomes are detailed in Figure 2.

FIGURE 2.

An example forest plot for the covariate impact on AUC in a hypothetical analysis (same as in Figure 1). The rectangles in the left panel indicate the seven possible outcomes, as described to the right. The rectangles on the left and text on the right are colored to indicate each of the seven cases. The two cases that are conclusive (4 and 7) are indicated with one or two asterisks. AUC, area under the curve; CL, clearance; EM, extensive metabolizer; IM, intermediate metabolizer; PM, poor metabolizer; UM, ultra‐fast metabolizer; WT, weight.

The problem is that the point estimates, the confidence interval, the reference line, and reference area are influenced by accumulated effects of study design and conduct aspects, data analysis choices and assumptions, and choices made when creating the forest plot, whereas these various choices and assumptions are not apparent from the plot itself.

This is illustrated in Figure 3 where four different forest plots for clearance (CL) are shown. All panels are based on the same (synthetic) source data. Panels a and b are based on the same model, but the conclusions are different. From panel a, it would be tempting to state that CYP2D2 poor metabolizer (PM), intermediate metabolizer (IM), and ultra‐fast metabolizer subjects would have clearance (CL) values that are different from the CL in CYP2D6 extensive metabolizer (EM) subjects in a clinically relevant way. Similarly, panel a gives the impression that it is likely that body weight should be considered when selecting the dose, and that there may be a difference depending on if the drug is administered with or without food. Panel b, on the other hand, gives a different picture. From this panel, it is doubtful if CYP2D6 IM subjects have a CL that is different from CYP2D6 EM subjects in a clinically meaningful way, body weight does not seem to predict clinically meaningful differences in CL, nor does drug intake with or without food. The apparent differences depends on the differences in reference area width (panel a uses a relative change of 0.8–1.25 whereas panel b uses an absolute difference in CL between 10–20 L/h), different values of body weight used for predicting the covariate effect (panel a uses the 10th and 90th percentile of the observed body weight whereas panel b uses weight values that are unrelated to the observed data but presumably interesting from a communication perspective) and that the reference line is based on a fasted state instead of a fed state. These differences are not obvious at a quick glance and the difference in reference area width is hard to spot because the parameter value is given on a relative scale.

FIGURE 3.

Forest plots for the covariate effects on CL. The same source data was used for all four panels. Panels (a) and (b) were generated from the same model but using different input to the Forest plot. Panels (c) and (d) were generated using different models but using the same input to the Forest plot. See text for a detailed discussion. CL, clearance; CRCL, creatinine clearance; EM, extensive metabolizer; FREM, full random effects model; IM, intermediate metabolizer; PM, poor metabolizer; SCM, stepwise covariate method; UM, ultra‐fast metabolizer; WT, weight.

Panels c and d in Figure 3 give another example of when it is not obvious from the plot what lies behind it. Panel c visualizes the statistically significant covariates on CL from a stepwise covariate method (SCM) ⁷ , ⁸ analysis, whereas panel d visualizes a subset of the covariates included in a full covariate model analysis. ⁹ Neither of the plots gives sufficient information to allow the viewer to evaluate the results properly (for example, if there were other covariates included in the analysis and if covariates were considered on other parameters) and visually the two plots are very similar despite the difference in covariate modeling methodology.

Panels a and b in Figure 3 illustrate that choices made during the creation of the forest plot can influence the conclusions, whereas panels c and d show that different and potentially important data analysis choices may not be obvious from the resulting forest plots. What is not illustrated in Figure 3, but that also may have been different between the panels without changing the visual appearance of the plots, are differences in study design and conduct. For example, the model underlying panel a may have been based on data from patients, whereas the model used for panel b may have been derived using data from healthy volunteers. There is no way to tell from the forest plot itself.

It is necessary that both the creator and viewers of forest plots are aware of the impact of the underlying assumptions and choices. Transparent communication by the creators and informed judgment by viewers is essential to avoid misinterpretations and to maximize the usefulness of forest plots.

FACTORS THAT INFLUENCE THE INTERPRETATION OF FOREST PLOTS

Study design and conduct choices

Forest plots are based on model predictions, and the properties and limitations of models are dependent on the data they are derived from. This means that the design of the studies the data come from as well as how these studies were conducted will influence any forest plots used to illustrate the leanings from the analysis of the data.

There are many factors that should be kept in mind when evaluating the appropriateness of the available data for the target inference. For instance, does the study design (e.g., number of subjects and observations, covariate ranges, etc.) offer sufficient detail for the planned analysis? Is the selected study population relevant for the intended inference? Are all the patient characteristics available for the analysis and were they collected with an appropriate level of exactness and completeness? Is the data a result of studies designed to investigate the impact of the covariates or are the studies observational with respect to the covariates? Are there potential biases introduced by non‐random patient dropouts or other data imbalances created by the study design and conduct? ¹⁰

These, and other aspects of similar nature, are often beyond the control of the pharmacometrician doing the data analysis, but it is still important to keep them in mind and evaluate if they have an undue impact or create restrictions on the model and, therefore, ultimately on the forest plots.

Data analysis choices and assumptions

Choice of covariate modeling method

The inclusion of covariates in pharmacometric models can be done in different ways ¹⁰ and the choice of method will influence on which covariates there will be quantifiable information available to the forest plot, and, to some extent, the quality of the quantifiable information.

Stepwise methods, like the SCM, ⁸ add and remove parameter‐covariate relations to the model sequentially based (typically) on statistical significance. This means that it is only the parameter‐covariate relations that meet these criteria that will have a quantifiable impact of the parameter in the forest plot. A potential problem with these methods is selection bias, which means that the estimated coefficients are biased away from zero (i.e., that the impact of a covariate may appear larger than it is). In practice, however, this is only relevant in situations when the data are uninformative for the parameter‐covariate relation and, consequently, that the power to detect the covariate is low. ¹¹

With the full covariate model (FCM) method ⁹ , ¹² the parameter‐covariate relations of interest are prespecified and are included in the model irrespective of statistical significance. However, to ensure (nearly) independent covariate effects (see below), covariates must be excluded from the prespecified set of relations if their correlation to other covariates are larger than a certain threshold (suggested to be a correlation < |0.3|). ⁹ This means that it is unlikely that all the originally prespecified parameter‐covariate relations will have a quantifiable impact in the forest plot.

A third alternative is the full random effects model (FREM) ¹² approach. With this method covariates are also prespecified but, in contrast to the FCM method, no covariates must be excluded due to correlations with other covariates. This means that all the originally prespecified covariates will have a quantifiable impact in the forest plot. Another difference is that with FREM all covariates will implicitly be included on all parameters, which removes the need to think in terms of and choosing the parameter‐covariate combinations when setting up the search scope (SCM) or prespecify the covariates to include in the model (FCM).

The above only discusses three possible covariate modeling methods. There are other methods available as well and more are likely to be developed in the future. ¹⁰ The point is, though, that the ability to make quantitative statements about a covariate in a forest plot is dependent on the choice of covariate modeling method. Consequently, viewers of a forest plot must know which method was used and understand its properties to draw informed conclusions.

Handling of correlated covariates

Correlations between the covariates in a model can have a large impact on the interpretation of a forest plot.

Forest plots visualize the marginal covariate effects (i.e., the impact of the outcome parameter when the value of one covariate changes and the rest are kept at their reference value). This requires that the covariates are uncorrelated (or independent) to avoid an under prediction of the impact of the covariate. Imagine two covariates that are so highly correlated that they are almost identical. Because the covariates are so similar, they will share the available information evenly and each covariate will explain about 50% of the variability and the estimated coefficients will be about half the size they would be compared to if the coefficient was estimated by itself without the correlated covariate. Less correlated covariates will share the available information as a function of the size of the correlation. (Note, though, that this may limit the causal inference from the model. Please consult Rogers et al. ¹³ for a more detailed discussion.)

In practice, this means that it is not appropriate to change only one out of two (or more) correlated covariates in the model to obtain the marginal effects. One option is to condition on one covariate and then change the other, for example, to create separate forest plots for men and women to explore the impact of body weight.

It is the creator of the forest plot's responsibility to communicate the covariate correlation structure in the data (plot or table) and to ensure that correlations are appropriately handled.

Having independent interpretable covariate effects is why it is important with the FCM method to keep pairwise correlations below a certain threshold. ⁹

With stepwise methods there is somewhat of an automatic filter for correlated covariates. Once a covariate has entered the model, it is unlikely that a correlated covariate (i.e., a predictor for the same source of variability), will be selected in a subsequent step, and the actual covariate that is selected will be determined by what best fits the data. However, there are no guarantees, so it is important to be aware of the correlation structure of the covariates in the model to be able to interpret the forest plot.

A useful feature of FREM models is that they can be reformulated to any smaller covariate model that is a subset of the full covariate model, with correctly sized coefficients and parameter uncertainties, ¹² without having to re‐estimate the parameters. This makes it possible to efficiently create univariable covariate models (i.e., models that only include one covariate), and use these to visualize the marginal effects in the forest plots without having to worry about correlations. This is equivalent to estimate the impact of each covariate individually in separate models but without the need to fit the model multiple times.

Correlations can also be used to glean information about covariates that are not part of the model. This is used by empirical forest plots, which offer a way to visualize the potential effect of covariates that were not included in the model and the approach relies on the correlations between the covariates to do this. Further details on empirical forest plots and how they can be generated is provided in the Appendix S2.

Selection of the parameter‐covariate relations to consider in the analysis

One of the most fundamental choices in covariate modeling is whether a covariate is included in the analysis or not. Even if information about the covariate was collected in the study, it can be decided to not include it in the covariate analysis.

A similar choice applies to the choice of parameters to consider in the covariate analysis. Sometimes it is decided to limit the covariate search to a subset of the model parameters, for example CL. This is typically done to reduce the time required for the covariate modeling, or perhaps based on a pure lack of interest in the other parameters. However, the parameters in a model are not independent so if a covariate is included on just one parameter and the covariate also (truly) have an impact on another parameter, it is likely that the estimation of the coefficient on the parameter will be influenced by the lack of the covariate on the second parameter (for example, including body weight on CL while not including it on volume of distribution [V]). It is hard to predict the magnitude and direction of this omission bias, but it will impact the estimated model coefficients and therefore the forest plots.

The choice of covariate modeling method also has a bearing on this. In an SCM analysis, it is necessary to explicitly state the search scope, that is, the parameter‐relations to be considered and there is a direct link between the size of the search scope and overall runtime for the covariate search. Excluding parameter‐covariate relations that are perceived as less interesting is a way to manage the runtimes. Because the FCM method also has a parameter‐covariate focus in the prespecification of the covariate model, excluding some combinations may be a way to decrease the model runtime and/or increase model stability. Regardless of the reason for excluding some parameter‐covariate combinations may lead to the omission bias problem described above.

In the FREM method, there is no need to exclude covariates due to correlations, so no subjective choices are needed in that regard, and, in principle, it is not possible to exclude the covariate on one or a few parameters while including it on others. However, the FREM is not without subjective choices. It is possible to include covariates on parameters prior to the actual FREM modeling, and it is not necessary to involve all parameters in the FREM specification, ¹² so there is still a risk for omission bias. As with all methods, there is always the (subjective) choice to not include a covariate in the analysis even if it is available.

As a viewer of forest plots, it is worth keeping in mind that the lack of presence of a covariate in a forest plot is not the same as a lack of impact of the covariate. It is the forest plot creator's responsibility to clearly communicate which covariates that were included in the analysis and on what model parameter.

Covariate model parameterization

An obvious choice during model development is the covariate model parameterization. Three common choices are the linear (Equation 1), the power (Equation 2), and the exponential (Equation 3) models.

$$P_i={\theta }_1\left(1+{\theta }_2\left({\mathrm{Cov}}_i-\overline{\mathrm{Cov}}\right)\right)$$ (1)

$$P_i={\theta }_1{\left(\frac{{\mathrm{Cov}}_i}{\overline{\mathrm{Cov}}}\right)}^{{\theta }_2}$$ (2)

$$P_i={\theta }_1{e}^{\left({\theta }_2\left({\mathrm{Cov}}_i-\overline{\mathrm{Cov}}\right)\right)}$$ (3)

where $P_i$ is the individual parameter prediction based on the covariate $\mathrm{COV}$, θ ₁ is the typical value of the parameter, ${\theta }_2$ is the coefficient that governs the size of the covariate effect, and $\overline{\mathrm{COV}}$ is the covariate value of the typical subject (often the median).

Figure S3 in Appendix S3, demonstrates the comparison among three models fitted to artificial data. The artificial data were generated using a power model relationship with varying weights (WTs) from 50 to 140 kg. The figure indicates that the models perform similarly for most practical applications within the 10th and 80th percentiles. However, differences start to appear for extreme WT values. Particularly, the models show notable divergence when extrapolating WT values outside the observed range. Although the linear and power models exhibit consistent behavior compared to the exponential model in this example, it is not possible to generalize these relative differences as the model's performances will vary based on the true, underlying, relationship. The implication is that careful consideration of the covariate model's parameterization is important when using it to extrapolate beyond the observed covariate range. Further details can be found in Appendix S3.

Method of handling missing covariates

The way missing covariate data is handled can significantly influence the estimated covariate effects. ¹⁴ For minor and balanced instances of missing covariates, using mean or median imputation may be suitable. This method replaces the missing value with the mean or median of the observed values, thereby preserving the overall distribution.

However, when the fraction of missing covariates is large or the pattern of missingness is imbalanced, the situation is more complicated. This may, for example, occur when the importance of a biomarker emerges during a drug development program so that only the later trials collect information about it. This can lead to large fractions of missing covariate data, with potential confounding aspects, for example, if the studies in which the covariate was collected only contains patients and the studies in which it was not collected only contains healthy volunteers. Conducting a sensitivity analysis can provide insight into the implications of missing data on the robustness of the findings. This is particularly important when a large amount of missing data is replaced using mean imputation, as this could result in a bias toward zero in the covariate coefficient, effectively diminishing the perceived impact of that covariate.

In scenarios with a substantial degree of missing data, it will be necessary to handle it in other ways than the (within pharmacometrics) traditional mean or median imputation. One possibility is to use a multiple imputation strategy. ¹⁴ With this approach, multiple data sets are created, in which the missing data is imputed based on the non‐missing information. The analysis is then repeated for each of the imputed data sets, and, in the context of forest plots, the results would be combined into a single set of covariate coefficients and uncertainties that would be used to generate the input to the forest plot.

Another strategy to use when there is a high degree of missing data is FREM. The FREM handles missing covariates implicitly without a need for any imputation or analysis across multiple data sets and has been shown to be able to handle large fractions of missing data much better than mean or median imputation. ¹⁵

Given the potentially large impact missing covariate data may have on the interpretation of forest plots it is important that the receiver of the forest plots is provided with the means of taking this aspect into account. At the very least, the information about the degree of missing covariate data, possibly stratified on important design variables, such as study, should be provided.

Choice of method for uncertainty estimation

A central component of the interpretation of forest plots are the confidence intervals around the predicted covariate effect. These confidence intervals reflect the uncertainties in the primary parameter estimates from the model, for example, CL and V in a PK model.

Parameter uncertainty can be obtained in many different ways, for example, based on the variance–covariance matrix of the estimates from a software like NONMEM, parametric or nonparametric bootstrap analysis, ¹⁶ sampling importance resampling (SIR), ¹⁷ and a Bayesian posterior. They all provide estimates of uncertainty but will have different properties. One main difference between the methods is if the uncertainty is based on normality assumptions (which will result in symmetric uncertainty for additive covariate effects) or not. Examples of the first category are the variance–covariance matrix‐based methods and parametric bootstraps. Methods that do not rely on normality assumptions are the nonparametric bootstrap and SIR.

Another challenge lies in the translation of the combined uncertainties of the covariate coefficients involved into the confidence intervals of the (potentially secondary or derived) metrics of interest, for example, AUC and the maximum concentration (C _max). A direct approximate method to use is the delta rule ¹⁸ but a more general approach is to use numerical methods based on multivariate sampling from the variance–covariance matrix. A unified framework for computing the uncertainty for forest plots based on this principle is provided in the Appendix S4.

The problem, from a communications perspective, is that it is not possible to know what method that was used to estimate the uncertainty (and thereby knowing its underlying properties) by just looking at the confidence intervals in a forest plot. As pointed out above, some methods will depend on normality assumptions whereas others do not. The appearance of the confidence intervals, on the other hand, will depend on the covariate model parameterization in combination with the distribution of the parameter uncertainty. For example, it is only in the specific case when the covariate effects (or transformations of the covariate effects) using a symmetrical uncertainty distribution that the confidence intervals will appear symmetric. In most other cases, the confidence intervals will be asymmetric. The method of deriving the uncertainty must be communicated in some other way than the visual appearance of the confidence interval, for example, in the method section of the paper or report.

Choices made during the creation of the forest plot

Choice of parameter to visualize

A fundamental choice in creating a forest plot is the parameter to visualize the covariate impacts for. The impact on any primary or secondary parameter can be visualized, but from a decision‐making point of view it is most likely that one or more secondary parameters (e.g., AUC and C _max in a PK analysis) are most relevant.

Choice of covariates to visualize

Just as it is possible to exclude available covariates from the analysis (see above), it is also possible to choose to not include covariates in the forest plot even if they were included in the analysis. Figure 3d provides an example of when only a subset of the covariates included in the analysis is shown in the forest plot. There are many reasons for not including all covariates that were part of the analysis in the forest plot. Perhaps there is a desire for simplification in case there are many covariates included in the model, or to only visualize the significant covariates, or perhaps some covariates are not regarded as relevant for future studies or for the current question. Regardless of the reasons for not including all covariates, it is important that the viewer is informed that they are looking at a subset of the covariates that were part of the analysis, especially if there are non‐ignorable correlations between the covariates in the model that may influence the interpretation of the forest plot (see above).

To show only a subset of the available covariates in a forest plot can be helpful when addressing specific questions. The potential distraction of irrelevant covariates is avoided, and the focus can be on the question at hand. However, if the purpose of a forest plot is to illustrate the influence of covariates in the “final” model, then it is better to include all covariates that were part of the analysis, regardless of statistical significance (in the case of an SCM‐based covariate analysis), as is shown in Figure 4. The left panel shows all covariates, regardless of statistical significance, that was included in an SCM analysis, whereas the right panel shows the FREM model for the same data set. The benefit with including all covariates is that it becomes directly clear to the viewer which covariates were considered in the analysis and how they impact the parameter. When forest plots are used in this descriptive manner it is best if the covariate values on the y‐axis are based on the observed data to minimize the impact of subjective choices.

FIGURE 4.

Forest plots for CL based on an SCM analysis (left) and a FREM analysis (right). All covariates included in the analysis are shown, even if they were not statistically significant (SCM). CL, clearance; CRCL, creatinine clearance; FREM, full random effects model; SCM, stepwise covariate method.

In a modeling report, the best place to put the descriptive forest plots is in the Results section, whereas the more subjective forest plots should be presented in a section in which the context makes it clear that they visualize interpretations and hypotheses.

Choice of the covariate values to evaluate

The choice of covariate values to use for predicting the impact of the covariate is very important. For categorical covariates, the choice is obvious – simply use all unique values, for example, predict for all CYP2D6 genotype categories as in Figure 1. Continuous covariates are more complicated. With a wide separation of the values used for prediction, the impact of the covariate will (visually) appear larger than if the covariate values are less separated. On the other hand, to create comparability with predictions based on categorical covariates, which represents the extremes in the distribution, it is reasonable to use values from the edges of continuous covariate distributions as well.

Another aspect to consider is learning and communicating about yet unstudied situations (e.g., extrapolations beyond the observed covariate range). There is nothing that prohibits the use of covariate values outside the observed range or combine covariate values to predict unobserved combinations of covariates, for example, the elderly and CYP2D6 PM women. Some of the challenges of extrapolations have been mentioned above but a further danger arises if a forest plot mix observed and extrapolated covariate values. Visually it is not possible to see that some covariate values are inside the observed range while others are selected to illustrate interesting what‐if scenarios.

We suggest using one of two approaches for selecting covariate values for a specific forest plot. The first approach is to select the covariate values objectively based on the observed covariate distribution, for example, the 10th and 90th percentiles. This avoids extrapolation, reflects the observed covariate distribution and is robust because the predictions will not depend on the most extreme subjects. One thing to be aware of, though, is situations where the observed covariate data are subject to artificial limits, for example, upper or lower detection limits. An example is shown in Figure 1, where the values of creatinine clearance (CRCL) in the analysis data set was used to compute the 10th and 90th percentile. The problem with this is that CRCL was capped at 150 mL/min to manage unrealistically high values (a problem with CRCL computed according to the Cockroft‐Gault formula). Because this capping applied to more than 10% of the subjects, the 90th percentile covariate value at which CRCL was artificial and not based on a real observation. This may be OK, but it should be clearly communicated to the receiver of the forest plot that some predictions are based on censored covariate values.

The objective approach described above is suitable for a descriptive communication of the covariates in a model but may not be completely relevant to the questions being asked of the model.

The second approach for selecting covariate values is to pick values that address the relevant questions or relevant situations, for example, to predict the expected exposure at relevant values of renal function, even if these values were not part of the observed covariate distribution. An example is CRCL in Figure 5. In this plot, the values for the covariate were selected to reflect different kidney function classes rather than the observed distribution of CRCL values. The possibility to generate outcomes for any value of a covariate is of course one of the great strengths of a model‐based analysis and can be used to produce very useful and informative representations of the current knowledge of the impact of covariates. However, it is important to keep extrapolation in mind. The 30 mL/min CRCL level in Figure 5 is outside the observed range in the underlying data set, which may or may not be acceptable. The important thing is that any extrapolations of this sort must be clearly communicated to the receiver of the forest plot. (In Figure 5, the extrapolation is indicated by an asterisk.)

FIGURE 5.

Forest plot of the impact of CRCL on CL. The CRCL values were selected independently of the observed distribution to reflect kidney function classes. The asterisk at CRCL 30 mL/min indicates that this value is outside the observed data range. CL, clearance; CRCL, creatinine clearance; EM, extensive metabolizer; IM, intermediate metabolizer; PM, poor metabolizer; SCM, stepwise covariate method; UM, ultra‐fast metabolizer; WT, weight.

Both approaches have value, but the important thing is to not mix them in the same forest plot (or, as in Figure 5, make it transparent to the viewer that a mix of values are used). It is better to use different forest plots for the different purposes, one for the descriptive communication of the covariate model and another forest plot to answer what‐if questions and be clear in the description of the plots how the covariate values were selected (see below).

Choice of the reference line/reference covariates

A central part in the interpretation of a forest plot is if the confidence interval around a predicted covariate effect overlaps the reference case, indicated by the reference line. If the confidence interval overlaps the reference line, it indicates that the covariate is not statistically different from the reference case. Therefore, it is important that the reference case is relevant to the situation.

The choice of reference case can be made based on the available data, for example, as the “typical individual” in the model, which is often defined as an individual with the median value of all covariates. This offers a degree of objectiveness to the choice of reference case but may not be sufficiently easy to communicate (in the case of many covariates) or may not reflect what is perceived as a relevant comparison subject. Alternatively, the reference case may be defined independently of the data to reflect what is a relevant comparison case, for example, a non‐smoking male patient that weighs 75 kg, even if that combination of covariate is far from the typical patient in the data set. However, from a forest plot perspective, it is not necessarily clear how the reference case was defined. It is up to the creator of the plot to provide this information and up to the viewer to appropriately take this into account.

The reference line is often regarded as fixed and known without uncertainty. However, consider Equation 1, the uncertainty around the predicted point estimate is clearly affected by the uncertainty in both θ ₁ (the typical value of the parameter) and θ ₂ (the covariate coefficient). If the reference line is defined as the “typical individual,” as discussed above, then it is logical that also the reference line should be associated with the uncertainty related to θ ₁. Similarly, if the forest plot presents the covariate impact on a relative scale, it may be reasonable that the confidence interval around the covariate effect also reflects the uncertainty in the denominator of the relative effect (i.e., θ ₁). This was done in Figure 1 and can be seen in the covariate effects for the fed state and CYP2D6 EM, which have confidence intervals despite being part of the definition of the reference line. Because it is not always obvious from the forest plot if the uncertainty in the reference is included or not, it is important that the creator of the forest plot provide this information in the description of the plot. Further details of how this uncertainty can be computed is provided in Appendix S4.

It is also possible to use different reference cases for each covariate group, which is useful if one wants to compare the impact of covariates in one subgroup of subjects to another, for example, sex in healthy volunteers to patients. In this case, one might use the healthy male volunteers as the reference for the impact of men in patients and the healthy female volunteers as the reference for the impact of women in patients, instead of using the typical subject in the data set as the reference to the overall impact of sex across both healthy volunteers and patients. This can be implemented using the possibility in PMXForest to use a varying reference across covariate groups and conditioning (see below). However, for this to be comparable to a classical statistical subgroup analysis, it needs to be combined with an empirical forest plot approach (see above).

Choice of covariates to condition the predicted parameter values on

When generating the predictions for forest plots it is possible to take multiple covariates into account and this can be used to condition an entire forest plot on a particular value of a covariate. This type of conditioning can, for example, be to generate all covariate effects conditional on a fasted state even if the typical individual in the model is defined as in the fed state. Obviously, it is necessary to clearly communicate such conditioning to the viewer of the forest plot. Note also that it is not necessary to keep the conditioning and reference line setting in sync. They can be specified independently of each other.

The relationship between conditioning and reference line is illustrated in Figure 6. In the left plot, no specific conditioning is used to generate the predictions, that is, the covariates not being investigated on a particular line in the plot are set to the typical individual values in the data set (i.e., body weight of 88 kg, CYP2D6 EM, or the fed state), and the reference line is based on the same set of covariate values.

FIGURE 6.

Illustration of different conditioning and reference line combinations. See text for details. CI, confidence interval; CL, clearance; SCM, stepwise covariate method.

In the middle plot in Figure 6, the predictions are conditioned on the fasted state, meaning that all predictions are generated assuming the fasted state, except when the impact of the fed state is predicted. The latter means that the top panels in the left and middle plots are the same. However, the middle and bottom panels are different because the predictions are generated using the fed state in the left plot and the fasted state in the middle plot. The other covariates are handled as in the left plot. The reference line in the middle plot is the same as in the left plot, that is, body weight of 88 kg, CYP2D6 EM, and the fed state. In comparison to the left plot, the predictions in the middle and bottom panels are shifted to the right relative to the reference line.

In the right plot in Figure 6, the conditioning is the same as in the middle plot, but the reference line is based on a body weight of 88 kg, CYP2D6 EM, and the fasted state, that is, covariate values that match the conditioning.

In many cases, it may seem reasonable to change the covariates that underlie the reference line to match the conditioning, but it may also make sense to not do so. For example, there may be a reference subject (reference line) that is generally accepted but the test conditions of interest may be different. This is the situation emulated in Figure 6. Imagine if most of the studies prior to the current one were carried out under fasted conditions and the new study was carried out under fed conditions, it then makes sense to relate the new results (under fed conditions) to the previous results (fasted conditions) and not the other way around.

From a communication perspective, it is important to clearly communicate the conditioning settings (and reference line settings, see above) to allow the viewer of the plot to clearly understand what is depicted.

Choice of the reference area width

Including a reference area in a forest plot to signify clinical relevance enhances its utility for decision making by providing a direct comparison of predicted covariate effects to clinically important thresholds. However, it is important that the reference area accurately represents what is relevant, as an incorrect representation can lead to misinterpretations and wrong decisions. The clinical relevance limits should ideally reflect both efficacy and safety considerations and therefore requires cross‐functional discussions. It is also important to note that during a drug development program, such information may not always be available or may evolve as new data emerges. Consequently, clear communication regarding the basis for the reference area is essential, and viewers should consider this when evaluating the plot.

Choice of the width of the confidence interval

The confidence intervals are very important for the usefulness of forest plots. The impact of the method used for deriving these were discussed above. An additional choice that must be made is the width, or coverage, of these intervals. When the uncertainty is based on the estimated variance covariance matrix there is little additional computational cost in deriving wide confidence intervals, for example, 95% or 99% intervals. For numerical approaches, such as the nonparametric bootstrap and SIR, it is more costly in terms or run times to derive wider intervals. ¹⁹

It may be tempting to aim for a wide confidence interval to use forest plots for hypothesis testing with commonly used significance levels (e.g., p < 0.05). However, given the many assumptions underlying forest plots it is doubtful if this is a reasonable endeavor and, for that to be successful, the plots need to be constructed very carefully. In addition, from a hypothesis testing point of view in the context of full covariate models, it is also necessary to keep the problem of multiple testing in mind, as highlighted by Xu et al. ²⁰ A more pragmatic use of forest plots is to illustrate the main trends and for that application it is sufficient with less wide confidence intervals (e.g., 80%).

Although it is technically feasible to use forest plots for visualizing between‐subject variability, we strongly advise against it. The standard use of the error bars in forest plots is to represent uncertainty (i.e., confidence intervals), not to represent between‐subject variability (i.e., prediction intervals), and blurring the use of the error bars leads to a significant risk of confusion for the viewer. It is better to maintain consistency in the graphical elements of the forest plots to minimize the risk for misinterpretations. Please note, though, that it is not the use of, and explorations supported by between‐subject variability we recommend against, only the use of forest plots for illustrating this source of variability (for the reasons given above).

BEST PRACTICES AND CHECKLISTS

Considering the assumptions and choices discussed above, one might question the practicality of forest plots. However, despite their complexity, they are valuable as compact and efficient communication tools for covariate effects in pharmacometric models. What is needed to avoid misinterpretation and incorrect conclusions is a collaborative mindset in both the creator and viewer of the forest plots. The creator must provide all the necessary information needed by the viewer to understand how the plots can be used and what conclusions they support. The viewer, on the other hand, must understand and be willing to consider the fact that forest plots are apex result presentations that rests on a long chain of assumptions and choices.

To facilitate this necessary collaboration between the creator and the viewer of the forest plots, we have compiled a list of fundamental best practices for creating forest plots as well as checklists for both the creator and the viewer. We are also proposing how the caption to forest plots should be structured.

Best practices for creating forest plots

The following are the best practices for the individuals in charge of creating forest plots:

1. Keep forest plots in mind during the planning stage of the analysis to ensure the methodologies align with the intended inferences.

2. Offer a broad overview of the limitations posed by the data and the analysis. Clarify what inferences are appropriate based on the provided forest plots.

3. Ensure that the covariates included in the analysis are independent or use a method that can manage correlated covariates.

4. Avoid mixing objective and subjective covariate values in the same forest plot.

5. To avoid misunderstandings, do not use forest plots to visualize between subject variability.

6. Be comprehensive in visualizing all covariates included in the analysis, even those that are not statistically significant. If some covariates are omitted from the forest plot, clearly state why.

7. Provide an exhaustive explanation of the methodologies, decisions, and assumptions that affect the interpretation of the forest plot to facilitate a technical review.

8. Leverage the “Checklist for the creator of forest plots” as a tool to ensure the viewer is given all necessary details for understanding the plot.

Checklist for the creator of forest plots

The following list is a complement to the best practices. The intention is to help the creator of forest plots to provide the information that is needed outside the figure caption.

• Specify any limitations due to study design and conduct.

• Specify the covariate modeling method.

• Specify how correlated covariates were handled during the analysis.

• Specify how much missing covariate information there was and how it was handled in the modeling.

• Specify which covariates that were included in the covariate analysis and which were left out.

• Specify which parameters that were subject to covariate modeling even if only one or a subset of them are visualized.

• Specify the covariate model parameterization.

• Specify how the uncertainty in the parameter estimates was derived.

Structured caption to forest plots

As we have seen, there is a lot of information that needs to be communicated for an informed interpretation of a forest plot. Some of this information can and should be included in the caption to the plot. Figure 7 provides a list of the information we believe should be provided in the figure caption and illustrates how the caption can be organized to include it.

FIGURE 7.

Illustration of how the caption to a forest plot can be organized to include as much relevant information as possible. The type of information is listed on the left‐hand side and an example forest plot with associated caption. (The plot is intentionally made small since the focus is on the caption.) The text in the caption is colored according to the corresponding information on the left. AUC, area under the curve; CRCL, creatinine clearance; EM, extensive metabolizer; IM, intermediate metabolizer; NCI, National Cancer Institute; PM, poor metabolizer; UM, ultra‐fast metabolizer.

Checklist for interpreting forest plots

The following checklist is intended for the viewer of the forest plots, to provide a reminder of the factors that need to be considered before conclusions can be drawn.

• Is the study design informative enough for the intended analysis?

• Is the study population relevant for the intended inference?

• Are all relevant covariates available for the analysis?

• Are all covariates collected with the appropriate level of exactness and completeness?

• Are the covariates in the model independent (low correlation)?

• Are there available covariates that were not included in the analysis?

• Are there potential biases introduced by non‐random patient dropouts?

• Were the covariates included based on a full model approach or based on statistical significance (e.g., SCM)?

• Are there parameters that were omitted from the covariate analysis?

• Will the covariate model parameterization impact the forest plot interpretation?

• Will the handling of missing covariates impact the forest plot interpretation?

• How was the uncertainty assessed?

• Is the width of the confidence interval clear?

• Are there covariates that were part of the analysis that are not shown in the forest plots? Why?

• How were the covariate values to visualize chosen?

• How was the reference line computed?

• Is there an objective rationale for the width of the reference area?

SUMMARY AND CONCLUSIONS

Forest plots are visually appealing and efficient communication instruments for covariate effects in pharmacometric models. However, they are very sensitive to study design and conduct choices, and choices and assumptions made by the analyst during the analysis and while creating the graph. To avoid incorrect conclusions, it is necessary that these choices are clearly communicated to the viewer, and that the viewer clearly understands what the impact these choices will have for the conclusions that can be drawn.

The generation of forest plots involves many possibilities and depends on a comprehensive articulation and understanding of the technical details needed for a quantitative interpretation of the results. It can therefore be argued that forest plots should first and foremost be regarded as a communication instrument for the combined outcome of practicality driven analytical choices and the signals present in the available data, rather than a tool for objective and unquestionable inference. This does not mean that forest plots are not useful, though. The compact view of the results of potentially complex covariate model development procedures and likewise potentially complex dependencies of the biological and physiological variables on the metric of interest, will still reveal important trends in a clear and easily understandable manner.

In conclusion, the interpretation of covariate effects using forest plots is influenced by choices made by the creator of the forest plots. Viewers must be aware of these choices and actively consider them to ensure informed interpretation, that is, consider the reasons a covariate appears to be relevant and to keep in mind that the lack of an apparent relevant effect does not equate a lack of true impact. Transparent reporting by creators and informed judgment by viewers creates a collaborative approach that enhances the reliability of conclusions drawn from forest plots.

CONFLICT OF INTEREST STATEMENT

The authors declared no competing interests for this work.

Supporting information

Appendix S1

Appendix S2

Appendix S3

Appendix S4

Jonsson EN, Nyberg J. Using forest plots to interpret covariate effects in pharmacometric models. CPT Pharmacometrics Syst Pharmacol. 2024;13:743‐758. doi: 10.1002/psp4.13116