Accounting for missing data in public health research using a synthesis of statistical and mathematical models

Abstract

Introduction

Accounting for missing data by imputing or weighting conditional on covariates relies on the variable with missingness being observed at least some of the time for all unique covariate values. This requirement is referred to as positivity, and positivity violations can result in bias. Here, we review a novel approach to addressing positivity violations in the context of systolic blood pressure.

Methods

To illustrate the proposed approach, we estimate the mean systolic blood pressure among children and adolescents aged 2–17 years old in the USA using data from the 2017–2018 National Health and Nutrition Examination Survey (NHANES). As blood pressure was not measured for those aged 2–7, there exists a positivity violation by design. Using a recently proposed synthesis of statistical and mathematical models, we integrate external information with NHANES to address our motivating question.

Results

With the synthesis model, the estimated mean systolic blood pressure was 100.5 (95% CI 99.9 to 101.0), which is notably lower than either a complete-case analysis or extrapolation from a statistical model. The synthesis results were supported by a diagnostic comparing the performance of the mathematical model in the positive region.

Discussion

Positivity violations pose a threat to quantitative medical research, and standard approaches to addressing non-positivity rely on restrictive untestable assumptions. Using a synthesis model, like the one detailed here, offers a viable alternative.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Positivity is an important underlying assumption for many methods to address systematic errors, or biases, in epidemiologic research.

• When there is non-positivity, epidemiologists have had to modify their scientific questions or rely on strong parametric modelling assumptions to extrapolate.

WHAT THIS STUDY ADDS

• A recently proposed method to address positivity violations based on a synthesis of statistical and mathematical models is reviewed and illustrated in the context of systolic blood pressure, missing data and non-positivity.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The method reviewed in this article offers a way forward when the positivity assumption is not met, which does not require modifying the motivating epidemiologic question or having to rely on extrapolation.

Introduction

Suppose we are interested in estimating the mean systolic blood pressure (SBP) among children and adolescents aged 2–17 in the USA between 2017 and 2018. To estimate this mean, we use data from the National Health and Nutrition Examination Survey (NHANES), a nationally representative survey of children in the USA.¹ However, SBP measurements are partially missing. While we could restrict our data to those with measured SBP (ie, conduct a complete-case analysis), this approach may be biased when there is a variable predictive of both SBP and missingness of SBP.² Instead, methods like weighting and imputation can be used to correct for missing outcome data.³ In the absence of strong parametric modelling assumptions, these methods assume that SBP is measured for at least some people for each unique value of the measured variables. For example, if missingness depends on age (which is related to SBP), then these methods assume that SBP has a non-zero probability of being measured for each age. This assumption is referred to as positivity and can lead to bias when it is violated.24,6 Existing approaches to address non-positivity require modifying the scientific question or extrapolating from a statistical model. In recent work, Zivich and colleagues introduced a new approach that avoids the issues of these existing approaches.^{7 8} Here, we review the problem posed by non-positivity and illustrate this novel alternative with publicly available data and corresponding code in R and Python to replicate the analyses.⁹

Methods

Here, we consider the following variables collected by NHANES. The variable of interest, SBP (mm Hg), was measured up to three times. In this analysis, we took the mean of the available SBP measurements, and SBP was set to missing if fewer than two measurements were available. Sampling weights were applied for inference to the US population. Additional covariates included age (years), height (centimetres), weight (kilograms) and gender (male, female).

In the NHANES data, SBP was missing for 44% of children. Unless SBP and the probability of missing SBP are independent, a complete-case analysis (ie, restricting to only those with measured SBP) can be substantially biased.² To help make these ideas more precise, we introduce some notation. Let $X$ be age and $Y$ be SBP, and $R=1$ indicate that SBP was observed. Our parameter of interest, the mean SBP for children and adolescents, can be expressed as $\mu =E\left[Y\right]$, where $E\left[\cdot \right]$ is the expected value function. However, SBP is only observed for those with $R=1$. For a complete-case analysis to be unbiased, SBP and missingness must be marginally independent or exchangeable,³ which can be expressed as $E\left[Y\right]=E\left[Y|R=1\right]$ and indicates that those with a measured SBP are a random sample of all children and adolescents. This assumption would be valid if NHANES randomly determined which participants had their SBP measured. However, this assumption is not consistent with the NHANES design.

With missing data, a marginal exchangeability assumption is often unreasonable, as investigators rarely have control over non-response. However, weaker assumptions regarding the missing data mechanism can be made. Rather than marginal exchangeability, we can instead assume conditional exchangeability. For didactic purposes, consider whether missingness was independent of SBP conditional on age. The conditional exchangeability assumption can then be expressed as $E\left[Y|X=x\right]=E\left[Y|X=x,R=1\right]$ for all ages $x$ from 2 to 17. This assumption states that those with a measured SBP are a random sample within each stratum of age from 2 to 17, allowing those with a measured SBP to ‘stand in’ for those with an unmeasured SBP that are the same age. Implicitly, this exchangeability assumption comes with the positivity assumption that SBP has a non-zero probability of being measured for each unique age in our population.^{4 10} This assumption can be written as $\mathrm{Pr}\left(R=1|X=x\right)\S amp;gt;0$ for all ages $x$ from 2 to 17. To illustrate why positivity is important, note that if everyone of a given age does not have a measured SBP, then there would be no one with a measured SBP to stand in for them.

Given conditional exchangeability with positivity, we can express the parameter of interest, $\mu$, in terms of the data we observed. Specifically, we can show that

$$E\left[Y\right]={\sum }_{x}E\left[Y|X=x,R=1\right]\mathrm{Pr}\left(X=x\right)$$

following from the law of total expectation over all ages $x$ from 2 to 17 and exchangeability with positivity. As the right-hand side is expressed in terms of the observed data, this result provides a recipe for estimating $\mu$. Specifically, it motivates a direct maximum likelihood estimator, also referred to as g-computation or imputation.^{2 3} To apply this method, one first estimates a regression model for SBP given age among all observations with complete data and the NHANES sampling weights. Then, using this estimated model, one predicts SBP given age for all children in the dataset. Effectively, this approach fills in the missing values for all observations. The mean SBP is then estimated by taking the sample-weighted average of the predicted SBP.

Non-positivity

Positivity is violated in our example, as NHANES did not measure SBP for any participants <8 years old. To reiterate why non-positivity is an issue, suppose we considered the following model for SBP given age

$${\displaystyle {\displaystyle E\left[Y|X,R=1;\beta \right]={\beta }_{2}I\left(X=2\right)+{\beta }_{3}I\left(X=3\right)+\dots +{\beta }_{17}I\left(X=17\right)}}$$

where $I\left(\cdot \right)$ is the indicator function. In this model, there is a unique coefficient for each age (ie, the model is saturated), so no parametric modelling assumptions are imposed. Non-positivity means that there is no one with SBP measured for ages <8, so the coefficients from ${\beta }_2$ through ${\beta }_7$ cannot be estimated using the NHANES data alone.

Existing approaches

To proceed when there is non-positivity, we could modify the research question. Here, we could consider estimating the mean SBP in children aged 8–17 instead, represented mathematically as $E\left[Y\mid X\ge 8\right]$. While valid, this revision addresses a different research question and thus may limit the utility of this data analysis if the interest is really in the 2–17 age range. Instead, consider another approach to address the original research question. One could incorporate structural or parametric assumptions into a statistical model and use that model to extrapolate to ages with unmeasured SBP. For example, we might consider the following linear model

$$E\left[Y|X,R=1,;\beta \right]={\beta }_0+{\beta }_{1}X$$

which imposes a linear relationship between age and SBP. This model allows us to fill in SBP values of those aged 2–7 using a model fit to those aged 8–17. Use of this regression model requires two important assumptions. First, the model assumes that there is a linear relationship between SBP and age among those 8–17. This assumption is possible to assess and relax with the available data. Second, the use of this model assumes that the linear relationship extends to those aged 2–7. In other words, the model fit to those aged 8–17 is used to extrapolate to those aged 2–7. The assumption licensing this extrapolation cannot be assessed with the available NHANES data.

Proposed approach

To avoid the restrictive modelling assumptions of the extrapolation approach while still addressing the motivating question, we review a method based on a synthesis of statistical and mathematical models.⁷ To motivate this approach, first note that we can divide the data into two parts: one where positivity is met and one where positivity is not met. As shorthand, let $X^{*}=1$ if a NHANES participant’s age was between 8 and 17 and $X^{*}=0$ otherwise. Using this additional notation and the idea of dividing the data regarding whether positivity is met, we can rewrite $\mu$ as

$${\displaystyle {\displaystyle E\left[Y\right]=E\left[Y|X^{\ast }=1\right]Pr\left(X^{\ast }=1\right)+E\left[Y|X^{\ast }=0\right]Pr\left(X^{\ast }=0\right)}}$$

which is a weighted average of the means in the positive and non-positive regions. Since $X^{*}$ is observed for all participants, $\mathrm{Pr}\left(X^{*}=1\right)$ and $\mathrm{Pr}\left(X^{*}=0\right)$ can be directly estimated from the data. Therefore, how $E\left[Y|X^{*}=1\right]$ and $E\left[Y|X^{*}=0\right]$ can be learnt is the remaining task. The core idea of the synthesis model is to rely on a statistical model to address missingness in the region with positivity (ie, $E\left[Y|X^{*}=1\right]$) and use a mathematical model to fill in missing values using external information in the non-positive region (ie, $E\left[Y|X^{*}=0\right]$).

Starting with $E\left[Y|X^{*}=1\right]$, recall that this is the region with positivity, which means standard statistical methods for missing data can be applied. Specifically, we can rely on the previous conditional exchangeability with positivity assumptions limited to those aged 8–17. These assumptions allow us to use the previously described g-computation procedure limited to those aged 8–17. Here, we use the following saturated model

$${\displaystyle {\displaystyle E\left[Y|X,X^{\ast }=1,R=1,\beta \right]={\beta }_{8}I\left(X=8\right)+{\beta }_{9}I\left(X=9\right)+\dots +{\beta }_{17}I\left(X=17\right)}}$$

which places no parametric constraints on the relationship between age and SBP for those aged 8–17. Unlike the extrapolation approach for dealing with non-positivity, this statistical model does not require there to be a linear relationship between age and SBP for those ages 8–17.

Now, we turn the attention to $E\left[Y|X^{*}=0\right]$. To reiterate, SBP was never measured for $X^{*}=0$ in the data. Therefore, we need to look outside the available NHANES data to make progress. To start, one might consider plugging in extreme but plausible values for the mean among 2–7 year olds (eg, 70–120 mm Hg).⁸ Plugging in these values provides a range of plausible values for $\mu$, often referred to as bounds.¹¹ These bounds provide a simple sensitivity analysis that only relies on the assumption that the true $E\left[Y|X^{*}=0\right]$ lies somewhere within the chosen range. How $\mu$ changes across the range of plausible values can also be plotted to further explore the bounds.

When additional external information is available, we can move beyond a sensitivity analysis and instead build a mathematical model to fill in the missing values. Here, values for those aged 2–7 are imputed based on published SBP distributions for US children and adolescents.¹² The imputed values from the mathematical model come from random draws of age, gender and height percentile-specific SBP distributions. Here, we assume normal distributions under the assumption that the mean is equal to the median, and the SD is approximated from the 90^th percentile. Use of this mathematical model to fill in missing values of SBP is justified by the assumption that the given external information is accurate for our context. Here, accurate external information for our context means that the source population for the published SBP descriptive statistics and the NHANES population are similar for all other predictors of SBP besides age, gender and height. This assumption would be violated, for example, if weight was distributed differently across data sources. As NHANES and the published SBP distributions are both intended to be representative of the US population, we expect any differences to be small.

To apply the synthesis model while also accounting for variability in both the statistical and mathematical models, we use a resampling procedure that incorporates the NHANES sample weights. The algorithm for this procedure is described in online supplemental appendix 1. Briefly, the NHANES data is resampled with replacement. The parameters of the statistical model are re-estimated and used to estimate $E\left[Y|X^{*}=1\right]$. Similarly, new imputed values are drawn from the mathematical model in order to estimate $E\left[Y|X^{*}=0\right]$. These values are then used to estimate $\mu$. This process is repeated a large number of times (ie, 10 000), and the different estimates of $\mu$ are summarised by the median as the point estimate and the 2.5th and 97.5th percentiles as the 95% CI. This procedure accounts for the sampling uncertainty in NHANES as well as the uncertainty in the mathematical model parameters. However, this procedure does not account for clustering in the NHANES design, so this procedure is likely an underestimate of the uncertainty.

As mentioned, the validity of this approach is premised on the mathematical model being built from accurate external information. While there are substantive reasons to believe this assumption to be reasonable, the external information allows us to indirectly check the validity of this assumption. Here, the external information also included SBP distributions for those aged 8–17. Using this additional information, we compare the age-specific SBP values for those aged 8–17 between the statistical and mathematical models. To incorporate sources of uncertainty, the resampling procedure is used to compare the difference between the age-specific SBP averages by model. While we believe such a check to have utility, it should be recognised that this diagnostic cannot assess the applicability of the mathematical model for the non-positive region.

In practice, researchers may wish to incorporate more than one variable in their imputation model. For example, one could instead assume exchangeability conditional on age, gender, height and weight within the positive region. In online supplemental appendix 2, we show how the synthesis approach can be applied with a parametric statistical model to account for these variables in the positive region. As alluded to earlier, the use of parametric statistical models relies on the assumption that the model is correctly specified. When this assumption is not met, estimates of $\mu$ may be biased. In online supplemental appendix 3, we describe an augmented inverse probability weighting estimator for synthesis models that relies on less restrictive parametric modelling assumptions.

Research ethics

Ethical approval for this study was not required, as this was a secondary data analysis of deidentified data, and thus not considered to be human subject research under US federal regulation 45 CFR 46.102(e)(1). The design of NHANES was originally approved by the National Center for Health Statistics Ethics Review Board.

Results

Those aged 2–7 made up 34% of the NHANES weighted sample (online supplemental appendix table 1). Among those aged 8–17, SBP was missing for 8%. In a complete-case analysis, the estimated mean SBP was 104.7 (95% CI 104.1 to 105.3). These results are questionable, given what is known about the missing data and the relation between age and SBP. This estimate is more reflective of the mean SBP among those aged 8–17. With a linear extrapolation from the statistical model (figure 1), the estimated mean SBP among children and adolescents was noticeably lower (mean: 101.6, 95% CI 100.8 to 102.4). The difference between the complete-case analysis and extrapolation aligns with expectations, as those under 8 years old are expected to have lower SBP. If the mean SBP for those aged 2–7 was bound between 70 and 120, the bounds for the overall mean SBP would have ranged from 92.7 to 109.9 (95% CI 91.9 to 110.5). Figure 2 shows how $\mu$ varies across this range. With a mathematical model, the estimated mean SBP was lower than the extrapolation approach, at 100.5 (95% CI 99.9 to 101.0). This difference arises from the linear model imputing a higher mean SBP for lower ages relative to the mathematical model (figure 1). While the extrapolation and synthesis model results may not appear substantially different, the observed 1.1 mm Hg difference is 2.9 times the estimated SE of the extrapolation approach. When incorporating gender, height and weight into the outcome model, results for the synthesis model were unchanged (online supplemental appendix 2). However, the extrapolation approach results moved closer to the synthesis approach. Results did not substantially change further when using an augmented inverse probability weighting estimator (online supplemental appendix 3).

Figure 1. Systolic blood pressure by age from observations in the 2017–2018 National Health and Nutrition Examination Survey (NHANES) and projections from the mathematical model. Grey dots indicate the observed systolic blood pressure values by age (uniformly jittered for visualisation) from NHANES. Systolic blood pressure was not measured for those younger than 8 years old. The blue diamonds indicate the age-specific means. The blue line indicates a linear model fit to the observed data and projected for those under 8. The red shaded region indicates the distributions of 20 000 simulated observations from the mathematical model, with the red diamonds indicating the corresponding mean.

Figure 2. Sensitivity analysis for non-positivity on the mean systolic blood pressure in the 2017–2018 National Health and Nutrition Examination Survey. The black line indicates how the mean systolic blood pressure for children aged 2–17 (y-axis) varies under differing input values for the mean in the non-positive region (ie, children aged 2–8) across the range 70–120 mm Hg (x-axis). The coloured horizontal lines represent the results from the complete-case and extrapolation analyses and show how they relate to the sensitivity analysis. Shaded regions denote the 95% CIs.

When examining the validity of the mathematical model using the aforementioned diagnostic, the distribution of predicted SBP from the mathematical model overlaps with the observed SBP values for each age (figure 3). When comparing the differences between means and incorporating uncertainty, the estimated means from either approach were reasonably close to zero (figure 4). However, the statistical model results seemed to be consistently smaller than the mathematical model for those aged 15–17. Altogether, these results provide some support for the validity of the mathematical model in the positive region, which helps to support the belief that the mathematical model is also reasonable for the uncheckable non-positive region.

Figure 3. Comparison of statistical and mathematical model values for systolic blood pressure by age in the positive region. Caption: Grey dots indicate the observed systolic blood pressure values by age (uniformly jittered for visualisation) from NHANES. Blue diamonds indicate age-specific means from the statistical model. The red shaded region indicates the distributions of 20 000 simulated observations at each age from the mathematical model, with the red diamonds indicating the corresponding mean. NHANES, National Health and Nutrition Examination Survey.

Figure 4. Differences between mean systolic blood pressure comparing statistical versus mathematical models and corresponding 95% CIs. Circles indicate the difference between the statistical model and mathematical model for each of the means among those in the positive region. Vertical lines indicate the corresponding 95% CIs.

Discussion

Non-positivity threatens our ability to address important public health questions.⁶ While described here in the context of missing data, the positivity assumption also appears when addressing other biases, like confounding and selection bias.⁴ Historically, non-positivity has pushed public health and medical researchers to modify their questions or make restrictive, untestable assumptions.⁷ Here, we illustrated an alternative in the context of missing data on SBP using publicly available data from NHANES. The proposed approach is based on integrating the available data with external information through a synthesis of statistical and mathematical models. Importantly, this approach does not require modifying the motivating question or reliance on restrictive modelling assumptions, unlike competing methods.

There are a number of limitations to our analysis, and the NHANES example provided here should be viewed only as illustrative. Regarding variance estimation, the approaches used here did not account for the use of cluster sampling in NHANES, so the variance may be underestimated. Extending the resampling algorithm to include additional complexities, like clustering, is an area for future research. Regarding the mathematical model, a fairly simple model was used here to directly impute SBP values. Simple models like this may not always be feasible. Instead, intermediate processes may need to be modelled. For example, imputing missing SBP data for adults on hypertensive medication with a mathematical model may instead involve modelling active drug concentrations and subsequent physiological responses.¹³ Designing and building complex mathematical models should follow best practices14,16 and evaluate diagnostic procedures whenever possible. Future work should illustrate the use of these more advanced models. Finally, non-positivity may occur for more than a single variable. While the essential concepts of synthesis modelling still apply, illustrating their use in these contexts remains needed.

Data availability statement

Data are available in a public, open access repository.