Quantitative Assessment of Systematic Bias: A Guide for Researchers

Abstract

Observational research provides valuable opportunities to advance oral health science but is limited by vulnerabilities to systematic bias, including unmeasured confounding, errors in variable measurement, or bias in the creation of study populations and/or analytic samples. The potential influence of systematic biases on observed results is often only briefly mentioned among the discussion of limitations of a given study, despite existing methods that support detailed assessments of their potential effects. Quantitative bias analysis is a set of methodological techniques that, when applied to observational data, can provide important context to aid in the interpretation and integration of observational research findings into the broader body of oral health research. Specifically, these methods were developed to provide quantitative estimates of the potential magnitude and direction of the influence of systematic biases on observed results. We aim to encourage and facilitate the broad adoption of quantitative bias analyses into observational oral health research. To this end, we provide an overview of quantitative bias analysis techniques, including a step-by-step implementation guide. We also provide a detailed appendix that guides readers through an applied example using real data obtained from a prospective observational cohort study of preconception periodontitis in relation to time to pregnancy. Quantitative bias analysis methods are available to all investigators. When appropriately applied to observational studies, findings from such studies can have a greater impact in the broader research context.

The Value of Observational Research

Observational epidemiology plays a critical role in generating insights about potential causal relationships and identifying opportunities for future research (Glymour and Bibbins-Domingo 2019). Observational methods require fewer resources than most clinical or randomized controlled trials and can be implemented even when ethical issues prevent intervention research (Harper 2019). Furthermore, observational epidemiology can often leverage existing sources of data, garner large study sizes, and generate evidence faster than trials.

Observational studies have received many criticisms, the most important of which are limitations in their ability to adequately address systematic errors that arise from bias in variable measurement, sample selection, or residual or unmeasured confounding (Harper 2019). As such, close attention to potential sources of systematic error and rigorous evaluations of their potential influence upon observed results are key to validly estimating causal effects from observational data.

Quantitative bias analysis (QBA) is a set of methodological techniques developed to estimate the potential direction and magnitude of systematic error operating on observed associations between exposures and outcomes. The foundational methods were described decades ago (Greenland 1996) with advances described more recently (Fox et al. 2021), yet they remain uncommon despite general acceptance (Petersen et al. 2021). Similarly, although there are some examples in the dental research literature (Polzer et al. 2012; Alshihayb et al. 2020, 2021; Bond et al. 2023), QBA methods have yet to be readily adopted despite their clear relevance and direct calls for their use (Mittinty 2020, 2021).

We aim to demonstrate the utility of QBA methods with the goal of minimizing barriers to adoption. Our objectives are 3-fold: 1) illustrate the opportunity for application of QBA to observational studies, 2) provide a step-by-step guide to their application and an applied example using real data (see Appendix), and 3) highlight challenges and opportunities for QBA in oral epidemiology.

Quantitative Bias Analysis—An Introduction

The primary purpose of QBA is to facilitate a quantitative assessment of residual systematic error on observed results. Systematic error, as distinct from random error, can be caused by biases in study design and conduct, the most common sources of which are confounding, selection bias, and information bias. These key terms are defined as follows:

• Random error: Error caused by chance or random variation, often summarized using frequentist confidence intervals (Rothman et al. 2008; Fox et al. 2021). The more random error, the less precision an observed estimate of effect will have (as indicated by the width of the confidence interval). Random error decreases with increasing study size (Fox et al. 2021).

• Systematic error: Bias in observed estimates of effect due to issues in measurement or study design, or the uneven distribution of risk factors for the outcome across exposure groups, primarily caused by confounding, selection bias, or information bias (Rothman et al. 2008). It is the opposite of validity and does not decrease with increasing study size (Fox et al. 2021). Common sources in observed associations include the following:

  • Confounding: Bias resulting from the mixing of any actual exposure–outcome effects with other factors that also affect the outcome such that the unexposed group experience cannot stand in for what would have happened to the exposed group had they, counter to fact, been unexposed (Rothman et al. 2008).
  • Selection bias: Bias due to selection procedures, factors influencing study participation, and differential loss to follow-up (Hernán et al. 2004; Rothman et al. 2008).
  • Information bias: Bias due to systematic errors in the measurement of analytic variables (exposures, outcomes, and confounders) (Rothman et al. 2008).

QBA methods require the specification of bias parameters, which are quantitative estimates of features of the bias (e.g., sensitivity and specificity for measurement error). Selecting an appropriate approach involves balancing the rationale for its use, the information available to inform the analysis, and the computational intensity of the method (Fox et al. 2021).

Figure 1 depicts a flowchart of available techniques according to their complexity. The approach to assigning bias parameters defines each method. In some contexts, it is sufficient to choose a single value for a bias parameter estimate. In many circumstances, however, it is preferable to incorporate uncertainty about a given bias parameter using multiple estimates of the parameter’s value. These choices inform the appropriate modeling method, defined below:

Figure 1.

Flowchart of quantitative bias analysis (QBA) methods by complexity. The flowchart organizes available QBA methods according to their capabilities and complexity. The objectives of the QBA, as well as the available input to inform the model, can direct the method selection.

• Simple bias analysis: Uses single parameter values to estimate the impact of a single source of systematic bias on an estimate. Requires summary-level data (e.g., 2-by-2 table relating the exposure and outcome) to compute. The output is a single bias-adjusted estimate.

• Multidimensional bias analysis: Uses multiple sets of bias parameters to estimate the impact of a single source of systematic error. It is a series of simple bias analyses and requires summary-level data. Useful in contexts where there is uncertainty about parameter values (e.g., validation data not available). The output is a set of bias-adjusted estimates.

• Probabilistic bias analysis: Requires specification of a probability distribution around bias parameter estimates. Bias parameter values are randomly sampled from the distribution over multiple simulations and used to probabilistically bias-adjust the observed data, the results of which are summarized in a frequency distribution of revised estimates. Can use individual-level (data set with 1 row per observation) or summary-level data.

Implementing Bias Analysis: A Step-by-Step Guide

What follows is an implementation guide in which we provide a high-level overview of each step involved in conducting QBA. In the accompanying Appendix, we detail an applied example based on an evaluation of the published association between preconception periodontitis and time to pregnancy (Bond et al. 2021). We strongly encourage interested readers to reference Lash et al. (2014) for a more in-depth exploration of QBA and the book Applying Quantitative Bias Analysis to Epidemiologic Data (Fox et al. 2021), which also includes publicly available resources.

Step 1. Determine the Need for QBA

To determine whether QBA is advisable, the context of observed findings must be considered—specifically, whether results are consistent with the existing literature, while also weighing the potential for systematic error in that literature. With a large body of consistent literature in which there are no concerns about systematic error arising from confounding, selection bias, or information bias, QBA may not be necessary. If, however, a study is not aligned with prior findings or there are concerns about systematic error, QBA may provide useful insights (Lash et al. 2014). QBA is especially important in contexts where the explicit goal of the research is to draw causal inferences and studies in which the influence of random error has been minimized (e.g., meta-analyses or large studies). In addition, a detailed understanding of the study design that produced the findings is needed. Directed acyclic graphs (DAGs) can be a helpful tool for identifying and communicating the hypothesized bias structures (Greenland et al. 1999; Akinkugbe et al. 2016). Researchers should consider creating and displaying a DAG depicting the relationships between analysis variables and their measurements (see Fig. 2) (Fox et al. 2021).

Figure 2.

Depiction of a directed acyclic graph (DAG) framework to display bias structure. DAGs provide a framework within which causal relationships between studied variables can be evaluated, and potential implications of design choices can be assessed. Arrows indicate directional relationships between the elements listed in each node.

Step 2. Select the Biases to Be Addressed

Selection of which biases to address should be informed by the ultimate goals of the QBA. If it is to depict any possible source of bias, the approach will be different than if it is an in-depth evaluation of 1 source of bias. In addition to DAGs, researchers can apply simple bias analysis to quickly assess the potential influence of sources of errors, thereby informing whether they choose to include those biases in a more robust method.

Step 3. Select a Method to Model Identified Biases

Selecting a modeling approach requires balancing computational complexity with a realistic view of the potential impact of bias on the observed estimate. As previously mentioned, simple bias analysis is relatively easy to implement; however, it does not incorporate uncertainty around the bias parameters. Multidimensional bias analysis requires more parameter estimates but is still simple to implement and can account for some uncertainty in the bias parameters. Probabilistic bias analysis enables the incorporation of more uncertainty in the model inputs and the modeling of combined and individual effects of multiple sources of bias, rendering its implementation more complex. An additional consideration is the availability of individual-level or summary-level data for the QBA. Summary-level data involve applying bias parameters to a summary 2 × 2 table of the study data, while individual-level data involve applying the bias parameters individually to each observation. Use of individual-level data also allows for confounder adjustment in bias-adjusted effect estimates, which may facilitate a more accurate comparison to original study findings.

Step 4. Identify Sources of Information for Bias Parameter Estimates

Bias parameters are values that characterize the bias and relate the observed data to the expected true data through a bias model. They are specific to the type of bias being modeled and include the following:

• Information bias: Sensitivity and specificity of key analytic variables (exposure, outcome, confounders), including defining whether the measurement error is differential or nondifferential with respect to other analytic variables

• Selection bias: Estimates of participation rates from the target population within all levels of the exposure and outcome in the analytic sample

• Unmeasured confounding: Prevalence of the unmeasured confounder among the exposed and unexposed, as well as the estimated strength of the association of the confounder with the outcome

The bias parameters are never known with certainty but must be estimated. Identifying appropriate sources of information for these parameters is crucial to a valid QBA. Investigators can leverage data from internal or external validation studies, although internal validation studies are typically preferable. Internal validation sources could include a brief survey sent to nonparticipants to assess parameters for selection bias, a subset of study participants in which a confounder was measured, or a validation study in which the sensitivity and specificity of a measure were calculated against a gold standard in a subset of the study population (Lash et al. 2014). Research using clinical assessments of oral health status should consider whether an alternative clinical measure could serve as a superior “gold-standard” measure of the construct under study. In many cases, however, internal validation data are not available. In these cases, researchers should identify external estimates of bias parameters and evaluate applicability to their study population. In the absence of informational sources, educated guesses, informed as much as possible by the published literature, are necessary.

Step 5. Implementation

For QBA of summary-level data, resources are available, including Excel workbooks freely available online (Fox et al. 2023a) and described in detail (Fox et al. 2021). When first starting out with bias analysis, we recommend using these resources to increase familiarity with the methods and principles. For more complex bias analysis, including analysis of individual-level data or multiple biases simultaneously, simulation methods using statistical software such as SAS or R are required. SAS and R code are also available online (Fox et al. 2023a) and described in a recent publication (Fox et al. 2023b).

The implementation of simple and multidimensional bias analyses is the same, with multidimensional being a series of simple bias analyses. Probabilistic bias analysis uses Monte Carlo simulations to draw randomly from bias parameter distributions. These randomly drawn values of the bias parameters are used to probabilistically simulate adjustments to the observed data, facilitating calculation of a “bias-adjusted” effect estimate. Random error can be further incorporated by multiplying the standard error for each bias-adjusted effect estimate by the natural log of the standard error of the effect estimate. This is done automatically in the above-referenced Excel sheet or can be coded in statistical software. The bias-adjusted estimates are then summarized over all simulations using the median as a point estimate and a 95% simulation interval spanning from the 2.5th to the 97.5th percentile of the distribution of bias-adjusted results. These results provide an estimate of the magnitude and direction of modeled systematic error on the observed results while also accounting for the uncertainty in the parameter estimates and random error (see Appendix for more details on simulation methods).

Step 6. Presentation and Interpretation

Additional tables and figures are often necessary to effectively communicate QBA results, particularly when simulation methods are employed. We recommend that QBA results be presented alongside the original findings for appropriate context (see Appendix Fig. 3, Appendix Table 1). When interpreting QBA results, it is important to acknowledge that it cannot generate “corrected” effect estimates. First, modeling every possible source of bias is likely impossible. Second, bias analysis is inherently limited by the bias parameters used. Even when high-quality, internal validation studies exist, uncertainty surrounding those estimates remains. Any presentation of QBA results should specify that the validity of the results depends on the validity of selected bias parameters.

Limitations of QBA

QBA is a valuable tool that can provide estimates of the magnitude and direction of the influence of systematic error on observed effect estimates, but it is not a panacea. Our accompanying example demonstrates the feasibility of applying QBA to studies evaluating the oral–systemic connection and highlights some noteworthy challenges. One of the biggest challenges we encountered was finding bias parameter estimates for the sensitivity and specificity of our exposure: self-reported periodontitis. Questionnaire items were based on clinically validated questions developed by the Centers for Disease Control and Prevention in collaboration with the American Academy of Periodontology for large-scale periodontitis surveillance efforts (Eke et al. 2013), yet validation is often reported in the context of multivariable models that include multiple questions and demographic factors (Slade 2007; Eke et al. 2013). Multivariable models do not provide sensitivity and specificity estimates appropriate for an evaluation of a single question. Therefore, validation studies that report the sensitivity and specificity of individual measures alongside any multivariable models may provide useful information for future QBA efforts, particularly in a diverse array of populations. Validation studies conducted within the observational cohort under study are ideal for QBA efforts (Fox et al. 2021) as external validation studies present the possibility that the parameters will not be fully compatible with the data under study, which can necessitate subjective adjustments to the parameters (see Appendix).

QBA is also limited by the accuracy of the assumptions made in the selected approach. For instance, in our applied example, we assumed differential exposure misclassification. However, a sensitivity analysis in which we modeled nondifferential exposure misclassification found substantially different results. Because QBA often requires researchers to use educated guesses for some inputs or assumptions, we recommend the use of sensitivity analyses to test the impact of these assumptions.

Despite its limitations, QBA enables a quantitative assessment of suspected sources of bias. Although it often requires assumptions, it makes these assumptions clear to readers, thereby enabling more robust and informed considerations of the possible influence of bias on observational findings.

Conclusion

QBA methods are available to all investigators and can be applied to any study type (i.e., cross-sectional, cohort, case control), although additional considerations may be necessary. Findings from observational studies can have greater impact when accompanied by a thorough and objective analysis of their validity, especially in the presence of known sources of systematic bias. We echo the calls of prior researchers that observational oral health studies would benefit from regular integration of QBA and further encourage their regular integration into the peer-review process (Fox and Lash 2017; Mittinty 2020, 2021).

Author Contributions

J.C. Bond, contributed to conception, design, data analysis and interpretation, drafted manuscript; M.P. Fox, contributed to conception, design, data analysis and interpretation, critically revised manuscript; L.A. Wise, contributed to data acquisition and interpretation, critically revised manuscript; B. Heaton, contributed to conception, design, data acquisition, analysis, and interpretation, critically revised manuscript. All authors gave final approval and agree to be accountable for all aspects of the work.