SAS and R code for probabilistic quantitative bias analysis for misclassified binary variables and binary unmeasured confounders

Abstract

Systematic error from selection bias, uncontrolled confounding, and misclassification is ubiquitous in epidemiologic research but is rarely quantified using quantitative bias analysis (QBA). This gap may in part be due to the lack of readily modifiable software to implement these methods. Our objective is to provide computing code that can be tailored to an analyst’s dataset. We briefly describe the methods for implementing QBA for misclassification and uncontrolled confounding and present the reader with example code for how such bias analyses, using both summary-level data and individual record-level data, can be implemented in both SAS and R. Our examples show how adjustment for uncontrolled confounding and misclassification can be implemented. Resulting bias-adjusted point estimates can then be compared to conventional results to see the impact of this bias in terms of its direction and magnitude. Further, we show how 95% simulation intervals can be generated that can be compared to conventional 95% confidence intervals to see the impact of the bias on uncertainty. Having easy to implement code that users can apply to their own datasets will hopefully help spur more frequent use of these methods and prevent poor inferences drawn from studies that do not quantify the impact of systematic error on their results.

Key Messages.

• Systematic error is ubiquitous in epidemiologic research but is rarely quantified using quantitative bias analysis.

• This may be due to the lack of easy to implement software and code to run such analyses.

• We have previously developed Excel spreadsheets to run such analysis, but they can only be run on summary data and take a long time to run.

• We present here short code in SAS and R that can be easily adapted to the user’s analysis and present examples of how they run.

Introduction

Systematic error from selection bias, uncontrolled confounding, and misclassification is ubiquitous in epidemiologic research. Unlike random error, which is nearly universally quantified, the presence and, perhaps, direction of systematic error is typically relegated to speculation in discussion sections, often long after conclusions have already been drawn. This relegation occurs despite the fact that simple methods for quantifying the impact of the bias on the direction, magnitude and amount of uncertainty have existed for decades.^1–4 We have previously noted that the demand^5–8 for bias analysis by editors, reviewers, and funders has lagged behind the availability of methods and the field has suffered as a result.

Quantitative bias analysis (QBA) is a type of sensitivity analysis used to quantify systematic error, in which the observed data and the expected true data^9–12 are linked by a bias model. Assuming a valid bias model, the QBA can produce bias-adjusted results and provide an estimate of the impact of the source of the bias. These results can be combined with random error to obtain estimates of total study error, which allows for more reliable conclusions to be drawn.¹³

In 2015 we developed a SAS macro to perform quantitative bias analysis to bias-adjust for misclassification of binary variables.¹⁴ We frequently get requests for support for the macro as it is complex, difficult to implement, and inefficient when applied to large datasets (the program is now available at https://sites.google.com/site/biasanalysis/short-code). The program is also far longer than it needs to be, increasing the chances that errors will occur when modifying to address any specific bias problem. We suspect this is part of the reason it is not more widely implemented. Finally, our macro only addressed bias due to misclassification, whereas relatively simple code can also be developed to address bias due to uncontrolled confounding.

Since this macro was developed, we have identified many shortcuts and better programming techniques for QBA and have found that adapting simple code is usually more effective than running the macro. We have also revised the methods we use for bias analysis, whether record level or summary level, as we describe in the second edition of our textbook¹⁵ to ensure that both approaches are closely aligned and to provide interval estimates with better coverage probabilities. In addition, both R (https://cran.r-project.org/web/packages/episensr/episensr.pdf) and Stata¹⁶ now have packages that allow for summary-level QBA to be run quickly and efficiently, yet no such simple approach for SAS exists and neither R nor Stata packages are available for record-level adjustments. To address these issues, we present here a simple method and accompanying SAS and R code that can be adapted to specific datasets and bias problems. The code is provided in the Supplementary Appendix (available as Supplementary data at IJE online) and a table describing each program is provided in Supplementary Appendix S1 (available as Supplementary data at IJE online).

Methods

General approach to probabilistic bias analysis

The objective of quantitative bias analysis is to adjust an observed estimate of effect and its variance for a specified bias by making assumptions about bias the parameters that control the degree of bias. For misclassification problems, this often means making assumptions about the sensitivity (Se) and specificity (Sp) of the misclassified variable (possibly within levels of another variable) and back-calculating what data would have been observed if the misclassification had been absent (assuming values assigned to the bias parameters are correct). For unmeasured confounding problems, this means specifying the probability of having the confounder within levels of the exposure and the strength of the effect of the confounder on the outcome. As with misclassification, the bias parameters can be used to back-calculate the data that would have been observed had the uncontrolled confounder been measured. We do not touch on selection bias as selection bias problems are often very specific to a study and as such it is difficult to create generic code. We refer the reader to our textbook for methods and examples.

Bias parameters are rarely known with certainty; validation studies are conducted with a finite number of participants, resulting in residual error in estimation of those parameters. Probabilistic bias analysis (PBA) uses Monte Carlo methods in which we sample values from a distribution for the bias parameters, which we then use as inputs in the bias analysis. This process is repeated many times to generate a distribution of results across the entirety of each of the input distributions to account for uncertainty in the bias parameters.

For example, we might believe sensitivity is most likely 0.9 but we are 95% certain the true sensitivity lies between 0.80 and 0.96. We could assign the sensitivity a beta(50,6) distribution to mathematically represent this belief as this would have a mean value of 50/(50 + 6)=0.89 and 95% of the distribution would be between 0.800 and 0.959. Sampling from those distributions we can use the chosen values in a single iteration of the bias-adjustment to estimate the data we would have seen in the absence of misclassification and then calculate a bias-adjusted effect estimate. We repeat this process hundreds of thousands of times, saving each bias-adjusted estimate. We summarize the distribution of bias-adjusted estimates using the median as a measure of central tendency (a point estimate) and the 2.5^th to the 97.5^th percentile as a simulation interval to indicate where 95% of the distributions lies, though we note that any quantiles or other summary statistics could be used. This interval accounts only for systematic error; it describes the distribution of estimates adjusted for the bias given the assumptions about the bias parameters.

We typically also want to incorporate conventional random error into results to obtain an interval accounting for both systematic and random error (i.e. total study error). There are several ways this can be done, including jack-knifing¹⁷ or bootstrapping,¹³ but here we use a simple approximate method. Assuming we are interested in relative effect estimates (which we will assume throughout) we subtract from the natural log of each bias-adjusted estimate a random standard normal deviate (a normal distribution with a mean of 0 and standard deviation of 1) multiplied by an estimate of the standard error of the natural log of the bias-adjusted estimate. This simulates the conventional random error around the bias adjusted estimate. We exponentiate this term to return a bias-adjusted estimate of effect. Finally, we summarize the resulting distribution of estimates, accounting for both systematic and random error, using the median as the measure of central tendency and the 2.5^th to 97.5^th percentile as a simulation interval describing the uncertainty in the results accounting for both systematic and random error.

Distributions to express uncertainty in the bias parameters

Our code requires specifying distributions for each bias parameter. This specification can be as simple as assuming a point mass distribution (i.e. setting the bias parameters to a fixed values) or as complicated as making distributional assumptions to reflect plausible values. SAS and R incorporate random draws from many distributions, which can be used to specify the uncertainty in the bias parameters. We provide examples using trapezoidal distributions (in which the user needs to specify a range of plausible values from a minimum to maximum value and a zone of confidence between a lower and upper mode) and beta distributions (useful when validation data are available as one can specify the first parameter in the beta distribution as the number of concordant values and the second parameter as the number of discordant values to obtain a distribution located on the proportion concordant). We encourage users to adapt the code to the most sensible distributions for the analysis.

Notes on the code

We have programmed the code to use a risk ratio, so if the user would prefer an odds ratio or some other estimator, the standard error formula would need to be edited in places noted in the code. The code can be adapted to any measure (relative or absolute) if it can be calculated from the data (or estimated in a regression model) and its standard error can be estimated as well.

Misclassification

Our approach to bias-adjusting for misclassification in has been described elsewhere in greater detail.¹⁵ We describe the algorithm in Supplementary Appendix S1. Briefly, we:

1. specify distributions for the bias parameters (sensitivity and specificity)

2. sample values from the specified distributions for sensitivity and specificity

3. convert these values to positive and negative predictive values.

4. use draws from a binomial distribution to apply the predictive values to individuals within the data to impute a dataset with a bias-adjusted exposure variable (done as adjusted cell counts with a binomial distribution for the summary level approach and bernoulli trials for each individual for the record level approach).

5. repeat this thousands of times using Monte Carlo methods to get a distribution of estimates bias-adjusted for misclassification and accounting for random error.

6. incorporate random error

7. summarize the results using the median, 2.5^th and 97.5^th percentiles of the distribution

The adjustment formulas are provided in Table 1 and in detail elsewhere.¹³

Table 1.

Algorithm to implement summary-level probabilistic bias analysis for misclassification

Step	Example
1. Sample bias parameters	$s{e}_1\sim \mathit{beta}(a_{se,1},b_{se,1})$ $s{e}_0\sim \mathit{beta}\left(a_{se,0},b_{se,0}\right)$ $s{p}_1\sim \mathit{beta}\left(a_{sp,1},b_{sp,1}\right)$ $s{p}_0\sim \mathit{beta}(a_{sp,0},b_{sp,0})$
2. Calculate bias-adjusted cell counts	$N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},1}=\frac{\left(Ob{s}_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},1}-N_1*\left(1-s{p}_1\right)\right)}{s{e}_1+s{p}_1-1}$ $N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},0}=\frac{\left(Ob{s}_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},0}-N_0*\left(1-s{p}_0\right)\right)}{s{e}_0+s{p}_0-1}$
3. Sample exposure prevalences	${\pi }_1\sim \mathit{beta}(N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},1},N_1-N_{ex{p}_1})$ ${\pi }_0\sim \mathit{beta}(N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},0},N_0-N_{ex{p}_0})$
4. Calculate PPV and NPV	$PP{V}_1=\frac{s{e}_1*{\pi }_1}{s{e}_1*{\pi }_1+\left(1-s{p}_1\right)*\left(1-{\pi }_1\right)}$ $NP{V}_1=\frac{s{p}_1*(1-{\pi }_1)}{\left(1-s{e}_1\right)*{\pi }_1+s{p}_1*\left(1-{\pi }_1\right)}$ $PP{V}_0=\frac{s{e}_0*{\pi }_0}{s{e}_0*{\pi }_0+\left(1-s{p}_0\right)*\left(1-{\pi }_0\right)}$ $NP{V}_0=\frac{s{p}_0*(1-{\pi }_0)}{\left(1-s{e}_0\right)*{\pi }_0+s{p}_0*\left(1-{\pi }_0\right)}$
5. Sample exposed and unexposed cell counts	$N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},1}\sim \mathit{binom}\left(Ob{s}_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},1},PP{V}_1\right)+\mathit{binom}\left(Ob{s}_{\mathit{unexp},1},\left(1-NP{V}_1\right)\right)$ $N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},0}\sim \mathit{binom}\left(Ob{s}_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},0},PP{V}_0\right)+\mathit{binom}\left(Ob{s}_{\mathit{unexp},0},\left(1-NP{V}_0\right)\right)$
6. Sample new log(RR)	$\mathit{log}\left(RR\right)\sim N\left(\mathit{log}\left(\frac{N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},1}/M_1}{N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},0}/M_0}\right),\frac{1}{N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},1}}-\frac{1}{M_1}+\frac{1}{N_{\hspace{0.17em}\mathit{\text{exp}}\hspace{0.17em},0}}-\frac{1}{M_0}\right)$

$a_{se,i},b_{se,i},a_{sp,i},b_{s{p}_i}$ are parameters explained in the text that parameterize the bias parameter distribution for sensitivity and specificity among cases (i = 1) and non-cases (i = 0).

$s{e}_i, s{p}_i$ are sensitivities and specificities among cases (i = 1) and non-cases (i = 0).

$Ob{s}_{\hspace{0.17em}\mathit{\text{exp}},i}$ is the number of observed exposed cases (i = 1) or non-cases (i = 0); $N_i$ is the total number of cases or non-cases; $N_{\hspace{0.17em}\mathit{\text{exp}},i}$ is the number of expected bias-adjusted exposed cases (i = 1) and non-cases (i = 0); $M_i$ is the number of exposed (i = 1) or unexposed (i = 0).

${\pi }_i$ is the prevalence of exposure among cases (i = 1) or non-cases (i = 0).

PPV, positive predictive value; NPV, negative predictive value; RR, relative risk; Se, sensitivity; Sp, specificity; binom, binomial.

We provide newly developed generic code for SAS and R code for exposure misclassification bias-adjustments (note we have also provided a link in the Supplementary Appendix to our website where updated versions can be downloaded). We provide code for exposure misclassification for summary level (Supplementary Appendices S2 and S3, available as Supplementary data at IJE online) and record level analyses (Supplementary Appendices S4 and S5, available as Supplementary data at IJE online) and summary level outcome misclassification (Supplementary Appendices S6 and S7, available as Supplementary data at IJE online) and record level (Supplementary Appendices S8 and S9, available as Supplementary data at IJE online), but algorithms for each follow the same logic, so we describe only exposure misclassification bias-adjustments. When implementing outcome misclassification adjustments in case-control studies, additional care has to be taken to adjust for the case and control sampling fractions (see Jurek et al.¹¹) We provide outcome misclassification code for summary level case-control studies in Supplementary Appendices S10 and S11 (available as Supplementary data at IJE online). The user must input values in the cells of the observed 2x2 table relating exposure to outcome and specify distributions for sensitivity and specificity of exposure classification within outcome levels. For non-differential misclassification, one only needs to set the sampled value of sensitivity among those with the outcome equal to that among those without the outcome and the same for specificity.

The distributions chosen are up to the user, using any draw from the many distributions SAS or R has pre-programmed or ones the user can program. Possible distributions include beta, uniform, truncated normal, triangular, trapezoidal, and log-normal. Examples of beta and trapezoidal distributions are provided in the code. In our previous work on QBA we have also found it necessary to ensure that there is a correlation between the two sensitivity distributions and the two specificity distributions to ensure that if we sample from the higher (or lower) end of one distribution, we are more likely to sample from the higher (or lower) end of the second. Here we specify a correlation of 0.8 between the two sensitivity and between the two specificity distributions. We also note that in misclassification problems, some values of sensitivity and specificity can lead to an adjusted dataset that includes negative cell counts that cannot then be used for effect estimation. These iterations are discarded from the analysis and the algorithms note how often this occurs. The user must decide whether to keep the distributions as is and report the number of discarded iterations or adjust the distributions (still reporting if any iterations are discarded). See Chapter 6, page 191–192 of our textbook for a complete discussion of options to handle impossible values in these bias model simulations¹⁵.

Summary-level vs record-level misclassification adjustment

The approach to individual record-level QBA for misclassification and summary-level QBA are nearly identical. The only difference is that the bias adjustments are conducted on individual records in the dataset (bias-adjusting each individual’s exposure or outcome). Because the bias adjustments are made to individual records, a regression model can be used for each iteration of the bias-adjusted data to generate a bias-adjusted and confounder-adjusted summary measure of effect. This allows for adjustment of numerous measured confounders or other complex modelling forms (e.g. accounting for censoring or missing data), which are difficult to do in summary-level bias analyses. We note, however, that our code creates systematic error adjusted estimates that do not adjust for the measured confounding.

For record-level bias-adjustments, as with summary-level analyses, the user must first provide the dataset, identify the exposure-disease relation and specify distributions to describe the uncertainty in the sensitivity and specificity (or predictive values, when directly measured). The main difference between summary-level and individual record-level PBA is that rather than estimating the adjusted cell counts using the predictive values, we instead simulate Bernoulli trials for each record to reclassify (or leave alone) the potentially misclassified variable. All other steps are the same, and accordingly, if no variables are controlled and no other model complexities are implemented (e.g. a time-to-event model where censoring must be accounted for or other confounders need to be adjusted for), such that the record-level and summary-level models are the same, they should produce the same results (as we show here).

The code requires the exposure and outcome to be dichotomous 0,1 variables. We label the observed exposure as ‘e_obs’ and the outcome as ‘d’. The user sets the number of desired iterations and distributions for the bias parameters, as with summary-level data. The code was designed to use log-linear regression to estimate relative risks (using ODS output in SAS) to be comparable to the analysis in the Supplementary Appendix, but can be adapted for any model. If the code is modified, the user must change the formula for the standard error of the conventional estimate for use in random error and total error simulations (noted in the code where modifications should be made for a different effect measure and its standard error).

Worked example

In the Supplementary Appendix example, we specified a probabilistic quantitative bias analysis for exposure misclassification using the 2 × 2 table and distributions for the bias parameters as shown in Table 2. The conventional observed risk ratio was 2.0. We assumed differential misclassification and set four distributions for the bias parameters. Sensitivity among those with the outcome was specified using a beta distribution (alpha = 25, beta = 3), indicating that our best guess for the sensitivity parameter was 0.89 (25/28) and we were 95% certain it was between 0.76 and 0.98. The sensitivity among those without the outcome was specified using a beta distribution (alpha = 45, beta = 7), indicating our best guess was 0.87 (45/52) and we were 95% certain the sensitivity was between 0.76 and 0.94. Specificity among those with the outcome was defined using a trapezoidal distribution with a minimum of 0.9, lower mode of 0.93, upper mode of 0.97 and maximum of 1.0, and specificity among those non-diseased was specified using a trapezoidal distribution with a minimum of 0.8, lower mode of 0.83, upper mode of 0.87 and maximum of 0.9. We further specified a correlation of 0.8 between the two sensitivity distributions and between the two specificity distributions. We note that users will often use the same type of distribution for all the bias parameters, we use beta and trapezoidal only to provide examples of the coding for each.

Table 2.

Exposure misclassification example data and bias parameters using summary-level and record-level probabilistic bias analysis

(a) Observed data
Observed data	Exposed	Unexposed
Diseased	40	20
Non-diseased	60	80
Total	100	100
Risk	0.4	0.2
Risk ratio	2.0

(b) Bias parameters
Bias parameter	Distribution	Alpha	Beta	Mean
Sensitivity | D+	Beta	25	3	0.893
Sensitivity | D−	Beta	45	7	0.865
Bias parameter	Distribution	Min	Mod1	Mod2	Max	Mean
Specificity | D+	Trapezoidal	0.9	0.93	0.97	1.0	0.95
Specificity | D−	Trapezoidal	0.8	0.83	0.87	0.9	0.85

D+, with the outcome D; D−, without the outcome D; Min, minimum; Mod1, lower mode; Mod2, upper mode; Max, maximum.

With the summary-level approach, we conducted 1 million simulations and it took about 30 seconds to run in SAS and 4 seconds in R. The results are presented at the top of Table 3. A total of 998 ,198 simulations produced bias-adjusted cell frequencies greater than 0, suggesting about 0.18% were discarded due to negative cell counts or 0 cells. Whereas the conventional point estimate was 2.0, the systematic error and total error distributions were centred on about 2.8, suggesting on average the bias was towards the null. The random error interval was narrower than the systematic error interval, showing more uncertainty from systematic error than from random error. Overall, although the point estimate shifted only moderately away from the null, we may be less certain in the effect of the exposure on the outcome, as the total error interval accounting for both systematic and random error is wider than the conventional 95% interval. At the same time, we may be more confident that the exposure increases the risk of the outcome relative to the unexposed condition, as the total error interval now reflects the uncertainty from systematic error, which otherwise would have been viewed as an unresolved threat to validity. We again note that the median value of the systematic and total error interval are roughly the same but it is only because we did not adjust for additional measured confounders in the total error interval.

Table 3.

Results from exposure misclassification example using summary-level and record-level probabilistic bias analysis

Analysis	Median	2.5th–97.5th percentile	Width of interval
Results of summary-level quantitative bias analysis
Random error	2	1.3–3.2	2.5
Systematic error	2.8	2.3–5.0	2.2
Total error	2.8	1.5–7.8	5.1
Results of record-level quantitative bias analysis
Random error	2	1.3–3.2	2.5
Systematic error	2.8	2.3–5.1	2.2
Total error	2.9	1.5–8.0	5.2

For the record-level approach, we specified an analysis using the same 2 × 2 table and input bias parameters from Table 2 for a misclassified exposure. We set distributions for each of the four bias parameters as in the first example. We ran the analysis to 100 ,000 simulations and it took roughly 20 minutes to run in SAS and 9 minutes in R. The increased time is a result of working with a much larger dataset (one record per individual per iteration rather than one record summarizing the full dataset per iteration) and because we need to run one regression model per iteration. Note that this is a small dataset, so larger datasets will take much longer to run. There are strategies to reduce the demand on computational resources,¹⁸ and distributing the simulations between multiple cores could also decrease the time considerably. The results are presented at the bottom of Table 3 and are nearly identical to the summary-level analysis.

Uncontrolled confounding

Our approach to implementing probabilistic bias analysis to simulate a binary unmeasured confounder has also been described elsewhere.^9,13 We describe the algorithm in Supplementary Appendix S2. Briefly, we:

1. specify distributions for the bias parameters (probability of having the confounder in the exposed and unexposed and the strength of effect of the confounder on the outcome);

2. sample values from specified distributions and conduct a simple bias analysis;

3. use the simple bias analysis to estimate the probability of having the confounder within levels of the exposure and outcome;

4. use these probabilities to sample from binomial distributions to impute the missing cell values for the record-level implementation of the PBA (done as adjusted cell counts with a binomial distribution for the summary-level approach and Bernoulli trials for each individual for the record-level approach);

5. repeat this process thousands of times using Monte Carlo methods to get a distribution of estimates adjusted for the uncontrolled confounder;

6. incorporate random error;

7. summarize the results using the median, 2.5th and 97.5th percentiles of the distribution.

The bias-adjustment formulas are provided in Table 4 and have been described in detail elsewhere.¹³

Table 4.

Algorithm to implement probabilistic bias analysis for uncontrolled confounding

	Total		Z = 1		Z = 0
	E = 1	E = 0	E = 1	E = 0	E = 1	E = 0
Case	a	b	A1	B1	A0	B0
Non-case	c	d	C1	D1	C0	D0
	m	n	M1	N1	M0	N0

Step	Example
1. Sample bias parameters	$p_1\sim \mathit{beta}\left(a_{p_1},b_{p_1}\right)$ $p_0\sim \mathit{beta}\left(a_{p_0},b_{p_0}\right)$ ${RR}_{CD}\sim \mathit{Trapezoid}(\mathit{min},\mathit{mode}1,\mathit{mode}2,\mathit{max})$
2a. Calculate expected cell counts	$M_1=m\cdot p_1$ $N_1=n\cdot p_0$ $E\left(A_1\right)=\frac{R{R}_{CD}{M}_{1}a}{R{R}_{CD}{M}_1+m-M_1}$ $E\left(B_1\right)=\frac{R{R}_{CD}{N}_{1}b}{R{R}_{CD}{N}_1+n-N_1}$ ${E(C}_1)=M_1-{E(A}_1)$ ${E(D}_1)=N_1-{E(B}_1)$
2b. Generate probability of having the confounder for all case/exposure combinations	Z=1   E=1       E=0
	Case   Pr(A1)=E[A1]/a  Pr(B1)=E(B1)/b
	Non-case Pr(C1)=E[C1]/c  Pr(D1)=E(D1)/d
3. Sample cell counts	$A_1\sim \mathit{binomial}\left(a,\mathrm{Pr}(A_1\right))$ $B_1\sim \mathit{binomial}\left(b,Pr\left(B_1\right)\right)$ $C_1\sim \mathit{binomial}\left(c,Pr\left(C_1\right)\right)$ $D_1 \sim \mathit{binomial}(d,Pr\left(D_1\right))$ $A_0=a-A_1$ $B_0=b-B_1$ $C_0=c-C_1$ $D_0=d-D_1$ Whereas the end result is the same, for record- level QBA, instead of sampling from binomial distributions, we apply the probabilities to individual records in the database
4. Calculate RR and se(log(RR))	Regression model or Mantel Haenszel to estimate $\widehat{RR}$ and $se\left(\mathit{log}\left(\widehat{RR}\right)\right)$
5. Sample new log(RR)	$\mathit{log}\left(RR\right)\sim N(\mathit{log}\left(\widehat{RR}\right),se(\mathit{log}\left(\widehat{RR}\right))$

PPV, positive predictive value; NPV, negative predictive value; RR, relative risk; Se, sensitivity; Sp, specificity; binom, binomial; QBA, quantatitive bias analysis.

Summary-level vs record-level misclassification adjustment

As with misclassification problems, the approach to simulating an unmeasured dichotomous confounder for a record-level analysis is like the summary-level approach, except that we simulate the confounder for each individual using Bernoulli trials with the probability equal to the appropriate exposure-outcome probability of having the confounder. If the Bernoulli trial is positive, the record is assigned the confounder, and if not, the record is assigned not to have the confounder. The new simulated confounder variable can be adjusted in the analysis along with any other variables the user specifies, and the process can be repeated to run as many simulations as the user wishes, with results summarized as for all other analyses.

Worked example

Code for summary-level bias analysis of an unmeasured binary confounder is available in Supplementary Appendices S12 and S13, available as Supplementary data at IJE online). The code again must be tailored to the user’s dataset by inputting the cells of the observed 2 × 2 table relating the exposure to the outcome. The user must also describe the uncertainty in the three parameters described above, using any probability distribution the user feels valid. We specified an analysis using the 2 × 2 table and input bias parameter for an unmeasured binary confounder using the same observed dataset from the previous example, as shown at the top of Table 5, such that the observed risk ratio is 2.0. We specified distributions for each of the three bias parameters: (i) the probability of having the confounder among the exposed, p(C+|E+) ∼ beta(10,20); (ii) the probability of having the confounder among the unexposed, p(C+|E−) ∼ beta(5,20); and (iii) the risk ratio relating the confounder to the exposure, RR_CE ∼ trapezoidal(min = 1.5, mod1 = 1.7, mod2 = 2.3, max = 2.5). We ran the analysis to 1 million simulations, and it took approximately 20 seconds to run in SAS and less than 1 second in R. Approximately 0.4% of all simulations were excluded by the algorithm because they resulted in impossible values (e.g. probabilities outside the range of 0–1). The results are presented at the top of Table 6. Compared with the conventional, the estimate bias-adjusted for systematic error is closer to the null (1.8 vs 2.0), suggesting the bias was, on average, away from the null. The total error interval is wider than the conventional interval (width 2.5 vs 2.7), suggesting that accounting for the bias should increase our uncertainty, but not by a large amount. For the record-level analysis (Supplementary Appendices S14 and S15, available as Supplementary data at IJE online), using the same dataset and input parameters, results are shown at the bottom of Table 6 for an unmeasured binary confounder. We ran the analysis to 100 ,000 simulations and it took about 30 minutes to run in SAS and 11 minutes in R. The results are almost identical to the summary-level analysis.

Table 5.

Uncontrolled confounding example data and bias parameters using summary- and record-level probabilistic bias analysis

(a) Observed data
Observed data	Exposed	Unexposed
Disease	40	20
No disease	60	80
Total	100	100
Risk	0.4	0.2
Risk ratio	2.0

(b) Bias parameters
Bias parameter	Distribution	Alpha	Beta	Mean
p(C+|E+)	Beta	10	20	0.33
p(C+|E-)	Beta	5	20	0.20
Bias parameter	Distribution	Min	Mod1	Mod2	Max
RRCE	Trapezoidal	1.5	1.7	2.3	2.5

p(C+|E+), probability of the confounder among those with the exposure; p(C+|E-), probability of the confounder among those without the exposure; RR_CE, relative risk related exposure to confounder; Min, minimum; Mod1, lower mode; Mod2, upper mode; Max, maximum.

Table 6.

Results of uncontrolled confounding example using summary- and record-level probabilistic bias analysis

Analysis	Median	2.5th–97.5th percentile	Width of interval
Results of summary-level quantitative bias analysis
Random error	2	1.3–3.2	2.5
Systematic error	1.8	1.5–2.2	1.4
Total error	1.8	1.1–3.0	2.8
Results of record-level quantitative bias analysis
Random error	2	1.3–3.2	2.5
Systematic error	1.8	1.5–2.1	1.4
Total error	1.8	1.1–3.0	2.8

We note that the code can be adapted to use other distributions for the bias parameters, with example code provided in Supplementary Appendices S16 and S17 (available as Supplementary data at IJE online).

Discussion

Quantitative bias analysis has increased in use over the past decade and although easy-to-use software now exists to run simple and multidimensional bias analyses, limited software exists to run probabilistic bias analyses, particularly for individual record analyses. We hope that the code provided here will further allow these methods to be used and developed and allow users to customize the code to their datasets with limited additional programming. The code presented here is an important improvement over our earlier SAS macro, in that the simplifications allow for more efficient run times on large datasets, and due to their simplicity, should allow more users to feel comfortable adapting them to their own work and customizing the code to their own specific bias problems. Further, this code adopts the most recent PBA methods.

It is worth noting that although the code is an improvement on the previous macro, allowing millions of simulations to run in seconds for summary-level analyses, for individual record-level analyses, implementation can still take hours (or even days) depending on the size of the dataset and number of simulations (we recommend upwards of 100, 000 to ensure the entirety of the bias parameter distributions is simulated). We offer several pieces of advice. First, given that coding errors can easily occur, we first recommend running simple sensitivity analyses (we provide free spreadsheets at [https://sites.google.com/site/biasanalysis/short-code] using the median of the bias parameter distributions to get a sense for the expected median result. This initial bias analysis will help the user identify problems should they arise before running lengthy simulations. Second, when running simulations, we recommend starting with 100 or fewer iterations to ensure the code is running appropriately before advancing to larger simulations that might run for days. If the large number of iterations is too much for the computing capacity available, we recommend running a smaller number at a time and saving the results into a dataset that can be combined with other runs of the simulation and summarized. Finally, most modern computers have multiple core processors, and some statistical software (such as R) can make use of multiple cores to improve performance; see also strategies for weighting individual records as a hybrid approach between summary-level and individual-level probabilistic bias analysis.^18,19

Our code is designed to deal with the problem of negative cell counts (or probabilities outside the range of 0–1) that can occur with misclassification problems and confounding problems. When values of Sensitivity and Specificity are chosen that lead to negative cell counts, it implies that the data are not compatible with those values of the bias parameters. The code is set to note this problem in three ways. First, we count the number of observations that produce valid bias adjustments and output this to allow the user to see how many simulations yielded problematic results. Second, we summarize results using only those simulations that produce valid results. Third, we plot histograms that display the distribution of the actual bias parameters used in the observations with valid results. The user can then check that these histograms agree with the density distributions they assigned. If more than a trivial proportion of simulations are discarded, the user should be cautious in their interpretation of results and more exact methods that do not discard impossible values (such as formal Bayesian approaches) may be needed.

Conclusion

It is our hope that providing simple accessible SAS and R code for summary-level and individual record-level quantitative bias analysis will allow for more rapid dissemination of the methods in the literature. Such approaches allow the user to account for both systematic and random error in interpreting study results and should allow us to reduce our over-confidence in study findings.^20,21 These approaches are flexible enough to allow users to adapt the code to more complex problems, and still allow newer users of QBA and less accomplished coders to implement the methods.

Ethics approval

As this was a methods paper using simulated data no IRB approval was sought.

Data availability

All data used are available in the accompanying code. Code can also be found at https://sites.google.com/site/biasanalysis/short-code.

Supplementary data

Supplementary data are available at IJE online.

Author contributions

M.P.F. conceived of the manuscript and drafted the first draft. R.M. wrote code for the manuscript and revised the manuscript. T.L. revised the manuscript and provided input on the code.

Funding

T.L. and R.M. were supported in part by a grant from the US National Library of Medicine (grant R01LM013049).

Conflict of interest

All three authors co-authored a textbook on quantitative bias analysis, for which they receive royalties. M.P.F. also teaches on these methods, for which he receives an honorarium. T.L. is a member of the Amgen Methods Advisory Council and advises on bias analysis methods, for which he receives consulting fees and travel support.