Nondifferential misclassification of outcome under (near-) perfect specificity: a simulation study

Abstract

Mismeasurement of a dichotomous outcome yields an unbiased risk ratio estimate when there are no false positive cases (perfect specificity) and when sensitivity is nondifferential with respect to exposure status. In studies where these conditions are expected, quantitative bias analysis may be considered unnecessary. We conducted a simulation study to explore the robustness of this special case to small departures from perfect specificity and stochastic departures from nondifferential sensitivity. We observed substantial bias of the risk ratio with specificity values as high at 99.8%. The magnitude of bias increased directly with the true underlying risk ratio and was markedly stronger at lower baseline risk. Stochastic departure from nondifferential sensitivity also resulted in substantial bias in most simulated scenarios; downward bias prevailed when sensitivity was higher among unexposed compared with exposed, and upward bias prevailed when sensitivity was higher among exposed compared with unexposed. Our results show that seemingly innocuous departures from perfect specificity (eg, 0.2%) and from nondifferential sensitivity can yield substantial bias of the risk ratio under outcome misclassification. We present a web tool permitting easy exploration of this bias mechanism under user-specifiable study scenarios.

Introduction

Systematic errors can bias associations measured by epidemiologic studies. Bias analysis methods permit rigorous quantitative evaluation and adjustment for systematic errors arising from measurement errors, selection bias, and confounding but are used infrequently in practice. Nondifferential and independent misclassification of a dichotomous outcome is a systematic error with an expected bias toward the null, except under certain conditions.^1-5 A special case arises when the specificity of outcome measurement is perfect (ie, there are no false-positive cases) and the sensitivity of outcome measurement is nondifferential (that is, equal for the exposed and unexposed groups). Under these conditions, the risk ratio associating exposure with disease is unbiased, since the numerator and denominator risks both lack false-positive cases, and the imperfect (but equal) sensitivities cancel by division (Figure 1).⁶ In studies where this special case is expected to hold, bias analysis methods may be deemed unnecessary if the risk ratio is the chosen measure of association.

Figure 1.

Mathematical illustration of unbiased risk ratio under misclassification of a dichotomous outcome with perfect specificity and nondifferential sensitivity.

There are 2 main concerns with relying on this special case of unbiasedness. First, because imperfect outcome measurement usually occurs at the individual level, differential sensitivity of outcome measurement may arise between the exposed and unexposed groups by chance alone. Second, investigators may be inclined to label specificity as perfect when validation data indicate an imperfect value that is close to 100% (eg, 99.5%).⁷ Herein, we carried out a simulation study to explore the impact of these 2 concerns on bias due to outcome misclassification under a variety of plausible conditions.

Methods

Overview of simulation approach

We simulated true and misclassified cohort data under different combinations of sensitivity (Se), specificity (Sp), defined true risk ratio (RR_T), exposure prevalence (P_E), and risk in the unexposed (R₀). We held the cohort size constant at 10000 subjects to reduce record loss due to emergence of empty (zero) cells under parameter combinations that generate sparse data (particularly in the cell for exposed cases). Table 1 summarizes the range of values assigned to each simulation parameter, comprising 288 unique simulation scenarios.

Table 1.

Summary of simulation parameters.

Parameters	Values	Levels
Study size (n)	10 000	1
Risk in unexposed (R0)	0.01, 0.10	2
Exposure prevalence (PE)	0.10, 0.50	2
True risk ratio (RRT)	1.0, 1.5, 2.0, 5.0	4
Sensitivity for outcome (Se)	0.5, 0.8, 1.0	3
Specificity for outcome (Sp)	0.990 to 1.000 by 0.002	6

Generating correctly classified data

For each of the 288 scenarios, we generated 10 000 individual data sets (iterations), allowing up to 200 000 attempts to compensate for deleted iterations that yielded data with empty cells. For each iteration, we generated a 2×2 table of correctly classified data assuming binomial error, where A is the number of exposed cases, B is the number of unexposed cases, N₁ is the total number of exposed (10 000^*P_E), and N₀ is the total number of unexposed (10 000 − N₁). A and B were defined as binomial random variables using scenario-specific simulation parameters:

A ~ binom(N₁, R₀^*RR_T).

B ~ binom(N₀, R₀).

Cell counts C = N₁ − A and D = N₀ − B are the number of exposed and unexposed noncases, respectively. We then calculated the study’s simulation-specific risk ratio without misclassification (which can differ slightly from RR_T) as,

RR_S = (A/N₁)/(B/N₀).

Generating data with misclassified outcome

For each iteration, we generated a misclassified data set by introducing nondifferential misclassification of outcome. Let j index exposure status, with 0 = unexposed and 1 = exposed. To calculate misclassified 2×2 table data, we first generated false-positive and false-negative frequencies, FP_j and FN_j, as follows:

FN₁ = A – binom(A, Se).

FN₀ = B – binom(B, Se).

FP₁ = binom(C, 1-Sp).

FP₀ = binom(D, 1-Sp).

We then derived misclassified case counts (represented by lowercase letters) and calculated the risk ratio under nondifferential misclassification of outcome (RR_M):

a = A – FN₁ + FP₁.

b = B – FN₀ + FP₀.

RR_M = (a/N1)/(b/N0).

Evaluation of simulation results

Definitions of iteration-level variables

We calculated multiple statistics to characterize results from each iteration of the simulation. We calculated the observed bias factor (BF) as the ratio of the misclassified and observed simulation-specific risk ratios (RR_M/RR_S), with and without log₁₀ transformation. We derived the simulation-specific realized values for sensitivity (Se_j) and specificity (Sp_j) of outcome classification using exposure stratum-specific frequencies of true cases, false negative cases, and false positive cases. We quantified the degree of departure from nondifferential sensitivity (Δ_Se) and specificity (Δ_Sp) by subtracting the value in the unexposed from the value in the exposed (ie, Δ_Se = Se₁ – Se₀). We also created indicator variables for the following conditions: (1) the observed simulation-specific risk ratio (RR_S) was within the 95% confidence interval of the misclassified risk ratio (RR_M); (2) the defined scenario-specific true risk ratio (RR_T) was within the 95% confidence interval of the misclassified risk ratio (RR_M); (3) the misclassified risk ratio was greater than the observed simulation-specific true risk ratio (RR_M > RR_S); (4) the misclassified risk ratio was greater than the defined scenario true risk ratio (RR_M > RR_T); (5) the misclassified risk ratio was between the null and the observed simulation-specific risk ratio (1 ≤ RR_M ≤ RR_S); (6) the misclassified risk ratio was between the null and the defined scenario-specific true risk ratio (1 ≤ RR_M ≤ RR_T); and (7) the misclassified risk ratio was less than the null (RR_M < 1).

Summarizing results

For each of the 288 scenarios defined by unique parameter combinations (Table 1), we tabulated summaries (geometric means and proportions) of the previously mentioned statistics over the 10 000 constituent iterations. We quantified the impact of departures from nondifferential sensitivity (Δ_Se) and specificity (Δ_Sp) on bias in the risk ratio by fitting linear regression models of log₁₀-transformed bias factors (dependent variable) on quantile-based categories of Δ_Se and Δ_Sp (independent variables).⁸ We first rounded Δ_Se and Δ_Sp to 4 decimal places, defined reference categories of Δ_Se = 0 and Δ_Sp = 0 (ie, zero after rounding), then parsed observations greater than and less than zero into quintiles. This yielded 11 categories for both Δ_Se and Δ_Sp: 5 levels below zero, a zero category, and 5 levels above zero. We modeled this as a factor variable with the zero category serving as the referent. We present results for an illustrative subset of scenarios herein; full results are available in Figure S1 (scenarios with perfect specificity) and Figure S2 (scenarios with imperfect specificity).

Simulations and analyses were performed with R version 4.3.1 (R Foundation for Statistical Computing, Vienna, Austria). Simulation code is available on GitHub (https://github.com/tpahern/ndmd-sim). We also developed a Shiny web application⁹ to facilitate interaction with our simulation-especially for those who are interested in exploring parameter combinations that do not appear in this article, or who wish to explore scenarios with different cohort sizes and true risk ratios in the protective domain (RR_T < 1). The application (available at both http://galaxy.med.uvm.edu:3838/thomasahern/ndmd-sim/ and https://tpahern.shinyapps.io/ndmd-sim/) includes intuitive controls for parameter input, an array of visualizations to convey results, and an option to download the custom simulated data set for further analysis.

Results

Table 2 summarizes key results of our simulation study for 48 selected scenarios (outcome sensitivity = 0.8 or 1.0; outcome specificity = 0.990 or 0.998; exposure prevalence = 0.1 or 0.5; true risk ratio = 1.0, 2.0, or 5.0; and baseline risk = 0.01 or 0.10). Summary results for all 288 scenarios are available online in Table S1. As expected, there was no bias in the risk ratio under perfect sensitivity and specificity (geometric mean bias factor = 1.00; see Table S1). Geometric mean bias factors also equaled 1 when specificity was perfect and sensitivity was imperfect, confirming that the special case for an unbiased risk ratio under outcome misclassification held up under expectation in our simulation data. Scenarios with positive true risk ratios (RR_T > 1) with imperfect specificity and imperfect sensitivity of outcome classification showed average bias toward the null. The magnitude of bias increased directly with RR_T and became more pronounced with decreasing Se and/or Sp, as expected. Exposure prevalence (P_E) had only minor impact on the magnitude of bias, but lower baseline risk (R₀) was associated with markedly stronger bias than higher baseline risk in most scenarios (Table 2).

Table 2.

Summary of simulation results under selected parameter combinations. Full results are available in the online supplement.

Scenario parameters				Baseline risk (R 0 ) = 0.01			Baseline risk (R 0 ) = 0.10
Sensitivity	Specificity	Exposure prevalence (P E )	True risk ratio (RR T )	Misclassified risk ratio (RR M ) a	Bias factor (RR M /RR S ) a	Proportion of RR M  95% CI containing RRT	Misclassified risk ratio (RR M ) a	Bias factor (RR M /RR S ) a	Proportion of RR M  95% CI containing RRT
0.8	0.990	0.1	1.0	0.97	1.026	0.95	0.99	0.999	0.95
			2.0	1.41	0.722	0.66	1.88	0.944	0.89
			5.0	2.74	0.552	0.02	4.55	0.910	0.54
		0.5	1.0	1.00	0.999	0.95	1.00	1.000	0.95
			2.0	1.44	0.716	0.32	1.89	0.944	0.82
			5.0	2.78	0.551	0.01	4.56	0.910	0.51
	0.998	0.1	1.0	0.96	0.999	0.96	1.00	0.999	0.95
			2.0	1.75	0.895	0.94	1.97	0.987	0.95
			5.0	4.18	0.837	0.85	4.90	0.980	0.93
		0.5	1.0	1.00	1.000	0.95	1.00	1.000	0.95
			2.0	1.81	0.901	0.91	1.98	0.989	0.95
			5.0	4.24	0.838	0.80	4.91	0.981	0.93
1.0	0.990	0.1	1.0	0.98	1.029	0.95	1.00	1.001	0.95
			2.0	1.48	0.755	0.70	1.90	0.954	0.90
			5.0	2.97	0.597	0.04	4.64	0.927	0.59
		0.5	1.0	1.00	1.000	0.95	1.00	1.000	0.95
			2.0	1.50	0.747	0.39	1.91	0.954	0.84
			5.0	3.00	0.596	0.01	4.64	0.927	0.58
	0.998	0.1	1.0	0.96	1.011	0.95	1.00	1.000	0.95
			2.0	1.79	0.919	0.94	1.98	0.990	0.95
			5.0	4.30	0.867	0.86	4.92	0.984	0.94
		0.5	1.0	1.00	1.001	0.95	1.00	1.000	0.95
			2.0	1.84	0.915	0.91	1.98	0.990	0.95
			5.0	4.36	0.866	0.82	4.93	0.984	0.93

^aGeometric mean over 10 000 iterations.

Impact of imperfect specificity on bias in the risk ratio

We observed substantial bias across the range of imperfect specificity values (ie, 0.990-0.998). While bias was less pronounced at higher values of imperfect specificity, it was still possible to observe substantial bias when specificity was as high as 99.8%—a value that, if observed in a validation study, may be deemed “close enough to 100%” to result in no bias of the risk ratio (Table 2, Table S1). For example, under a true risk ratio of 5 P_E = 0.1, R₀ = 0.01, a specificity of 99.8% yielded substantial bias of the risk ratio (BF < 0.9) across all values of sensitivity, ranging from BF = 0.87 under perfect sensitivity to BF = 0.76 under 50% sensitivity (Table 2 and Table S1). Under these scenarios, the 95% confidence interval around the misclassified risk ratio estimate (RR_M) contained the true risk ratio (RR_T) in 86% and 80% of iterations, respectively. Bias that displaced nonnull estimates (RR_T > 1) below the null was uncommon but increased with decreasing sensitivity when P_E equaled 0.1 and R₀ equaled 0.01, reaching a maximum at 14% of iterations when Se = 0.5, Sp = 0.990, and RR_T = 2.

Impact of stochastically differential sensitivity on bias in the risk ratio

In simulation scenarios with perfect specificity, stochastically differential sensitivity also resulted in substantial error in the risk ratio in most simulated scenarios. Negative values of Δ_Se (ie, quantiles below zero indicative of higher sensitivity among the unexposed compared with the exposed) resulted in downward error in the risk ratio (bias factors < 1), whereas positive values of Δ_Se resulted in upward error in the risk ratio (bias factors > 1). The magnitude of error was a function of overall sensitivity (greater bias with lower sensitivity), exposure prevalence (less bias with higher exposure prevalence), risk in the unexposed (less bias with higher baseline risk), the true risk ratio (less bias with higher risk ratios), and the magnitude of the departure from nondifferential sensitivity (Figure 2 and Figure S1). These results extend to scenarios with imperfect specificity, which introduces stochastic departures from both nondifferential sensitivity and specificity; plots visualizing the mutually adjusted impacts of Δ_{Se and} Δ_Sp on the risk ratio are available in Figure S2. The impact of Δ_Se and Δ_Sp at the iteration level are visualized with scatterplots in Figure S3. We note that larger cohort size will reduce the magnitude of departures from nondifferential sensitivity or specificity and therefore attenuate the contribution of such errors to deflections in the risk ratio. We illustrate this in Table S2 for selected simulation scenarios with perfect specificity and imperfect sensitivity, comparing cohort sizes of n = 10 000 (as used for our main simulation) and n = 1 000 000. Our Shiny application allows users to further explore the role of cohort size in different simulation scenarios.

Figure 2.

Visualization of coefficients from linear regression models of the log₁₀-transformed bias factor (log₁₀[IPR_M/IPR_C]) as a function of departure from nondifferential sensitivity (Δ_Se = Se_exposed-Se_unexposed) in simulation scenarios with perfect specificity. Vertical dotted lines show the reference values of zero (nondifferential Se). Categories to the left of these lines are increasingly negative, and categories to the right of these lines are increasingly positive (both divided into fifths of the scenario-specific distribution). Results are presented only for 2 illustrative simulation scenarios; full results are available in Figure S1.

Discussion

We conducted a simulation study to explore the effects of outcome measurement error on bias in the risk ratio. Specifically, we sought to investigate the consequences of small violations of the assumptions behind a commonly cited special case of outcome misclassification, wherein the risk ratio is unbiased when there are no false-positive cases, and the sensitivity of outcome measurement is nondifferential between exposure groups. This special case can be used with or without corroborating validation data to claim that risk ratio associations in a given study are unbiased.

Our results showed that even very slight departures from perfect specificity (0.2%) can yield substantial bias of the risk ratio and that stochastic departures from nondifferential sensitivity of outcome classification introduce bias, with the magnitude of bias increasing directly with the magnitude of the difference in sensitivity between exposure groups. The direction of bias depended on whether sensitivity was higher in the exposed or unexposed group, with greater sensitivities in the exposed groups producing bias away from the null and vice-versa. These findings are particularly relevant for studies in which the sensitivity is low, and therefore the potential for higher-magnitude differences between groups is increased.

Results of our simulation demonstrate that departures from perfect specificity as small as 0.2% (that is, a specificity of 99.8%) can yield substantial bias toward the null, especially in cases where the true risk ratio is large. When considering potential sources of bias and/or interpreting validation data, investigators must therefore be cautious in inferring that specificities near 100% will yield only negligible bias. Even when investigators base this assumption off validation data that estimate perfect specificity, confidence intervals around that estimate will often leave open the possibility of small deviations from a specificity of 100%. We reiterate that our results pertain only to risk ratio measures of association; when other measures of association are employed, bias due to outcome misclassification would behave differently.

Jurek et al previously simulated stochastic departure from nondifferential measurement error in the context of exposure misclassification, another scenario with an expected bias toward the null.¹⁰ They showed that applying imperfect sensitivity and specificity to individuals (as one would expect measurement error to operate in the real world) sometimes yielded estimates biased away from the null or biased across the null such that causal true associations appeared protective. As the Jurek et al study highlighted important nuances in nondifferential misclassification that led to adjustment of the usual heuristic surrounding it, our simulation results should motivate caution when arguing against the presence of bias due to outcome misclassification because of ostensibly perfect specificity and nondifferential sensitivity between exposure groups. We encourage authors to use our simulation web tool to explore bias arising in study scenarios that reflect conditions in their own work.

Supplementary material

Supplementary material is available at the American Journal of Epidemiology online.

Funding

This work was supported by an award from the US National Library of Medicine (R01 LM013049) to T.L. L.C. was supported by K99CA277580 from the US National Cancer Institute.

Conflicts of interest

No author declares a conflict of interest.

Data availability

All simulation code and the simulated data set upon which this report is based are freely available on the senior author’s GitHub repository: https://github.com/tpahern/ndmd-sim.