Addressing bias due to measurement error of an outcome with unknown sensitivity in database epidemiologic studies. A contribution from the ConcePTION project

Abstract

In epidemiologic database studies, the occurrence of an event is measured with error through an indicator whose specificity is often maximized, at the expense of sensitivity. However, if the indicator has low sensitivity, measures of occurrence are underestimated. In association studies, risk difference is biased, and risk ratio may be biased as well, in either direction, if the sensitivity is differential across exposure groups. In this work, we show that if an auxiliary screening indicator can be defined to complement the main indicator, estimates of the positive predictive value of both indicators provide tools to estimate the sensitivity of the primary indicator or a lower bound thereof. This mitigates bias in estimating the number of cases, prevalence, cumulative incidence, rate (particularly when the event is rare), and, in association studies, risk ratio and risk difference. They also allow testing for nondifferential sensitivity. Although direct estimation of sensitivity is often infeasible, this novel methodology improves evidence based on data obtained from reuse of existing databases, which may prove critical for regulatory and public health decisions.

Introduction

Electronic health care data sources are a key resource in assessing the occurrence of events in human populations, playing a major role in public health and regulatory decision-making. During the recent coronavirus crisis, calculation of background rates of events that may have later occurred as adverse reactions to the new vaccines were requested by the European Medicines Agency ahead of the vaccination campaign and, in March 2021, were used to rapidly assess potential causality of safety signals emerging from spontaneous reports.^1,2 In pregnancy studies, the occurrence of adverse maternal or infant outcomes in the general population and in populations exposed to medications can be used to provide signals on the safety of use by pregnant women, because clinical trials in this population are rarely conducted.³ Information on the prevalence of conditions such as hypertensive disorders during pregnancy is also important for pregnancy studies.⁴

In database studies, the occurrence of the event of interest is measured using an indicator. Due to the nature of such databases, the indicator may be affected by errors: false positives are cases that are detected by the indicator, but are not true cases; false negatives are true cases that are missed by the indicator. The degree of misclassification of an indicator is measured by its validity indices,⁵ including sensitivity (SE), which is the proportion of true positives among true cases in the population, specificity (SP), which is the proportion of true negatives among noncases in the population, and positive predictive value (PPV), which is the proportion of true positives among cases detected by the indicator. Guidelines in the conduct and reporting of database studies recommend to estimate validity indices,⁶ which would allow adjustment of observed rates using a straightforward correction (PPV/SE).⁷

However, validation is often not feasible due to cost, time constraints, or ethical reasons.⁸ When validation studies are conducted, they often estimate only the PPV. Indeed, estimation of the PPV requires extracting a sample of cases positive for the indicator and their assessment through comparison with a gold standard. On the contrary, the true outcome is not observed, and it is often a rare event, thus estimating the sensitivity requires large validation samples to get enough true positive cases.^9-15 Therefore, the rate of occurrence is overestimated if SE > PPV, or underestimated if adjusted only with the PPV or if PPV > SE.

A second problem is when the objective of the database study is to assess a causal relationship between an exposure and an outcome. In this case, misclassification of the outcome or the exposure may introduce bias both in the risk ratio and in the risk difference.¹⁶ Each validity index can be defined with respect to the exposure groups, and the SE or SP of the indicator of the outcome is said to be nondifferential if it does not depend on the exposure group. A commonly accepted idea in epidemiology is that nondifferential misclassification of the exposure skews the expected value of an estimator toward the null. Another established principle regarding misclassification of the outcome is that if the SE is nondifferential and the PPV is 1, then the RR is unbiased (for this reason, guidelines recommend choosing indicators that minimize false positives, so the PPV approaches 1). However, the mentioned results may not hold exactly, because even if the data generation process has nondifferential errors, observed errors can still be differential, especially in small populations.¹⁷ Furthermore, it has been demonstrated that when misclassification of exposure is not independent of misclassification of the outcome, even a small error can significantly bias the results in both directions, especially in scenarios with low disease prevalence.^18,19 In fact, even if we may assume that risk ratio is not biased, risk difference is prone to bias irrespective of whether SE is differential.

In this article, we build on a strategy suggested by Lanes and Beachler²⁰ to correct outcome misclassification, while we assume the exposure to be recorded without error. Causes for misclassification of exposure are classified in a recent study²¹ and include purchase over-the-counter, medicines prescribed by a different prescriber or to a different patient, and medicines prescribed but not collected or not ingested/administered. The assumption of perfect measurement of exposure holds when such cases are less likely to happen, such as when a medicine is newly introduced in the market, with an uncertain risk profile, and/or is costly, because prescription, dispensing, and administration are expected to be more strictly controlled by the health care system that is originating the health care data.

Alongside a primary outcome indicator with a high PPV, we suggest searching for an auxiliary screening indicator aimed at capturing all (or nearly all) the cases so it has near-perfect SE, while still being specific enough to exclude many noncases.^20,22 Estimating the PPV of both indicators then becomes feasible, possibly stratified per exposure.

We consider the situation when a screening algorithm is available. We have an estimate of PPV for both indicators, and we are able to make at least 1 in a range of assumptions. We first show it is possible to estimate the SE of the primary indicator or a lower bound thereof. Furthermore, in the case when an exposure is involved in the study and the PPV of both indicators is estimated across exposure strata, we provide a hypothesis test to verify whether the SE of the main indicator is nondifferential. A simulation study is conducted to verify the performance of the test in multiple scenarios. Finally, we demonstrate how to use the SEs estimated from the PPVs to adjust, or to obtain bounds on, the number of cases, risks, risk ratios, and risk differences.

Methods

Notation

We denote by Y the binary variable for the true value of the outcome of interest, with A being an observed, possibly misclassified, binary variable that is the primary indicator for Y, and by B being a screening indicator for the same outcome. Combining A and B yields further indicators, such as , namely the indicator retrieving all cases retrieved by either A or B, and , which is the intersection indicator retrieving all cases retrieved by both A and B.

We assume a framework with a finite target population with N subjects. For an indicator I, we denote by TP_I, FN_I, and FP_I the true positives, false negatives, and false positives, respectively, in the target population. Then, is the number of subjects in the study population for whom the outcome has occurred and is the number of subjects identified by I.

We indicate with the true, unobserved prevalence of Y; we indicate with the observed prevalence. Note that does not depend on the indicator.

The validity indices are defined for each indicator I as follows: ; ; and.

We focus on the case of a binary exposure, denoting with e the exposed and with the nonexposed. We denote by and the number of subjects in the study population who are in the exposed and nonexposed groups, respectively. Then, is the SE for the exposed and is the SE for the nonexposed. Similarly, is the SP for the exposed and is the SP for the nonexposed. An indicator I has nondifferential SE if and nondifferential SP if .

Lastly, we adopt the notation in which the “hat” symbol above a parameter (eg, ) indicates we are referring to a sample statistic rather than the population value of the parameter.

SE of indicators

Estimating the SE

We first consider the situation where B is a comprehensive screening indicator, namely, an indicator for which , though it can include false positives. The following formula, derived from the interrelation between validity indices as described in Appendix S1 and proven in Appendix S2, shows how the SE of the primary indicator can be derived from observed prevalences and PPVs:

(1)

As an example, consider a study on myocardial infarction under the assumption that all case patients who are not promptly admitted to a hospital die. In this case, A is the indicator retrieving hospital admissions with a specific diagnostic code of infarction, whereas B is the indicator retrieving hospitalization with unspecified codes of infarction or death for any cause.²⁰

If cannot be assumed to be 1, it is proven in Appendix S2 that the right-hand side of eqn (1) is an upper bound for the SE of A, namely,

(2)

If no case is retrieved by both A and B, namely , eqn (2) simplifies to

(3)

The approach summarized by eqns (1), (2), and (3) has the advantage of providing an estimate of the SE based on PPV; the drawback is that the estimator of the SE is based on the estimators of 3 PPVs, namely those of the primary indicator A, the screening indicator B, and their intersection . This could imply a loss of accuracy due to conveying multiple sources of uncertainty. Equations (1), (2), and (3) can be written using the absolute number of subjects identified by an indicator (N_I) in place of the observed prevalence (P_I). Thus, by way of illustration, eqn (3) becomes:

(4)

Example: Estimating an upper bound for the SE of an indicator of angioedema

This example was presented at the 38th International Conference on Pharmacoepidemiology and Therapeutic Risk Management.²³ Angioedema is a local, circumscribed edema due to increased plasma leakage from capillaries into the deep layers of the skin and mucous membranes. The Entresto study is a postauthorization safety study requested by the European Medicines Agency as part of the Risk Management Plan of Entresto, a medicinal product indicated for the treatment of heart failure. One of the objectives of the study was to estimate the incidence and relative risk (RR) of angioedema, which is a suspected adverse reaction to the medicinal product.²⁴

As part of the Entresto study, a validation study was conducted in the data source, accessed by ARS Toscana, to estimate the PPV and SE of an indicator A for angioedema (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] code 995.1). A screening indicator B was identified, retrieving cases of hypersensitivity (ICD-9-CM codes 374.82, 376.33, 478.25, 478.6, 478.75, 508.8, 708.0, 708.1, 708.8, 708.9, 782.3995.0, 995.2, 995.27). In the study population, A retrieved N_A = 34 cases and B retrieved N_B = 451 cases, whereas no case was found by both A and B; hence, . All cases retrieved by A were validated, and 12 were found to be true cases; hence, . A sample of 96 cases was validated from those retrieved by B, and 8 were found to be true cases; hence, . Therefore, from eqn (4),

Note that even though the PPV of B is much lower than the PPV of A, because its observed prevalence is much higher, the true cases among those retrieved by B may be a substantial share. This will be quantified in section ``Number of cases and risk''.

A test for nondifferential SE

The nondifferentiality hypothesis states that the SE of the primary indicator is identical across exposure groups, namely

(5)

To carry out the test for the hypothesis of nondifferential SE, hypothesis (5) can be rewritten as

(6)

To derive the test statistic, we estimate the sensitivities of A in the exposure strata by exploiting a screening indicator B. Specifically, in Appendix S3, we prove that, under the assumption of nondifferentiality of the SE of (ie, ), hypothesis (6) can be expressed in terms of observed prevalences () and PPV as follows:

(7)

The estimate of the PPV, denoted as , is achieved by comparing each subject with a gold standard, which provides the true classification of the outcome. The validation study must be carefully designed to provide accurate estimates of the PPVs, and the sampling process must consider each exposure stratum and each indicator.

Once the required PPVs are estimated, given that the observed prevalences P are known, it is straightforward to calculate the value of test statistic. To carry out the test, it is required to determine the distribution of the test statistic under the null. This is not straightforward; therefore, we proceed with the bootstrap method²⁵ and reject the hypothesis if the 95% CI does not include 0.

Simulation study

To assess the performance of the proposed nondifferentiality test, we conducted a simulation study over multiple scenarios in which the indicators’ validity indices and the population characteristics were varied. We generated 2 binary vectors: 1 representing the true outcome of interest Y and the other representing the exposure groups e and . The proportion of subjects exposed was fixed at 5%. We explored 3 scenarios for the true prevalence of the outcome (Y = 1) in the unexposed group: (rare), , and (common). Moreover, we considered 3 scenarios for the RR: = 1, 1.2, and 2.

Then, we generated 2 indicators of Y, denoted with A (primary) and B (screening), which are misclassified with specific error rates for each exposure group. was fixed to 0.85 or 0.95. In each scenario, remains equal in both exposure groups to comply with the assumption of nondifferentiality of the SE of .

The values of the SEs are chosen to produce an SE ratio in {0.6, 0.8, 1, 1.25, 1.67}, where values greater than 1 are reciprocals of those less than 1 (in fact, 1.25 = 1/0.6 and 1.67 = 1/0.8). In particular, when , then was set at 0.5 and varied in {0.3, 0.4, 0.5, 0.625, 0.835}; alternatively, when , then was fixed at 0.75 and varied in {0.45, 0.60, 0.75, 0.94} (in this case, the fifth value is missing because it is impossible to produce a SE ratio equal to 1.67).

To study the possible scenarios in real-life situations, we varied the intersection among A and B, considering 2 values for : 0 and 0.2. When was equal to 0, was set to 1 to create a nonintersection scenario in which no subject tests positive for both indicators. Nonintersection scenarios are plausible, as seen in the Entresto study described in Section ``Example: estimating an upper bound for the sensitivity of an indicator of angioedema''.

The SP was set to and for A and B, respectively, in both exposure groups; this implies that the PPV has a maximum of 90% for A and 50% for B. We further assumed that , that is, the number of false positives of A is independent of the number of false positives of B.

For each scenario, we generated a finite population of size 1 million with the aforementioned characteristics. Then, repeating 1000 times, we drew a validation sample and carried out the test. We considered 3 different sizes of the validation sample: 250, 500, and 750.

We carried out random sampling across indicators and exposure. Specifically, we sampled 40% of the subjects from those with A = 1, 40% from B = 1, and 20% from A = 1 and B = 1 jointly. Each of these groups has 50% exposed and 50% unexposed. In eqn (7), the PPV was replaced by its estimate , given by the proportion of true positives (Y = 1) in the subset of subjects flagged by the indicator (A = 1 or B = 1).

The nondifferentiality test was conducted using the bootstrap method.²⁵ Specifically, 500 bootstrap samples were generated from the validation sample. For each bootstrap sample, the test statistic was calculated, and its empirical distribution was derived. The 95% CI was then computed, and the null hypothesis was rejected if the interval did not encompass zero. The power of the test is the proportion of times the test rejected the null hypothesis, denoted as Pr_rej. When the null hypothesis is true (SE ratio = 1), Pr_rej is the empirical significance level of the test.

The simulation was conducted using R software. The code is freely available on GitHub at the following link: https://github.com/GiorgioLimoncella/NonDifferentialityTest.

The findings of the simulation study are summarized in Figure 1 and Figure 2, which show how the rejection rate Pr_rej varies according to the true SE ratio in the case when and , respectively; graphs are stratified by , , and the RR.

Figure 1.

Proportion of times when the null hypothesis is rejected (Pr_rej). Different scenarios (n = 18) are presented in which sensitivity (SE) , relative risk (RR), and vary. Each panel presents the power of the test power across 3 different sample sizes: 250, 500, and 750. is set to 0.5.

Figure 2.

Proportion of times when the null hypothesis is rejected (Pr_rej). Different scenarios (n = 18) are presented in which sensitivity (SE) , relative risk (RR), and vary. Each panel presents the power of the test power across 3 different sample sizes: 250, 500, and 750. is set to 0.75.

First, under the null hypothesis of a SE ratio equal to 1, the proposed test has a Pr_rej close to the significance level 0.05. Then, the power of the test depends on the factors of the simulation design in the expected way:

• positive association with the size of the validation sample (strong)

• positive association with the distance of the SE ratio from the null hypothesis (strong)

• positive association with the baseline SE in the unexposed (strong)

• negative association with the SE of the intersection (strong)

• positive association with the distance of the RR from 1 (moderate)

• negative association with the prevalence in the unexposed (weak)

Now we explore the cases when the power is large, namely, exceeding 0.8. For a validation sample of size 750, the power is always large if the SE ratio is 0.6 or 1.67, whereas, for values closer to 1, the power is large only for some configurations with . For a validation sample of size 500, the performance worsens, though the power is still large for most configurations if the SE ratio is 0.6 or 1.67. Finally, for a validation sample of size 250, the performance further worsens, so the power is large only for some configurations if the SE ratio is 0.6 or 1.4, whereas it is quite low if the SE ratio is closer to 1.

Adjusting the misclassification bias for number of cases, risk, risk ratio, and risk difference

In this section, we discuss a situation in which we have an observed outcome Y and have conducted validation studies for an indicator A with high SP and a screening algorithm B, obtaining estimates of the PPV for both indicators. The issue is how this information can be used to adjust measures of the number of cases and, as long as the PPVs have been estimated across exposure strata, to adjust the risk ratio and the risk difference.

In this section, we use the term “risk” to indicate a measure of risk where the numerator is the number of cases, and the calculation of the denominator does not depend on the detection of cases. This is the case for the prevalence, the cumulative incidence, and (with approximation) the rate of a rare event: in the latter case, indeed, the risk is the number of new cases divided by the person-time still at risk, which depends on the detection of cases. However if the event is rare, a small variation in the detection of new cases has a negligible impact on the person time at risk.

Number of cases and risk

The number of cases observed in the population N_A differs from the true number N_Y. If we know PPV_A and SE_A, a straightforward correction is possible:

(8)

This is because

Denoting with R_Y a measure of the risk for Y in the population and with R_A its measure through A, and dividing each member of eqn (8) by the common denominator, we obtain the following formula:

(9)

When we can assume that , the number of cases and the true risk are then obtained as follows:

(10)

and

(11)

In cases where it cannot be assumed that , a lower bound for the number of cases and for the risk is derived:

(12)

and

(13)

For example, in the Entresto study, we can adjust N_Y as follows. Because , the correction factor in is

(14)

Thus, the true number of cases is at least 1.45 times higher than N_A = 34:

(15)

Similarly, an estimate of risk for the study based on A would underestimate the true risk by at least 45%.

Risk ratio

The true risk ratio is based on the outcome Y:

(16)

In database studies, it is approximated using the observed indicator A:

(17)

Brenner and Gefeller⁷ proved the following relationship:

(18)

Note this formula also holds for rate ratios as long as the event is rare.

The common approach to adjust for measurement error is to assume that the SE is nondifferential across exposure groups () so that the last factor in eqn (18) disappears and the correction only needs the PPVs of A. If the screening algorithm B is available and it can be assumed that is nondifferential (), the statistical test introduced in the previous section can be used to test the hypothesis of nondifferential SE and make the results more robust. If the test fails, but it still can be assumed that is nondifferential, then the correction (18) becomes

(19)

Risk difference

The risk difference is defined as

By eqn (8), the risk difference can be calculated through the validity indexes of an indicator A as follows:

(20)

If A is chosen to have a very large PPV, this may come at the expense of the SE. shows that this may induce a significant underestimation of the risk difference: the PPV of the screening indicator B can help quantify this.

Multiple scenarios are possible depending on the assumptions that can be made on A and on the PPVs that are available or can be estimated. We first describe a simple scenario (scenario 1) and prove a lower bound for the underestimation of the risk difference based on PPV_B. We then discuss a set of formulas that can be used when this simple scenario is not valid (scenarios 2-5) and prove them in Appendix S4.

All scenarios, except the second, assume the existence of the screening indicator B. In scenarios 1 and 3, the condition that SE_A is nondifferential is required. This assumption can be justified on theoretical grounds or, if the PPV_B can be estimated across exposure strata, it can be verified through the nondifferentiality test of Section ``A test for non-differential sensitivity''.

Scenario 1: SE_A nondifferential () and , and . This is the case when indicator A has been chosen to be highly specific. The statement that PPV_A = 1 in both exposure strata may be an assumption or may be supported by a validation study. Replacing the assumptions in proves the following formula:

where RD_A is the observed risk difference

Hence, using the upper bound for SE_A give in eqn (3), we get the following lower bound for the estimator of the risk difference:

(21)

(22)

Note that this formula does not require that PPV_B is estimated across exposure strata.

Scenario 2: SE_A nondifferential (). This is a scenario in which the auxiliary indicator B is not available but estimation stratified by exposure of the PPV of A can be obtained. Then, the following lower bound can be used:

(23)

(24)

Scenario 3: SE_A nondifferential () and . This is the generalization of scenario 1 to the case when . Again, a lower bound can be provided where the PPV of B can be estimated in the general population (and, therefore, be retrieved from a previous study, if available):

(25)

If then:

(26)

Scenario 4:   nondifferential . This is the generalization of scenario 1 to the case when the assumption that there are no false positives in A cannot be made, and . Then, the PPV of A, B, and (if ) must be estimated across exposure strata, and the following lower bound holds:

(27)

If, on top of this, the assumption can be made, we obtain the equation:

(28)

Scenario 5: the PPV of A is greater than SE in the exposed group (). We finally give a formula that can be used when PPV of A and B are available only among nonexposed: when exposure is rare, such parameters can be assumed to be equal to those in the general population, that may be available from previous studies. The formula holds under the assumption that the PPV of A among exposed is higher than its SE, which is plausible if A can be built with convincingly high SP.

(29)

where .

Discussion

When an event Y is the outcome of a database study, it is common practice to choose a very specific indicator A for , and either assume that it has no false positives or, if validation is possible, to estimate PPV_A to account for false positives captured by A. In this article, we have elaborated on the notion of a screening indicator B aimed at capturing all (or nearly all) of the cases missed by A so that it has near-perfect SE while still being specific enough to exclude many noncases. We showed that, by including the estimation of PPV_B in the validation study, we can at least partially account for false negatives that A fails to identify. In Table 1, we summarize the opportunities provided by using a screening algorithm.

Table 1.

Scenarios for estimating sensitivity of A, cases, risk, risk ratio, risk difference, and testing for nondifferentiality.

	No screening algorithm	With screening algorithm
Population not stratified per exposure	PPV A  available		PPV A , PPVB,  available
Sensitivity of A, no. of cases risk	Not estimable	Unbiased	Upper bound
Underestimated		Unbiased	Reduced underestimation
Underestimated		Unbiased	Reduced underestimation
Population stratified per exposure	PPV A  available in both strata	PPV A , PPVB,  available in both strata
	–	a
Risk ratio Risk difference Differentiality of A	Unbiased if	Unbiased	Unbiased	Unbiased if
Biased	Unbiased	Lower bound in absolute valueb	Biased
Not testable	Testable	Testable	Not testable

Abbreviations: PPV, positive predictive value; SE, sensitivity.

^aIn this scenario, sensitivity across exposure groups can be estimated, and measures of association can be adjusted using bias analysis.

^bThis corresponds to scenario 4. Other scenarios are reported and allow for different improvements.

Recently, the estimation of the SE of indicators was recommended by regulatory authorities as a means to improve the quality of evidence generated for regulatory purposes.⁶ Although this recommendation is considered very arduous to comply with,⁸ we introduced a method that partially meets the recommendation.

When estimating the occurrence of Y, we provided a formula to adjust, or provide a lower bound for, the number of cases, the prevalence, the cumulative incidence, and the incidence rate of Y (the latter being valid only for rare events). For association studies, we introduced several tools to address differential misclassification.

Choosing an appropriate screening algorithm may depend on several factors. An extreme example is when , meaning B retrieves all units not identified by A (ie, covers the whole population; in this case, ). However, estimating the PPV of B in this scenario is not convenient, because the expense is similar to that of directly estimating the SE of A. Conversely, an overly specific algorithm may fail to detect any individuals not already identified by the main algorithm (no false negatives of A in B). Here, the screening algorithm provides no additional information to the main algorithm and is not useful for SE estimation.

A key factor to be taken into account is the sample size of the validation study which is, for feasibility, typically on the order of hundreds (this is why we chose this range in our simulations). To be useful, the screening indicator must identify at least a few true cases outside of A within the validation sample. To this end, SP_B should be large enough depending on the prevalence (this is why we set in our simulations). The choice of B, which has to be done before the validation study, should be based on the clinical plausibility that cases are found outside of A.

In our simulation studies, we explored different scenarios concerning the validation indices of B, even though we consistently remained within scenarios where prevalence and SP_B are linked by the relationship . The power of the test does not primarily depend on the specifics of B (as long as some true positives are identified), nor does the estimation of SE_A. The power of the test mainly depends on the SE ratio across exposure groups, the intersection between A and B, and the sample size. Regarding the estimation of SE, the uncertainty of the estimate depends on the uncertainty in estimating the PPVs, which, in turn, primarily depends on the sample size.

A set of important examples on how the screening indicator B can be defined was constructed in the Vaccine COVID-19 Monitoring Readiness (ACCESS) study¹ during the recent pandemic emergency. Ahead of the vaccination campaign, the ACCESS study was tasked by the European Medicines Agency to calculate background rates for 41 potential adverse events of special interest. Using the medical definitions of each event, the tool Codemapper,²⁶ which leverages the Unified Medical Language System, provided a comprehensive list of potentially relevant diagnostic codes. These codes were categorized by medical experts as “narrow” if they strictly imply the medical condition of interest, or as “possible” if they could also denote other conditions with similar clinical presentation. Such lists are stored in the Zenodo public repository; for example, see the definition of meningoencephalitis.²⁷ The list of codes tagged as narrow was used in ACCESS as algorithm A to identify background rates of the event of special interest. The possible codes list was not used. In future applications, it could serve as the screening algorithm B. Even if is not guaranteed (eg, patients with nonsevere adverse events of special interest may not access the health care system), validating a sample of cases detected by possible codes and using our formulas would still reduce the underestimation of the incidence when compared with the incidence estimated using only narrow codes. This would increase the number of expected cases in vaccinated cohorts, thereby reducing the risk of triggering false alerts of an adverse reaction to vaccines in an observed-to-expected analysis.^2,28

Our tools allow the application of the following strategy. When estimating the fitness-for-purpose of a data source to assess the occurrence of an event Y, a specific indicator A for Y can be sought alongside a screening indicator B. The SE of a A in the population can be estimated by validating both a sample of A and a sample of B, using the methodology described in Section ``Sensitivity of indicators''. This information can be used immediately to assess the number of cases of Y in the population and in subpopulations. Moreover, it can be used in a later stage when association studies are conducted with Y as an outcome.

If the risk ratio is the measure of association, knowledge of the SE of A in the population can be used to evaluate the impact of differential misclassification: if A has low SE in the population, the impact of differential SE can be high, because SE may become substantially higher in special populations, such as those exposed to a new medication.

Differential SE is not a trivial matter. In safety studies, it may occur when studying suspected adverse reactions to a medicinal product because a clinician assessing the symptoms of a patient may use their exposure to the medicine in the diagnostic process, and the knowledge that the outcome is a possible adverse reaction may lead to record a more specific diagnostic code. Our tools are based on estimates of the PPVs of A and B across exposure strata. We introduced a statistical test for nondifferential SE based on the assumption that is nondifferential and exposure is not misclassified.

Our test showed good power in detecting modest departures from nondifferentiality with a validation sample of size 500, retrieved from cases detected by A, B, and (if nonempty), independently on the prevalence of Y. This test can be used to decide whether association measures between Y and exposure are unbiased if adjusted by the PPV of A. If the test fails, we introduced a formula to adjust RRs and rate ratios for a rare Y, based on the same PPVs that were used for the test.

When applying our formulas to the case of association studies, we release the assumption that sensitiviy of A is nondifferential and replace it with the request that is nondifferential. This is automatically satisfied when B captures all true cases missed by A but can be assumed in other situations. Indeed, note that when the physician assesses a set of symptoms, use of medications, especially in case of a suspect adverse reaction, can be used in the process of diagnosis to obtain a more specific diagnosis. Those without the medication presenting the same symptoms would then be registered with a less specific diagnosis. This is a case when the SE of the specific indicator A would be differential, whereas the SE of would not. A practical example would be the case of angioedema and angiotensin-converting enzyme inhibitors: because angioedema is a known adverse reaction to angiotensin-converting enzyme inhibitors, users (ie, individuals in the exposed stratum) will be more likely to be recorded with the specific code (A), rather than being recorded as hypersensitive (B). Therefore, is expected to be higher than . However, symptoms are the same, so the overall likelihood of the patient to access the emergency room is the same across exposure strata, and patients missed by A are picked by B; therefore, the union of the 2 indicators is expected to have nondifferential SE.

The knowledge of P_A, P_B, and across exposure strata can be used to run a simulation of the power of our test, as done in Section ``A test for non-differential sensitivity'', and to decide the sampling strategy for a validation study stratified by exposure. If the test fails, the results from the validation study can also be used to adjust the estimates. In addition, a test for the equality of the SEs is useful even if the SEs can be estimated in the exposure groups separately, because this entails a loss of precision as compared with pooled estimation, in analogy to the case of comparing 2 means using a t test with unequal variances when the population variances are equal.²⁹

If the association measure of interest is the risk difference, then knowledge of the SE of A and the context of Y can be used to assess whether 1 of the scenarios of Section ``Risk difference'' applies to adjust for misclassification. If required by the scenario, a validation study stratified per exposure may also be conducted in this case.

A limitation of our test is that it does not account for errors in the exposure. Specifically, we did not consider the potential dependency between outcome misclassification and exposure misclassification. In observational studies, systematic differences in determining exposures may also affect the accuracy of diagnoses, even if only marginally. Research has shown that this scenario can lead to substantial bias in the results, even with minor levels of misclassification. In a future development, the test can be adapted to address also exposure misclassification.

In addition, we did not provide estimates of the uncertainty around our formulas. However, the uncertainty can always be assessed through simulation-based approaches such as the bootstrap. Another limitation is that, in the simulations, we did not explore a range of options for the SP of A and B. Such scenarios need further research.

Areas for further research could include extending the proposed method to nonrare events and hazard ratios, as well as incorporating the method in statistical modeling possibly with weighted estimation.

Conclusions

Whenever an auxiliary screening indicator can be defined to complement a primary indicator for an event, estimates of the PPV of both indicators provide tools to reduce the bias in estimating the number of cases, prevalence, cumulative incidence, rate (if the event is rare), and, in the case of association studies, risk ratio and risk difference. They also allow to test for nondifferential SE.

Although direct estimation of the SE is often infeasible, this novel methodology improves evidence based on data obtained from reuse of existing databases, which may prove critical for regulatory and public health decisions. Our work encourages in future studies a more widespread use of validation studies to support or reject assumptions on misclassification.

Funding

The ConcePTION project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement no. 821520. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and the European Federation of Pharmaceutical Industries and Associations.

Conflict of interest

None declared.

Disclaimer

The research leading to the results reporter here was conducted as part of the ConcePTION consortium. This article only reflects the personal views of the stated authors.