The State of Use and Utility of Negative Controls in Pharmacoepidemiologic Studies

Abstract

Uses of real-world data in drug safety and effectiveness studies are often challenged by various sources of bias. We undertook a systematic search of the published literature through September 2020 to evaluate the state of use and utility of negative controls to address bias in pharmacoepidemiologic studies. Two reviewers independently evaluated study eligibility and abstracted data. Our search identified 184 eligible studies for inclusion. Cohort studies (115, 63%) and administrative data (114, 62%) were, respectively, the most common study design and data type used. Most studies used negative control outcomes (91, 50%), and for most studies the target source of bias was unmeasured confounding (93, 51%). We identified 4 utility domains of negative controls: 1) bias detection (149, 81%), 2) bias correction (16, 9%), 3) P-value calibration (8, 4%), and 4) performance assessment of different methods used in drug safety studies (31, 17%). The most popular methodologies used were the 95% confidence interval and P-value calibration. In addition, we identified 2 reference sets with structured steps to check the causality assumption of the negative control. While negative controls are powerful tools in bias detection, we found many studies lacked checking the underlying assumptions.

This article is part of a Special Collection on Pharmacoepidemiology.

Abbreviations

ADR

adverse drug reaction

AUC

area under the receiver-operating-characteristic curve

CI

confidence interval

DiD

difference-in-differences

EU-ADR

Exploring and Understanding Adverse Drug Reactions

FDA

Food and Drug Administration

NCE

negative control exposure

NCO

negative control outcome

OMOP

Observational Medical Outcome Partnership

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

A major concern in conducting observational studies is the threat to internal validity. While adequate adjustments for unbalanced covariates such as propensity score matching allow an observational study to potentially mimic an experimental study, the presence of unmeasured confounding, selection bias, and measurement error nonetheless remain an ongoing challenge in observational studies.

The use of negative controls for identifying and addressing potential residual bias has gained attraction in drug effectiveness and safety research in recent years. In such studies, negative controls can be used either in place of an exposure (negative control exposure or NCE) or an outcome (negative control outcome or NCO) in an alternative hypothesis for examining potential bias. The premise underlying the use of negative controls to detect bias is that while negative controls should not have a causal association with the primary exposure or outcome of interest, they should share a same bias with the primary association of interest (1).

In recent years, there have been advancements in epidemiologic methods for applying negative controls in observational studies (1–9). Nevertheless, it is not well known to what extent these methodologies have been applied in pharmacoepidemiologic studies. In this work, we conducted a scoping review of the state of use, applications, and utility of negative controls in pharmacoepidemiologic studies. Findings from this work may inform the development of a guidance framework for future use of negative controls.

METHODS

We conducted a scoping review of published literature on pharmacoepidemiologic studies that deployed negative controls to detect or correct bias. Our review was based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) (10).

Identifying articles

The search strategy was initially developed by our librarian (E.F.G.) and modified by all team members (Z.Z., C.H.S., J.P., S.D.R., F.T., W.H., Y.M.). The search terms were chosen based on key concepts of focus in this review, including terms related to negative control, pharmacoepidemiology, and bias. We initiated the search in multiple electronic databases, including PubMed, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Cochrane Library, Scopus, and Dissertations and Theses Global, from inception to September 4, 2020. In addition, we manually searched the articles that cited the key method papers that are often cited for the negative control methods using Scopus. Any relevant articles that were identified from the included studies and not retrieved from the database searches were screened and reviewed for inclusion additionally. Based on our review of the literature, we used multiple key terms that were used for negative controls, including negative control, proxy outcome, bias indicator, and probe variable. We utilized the review management software Covidence (Covidence systematic review software; Veritas Health Innovation, Melbourne, Australia; https://www.covidence.org/) throughout the review process to manage the data and facilitate discussion among the investigators (see Web Appendix 1 for search strategies, available at https://doi.org/10.1093/aje/kwad201).

Selection of studies

We included studies that were published in English, had a focus on pharmacoepidemiology, and clearly expressed the use of negative controls to address bias in their analysis or added novel utilities of negative control methods. We excluded experimental studies either of a preclinical or clinical nature, or the ones that used negative controls as active comparators. Duplicate records were removed after comparing titles, published dates, authors, and journal fields. The selection criteria are provided in Web Appendix 1. Two reviewers (J.P. and C.H.S.) independently screened the titles and abstracts of all records retrieved from the database searches to determine studies to be further assessed for full-text screening. The full texts of the included articles from the abstract screening were reviewed by the same reviewers (J.P. and C.H.S.) independently for eligibility according to the inclusion criteria. Any discrepancies between the reviewers were resolved through further discussion, and if agreement was not reached, a third reviewer (Z.Z.) made the final determination throughout the selection process.

Data extraction and analysis

Three reviewers (J.P., C.H.S., and Z.Z.) independently extracted data from the texts of the included articles. The team members crosschecked each other’s data extraction. Disagreements among the reviewers on extracted information were resolved through discussion at team meetings.

We obtained descriptive statistics on the negative control use in the included studies and abstracted data on key methodological items including study characteristics such as publication year, study region, study design/pharmacovigilance method, type of data source, type and number of negative controls used, causal question under investigation (e.g., a drug-outcome association of interest), measure of association used, negative control assumptions, results of the negative control analysis, and negative control method used for bias correction or P-value calibration. Based on our quantitative and qualitative review of the extracted data, we identified distinct domains of negative control use in pharmacoepidemiologic research and summarized the studies based on our domain classification. We also summarized the approaches undertaken in the included studies for validating the assumptions of negative controls.

RESULTS

We initially identified 6,043 records from the database and the manual searches and selected 588 studies for further full-text assessment after the title and abstract screening. After the full-text review, we selected 184 papers for data extraction (Figure 1, showing a PRISMA flow diagram).

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for selection for inclusion in a review pharmacoepidemiologic studies involving negative controls (NCs). CENTRAL, Cochrane Central Register of Controlled Trials; CINAHL, Cumulative Index to Nursing and Allied Health Literature.

Reviewing key definitions and assumptions

Under the counterfactual or potential outcome framework, which models and contrasts potential outcomes under alternative exposure levels, possibly contrary to fact (6), there are 2 major assumptions necessary for negative controls and their use for bias detection (1, 6, 9). The first assumption (lack of causality or exclusion restriction assumption) is necessary to define negative controls and ensures lack of causal associations between the NCE and the primary outcome and between the primary exposure and the NCO. The second assumption necessitates that the negative control association has a similar underlying bias structure to that of the primary exposure-outcome association (U-comparability assumption) required for bias detection. In addition to these main assumptions specific to negative controls, there are other assumptions required for causal inference while acknowledging the presence of unmeasured confounding, mainly the consistency assumption and the latent ignorability assumption, as described elsewhere (9). While the exclusion restriction and the U-comparability assumptions are sufficient for bias detection, there may be additional assumptions required for bias correction. For example, for identification of the causal effect using the double negative control methods, there are identification conditions described as positivity and completeness assumptions (9). Directed acyclic graphs of negative controls and formal representations of these assumptions are provided in Web Appendix 2, including Web Figure 1.

Basic characteristics of the included studies

Table 1 shows the study characteristics. Out of 184 included studies, the majority were published in recent years (126, 68.5%; between 2017 and 2020). The most common study design was cohort study (n = 115, 62.5%), followed by case-control study (n = 12, 6.5%) and self-controlled case series (n = 9, 4.9%). Most of the studies used health-care administrative data (e.g., insurance claims) (n = 114, 62.0%) or electronic health record data (n = 62, 33.2%). Of the 184 studies, 54 (29.3%) used NCE, 91 (49.5%) used NCO, and 34 (18.5%) used both NCE and NCO in their analyses. Unmeasured confounding was the most frequently stated reason (n = 93, 50.5%) for use of negative controls in the included studies, followed by information bias (including misclassification, measurement error) (n = 10, 5.4%) or selection bias (n = 7, 3.8%).

Table 1.

Characteristics of the Included Studies in the Scoping Review for the State of Use and Utility of Negative Controls in Pharmacoepidemiologic Studies

Characteristic	Study (n = 184)
	No.	%
Publication year
Before 2012	12	6.5
2013–2016	46	25.0
2017–2020	126	68.5
Study region
North America	94	51.1
Europe	61	33.2
Asia	13	7.1
Others	4	2.2
Multiple regions (including more than 1 continent)	12	6.4
Study design
Cohort study	115	62.5
Case-control study	12	6.5
Self-controlled case series	9	4.9
Nested case-control study	7	3.8
Ecological study	5	2.7
Self-controlled cohort study	2	1.1
Case-crossover study	1	0.5
Cross-sectional study	1	0.5
Self-controlled risk interval study	1	0.5
Pharmacovigilance study
Disproportionality analysis	4	2.2
Sequence symmetry analysis	3	1.6
Sequential statistical testing	3	1.6
Multiple studies with different designs/methods	14	7.6
Reference set identification	5	2.7
Others	2	1.1
Type of NC
NCE	54	29.3
NCO	91	49.5
Both NCE and NCO	34	18.5
Not applicablea	5	2.7
No. of NCs used
Single (either NCE or NCO)	84	45.7
Multiple	94	51.0
Not applicablea/not reported	6	3.3
Type of biasb
Unmeasured confounding	93	50.5
Information bias (e.g., misclassification, measurement error)	10	5.4
Selection bias	7	3.8
Protopathic bias	2	1.1
Bias in general/not specified	80	43.5
Type of data source usedb
Health-care administrative data (e.g., insurance claims, national health system data)	114	62.0
Electronic health records	61	33.2
Research data (e.g., primary data)	26	14.1
Disease registry	11	6.0
Passive surveillance data for drug safety	7	3.8
Simulated data set	6	3.3
Public health survey data	4	2.2
Not applicablea	5	2.7

Abbreviations: NC, negative control; NCE, negative control exposure; NCO, negative control outcome.

^a Reference set identification studies with no empirical application of NC.

^b Categories are not mutually exclusive.

We identified different nomenclature used for negative controls. While most of the included studies used the term negative control or similar terms (e.g., negative outcome/exposure, control outcome/exposure), some studies used other terms, such as falsification endpoint/analysis, borrowed from the econometrics literature (10 studies). We identified 1 study that used the term indicator conditions to refer to negative control outcomes.

The utility and application of negative controls in pharmacoepidemiologic studies

Based on our review of the 184 studies, we identified 4 main domains for the application of negative controls in pharmacoepidemiologic studies. The first 3 domains directly address bias in the effect estimate of the exposure-outcome association and include: 1) detection of bias, 2) correction of bias (by correcting the point estimate or calibrating its confidence interval (CI)), and 3) calibration of P value. The fourth domain is not directly related to a particular exposure-outcome association; instead, it deploys negative controls to assess the performance of different pharmacoepidemiologic methods.

Detection of bias in the effect estimate of the exposure-outcome association.

Most of the studies (149 (81%)) used negative controls to detect potential bias in the effect estimate of the exposure-outcome association (4, 5, 7, 11–156). This can be done by testing the negative control association against the null hypothesis. Most studies indicated the presence of a bias by rejecting a null hypothesis if the effect estimate of the negative control association was statistically significant given the type I error of 5% (P < 0.05). However, we found some studies that concluded presence of bias if the direction of the point estimate of their negative control association was considered indicative of bias even when its P value was not statistically significant (14, 71, 116). Of the 149 studies included in this category, 63 (42%) reported detecting a presence of bias. For types of bias detected see Table 1 and Web Table 1.

Negative controls to correct the effect estimate and CI for bias.

We found 16 studies that used negative controls to obtain bias-adjusted estimates (point estimates or CIs) for the exposure-outcome association of interest (Table 2) (5, 7, 47, 48, 50, 53, 63, 70, 73, 74, 89, 107, 111, 126, 139, 154). Fourteen out of 16 studies were published between 2017 and 2020, which may suggest expansion of the use of negative controls for bias correction in pharmacoepidemiology in recent years. The most common practice for bias correction was a difference-in-differences (DiD) approach by subtracting (or dividing) the effect estimate of the negative control association from that of the primary association of interest. This approach requires a restrictive assumption of additive equi-confounding, which necessitates that the magnitude and direction of the bias for the negative control association be equal to that of the primary exposure-outcome association on the additive scale (157). For example, a group of studies examining drug-drug interactions (DDIs) obtained the bias-corrected effect estimates by taking a ratio of rate ratio (RR) (or ratio of reporting odds ratio (ROR)) from an association with suspected DDI and the RR from negative control associations with no suspected DDI (50, 73, 74, 154). The delta method was commonly used to construct the 95% CIs for the corrected estimate in these studies (50, 73, 154).

Table 2.

Pharmacoepidemiologic Studies That Used Negative Controls to Correct for (or Attenuate) Bias or Calibrate P Values

First Author, Year (Reference No.)	Study Design	Type of Data	Hypothesized Causal Association	Primary Study Effect Measure	NC(s) Used	Type of Bias as Reported by Study	Methods Used to Correct for Effect of Bias (in Point Estimates, CIs, or P Values)
Correction of Point Estimate and 95% CI
Johanson, 2012 (63)	Ecological study	Public health survey data; administrative claims; other publicly available data	Exposure: buprenorphine/naloxone; outcome: report of diversion (the percentage of applicants who reported knowing that buprenorphine/naloxone was sold on the street) and abuse (the percentage who reported knowing that it was used to get high)	Proportion	NCE: amitriptyline; PCE: methadone, oxycodone, and heroin	Measurement error	To correct for the point estimate, the following formulae was used: Relative abuse of buprenorphine/naloxone = (abuse of buprenorphine or naloxone – abuse of NCE) / (abuse of PCE – abuse of NCE)).
Greene, 2013 (47)	Cohort study	EHR	Exposure: use of oseltamivir; outcome: neuropsychiatric adverse events (ataxia, psychiatric, encephalitis, disturbance of consciousness) and non-neuropsychiatric events (arrhythmia, syncope, convulsions, movement disorder, stroke)	OR, risk difference	NCO: cellulitis, anemia, injury/trauma	Not specified	To reduce bias (bias attenuation), the history of NCOs was included in the propensity score matching model.
Han, 2017 (50)	SCCS	Administrative claims	Exposure: concomitant use of a precipitant of interest (vs. not receiving a precipitant) with secretagogues; outcome: serious hypoglycemia	Rate ratio	NCE: concomitant use of a precipitant of interest with metformin	Confounding bias by inherent hypoglycemic effects of the precipitants	The ratio of the semi-Bayes–adjusted rate ratio associated with the exposure to the semi-Bayes–adjusted rate ratio associated with the corresponding NCE was estimated. The delta method was used for 95% CI calibration (204).
Leonard, 2017 (74)	SCCS	Administrative claims	Exposure: discontinuation of the antihyperlipidemic drugs in the presence of ongoing warfarin therapy; outcome: composite of hospitalization for venous thromboembolism or ischemic stroke	IRR	NCE: discontinuation of pravastatin in the presence of ongoing warfarin therapy	Confounding bias by inherent effects on the outcome of the precipitants	To correct for the point estimate and 95% CI, ratio of IRR for the exposure vs. IRR for the NCE was calculated.
Gruber, 2018 (48)	Cohort study	Clinical trial data	Exposure: vaccine dose timing; outcome: Severe rotavirus gastroenteritis incidence	Risk difference, risk ratio	NCE: placebo	Confounding bias; administrative censoring	To correct for bias in the estimated risk difference, the effect estimate for the placebo arm was subtracted from the effect estimate for the vaccination arm (for the ratio outcomes, these estimates were divided). A nonparametric bootstrap with 2,000 sample draws with replacement was used to obtain the point estimates and empirical 95% CIs.
Schuemie, 2018 (5)	Southworth, Graham replications: cohort study; Tata case-control replication: case-control study; Tata SCCS replication: SCCS	Southworth, Graham replications: Administrative claims; Tata case-control, SCCS replications: EHR	Southworth replication: Exposure: use of dabigatran (vs. warfarin) in the Southwork and Graham replication and use of selective serotonin reuptake inhibitors in the Tata case-control and SCCS replications; outcome: gastrointestinal bleeding, in the Southwork and Graham replications and upper gastrointestinal bleeding in the in the Tata case-control and SCCS replications	Southworth replication: IRR; Graham replication: HR Tata case-control replication: OR; Tata SCCS replication: IRR	50 NCOs and 150 synthetic PCOs for each replication	Any residual bias (including confounding bias, misclassification, selection bias)	This study developed a methodology for empirical calibration of 95% CI of the effect estimate when a set of negative controls were used in observational studies. The calibration procedure first estimated the distribution of systematic error using the observed estimates for negative and positive controls, then generated calibrated 95% CIs considering both random and systematic error and assuming Gaussian distribution with a mean and log standard deviation linearly related to the true effect size. We refer to this method as “Schuemie’s empirical CI calibration method.”
Schuemie, 2018 (111)	Cohort study	Administrative claims	Exposure: use of duloxetine (vs. sertraline); outcome: stroke	HR	52 NCOs and 156 PCOs	Any residual bias (including confounding bias)	The study used Schuemie’s empirical CI calibration method (5).
Thorrington, 2018 (126)	Ecological study	Hospital admission data	Exposure: 24-month post-PCV period (vs. pre-PCV period); outcome: pneumonia, empyema, sepsis, otitis media	IRR	NCO: urinary tract infections, infections of the skin and subcutaneous tissue, disorders of the thyroid gland, diseases of the blood, and fractures	Bias arising from potential secular trend	For each outcome of interest, the age-specific ratio of the IRR for the outcome of interest vs. the geometric mean of the IRR for the NCOs was calculated. The minimum and maximum incidence rate ratio across all NCOs were used to represent uncertainty.
Leonard, 2019 (73)	SCCS	Administrative claims	Exposure: concomitant use of clopidogrel and a precipitant drug of interest (clopidogrel-precipitant pairs); outcome: gastrointestinal bleeding or intracranial hemorrhage	Rate ratio	NCE: concomitant use of pravastatin and a precipitant drug of interest (pravastatin-precipitant pairs)	Confounding bias by inherent bleeding effects of the precipitant drug	The ratio of the semi-Bayes–adjusted rate ratio associated with the exposure to the semi-Bayes–adjusted rate ratio associated with the corresponding NCE was estimated. The delta method was used for 95% CI calibration (204).
Hripcsak, 2020 (53)	Cohort study	Administrative claims; EHR	Exposure: use of chlorthalidone (vs hydrochlorothiazide); outcome: multiple cardiovascular outcomes (e.g., acute myocardial infarction, hospitalization for heart failure, etc.)	HR	76 NCOs and 228 PCOs (3 PCOs are generated for each NCO, with true relative risk 1.5, 2, and 4).	Any residual bias	The study used Schuemie’s empirical CI calibration method (5).
Lane, 2020 (70)	Cohort study	Administrative claims; EHR	Exposure: initiation of hydroxychloroquine (vs. initiation of sulfasalazine); outcome: 16 severe adverse events (e.g., gastrointestinal bleeding, acute renal failure, acute pancreatitis, myocardial infarction, stroke, etc.)	HR	NCO: 67 conditions	Confounding bias	The study used Schuemie’s empirical CI calibration method (5).
Rodgers, 2020 (107)	Cohort study	EHR	Exposure: use of thiazolidinediones (vs. sulfonylureas); outcome: edema, weight gain	HR	NCO: gastrointestinal side effects	Confounding bias	To correct for bias, the ratio of the HR for the post period vs. the HR of the prior period was calculated. They verified the removal of bias by replicating the same bias correction method for the NCO (ratio of the HRs for the post vs. prior periods for the NCO). The standard error for the estimates was obtained by bootstrapping.
Shi, 2020 (7)	Cohort study	Integrated health system data	Exposure: receipt of DTaP-IPV-Hib vaccine (vs. DTap containing comparator vaccine); outcome: fever; NCO: injury or trauma; NCE: ringworm	Relative risk	1 NCO and 1 NCE	Confounding bias	The study developed a methodology for correcting for a categorical unmeasured confounding. The methodology uses an NCE and an NCO to build a semiparametric model and propose multiply robust estimator for the average treatment effect. The study demonstrated the application of their method in a pharmacoepidemiologic vaccine safety study and showed that the multiply robust estimator provided a smaller bias and protected against the model misspecification.
Zhou, 2020 (154)	SCCS	Administrative claims	Exposure: concomitant use of a precipitant of interest (vs. not receiving a precipitant) with anticoagulants (anticoagulant-precipitant pairs); outcome: thromboembolism (a composite outcome of stroke and venous thromboembolism)	Rate ratio	NCE: concomitant use of a precipitant of interest with pravastatin (pravastatin-precipitant pairs)	Confounding bias by inherent effect of the precipitant on the primary outcome	The ratio of the semi-Bayes–adjusted rate ratio associated with the exposure to the semi-Bayes–adjusted rate ratio associated with the corresponding NCE was estimated. The delta method was used for 95% CI calibration (204).
P-Value Calibration
Schuemie, 2014 (4)	Cohort study (example 1); case-control study (example 2); SCCS (example 3)	Administrative claims (example 1); EHR (examples 2 and 3)	Exposure: use of isoniazid (example 1); use of selective serotonin reuptake inhibitors (examples 2 and 3); outcome: acute liver injury (example 1); upper gastrointestinal bleeding (examples 2 and 3)	OR	37 NCEs (example 1); 67 NCEs (examples 2 and 3)	Most forms of bias (including confounding bias, misclassification, selection bias)	This study developed a methodology for calibrating the P value of the effect estimate when a set of negative controls were used to account for bias. The method first derived an empirical null distribution from the observed effect estimates for the negative controls, then generated calibrated P values assuming Gaussian distribution to the estimates and taking into account the sampling error of each estimate. We refer to this method “Schuemie’s empirical P-value calibration method.”
Duke, 2017 (38)	Cohort study	Administrative claims; EHR	Exposure: use of levetiracetam; outcome: risk of angioedema	HR	100 NCOs	Any residual bias	The study used Schuemie’s empirical P-value calibration method (4).
Yuan, 2018 (151)	Cohort study	Administrative claims	Exposure: use of sodium glucose co-transporter 2 inhibitors (overall), and canagliflozin (specifically) (vs. non-sodium glucose co-transporter 2 inhibitor antihyperglycemic agents); outcome: below-knee lower extremity amputation	HR	NCO (number not reported)	Confounding bias	The study used Schuemie’s empirical P-value calibration method (4).
Schuemie, 2019 (112)	Case-control studies	Administrative claims	Exposure: use of isotretinoin (Crockett study replication); use of dipeptidyl peptidase-4 inhibitors (Chou study replication); outcome: ulcerative colitis (Crockett study replication); acute pancreatitis (Chou study replication)	OR	35 NCEs	Any residual bias	The study used Schuemie’s empirical P-value calibration method (4).
Kim, 2020 (65)	Cohort study	Administrative claims; EHR	Exposure: use of alendronate (vs. use of raloxifene); outcome: hip fracture, vertebral fracture, esophageal cancer, osteonecrosis of the jaw	HR	147 NCOs	Confounding bias	The study used Schuemie’s empirical P-value calibration method (4) to construct the empirical null distribution and compare it with the theoretical null distribution to check the presence of bias.
You, 2020 (148)	Cohort study	Administrative claims	Exposure: ACEIs/ARB + CCB vs. ACEI/ARB + TZD vs. CCB+TZD; outcome: all-cause mortality, myocardial infarction, heart failure, stroke, a composite outcome of major adverse cardiac and cerebrovascular events	HR	39 NCOs	Any residual bias	The study used Schuemie’s empirical P-value calibration method (4).
Both Bias Correction With Regard to Point Estimate and 95% CI and P-Value Calibration
Weinstein, 2020 (139)	Cohort study	EHR	Exposure: use of any paracetamol (vs. use of ibuprofen); outcome: diagnoses of gastrointestinal bleeding, myocardial infarction, ischemic or hemorrhagic stroke, or acute or chronic renal disease	HR	39 NCOs and 147 synthetic PCOs (3 for each NCO)	Channeling bias	The study used Schuemie’s empirical CI and P-value calibration method (4, 5).
Morales, 2021 (89)	Cohort study	Administrative claims; EHR	Exposure: use of ACEIs and ARBs (vs. CCBs and thiazide or THDs); outcome: COVID-19 diagnosis; hospital admission with COVID-19; hospital admission with pneumonia; hospital admission with pneumonia, acute respiratory distress syndrome, acute kidney injury, or sepsis	HR	123 NCOs	Confounding bias	The study used Schuemie’s empirical CI and P-value calibration method (4, 5).

Abbreviations: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin-receptor blocker; CCB, calcium channel blocker; CI, confidence interval; COVID-19, coronavirus disease 2019; DTaP-IPV-Hib, diphtheria and tetanus toxoids and acellular pertussis adsorbed, inactivated poliovirus, and haemophilus B conjugate vaccine; EHR, electronic health records; HR, hazard ratio; IRR, incidence rate ratio; NC, negative control; NCE, negative control exposure; NCO, negative control outcome; OR, odds ratio; PCE, positive control exposure; PCO, positive control outcome; PCV, pneumococcal conjugate vaccine; SCCS, self-controlled case-series; TZD, thiazide-type diuretic.

Gruber et al. (48) used the DiD approach in examining the effect of different vaccine dose timing on rotavirus vaccine effectiveness. The authors used the placebo group in a randomized controlled trial as a negative control for measuring and correcting for bias (48). Rodgers et al. (107) demonstrated the utility of prior event rate ratio (PERR), a bias correction method based on the DiD approach, to correct for bias using the same outcome measured in a negative control period. The study used PERR-adjusted hazard ratio (HR_PERR), defined as the ratio between the hazard ratio for the study period and the hazard ratio for period before exposure, in adjusting for time-invariant unmeasured confounding for estimating the association between second-line antidiabetics and known side effects (edema and weight gain) (107). Using the outcome measured in the prior period as an NCO, the authors ensured the similarity in the bias measured in the negative control association with regard to time-invariant factors (107). Other studies described the use of the DiD approach in the context of survival analysis and its conditions (158–160). Bootstrapping was used to estimate 95% CIs for the corrected effect estimates for these studies (48, 107).

In another group of studies, an approach of utilizing more than one negative control was taken in correcting the effect estimates to account for variations in bias estimates from different negative control associations (5, 53, 70, 89, 111, 126, 139). Schuemie et al. (5) formalized the idea of using multiple negative controls in bias estimation into the empirical calibration method for 95% CI, which constructs a parametric empirical null distribution of bias using multiple negative controls and synthetic positive controls to calibrate the effect estimate and 95% CI. Table 2 shows the methodological details of the studies that corrected effect estimates and CIs for bias.

In addition, Shi et al. (7) leveraged the use of double negative controls, one NCE and one NCO, for constructing a semiparametric model for a multiply robust estimator of the average treatment effect in the presence of a categorical unmeasured confounding. Their study showed the multiply robust estimator reduced bias in a vaccine safety study (7).

Negative controls to calibrate the P value.

We found 8 pharmacoepidemiologic studies that used the empirical calibration method to calibrate P value to correct for the type I error (erroneously rejecting the null hypothesis) (4, 38, 65, 89, 112, 139, 148, 151). Based on the same theoretical background as the CI calibration method, this method proposed by Schuemie et al. (4) uses a large number of negative controls to estimate the empirical null (bias) distribution using the maximum likelihood approach to calibrate P values and accounts for both random and systematic error in drawing a statistical inference (Table 2).

Negative controls to assess the performance of different methods used in drug safety and effectiveness studies.

We found 31 studies that deployed negative controls as a reference standard to compare the performance of different methods for drug safety and effectiveness studies (31, 60, 64, 86, 141, 161–186). The details of these studies are presented in Table 3. The first group of studies in this domain used negative controls specifically to evaluate and validate methods used to identify potential adverse drug reactions (ADRs) for pharmacovigilance purposes (161–186). The studies commonly used a large set of negative controls and known drug reactions (positive controls) to quantify the variations in discriminatory power of different pharmacovigilance methods in detecting safety signals. For example, one of the earliest studies of this kind by Schuemie et al. compared the performance of different pharmacovigilance methods and study designs to identify ADRs, including cohort methods, case-based methods, and other data-mining methods commonly used in spontaneous reporting systems for ADRs (182). The authors used a set of known drug-ADR pairs and negative control pairs to compare the discriminatory power of the aforementioned methods by measuring the area under the receiver-operating-characteristic curve (AUC) across 7 European electronic health record databases (182). Some studies in this group used this approach to examine the transferability of the methods and assess the performance of established ADR identification methods in new settings. For example, Pratt et al. (174) evaluated the consistency of the prescription sequence symmetry analysis method to detect ADRs across settings in 5 countries using positive and negative controls.

Table 3.

Studies That Used Negative Controls to Assess the Performance of Methods for Drug Safety and Effectiveness Studies

First Author, Year (Reference No.)	Description of the Use of NCs in the Study
Studies That Used NCs as Reference Standards in Evaluating the Performance of Pharmacovigilance Methods
Brown, 2007 (165)	The study assessed the performance of sequential analysis for active surveillance of ADR in large observational databases. The performance was measured by applying sequential analysis to 5 PC and 2 NC drug-outcome pairs in administrative data from 9 health plans participating in the HMO Research Network’s Center for Education and Research on Therapeutics. The signal was determined by testing the maximized sequential probability ratio against the null hypothesis, and the number of significant signals among PC and NC drug-outcome pairs was reported.
Brown, 2009 (166)	The study assessed the variation in the performance of sequential analysis with alternative analytical choices with regard to exclusion criteria and surveillance period, and the results were compared with the performance of sequential analysis with the original set of analytical choices, which was assessed in an earlier study (Brown et al. (165)). The same data source and methods from the previous study were used for the assessment including the choice of PC and NC drug-outcome pairs.
Ryan, 2012 (177)	The study evaluated the performance (measured with sensitivity, specificity, PPV, and AUC) of 8 different methods in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network, using 53 unique drug-outcome pairs (9 PC and 44 NC drug-outcome pairs). The methods evaluated were: 1) disproportionality analysis, 2) ICTPD, 3) SCCS, 4) case-control surveillance, 5) case-crossover, 6) observational screening, 7) high-dimensional propensity score, and 8) incident user design.
Schuemie, 2012 (182)	The study evaluated the performance (measured with AUC) of 10 different methods to detect ADR in health-care databases participating in EU-ADR, using a set of PCs and NCs were as reference standards. The methods evaluated in the study included: 1) spontaneous reporting system methods (PRR, ROR, GPS, BCPNN), 2) cohort methods (IRR, LGPS, Bayesian hierarchical model), 3) case-based methods (matched case-control, SCCS), and 4) LEOPARD.
DuMouchel, 2013 (162)	The study evaluated the performance (measured with AUC) of disproportionality analysis with different combinations of analytical choices (with regard to outcome definition, disproportionality metric, stratification, time at risk across) for 4 health outcomes (acute liver failure, acute MI, acute renal failure, and upper GI bleeding) in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network and 6 simulated data sets. A collection of 165 PC and 234 NC drug-outcome pairs were used as reference standards for evaluation.
Madigan, 2013 (168)	The study evaluated the performance of case-control design for ADR risk identification and compared AUC for the design with various combinations of analytical choices with regard to: 1) number of controls, 2) minimum time before outcome, 3) time at risk, and 4) cohort nesting within indication. The reference standards included 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver failure, acute MI, acute renal failure, upper GI bleeding). The performance was evaluated in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network.
Norén, 2013 (171)	The study evaluated the performance of the calibrated self-controlled cohort analysis with temporal pattern discovery as a tool for risk identification, by comparing AUC for different combinations of the design choices with regard to: 1) surveillance period and 2) control period. The method was applied to 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver failure, acute MI, acute renal failure, and upper GI bleeding). The performance was evaluated in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network.
Reich, 2013 (175)	The study evaluated the performance of different algorithms to define health outcomes of interest for 3 health outcomes (acute kidney injury, acute liver injury, and acute MI) for risk identification methods in observational data. The algorithms were tested across an array of analytical methods and design choices (including choices for preexposure control period and time at risk) in 4 US health-care databases participating in OMOP network. The methods were executed against predefined drug-outcome pairs (test cases) that included PCs and NCs. The performance was measured with AUC, bias, and minimal detectable relative risk.
Reich, 2013 (176)	The study assessed the effects of drug and outcome prevalence on the feasibility and performance for risk identification methods in observational data. The methods of interest (new user cohort design, case control design, SCCS, SCC, ICTPD, disproportionality analysis, and LGPS) were tested against PCs and NCs across 4 health outcomes (acute kidney injury, acute liver injury, acute MI, and upper GI bleeding) in 3 US health-care databases participating in the OMOP network. The performance was measured with AUC.
Ryan, 2013 (178)	The study evaluated the performance (measured with AUC, bias, and coverage probabilities) of new-user cohort method for detecting safety signals in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network and 6 simulated data sets. Different combinations of analytical choices were tested, including choices for: 1) required observation time prior to the exposure, 2) nesting within population with the indication of the target drug, 3) comparator population, 4) time-at-risk, 5) propensity score covariate selection strategy, 6) covariate eligibility window, 7) dimension to include as potential covariates, 8) additional covariates in the propensity score model, 9) propensity score trimming, and 10) analysis method. The method was tested against 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver failure, acute MI, acute renal failure, and upper GI bleeding).
Ryan, 2013 (179)	The study evaluated the performance of SCC method for detecting safety signals in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network and 6 simulated data sets. The performance metrics of AUC, bias (measured with relative risk estimates from the NC analysis) and coverage probabilities (proportion of the CIs that contained the true relative risk) for different combinations of design choices of the method were measured. Different combinations of design choices were tested, including choices for 1) exposure definition, 2) outcome definition, 3) time-at-risk, 4) control period. A set of 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver failure, acute MI, acute renal failure, and upper GI bleeding) were used as reference standards.
Ryan, 2013 (180)	The study compared the performance (measured with AUC, bias, MSE, and CI coverage probability) of 7 established pharmacovigilance methods including: 1) new user cohort, 2) case-control, 3) SCCS, 4) SCC, 5) disproportionality analysis, 6) temporal pattern discovery, and 7) LGPS. The methods were tested against 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver injury, acute MI, acute renal failure, and GI bleeding) in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network.
Ryan, 2013 (181)	The study used simulated health-care data to evaluate the performance (measured with AUC, bias, and MSE) of 7 observational designs for ADR identification. The tested methods included: 1) case-control, 2) new user cohort method, 3) disproportionality methods, 4) ICTPD, 5) LGPS, 6) SCC, and 7) SCCS. The methods were tested against 399 reference drug-outcome pairs (including 165 PCs and 234 NCs) across 4 health outcomes (acute liver injury, acute MI, acute renal failure, and GI bleeding) in 6 simulated data sets.
Schuemie, 2013 (183)	The study replicated the OMOP experiment in European databases and compared the performance of the established methods for risk identification, including: 1) case-control, 2) new user cohort method, 3) disproportionality methods, 4) ICTPD, 5) LGPS, 6) SCC, and 7) SCCS. The methods were tested against 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver injury, acute MI, acute renal failure, and GI bleeding) in 6 health-care databases (administrative claims and EHR) participating in the EU-ADR database network. The performance was measured with AUC and bias.
Schuemie, 2013 (184)	The study evaluated the performance (measured with AUC, bias, and coverage probabilities) of LGPS and LEOPARD in in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network and 6 simulated data sets in safety signal detection. Different combinations of analytical choices were tested, including choices for: 1) data source, 2) exposure definition, 3) run-in period, 4) carry-over period, and 5) shrinkage application. The method was tested against 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver failure, acute MI, acute renal failure, upper GI bleeding).
Suchard, 2013 (185)	The study evaluated the performance (measured with AUC, bias, and coverage probabilities) of SCCS for safety signal detection in 5 US health-care databases (administrative claims and EHR) participating in the OMOP network and 6 simulated data sets. Different combinations of design choices were tested, including choices for: 1) outcome definition, 2) multivariate adjustment, 3) at-risk window, 4) observation time-window definition, and 5) minimum observation length. The method was tested against 165 PC and 234 NC drug-outcome pairs across 4 health outcomes (acute liver failure, acute MI, acute renal failure, and upper GI bleeding).
Gagne, 2014 (167)	The study developed a semiautomated process for active safety monitoring for new drugs in distributed data environment for health-care data. Five PC and 2 NC drug-outcome pairs were defined and used for testing the validity of the process, and whether the system generated an alert for each drug-outcome pair was reported.
Pratt, 2015 (174)	The study evaluated the robustness and consistency of PSSA to detect ADRs across settings in 5 different countries. The method was applied to 1 PC and 2 NCs as test cases. The ASR and 95% CIs were estimated for the associations with ADRs for each country.
Backenroth, 2016 (163)	The study compared AUCs for the case-control method with different analytical choices (including choices for: 1) covariates used (none, demographics, phenome-wide association study (PHEWAS) groups plus demographics), 2) covariate selection method (none, or 1-step LASSO), and 3) estimation method (marginal odds ratio, 1 model per drug-outcome pair, or 1 model per outcome)) across 4 health outcomes (acute kidney injury, acute liver injury, acute MI, and GI ulcer hospitalization) in a US EHR database.
Marbac, 2016 (169)	The study proposed a model-based method for analyzing spontaneous reporting databases (Bayesian model selection in logistic regression), and compared the performance (measured with number of signals, rate of positive controls, rate of negative controls, rate of unknown signals) to the well-established approaches, including 4 disproportionality-based methods (PRR, ROR, reporting Fisher exact test, FDR-based GPS, Lasso-based logistic regressions, and model-based logistic regressions) across 4 health outcomes (acute MI, acute kidney injury, acute liver injury, and upper GI bleeding) in the French pharmacovigilance database. The OMOP reference set of test cases were used to evaluate the performance, which included 165 PC and 234 NC drug-outcome pairs.
Osokogu, 2016 (172)	The study assessed the performance (AUC) of 2 established signal detection algorithms (PRR and empirical Bayes geometric mean) in the pediatric population in data from the US FAERS. Thirty-seven PC and 90 NC drug-outcome pairs from a pediatric-specific Global Research in Pediatric reference set were used.
Hauben, 2017 (164)	The study estimated the effects of database restriction for oncology drug on the signal detection performance using FAERS data. Multi-item GPS analysis, a type of disproportionality analysis, was carried out in restricted and unrestricted data sets for a predefined oncology-specific reference set (including 638 PC and 67 NC drug-outcome pairs). The method performance was measured with sensitivity, specificity, PPV, NPV, signal/noise, F, and Matthews correlation coefficient.
Nishtala, 2017 (170)	The study assessed the performance of PSSA in the New Zealand prescription database for risk identification using 6 PC drug exposures and 6 NC drug exposures The adjusted sequence ratios and 95% CIs were estimated for the associations with ADRs.
Pierce, 2017 (173)	The study assessed the potential utility of social media data for monitoring drug safety signals and examined whether a safety signal appears in social media prior to being reported to a spontaneous reporting system. An analysis of social media posts was conducted for 10 safety signals (PCs) recently identified by FDA, and 6 NC drug-outcome pairs. The number of associations appeared in the social media data was reported for each of the pairs as an outcome.
Trinh, 2018 (161)	The study evaluated the performance of combining change-point analysis method with disproportionality analysis for detecting changes in safety signals in the French National Pharmacovigilance Database (Base Nationale de Pharmacovigilance or BNPV) and EudraVigilance database. The performance was assessed by testing the change-point analysis hypothesis against the null hypothesis for 39 PC and 56 NC drug-outcome pairs selected from the OMOP reference set in EudraVigilance database, and 2 PC drug-outcome pairs were used as test cases in BNPV database.
Thurin, 2020 (186)	The study assessed the performance of 3 case-based designs (SCCS, case-control, and case-population) for identifying risk of upper GI bleeding in the French National Healthcare System Database. Different combinations of design and analytical choices for each method were tested against the reference standards (22 PC and 42 NC drugs for upper GI bleeding), and the performance (measured with AUC, MSE, and coverage probability) was compared. The best-performing method was selected for calibration using the empirical P-value calibration method to improve the accuracy of the method.
Studies That Used NCs to Compare Different Methods for Bias Mitigation in Pharmacoepidemiologic Studies
McGrath, 2015 (86)	The study assessed the residual confounding associated with different analytical choices (marginal structural models with different specifications and sample restriction in multiple different ways) in evaluating influenza vaccine (comparing the vaccinated vs. unvaccinated) effectiveness in adult hemodialysis patients. The study used data from the US Renal Data System linked with data from a commercial dialysis provider to compare all-cause mortality between the groups during 3 influenza seasons. The outcome measured in pre-influenza season was used as a NC outcome to measure residual confounding. The study found that marginal structural models did not improve confounding controls even after adding additional variables. Restricting the sample to more homogeneous and healthier populations (e.g., by applying stricter survival requirements) moved the results toward the null in pre-influenza seasons, indicating reduction of residual bias.
Davies, 2017 (31)	The study applied 3 techniques (NC outcome, NC populations, and tests of covariate balance) to assess the relative bias of the effect estimates by 2 analytical methods: 1) instrumental variable (physicians’ prescribing preferences) approach, and 2) a conventional regression. The Clinical Practice Research Datalink was used to investigate the effect of smoking cessation therapies (varenicline vs. nicotine replacement products) on suicide and self-harm, depression, and urinary tract infection was used as the NC outcome. Bias component plot was applied to illustrate the relative bias of the 2 methods.
Weinstein, 2017 (141)	The study compared the residual bias associated with 6 different analytical methods (multivariate outcome modeling with 3 different specifications, publication variable PS adjustment, large-scale PS adjustment, and large-scale PS adjustment with large-scale outcome modeling) to adjust confounding in evaluating safety of frequently used over-the-counter medications (paracetamol vs. ibuprofen) using data from the Clinical Practice Research Datalink. The study used 31 NC outcomes to assess the bias and reported the proportion of NC outcomes that were statistically significant as an indicator of residual bias. While using multivariate outcome modeling alone and the publication variable PS adjustment resulted in residual bias (fraction significant > 5%), the other 2 large-scale PS adjustment methods resulted in no significant NC results.
Kaiser, 2018 (64)	The study assessed potential biases associated with different analytical choices in estimating the association (HR) between statin use and incident MI using a retrospective cohort design. Six analytical variations were compared for potential bias: 1) crude analysis, 2) restriction to those eligible for statins, 3) multivariable adjusted among eligible population, 4) restriction to eligible new users, 5) multivariable adjusted among eligible new users, and 6) PS-matched among eligible new users. Noncardiovascular mortality was used as an NC outcome. HRs from negative control analysis were compared with HRs from the primary analysis to assess bias. Data from participants in the Cardiovascular Health Study from 1989 to 2004 were used.
Izurieta, 2019 (60)	The study assessed the impact of using multiple imputation in reducing potential bias due to residual confounding in a HZV effectiveness study (comparing the vaccinated vs. unvaccinated) using 13 NC outcomes. The authors had previously published a HZV effectiveness study using Medicare claims data and checked the residual bias using 13 NC outcomes in the original study (59). This study additionally used and linked the MCBS to the Medicare claims to impute 3 new MCBS variables, potential confounders missing in the original analysis. The same 13 NC outcomes from the previous study were used to detect residual confounding in the imputation analysis and compare the results with the original analysis. The study found similar HR estimates compared with the original study across the 13 NC outcomes (CIs overlapping). The point estimates shifted toward the null for 8 out of 13 outcomes, and the CIs were wider for the imputation analysis.

Abbreviations: ADR, adverse drug reaction; ASR, adjusted sequence ratio; AUC, area under the receiver operating characteristic curve; BCPNN, Bayesian confidence propagation neural network; CI, confidence interval; EHR, electronic health records; EU-ADR, Exploring and Understanding Adverse Drug Reactions; FAERS, Food and Drug Administration’s Adverse Event Reporting System; FDA, US Food and Drug Administration; FDR, false discovery rate; GI, gastrointestinal; GPS, γ Poisson shrinker; HMO, health maintenance organization; HR, hazard ratio; HZV, herpes zoster vaccine; ICPTD, information component temporal pattern discovery; IRR, incidence rate ratio; LASSO, least absolute shrinkage and selection operator; LEOPARD, longitudinal evaluation of observational profiles of adverse events related to drugs; LGPS, longitudinal γ Poisson shrinker; MCBS, Medicare Current Beneficiary Survey; MI, myocardial infarction; MSE, mean squared error; NC, negative control; NPV, negative predictive value; OMOP, Observational Medical Outcomes Partnership; PC, positive control; PPV, positive predictive value; PRR, proportional reporting ratio; PS, propensity score; PSSA, prescription sequence symmetry analysis; ROR, reporting odds ratio; SCC, self-controlled cohort; SCCS, self-controlled case series; US, United States.

The other group of 5 studies in this domain (lower section of Table 3) used negative controls to compare the performance of different analytical methods to mitigate bias in pharmacoepidemiologic studies (31, 60, 64, 86, 141). For example, Davies et al. (31) used an NCO and an NCE to measure and compare the relative bias in the effect estimates from 2 different analytical methods (instrumental variable vs. conventional regression) in assessing the comparative safety of smoking cessation therapies. Weinstein et al. (141) used 31 NCOs to compare residual bias resulting from the 6 different confounding adjustment methods in evaluating comparative safety of frequently used over-the-counter analgesics. The authors reported the proportion of statistically significant NCOs as an indicator of bias (141).

Validating the assumptions of negative controls in pharmacoepidemiologic studies

We identified 3 common strategies for selecting negative control associations satisfying the first assumption (lack of causality assumption) in the included studies: 1) logically impossible association (e.g., causally implausible timing of the event), 2) clinically implausible association, and 3) association with no previous evidence of potential causal association. The use of a prespecified set of drug-outcome pairs classified as either positive or negative controls based on prior scientific evidence has become common practice in validation of pharmacovigilance methods. The Exploring and Understanding Adverse Drug Reactions (EU-ADR) reference set was developed as part of the EU-ADR project and composed of 44 positive and 50 negative control drug-outcome associations across 10 outcomes that were considered important from a pharmacovigilance perspective (187). The EU-ADR researchers started the selection by identifying drugs with enough exposure in the EU-ADR database network and checked the lack of causality for each drug-outcome pair against 3 resources: 1) information from published literature and drug product labels, 2) review of a spontaneous reporting system, and 3) manual verification by expert physicians. The Observational Medical Outcome Partnership (OMOP) reference set was another commonly used set as a reference standard in the OMOP studies (188, 189). This set included 165 positive control and 234 negative control associations across 4 common adverse outcomes (acute kidney injury, acute liver injury, acute myocardial infarction, and upper gastrointestinal bleeding) and similarly checked the lack of causality using evidence from: 1) Food and Drug Administration (FDA) structured product labeling; 2) systematic literature reviews by Tisdale et al. (190) and 3) manual search of the literature. More recently, a semiautomated process for identifying negative controls, in which the algorithm automatically combines up-to-date evidence from various sources (spontaneous reports, scientific literature, and both American and European product labeling), has been introduced (191). We also identified reference sets that were built for specific population groups (e.g., pediatric population) (172, 192, 193) or therapeutic areas (e.g., oncology, vaccine) (164, 193). More information on these reference sets is presented in Web Table 2.

For the second assumption of negative controls (shared bias structure assumption), while there is no empirical test to check this assumption, we found 8 studies that explicitly provided, and compared, the distributions of major measured covariates/confounders between the primary exposure or outcome of interest and their corresponding NCE or NCO (Table 4) (34, 50, 57, 72, 83, 104, 128, 154). While the true similarity in the bias structure of the exposure-outcome association and the negative control association remains unobserved, the premise behind this approach is that showing similar distributions and effects of observed confounders can provide some evidence for potential similarities in unmeasured confounders. Out of the 8 studies, one study provided additional steps to improve the comparability of the bias structure. Dillon et al. (34), in examining the effects of gaps in antihypertensive medication adherence on injurious falls using the gaps for antithrombotic medication adherence as an NCE, checked the distributions of measured covariates between the exposure and NCE samples and found significant differences for gender, comorbidities, and rate of regular medication use between the samples. Accordingly, the authors reweighted the NCE sample using inverse probability weighting to adjust for differences in the covariate distributions (34). The details of all 8 studies are provided in Table 4.

Table 4.

Pharmacoepidemiologic Studies that Provided Analysis for Validating the Assumptions of Negative Controls

First Author, Year (Reference No.)	Study Design	Type of Data Used	Hypothesized Causal Association	Primary Study Effect Measure	Type of NC	Types of Bias as Reported by Study	Statistical Approach Used to Test the Underlying Assumptions of NC
Ivers, 2015 (57)	Case-control study	Primary data collected for the study	Exposure: receipt of oral inactivated bivalent whole-cell cholera vaccine; outcome: acute watery diarrhea with a stool sample positive for cholera	Relative risk	NCO: acute watery diarrhea with a stool sample that tested negative for cholera	Not specified	Study showed the distributions of factors related to clinical presentation and treatment for cholera (time from symptom onset to admission, serotype, dehydration stage at presentation, treatment received at clinic, volume of oral rehydration solution given in clinic, volume of intravenous fluid given in clinic, admitted overnight to the cholera treatment unit, duration of stay at cholera treatment unit, and the discharge outcome type), age, gender, participation response rate, earthen floor in home, ever attended school, electricity in house, number of people in household, whether agriculture is the main source of income, whether main toilet is a latrine, ever admitted to a cholera treatment unit overnight, household member with cholera in the previous week, household member ever spent a night in a cholera treatment unit, and household member with diarrhea in the previous week across the NCO and the main outcome groups.
Lazarus, 2016 (72)	Cohort study	Main study: data from an existing cohort study; replication study: integrated health system data	Exposure: use of proton pump inhibitors; outcome: chronic kidney disease	HR	NCE: use of histamine H2-receptor antagonists	Not specified	Study provided the distributions of age, gender, race, education, health insurance, annual household income, estimated glomerular filtration rate, ratio of urinary albumin to creatinine, smoking, body mass index, systolic blood pressure, prevalent medical conditions, and concomitant medication use for the NCE group along with the primary exposure and the control group.
Han, 2017 (50)	SCCS	Administrative claims	Exposure: concomitant use of a precipitant of interest (vs. not receiving a precipitant) with secretagogues; outcome: serious hypoglycemia	Rate ratio	NCE: concomitant use of a precipitant of interest with metformin	Confounding bias by inherent hypoglycemic effects of the precipitants	Study provided the distributions of gender, median age, and proportion of cases receiving antidiabetics other than secretagogues in the past 30 days before events across the NCE and the primary exposures.
Dillon, 2019 (34)	Cohort study	Pharmacy dispensing data, primary data collected for the study	Exposure: gaps in antihypertensive medication adherence; outcome: injurious fall	Relative risk	NCE: gaps in antithrombotic medication adherence	Confounding bias resulting from healthy adherer bias	Study checked for the distribution of covariates across the exposure (antihypertensive) sample and the NCE (antithrombotic) samples. They found statistically significantly different distributions for gender, comorbidities, and rate of regular medication use between the exposure and the NCE samples. Accordingly, the NCE sample was reweighted using inverse probability weighting, with weights being the ratio of the predicted probabilities using 2 probit regressions, one with and one without statistically different participant characteristic variables of gender, comorbidities, and rate of regular medication use.
Markovic, 2019 (83)	Cohort study	Data from an existing cohort study	Exposure: maternal exposure to NSAIDs during pregnancy; outcome: neurodevelopmental outcomes (attention problem score) in children	Mean difference	NCE: maternal NSAIDs use before pregnancy; NCO: somatic complaints in children	Confounding bias	Study provided the distributions of child factors (gender, birth weight, gestational age at birth, and Apgar score at 5 minutes after birth) and maternal factors (age, ethnicity, education, family income, body mass index, smoking during pregnancy, alcohol use during pregnancy, psychopathology symptoms, cognitive ability score, and concomitant-medication use) for the NCE and the exposure.
Ray, 2019 (104)	Case-control study	EHR	Exposure: receipt of inactive influenza vaccine; outcome: positive test result for any influenza virus	OR	NCO: positive test result for respiratory syncytial virus	Confounding bias	Study provided the distributions of age, gender, Diagnostic Cost Group risk score, influenza vaccination in previous season, year, number of days from September 1 to vaccination, and number of days from vaccination to influenza test for the NCO and the primary outcome. The study also mentioned that the known differential risk factors between the outcome and the NCO are related to age, and that the study adjusted for age and removed ages below 2 years old. Study also cited earlier research that found similar characteristics between respiratory syncytial virus and influenza A in relation to demographic characteristics, chronic illnesses, residence in long-term care facilities, and smoking.
Tien, 2020 (128)	Cohort study	Primary data collected for the study	Exposure: chlorhexidine gluconate bathing (vs. over-the-counter non–chlorhexidine-based antibacterial soap or cleansing lotion); outcome: gram-positive cocci-related, skin flora–related, or central line-associated bloodstream infection	HR	NCO: gut-origin bacteremia	Confounding bias, participation bias	Study provided the results of regression analysis for the NCO with the same set of covariates that were used in the regression analysis for the primary outcome. The covariates included 2% chlorhexidine daily bathing, age, gender, diagnosis of hematological, relapsed/refractory diseases, postchemotherapy neutropenia, autologous peripheral blood stem cell transplantation, chemotherapy, central venous catheter use, portacath use, oral ciprofloxacin/levofloxacin for prophylaxis, oral sulfamethoxazole/trimethoprim for prophylaxis, multidrug-resistant organism carriage, diabetes, and hospital mortality. Results for both the univariate and multivariate analyses were provided across the primary outcome and the NCO. In the univariate analysis of the primary outcome, 2% chlorhexidine daily bathing, post-chemotherapy neutropenia, central venous catheter use, and oral ciprofloxacin/levofloxacin for prophylaxis were statistically significant. In the univariate analysis of the NCO, 2% chlorhexidine daily bathing, post-chemotherapy neutropenia, chemotherapy, and oral ciprofloxacin/levofloxacin for prophylaxis were statistically significant.
Zhou, 2020 (154)	SCCS	Administrative claims	Exposure: concomitant use of a precipitant of interest (vs. not receiving a precipitant) with anticoagulants (anticoagulant-precipitant pairs); outcome: thromboembolism (a composite outcome of stroke and venous thromboembolism)	Rate ratio	NCE: concomitant use of a precipitant of interest with pravastatin (pravastatin-precipitant pairs)	Confounding bias by inherent effect of the precipitant on the primary outcome	Study provided the distributions of gender, age, and baseline medical conditions (prior diagnoses of atrial fibrillation, stroke, serious bleeding, cirrhosis, chronic kidney disease, and hypertension), prior prescription of nonsteroidal antiinflammatory drugs in the 30 days prior to initiation of oral anticoagulant or pravastatin, and prior prescription of antiplatelet agents in the previous 30 days) across the NCE and the primary exposures.

Abbreviations: EHR, electronic health records; HR, hazard ratio; NC, negative control; NCE, negative control exposure; NCO, negative control outcome; NSAID, nonsteroidal antiinflammatory drug; OR, odds ratio; SCCS, self-controlled case series.

DISCUSSION

In this scoping review, we identified 184 pharmacoepidemiologic studies that reported the use of negative controls in their design and analysis or added novel utilities of negative control methods. We identified 4 domains for the use of negative controls: 1) detection of bias, 2) correction for bias in the effect estimates, 3) calibration of P value, and 4) assessing the performance of different methods in drug safety and effectiveness studies.

Among the methodological developments of negative controls, our study revealed that the methods for empirical calibration of P value or 95% CIs were used more than other methods in the field (4, 5). An advantage of the empirical calibration methods is that they do not require the assumption of equi-confounding, which is often unverifiable in practice. Applying this methodology requires defining multiple negative controls for the primary exposure-outcome association, which can be a challenging task for researchers. Thus the OMOP researchers provided a semi-automated process for selecting negative controls (191). One important condition for using the method is that the confounders of negative control associations should be drawn from the same distribution as confounders of the primary drug-outcome association. In addition, identifying negative control drugs is more challenging for case-control designs, as some of these drugs might cause confounding by indication (4). Also, a large amount of heterogeneity of bias estimates of negative controls may lead to increased type II error reducing the power to detect the true safety and efficacy signal (194).

As the use of observational data, for example during the coronavirus disease 2019 (COVID-19) pandemic, gains momentum, the importance of negative controls as a tool to address unmeasured confounding becomes more salient. Two recent studies during the COVID-19 pandemic investigated the effectiveness of mRNA vaccines using observational data, and both studies used negative controls to address unmeasured confounding in the data set (195, 196). These studies were published after our review’s article inclusion period and hence were not included.

While use of negative controls has been primarily for addressing unmeasured confounding in pharmacoepidemiologic studies, they have been used for other types of biases, such as selection bias or measurement error. In addition, under recent developments in the proximal causal inference framework, negative controls are viewed as a version of proxy variables for sources of bias (197). The proximal causal inference framework expands the potential utility of proxy variables in bias correction by acknowledging the presence of different sources of bias with imperfect information (e.g., unmeasured common cause or descendent, unmeasured mediator, censored data) using proxies to make inferences on these latent variables of interest (197–199).

Methods for bias correction have evolved from the bias attenuation work by Flanders et al. (3), which demonstrated the effect of including future exposure in time-series analysis in reducing the amount of bias. This work was further extended by Miao and Tchetgen Tchetgen (200) for further reduction and even elimination of bias by including more exposures in the future in time-series analysis under some untestable parametric assumptions. A recent study provided a selective review of major methodological developments of negative controls in broader epidemiology (9). Our review discovered that many of these new developments for bias correction, such as the control outcome calibration approach (COCA) (6), a method based on scale-invariant generalization of DiD (157), and methods of double negative controls (7, 8, 201), have not yet been widely adopted in pharmacoepidemiologic studies.

Despite the growing attention to negative controls, there is no standardized guidance or statistical method to test the underlying assumptions of negative controls. While identifying unmeasured confounders without any restriction or assumption may be unlikely, recent work by Kummerfeld et al. (202) provided an empirical test for validating a triplet of disconnected negative controls. In the absence of empirical tests, directed acyclic graphs and other structural models that require subject matter knowledge to justify the causal associations can provide a proxy, yet a formal justification for bias. Future research to provide a formal approach to test for the assumptions of negative controls is warranted.

Our study had some limitations. Although we conducted the literature search and selection of articles in a comprehensive and systematic manner, it is possible that some articles might have been missed given the nature of the topic. For example, it is possible that other nomenclature used interchangeably with negative controls might not have been included in our search strategy. A recent study published after our search was conducted, for instance, mentioned the use of falsification endpoint in some negative control studies (9). While we identified some studies using the term falsification endpoint, including every possible nomenclature in our search was not logistically feasible given the substantially large number of article outputs, and we do not believe addition of other terms would change the identified domains for utility of negative controls in pharmacoepidemiologic studies beyond our results. Negative controls are often used as a sensitivity or secondary analysis, which might not be reported in detail in the fields searchable in the databases. A previous study showed that many studies do not report use of a specific methodology in the title and abstract (203). We attempted to minimize the risk of missing a relevant article by manually reviewing the reference lists of the selected articles. Finally, to our knowledge, there is no standardized checklist for the use and report of negative control analyses in observational studies, thus we were not able to grade the study quality of the included articles.

CONCLUSION

Our study laid out a holistic overview of the utility of negative controls to provide guidance for different applications of negative controls in future pharmacoepidemiologic studies. While the primary use of negative controls has been to detect bias, we identified other utility domains for their application, such as assessing the performance of epidemiologic methods of identifying ADRs. Despite considerable methodological advancements in the use of negative controls for bias correction or P-value calibration in recent years, only a few studies have applied these methods for their research questions. Finally, while the validity of negative control use is based on the underlying assumptions, we found that many studies selected their negative controls without efforts made toward validating the underlying assumption. Violation of the underlying assumptions can lead to biased estimates.