Table 2.
Pharmacoepidemiologic Studies That Used Negative Controls to Correct for (or Attenuate) Bias or Calibrate P Values
| First Author, Year (Reference No.) | Study Design | Type of Data | Hypothesized Causal Association | Primary Study Effect Measure | NC(s) Used | Type of Bias as Reported by Study | Methods Used to Correct for Effect of Bias (in Point Estimates, CIs, or P Values) |
|---|---|---|---|---|---|---|---|
| Correction of Point Estimate and 95% CI | |||||||
| Johanson, 2012 (63) | Ecological study | Public health survey data; administrative claims; other publicly available data | Exposure: buprenorphine/naloxone; outcome: report of diversion (the percentage of applicants who reported knowing that buprenorphine/naloxone was sold on the street) and abuse (the percentage who reported knowing that it was used to get high) | Proportion | NCE: amitriptyline; PCE: methadone, oxycodone, and heroin | Measurement error | To correct for the point estimate, the following formulae was used: Relative abuse of buprenorphine/naloxone = (abuse of buprenorphine or naloxone – abuse of NCE) / (abuse of PCE – abuse of NCE)). |
| Greene, 2013 (47) | Cohort study | EHR | Exposure: use of oseltamivir; outcome: neuropsychiatric adverse events (ataxia, psychiatric, encephalitis, disturbance of consciousness) and non-neuropsychiatric events (arrhythmia, syncope, convulsions, movement disorder, stroke) | OR, risk difference | NCO: cellulitis, anemia, injury/trauma | Not specified | To reduce bias (bias attenuation), the history of NCOs was included in the propensity score matching model. |
| Han, 2017 (50) | SCCS | Administrative claims | Exposure: concomitant use of a precipitant of interest (vs. not receiving a precipitant) with secretagogues; outcome: serious hypoglycemia | Rate ratio | NCE: concomitant use of a precipitant of interest with metformin | Confounding bias by inherent hypoglycemic effects of the precipitants | The ratio of the semi-Bayes–adjusted rate ratio associated with the exposure to the semi-Bayes–adjusted rate ratio associated with the corresponding NCE was estimated. The delta method was used for 95% CI calibration (204). |
| Leonard, 2017 (74) | SCCS | Administrative claims | Exposure: discontinuation of the antihyperlipidemic drugs in the presence of ongoing warfarin therapy; outcome: composite of hospitalization for venous thromboembolism or ischemic stroke | IRR | NCE: discontinuation of pravastatin in the presence of ongoing warfarin therapy | Confounding bias by inherent effects on the outcome of the precipitants | To correct for the point estimate and 95% CI, ratio of IRR for the exposure vs. IRR for the NCE was calculated. |
| Gruber, 2018 (48) | Cohort study | Clinical trial data | Exposure: vaccine dose timing; outcome: Severe rotavirus gastroenteritis incidence | Risk difference, risk ratio | NCE: placebo | Confounding bias; administrative censoring | To correct for bias in the estimated risk difference, the effect estimate for the placebo arm was subtracted from the effect estimate for the vaccination arm (for the ratio outcomes, these estimates were divided). A nonparametric bootstrap with 2,000 sample draws with replacement was used to obtain the point estimates and empirical 95% CIs. |
| Schuemie, 2018 (5) | Southworth, Graham replications: cohort study; Tata case-control replication: case-control study; Tata SCCS replication: SCCS | Southworth, Graham replications: Administrative claims; Tata case-control, SCCS replications: EHR | Southworth replication: Exposure: use of dabigatran (vs. warfarin) in the Southwork and Graham replication and use of selective serotonin reuptake inhibitors in the Tata case-control and SCCS replications; outcome: gastrointestinal bleeding, in the Southwork and Graham replications and upper gastrointestinal bleeding in the in the Tata case-control and SCCS replications |
Southworth replication: IRR; Graham replication: HR Tata case-control replication: OR; Tata SCCS replication: IRR | 50 NCOs and 150 synthetic PCOs for each replication | Any residual bias (including confounding bias, misclassification, selection bias) | This study developed a methodology for empirical calibration of 95% CI of the effect estimate when a set of negative controls were used in observational studies. The calibration procedure first estimated the distribution of systematic error using the observed estimates for negative and positive controls, then generated calibrated 95% CIs considering both random and systematic error and assuming Gaussian distribution with a mean and log standard deviation linearly related to the true effect size. We refer to this method as “Schuemie’s empirical CI calibration method.” |
| Schuemie, 2018 (111) | Cohort study | Administrative claims | Exposure: use of duloxetine (vs. sertraline); outcome: stroke | HR | 52 NCOs and 156 PCOs | Any residual bias (including confounding bias) | The study used Schuemie’s empirical CI calibration method (5). |
| Thorrington, 2018 (126) | Ecological study | Hospital admission data | Exposure: 24-month post-PCV period (vs. pre-PCV period); outcome: pneumonia, empyema, sepsis, otitis media | IRR | NCO: urinary tract infections, infections of the skin and subcutaneous tissue, disorders of the thyroid gland, diseases of the blood, and fractures | Bias arising from potential secular trend | For each outcome of interest, the age-specific ratio of the IRR for the outcome of interest vs. the geometric mean of the IRR for the NCOs was calculated. The minimum and maximum incidence rate ratio across all NCOs were used to represent uncertainty. |
| Leonard, 2019 (73) | SCCS | Administrative claims | Exposure: concomitant use of clopidogrel and a precipitant drug of interest (clopidogrel-precipitant pairs); outcome: gastrointestinal bleeding or intracranial hemorrhage | Rate ratio | NCE: concomitant use of pravastatin and a precipitant drug of interest (pravastatin-precipitant pairs) | Confounding bias by inherent bleeding effects of the precipitant drug | The ratio of the semi-Bayes–adjusted rate ratio associated with the exposure to the semi-Bayes–adjusted rate ratio associated with the corresponding NCE was estimated. The delta method was used for 95% CI calibration (204). |
| Hripcsak, 2020 (53) | Cohort study | Administrative claims; EHR | Exposure: use of chlorthalidone (vs hydrochlorothiazide); outcome: multiple cardiovascular outcomes (e.g., acute myocardial infarction, hospitalization for heart failure, etc.) | HR | 76 NCOs and 228 PCOs (3 PCOs are generated for each NCO, with true relative risk 1.5, 2, and 4). | Any residual bias | The study used Schuemie’s empirical CI calibration method (5). |
| Lane, 2020 (70) | Cohort study | Administrative claims; EHR | Exposure: initiation of hydroxychloroquine (vs. initiation of sulfasalazine); outcome: 16 severe adverse events (e.g., gastrointestinal bleeding, acute renal failure, acute pancreatitis, myocardial infarction, stroke, etc.) | HR | NCO: 67 conditions | Confounding bias | The study used Schuemie’s empirical CI calibration method (5). |
| Rodgers, 2020 (107) | Cohort study | EHR | Exposure: use of thiazolidinediones (vs. sulfonylureas); outcome: edema, weight gain | HR | NCO: gastrointestinal side effects | Confounding bias | To correct for bias, the ratio of the HR for the post period vs. the HR of the prior period was calculated. They verified the removal of bias by replicating the same bias correction method for the NCO (ratio of the HRs for the post vs. prior periods for the NCO). The standard error for the estimates was obtained by bootstrapping. |
| Shi, 2020 (7) | Cohort study | Integrated health system data | Exposure: receipt of DTaP-IPV-Hib vaccine (vs. DTap containing comparator vaccine); outcome: fever; NCO: injury or trauma; NCE: ringworm | Relative risk | 1 NCO and 1 NCE | Confounding bias | The study developed a methodology for correcting for a categorical unmeasured confounding. The methodology uses an NCE and an NCO to build a semiparametric model and propose multiply robust estimator for the average treatment effect. The study demonstrated the application of their method in a pharmacoepidemiologic vaccine safety study and showed that the multiply robust estimator provided a smaller bias and protected against the model misspecification. |
| Zhou, 2020 (154) | SCCS | Administrative claims | Exposure: concomitant use of a precipitant of interest (vs. not receiving a precipitant) with anticoagulants (anticoagulant-precipitant pairs); outcome: thromboembolism (a composite outcome of stroke and venous thromboembolism) | Rate ratio | NCE: concomitant use of a precipitant of interest with pravastatin (pravastatin-precipitant pairs) | Confounding bias by inherent effect of the precipitant on the primary outcome | The ratio of the semi-Bayes–adjusted rate ratio associated with the exposure to the semi-Bayes–adjusted rate ratio associated with the corresponding NCE was estimated. The delta method was used for 95% CI calibration (204). |
| P-Value Calibration | |||||||
| Schuemie, 2014 (4) | Cohort study (example 1); case-control study (example 2); SCCS (example 3) | Administrative claims (example 1); EHR (examples 2 and 3) | Exposure: use of isoniazid (example 1); use of selective serotonin reuptake inhibitors (examples 2 and 3); outcome: acute liver injury (example 1); upper gastrointestinal bleeding (examples 2 and 3) | OR | 37 NCEs (example 1); 67 NCEs (examples 2 and 3) | Most forms of bias (including confounding bias, misclassification, selection bias) | This study developed a methodology for calibrating the P value of the effect estimate when a set of negative controls were used to account for bias. The method first derived an empirical null distribution from the observed effect estimates for the negative controls, then generated calibrated P values assuming Gaussian distribution to the estimates and taking into account the sampling error of each estimate. We refer to this method “Schuemie’s empirical P-value calibration method.” |
| Duke, 2017 (38) | Cohort study | Administrative claims; EHR | Exposure: use of levetiracetam; outcome: risk of angioedema | HR | 100 NCOs | Any residual bias | The study used Schuemie’s empirical P-value calibration method (4). |
| Yuan, 2018 (151) | Cohort study | Administrative claims | Exposure: use of sodium glucose co-transporter 2 inhibitors (overall), and canagliflozin (specifically) (vs. non-sodium glucose co-transporter 2 inhibitor antihyperglycemic agents); outcome: below-knee lower extremity amputation | HR | NCO (number not reported) | Confounding bias | The study used Schuemie’s empirical P-value calibration method (4). |
| Schuemie, 2019 (112) | Case-control studies | Administrative claims | Exposure: use of isotretinoin (Crockett study replication); use of dipeptidyl peptidase-4 inhibitors (Chou study replication); outcome: ulcerative colitis (Crockett study replication); acute pancreatitis (Chou study replication) | OR | 35 NCEs | Any residual bias | The study used Schuemie’s empirical P-value calibration method (4). |
| Kim, 2020 (65) | Cohort study | Administrative claims; EHR | Exposure: use of alendronate (vs. use of raloxifene); outcome: hip fracture, vertebral fracture, esophageal cancer, osteonecrosis of the jaw | HR | 147 NCOs | Confounding bias | The study used Schuemie’s empirical P-value calibration method (4) to construct the empirical null distribution and compare it with the theoretical null distribution to check the presence of bias. |
| You, 2020 (148) | Cohort study | Administrative claims | Exposure: ACEIs/ARB + CCB vs. ACEI/ARB + TZD vs. CCB+TZD; outcome: all-cause mortality, myocardial infarction, heart failure, stroke, a composite outcome of major adverse cardiac and cerebrovascular events | HR | 39 NCOs | Any residual bias | The study used Schuemie’s empirical P-value calibration method (4). |
| Both Bias Correction With Regard to Point Estimate and 95% CI and P-Value Calibration | |||||||
| Weinstein, 2020 (139) | Cohort study | EHR | Exposure: use of any paracetamol (vs. use of ibuprofen); outcome: diagnoses of gastrointestinal bleeding, myocardial infarction, ischemic or hemorrhagic stroke, or acute or chronic renal disease | HR | 39 NCOs and 147 synthetic PCOs (3 for each NCO) | Channeling bias | The study used Schuemie’s empirical CI and P-value calibration method (4, 5). |
| Morales, 2021 (89) | Cohort study | Administrative claims; EHR | Exposure: use of ACEIs and ARBs (vs. CCBs and thiazide or THDs); outcome: COVID-19 diagnosis; hospital admission with COVID-19; hospital admission with pneumonia; hospital admission with pneumonia, acute respiratory distress syndrome, acute kidney injury, or sepsis | HR | 123 NCOs | Confounding bias | The study used Schuemie’s empirical CI and P-value calibration method (4, 5). |
Abbreviations: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin-receptor blocker; CCB, calcium channel blocker; CI, confidence interval; COVID-19, coronavirus disease 2019; DTaP-IPV-Hib, diphtheria and tetanus toxoids and acellular pertussis adsorbed, inactivated poliovirus, and haemophilus B conjugate vaccine; EHR, electronic health records; HR, hazard ratio; IRR, incidence rate ratio; NC, negative control; NCE, negative control exposure; NCO, negative control outcome; OR, odds ratio; PCE, positive control exposure; PCO, positive control outcome; PCV, pneumococcal conjugate vaccine; SCCS, self-controlled case-series; TZD, thiazide-type diuretic.