Table 5.

Recommendations for conducting sensitivity analyses in observational studies

Recommendation 1: Consider the potential unmeasured confounders or unaddressed bias that may alter the study’s underlying assumptions
Explanation
• In observational studies using RCD, unmeasured confounders or unaddressed bias often threaten the validity of the inferences • Carefully considering the potential unmeasured confounders or unaddressed bias and testing how these violations may change the effect estimates is essential to strengthen the validity of the inferences • Typically, an observational study’s underlying assumptions can be altered along three dimensions: study definition, study design, and analysis modeling
Example
• A real-world cohort study investigated the effect of dipeptidyl peptidase-4 (DPP-4) inhibitors on the incidence of inflammatory bowel disease (IBD) among patients with type 2 diabetes [18]. In this study, the authors conducted sensitivity analyses including the following three dimensions: ✓ Using alternative exposure definition: a stricter exposure definition that DPP-4 inhibitor use was redefined as receipt of at least four prescriptions (patients were considered exposed after first prescription in the primary analysis), and alternative outcome definition, restricting in IBD events to those supported by clinically relevant events (diagnoses of IBD with or without clinically relevant events in database for the primary analysis); ✓ Changing the study design: with alternative eligibility of study population, excluding patients treated with thiazolidinediones at any time before cohort entry (prior use of thiazolidinediones was not considered in the primary analysis); ✓ Using alternative analysis modeling: such as performing a competing risk analysis by death from any cause using the Fine and Gray sub-distribution model (using time-dependent Cox model in the primary analysis). The marginal structural model was used for testing the potential impact of time-varying confounding and the rule-out method was applied to test for unmeasured confounding
Recommendation 2: Carefully select and clearly report the approaches used to conduct sensitivity analyses
Explanation
•Select appropriate approaches to provide quantitative assessments of the robustness of the primary analysis to violations of the assumption of no unmeasured confounders or unaddressed bias; •As observational studies are always subjected to unmeasured confounding, methods such as E-value and negative control are important to apply to assess the impact; • The sources of uncertainty being tested in each sensitivity analysis should be clearly stated; •The basic rationales and implementations of alternative approaches should be provided
Example
• A cohort study by Crellin et al. investigated association between trimethoprim use for urinary tract infection and risk of adverse outcomes in older patients [46]. In the Method section, the uncertainty being tested, methods, and rationales for the sensitivity analyses were provided as follows: “Finally, to ensure that we were comparing similar groups (to reduce confounding by indication), we examined the risks of all three outcomes after propensity score weighting (inverse probability of treatment weighting) of trimethoprim and amoxicillin users (full details in web appendix 1). In inverse probability of treatment weighting, patients are reweighted according to the inverse of their probability of receiving the treatment they actually received.”
Recommendation 3: Clearly report the sensitivity analysis results and compare the differences between primary and sensitivity analyses
Explanation
• Clearly report effect point estimates and confidence intervals for each sensitivity analysis; • Systematically evaluate the consistency between primary and sensitivity analyses by comparing the point estimates, confidence intervals, and statistical significance
Example
• Filion et al. summarized point estimates and confidence intervals of adjusted hazard ratios for the primary and sensitivity analyses in one table [47]. As clearly shown in one column, point estimates and statistical uncertainty were easy to compare; • Shapiro et al. summarized point estimates and confidence intervals of adjusted hazard ratios for the primary and sensitivity analyses in one figure [48]. Presenting a forest plot with the number of point estimates and 95% confidence interval is a good reporting example
Recommendation 4: Highlight the differences and discuss the reason for the divergence when primary and sensitivity analyses yield different inferences
Explanation
• When sensitivity analyses result in changed effect estimates, researchers should highlight the differences and discuss the potential reason for the inconsistency; • The investigators should also reconsider the underlying assumptions, and interpret the findings with caution
Example
• A cohort study by Abrahami et al. investigated the use of incretin-based drugs and risk of cholangiocarcinoma among patients with type 2 diabetes [18]. In the Results section, the authors highlighted the inconsistent results between sensitivity and primary analyses: “The sensitivity analyses led to generally consistent results (supplementary tables 4–13), except for the lagged analyses with hazard ratios ranging from 1.31 to 1.62 for DPP-4 inhibitors and 1.42 to 2.38 for GLP-1 receptor agonists. The stricter exposure definition generated hazard ratios that excluded the null for both DPP-4 inhibitors (1.77, 95% confidence interval 1.01 to 3.11) and GLP-1 receptor agonists (2.46, 1.04 to 5.85).” • Suchard et al. assessed the effectiveness and safety of first-line antihypertensive drug classes [49]. Two different analysis strategies, on-treatment analysis vs. intention-to-treatment analysis, yielded differential results ✓ The following difference was noted: “On-treatment time results in shorter follow-up than intention to treat. As expected, we saw blunted estimates of differential effectiveness and risks between drug class new users under an intention-to-treat design.” ✓ The authors explained the reason for the difference: “On-treatment follow-up also helps to assess differential adherence to initial treatment. Except in the Columbia University Medical Center database, median on-treatment time was modestly shorter (0–38 days) for thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors new users. Such differences, if meaningful, are also less likely to confound on-treatment estimates where time-at-risk ends with treatment discontinuation. Further, claims databases reported drug fulfilment whereas electronic health records reported prescriptions. Because fulfilment more directly reflects actual drug taking, one might expect differential adherence to generate notable effect estimate differences across data sources; we did not observe such differences in comparing thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors new users.” ✓ The results were interpreted as follows: “We found, however, that patients initiating treatment with a thiazide or thiazide-like diuretic had a significantly lower risk of seven effectiveness outcomes, including acute myocardial infarction, hospitalisation for heart failure, and stroke, as compared with angiotensin-converting enzyme inhibitors new users, while patients remain on-treatment with their initial drug class choice.”