Comparison of self-controlled designs for evaluating outcomes of drug-drug interactions: simulation study

Abstract

Background

Self-controlled designs, both case–crossover and self-controlled case series, are well suited for evaluating outcomes of drug–drug interactions in electronic healthcare data. Their comparative performance in this context, however, is unknown.

Methods

We simulated cohorts of patients exposed to two drugs: a chronic drug (object) and a short-term drug (precipitant) with an associated interaction of 2.0 on the odds ratio scale. We analyzed cohorts using case–crossover and self-controlled case series designs evaluating exposure to the precipitant drug within person–time exposed to the object drug. Scenarios evaluated violations of key design assumptions: (1) time-varying, within-person confounding; (2) time trend in precipitant drug exposure prevalence; (3) non-transient precipitant exposure; and (4) event-dependent object drug discontinuation.

Results

Case–crossover analysis produced biased estimates when 30% of patients persisted on the precipitant drug (estimated OR 2.85) and when the use of the precipitant drug was increasing in simulated cohorts (estimated OR 2.56). Self-controlled case series produced biased estimates when patients discontinued the object drug following the occurrence of an outcome (estimated incidence ratio (IR) of 2.09 [50% of patients stopping therapy] and 2.22 [90%]. Both designs yielded similarly biased estimates in the presence of time-varying, within-person confounding.

Conclusion

In settings with independent or rare outcomes and no substantial event-dependent censoring (<50%), self-controlled case series may be preferable to case–crossover design for evaluating outcomes of drug–drug interactions. With heavy event-dependent drug discontinuation, a case–crossover design may be preferable provided there are no time-related trends in drug exposure.

INTRODUCTION

Drug–drug interactions are an increasingly important clinical and public health concern as individuals with multiple chronic conditions are living longer and taking more drugs. Pre-approval screening for potential drug interactions is an important step in the development of new medications; however, the impact of putative interactions on patient health outcomes is usually not quantified before drugs are marketed and used in large populations of patients. Electronic healthcare databases provide a valuable opportunity for providing real world evidence on the clinical impact of drug interactions.

Self-controlled designs are particularly promising for high-throughput screening of large databases to quantify adverse health outcomes of potential drug–drug interactions. As observational data reflect treatment decisions that are not random, control of confounding by factors that drive treatment selection and that are also associated with the outcome of interest is essential and often presents a critical challenge in observational research. In contrast to other observational methods, self-controlled designs, in which comparisons are made within individuals, have as a strength the inherent control for all potential confounders that are stable over the study period, including those that are unmeasured and unknown.^1–3

Self-controlled designs comprise two main classes of methods, which can be thought of as outcome-indexed (e.g., the case–crossover) and exposure-indexed (e.g., the self-controlled case series) designs. In recent studies, both designs have been used to identify and quantify outcomes of drug–drug interactions in electronic healthcare databases.^4–6 Both designs have also been recommended for pharmacoepidemiologic screening of electronic healthcare data to identify outcomes of interacting drugs.^7,8 However, these designs rely on different assumptions. While both designs require an assumption of no within-person confounding by factors associated with both exposure and outcome, the conventional self-controlled case series method, which makes use of exposure information both before and after an outcome, requires that the occurrence of an event does not influence subsequent drug exposure or the end of observation.⁹ The unidirectional case–crossover design with right-censoring at an outcome, which is the conventional implementation used in studies of drug effects, requires no time trends in the exposure prevalence.³ It is unknown to what extent violations of these assumptions result in bias in observational studies of drug–drug interactions. These designs also have not been formally compared for their ability to identify and quantify the effects of drug–drug interactions.

The objective of this study was to use simulated data to compare the performance of the unidirectional case–crossover design and the bidirectional self-controlled case series design in the context of two interacting drugs. We specifically focused on scenarios that involved violation of key assumptions of these designs, namely the presence of time-varying, within-person confounding, event-dependent drug discontinuation, and changing drug exposure prevalence.

METHODS

Simulation setup

We simulated a setting in which patients were exposed to two drugs: a chronic drug that increased the occurrence of an adverse outcome (object drug) and a short-term drug (exposure duration 14 days; precipitant drug) that had no effect on the outcome in the absence of object drug exposure (no direct effect), but increased the odds of the outcome twofold when both drugs were used concomitantly. The details of the simulation are presented in the eAppendix. Briefly, for each patient, we generated day-level exposure status for each of the two drugs to emulate data available in longitudinal electronic healthcare databases. On average, patients were exposed to the object drug for 130 days. Daily probability of outcome was calculated as a function of exposure to the drugs and was used to generate a binary outcome (Y_ij) for day j during follow-up for person i:

$$\mathrm{Logit}\left(\mathrm{Pr}\left[{\mathrm{Y}}_{\mathrm{ij}}=1\right]\right)=-8+\ln \left(2.5\right){\mathrm{Obj}}_{\mathrm{ij}}+\ln \left(1.0\right){\mathrm{Prec}}_{\mathrm{ij}}+\ln \left(2.0\right){\mathrm{Obj}}_{\mathrm{ij}}*{\mathrm{Prec}}_{\mathrm{ij}}$$

where Obj_ij and Prec_ij are indicators for exposure to the object drug and precipitant drug, respectively, for person i, day j. The effects of drugs on the outcome were constant – i.e., no cumulative or time-varying effects were modeled. Patients could have multiple outcomes, which were independent.

For each scenario, we generated 1000 cohorts of 30,000 patients each and analyzed them using the case–crossover design and the self-controlled case series design, both restricted to analyses within person–time exposed to the object drug (Figure 1). Self-controlled designs that evaluate only person–time exposed to the object drugs inherently control for confounding associated with the object drug and have been recommended for observational studies of drug–drug interactions, particularly for screening purposes.¹⁰ The relation of interest in such designs is the association of the precipitant drug with the outcome, which can be interpreted as the effect of the interaction in the absence of a direct effect of the precipitant on the outcome.¹⁰

Figure 1.

Case–crossover and self-controlled case series analyses

Scenarios

A total of fifteen scenarios were simulated evaluating violation of four major assumptions to various extents: (1) presence of time-varying, within-person confounding; (2) population time trend (linear increase) in precipitant drug utilization; (3) therapy with a non-transient precipitant; and (4) event-dependent discontinuation of the object drug. In all scenarios, the true effect of the object drug on the outcome was set to odds ratio (OR) of 2.50, the effect of the precipitant drug on the outcome was set to OR of 1.0 and the effect of the interaction was set to the ratio of ORs of 2.0. More detailed description of simulations with the code is presented in the eAppendix.

We did not introduce any violation of assumptions in the base case scenario. To evaluate the impact of time-varying, within-person confounding, we generated two scenarios, where either 40% or 80% of patients were exposed to a confounder that increased the odds of the outcome 1.75-fold (OR 1.75), was associated with exposure to the precipitant drug, and had the duration of 5 days. In these scenarios, exposure to the precipitant drug was simulated as a function of the confounder such that the expected prevalence of the confounder was 43% among patients exposed to the precipitant drug and 13% among unexposed patients in the scenario with 40% overall confounder prevalence, and 86% among exposed versus 28% among unexposed patients in the scenario with 80% overall confounder prevalence. For patients who were exposed to both the covariate and the precipitant drug, the start of the precipitant drug exposure corresponded to the start of covariate exposure.

For the scenario with a population time trend in precipitant drug exposure, we simulated a linear increase in the precipitant drug initiation over time, such that the number of patients initiating the drug at the end of the study period was double the number initiating at the study midpoint (see eFigure for precipitant drug exposure prevalence over the course of the study period).

To evaluate the performance of designs in the presence of non-transient precipitant drug exposure, we simulated two separate scenarios. In the first scenario, duration of exposure to the precipitant drug was 90 days for everybody, although patients who were censored while on treatment or who entered the cohort exposed, could have shorter observed treatment durations. In the second scenario, 70% of patients were exposed for 30 days while the remaining 30% did not discontinue therapy and were censored while exposed, a likely scenario for chronic drugs intended as life-long treatment, but with high degree of patient non-adherence.

We evaluated the impact of event-dependent drug discontinuation by simulating 10%, 50%, or 90% of patients discontinuing the object drug on the occurrence of an outcome (three separate scenarios). In addition, as a sensitivity analysis, we evaluated whether the impact of event-dependent drug discontinuation differed in situations where the precipitant drug is not a short-term therapy by simulating object drug discontinuation for each scenario of non-transient precipitant drug exposure (90-day exposure and 30% of patients persisting on precipitant drug therapy).

Analyses

We included only patients who experienced outcomes while exposed to the object drug in the analyses. Because outcomes were independent, we included all outcomes in our main analysis; however, we also conducted analyses using the first outcome only since this is a common approach in situations when independence of outcomes cannot be assumed.

Case–crossover

Logistic regression, stratified on individual, was used to compare the odds of exposure to the precipitant drug on the date of the outcome to the odds of exposure on a referent day 30 days preceding the outcome (Figure 1). Patients were required to be enrolled and exposed to the object drug for at least 30 days prior to the outcome to ensure that both the outcome and referent days were within object-drug exposed person–time and no confounding due to changing exposure to the object drug is introduced. Outcomes that did not satisfy this criterion (e.g., occurred during the first 30 days of object drug exposure) were excluded.

Self-controlled case series

The observation period started on the first day of exposure to the object drug and ended on the last day of object drug exposure (Figure 1). All outcomes were counted. The incidence during the time exposed to the precipitant was compared to the incidence during the time not exposed to the precipitant using the standard self-controlled case series method from the SCCS R package (version 1.0) that is fit using conditional logistic regression, stratified on each event.^11,12

Performance metrics

For each scenario and design we report the estimated effect on the log scale, which is the mean of log estimates from 1,000 simulation iterations. We calculated bias as the mean difference between the estimated effect and the true effect on the log scale across 1,000 simulation iterations. We calculated mean squared error as the sum of the variance of the log estimates (across 1,000 simulations) and the squared bias. In addition, we also calculated the OR in the case–crossover analysis and incidence ratio (IR) in the self-controlled case series analysis as the exponential of the mean estimated effect on the log scale.

We performed all simulations and analyses using R statistical software (RStudio version 3.2.2).

RESULTS

Table 1 presents the estimates, biases and MSEs on the log scale, along with odds ratio estimates for the case–crossover and incidence ratio estimates for the self-controlled case series, across all scenarios. When no assumptions were violated, both designs produced unbiased estimates, with the self-controlled case series method having slightly better average precision (Table 1).

Table 1.

Comparison of parameter estimates from the case–crossover and the self-controlled case series designs across all scenarios

Scenarios	Case–crossover				Self-controlled case series
	Log estimatea	Bias	MSE	OR	Log estimatea	Bias	MSE	IR
Base case (no violation of assumptions)	0.70	0.01	0.04	2.02	0.70	<0.01	0.01	2.01
Time-varying confounder (prevalence 40%)	0.82	0.12	0.06	2.27	0.80	0.11	0.02	2.22
Time-varying confounder (prevalence 80%)	0.91	0.22	0.09	2.49	0.90	0.21	0.05	2.46
Time trend (linear increase) in precipitant drug exposure prevalence	0.94	0.25	0.10	2.56	0.69	−0.01	0.01	1.98
90-day precipitant	0.70	0.01	0.02	2.02	0.70	<0.01	<0.01	2.00
Precipitant drug with persistent useb	1.05	0.36	0.15	2.85	0.69	<0.01	0.01	1.99
Object drug discontinuation (10%)	0.71	0.01	0.04	2.03	0.70	0.01	0.01	2.02
Object drug discontinuation (50%)	0.71	0.01	0.05	2.03	0.73	0.04	0.02	2.09
Object drug discontinuation (90%)	0.71	0.01	0.04	2.03	0.80	0.10	0.03	2.22

^aTrue value is 0.69 (OR of 2.0).

^bAmong patients exposed to the precipitant drug, 70% were exposed for 30 days and 30% stayed exposed through the end of the study (persistent use). Bias calculated as the mean difference between the log estimate and the true value (averaged across 1,000 iterations); MSE – mean squared error; OR – odds ratio; IR – incidence ratio. Both OR and IR are calculated as exponential of the mean log estimate.

In the presence of within-person confounding, the analyses were biased to the same extent, and the magnitude of bias increased with the increasing prevalence of the confounder. Thus, when 40% of the population were exposed to the confounder, the case–crossover yielded an estimated OR of 2.27 while the IR estimate from self-controlled case series was 2.22. When 80% of patients were exposed to the confounder, the case–crossover estimate was 2.49 and the self-controlled case-series 2.46.

As expected, in the scenario where there was a linear increase in the precipitant drug initiation over the study period, the case–crossover design yielded biased estimates (bias of 0.25; OR of 2.56), while the self-controlled case series design was not affected (bias = −0.01).

Both designs produced unbiased estimates when the precipitant drug exposure lasted 90 days; however, in the scenario where 30% of patients persisted on the precipitant drug therapy, the case–crossover design produced biased estimates (bias 0.36; OR of 2.85), while the self-controlled case series yielded unbiased estimates (bias < 0.01).

Finally, event-dependent object drug discontinuation had no impact on the case–crossover analysis but led to biased estimates for the self-controlled case series method. The bias increased with increasing percentage of patients discontinuing the object drug (Table 1) but was still relatively small (≤ 0.10; estimated IR of 2.09 with 50% of patients discontinuing the object drug and 2.22 with 90%) in the main scenarios with a 14-day precipitant drug. However, in situations when the precipitant drug was not used transiently, the bias due to object drug discontinuation was much larger (eTable 1). When precipitant drug duration was 90 days, estimated IR was 2.22 when 50% of patients discontinued the object drug and 2.64 when 90% discontinued. When 30% of patients persisted on the precipitant drug therapy, 50% discontinuation led to estimated IR of 2.72 and 90% to 4.04 (eTable 1).

Since most patients experienced only one outcome, analyses limited to the first outcome produced similar results to analyses that included all outcomes (eTable 2). Nevertheless, we observed a small increase in bias when only first outcomes were included in case–crossover analyses, as compared to analyses of all outcomes. This increase in bias was not observed in self-controlled case series analyses, and for most scenarios self-controlled case series analyses of first and all outcomes yielded identical results. However, in scenarios with increasing precipitant drug exposure prevalence over time, self-controlled case series analyses of first outcomes yielded estimates that were slightly biased downward (bias of −0.03, IR 1.94 when 30% of patients persisted on therapy and bias of −0.03, IR 1.95 when precipitant drug use was increasing in simulated cohorts), even though their respective analyses that included all outcomes yielded unbiased estimates (bias ≤ 0.01).

DISCUSSION

Self-controlled designs are attractive for pharmacoepidemiologic investigations of outcomes caused by drug–drug interactions due to their inherent control for all confounders that are stable over the observation period, and because exposure to interacting drugs is often transient in real world and outcomes are abrupt. However, these designs are still susceptible to time-varying, within-person confounding and rely on design-specific assumptions that may be violated in the context of interacting drugs (Table 2). In this paper, we investigated the impact that violation of these assumptions had on the performance of two self-controlled designs, the case–crossover and the self-controlled case series.

Table 2.

Key assumptions of the case–crossover and the self-controlled case series (SCCS) methods^a and consequences of their violations in the context of interacting drugs

Design		Key Assumptions	Consequences of violation
Case–crossover	SCCS
✓	✓	No within-person, time-varying confounding (e.g., confounding by drug indication)	Both designs are affected to the same extent. Magnitude of bias depends on the magnitude of the confounder–outcome association, confounder–exposure association, and prevalence of the confounder.
✓		Constant drug exposure probability in population: no time-trends in the exposure	Biased estimates with the case–crossover design
✓		Transient drug exposure	Evaluating persistent exposures leads to upward bias in the case–crossover design; no bias when prolonged, but finite exposures are evaluated
	✓	Outcome does not affect subsequent drug exposure	We did not evaluate violation of this assumption independent of event-dependent censoring in the current study
	✓	Observation time is not affected by the outcome	In SCCS design nested within person–time exposed to an object drug, discontinuation of the object drug following an outcome will lead to censoring and biased estimates, particularly when discontinuation occurs in more than 50% of the patients. Bias may be larger when chronic precipitant drugs are evaluated.
	✓	Independent recurrent events or rare if independence cannot be assumed	We did not evaluate violation of this assumption in the current study

^aAs conventionally implemented in studies of drug effects: case–crossover with right censoring at an outcome (unidirectional) and SCCS that makes use of exposure information before and after outcome(s) of interest.

Our simulations showed that both case–crossover and self-controlled case series designs had good performance when no assumptions were violated and both were affected to the same extent by time-varying, within-person confounding. As expected, the self-controlled case series method produced biased results in the presence of event-dependent drug discontinuation; however, in the situation with a 14-day precipitant, bias was substantial only when 90% of patients discontinued the object drug following the outcome occurrence. These findings are consistent with at least one prior study that found that, when event-independence of exposure or observation periods are found to be questionable in a self-controlled case series analysis, estimates may still exhibit very little bias.⁹ Nevertheless, if an event is a contraindication to continuing therapy, or many patients decide to stop therapy because of the adverse event, discontinuation can be substantial. Researchers should also be aware that in a drug–drug interaction study nested within person–time exposed to an object drug, discontinuation of the object drug represents event-dependent censoring, not event-dependent exposure termination, while discontinuation of both drugs will lead to violation of both assumptions, i.e., event-dependent censoring and event-dependent exposure termination. While we did not observe substantial bias due to object drug discontinuation with a short-term precipitant drug, the bias was much larger when the precipitant drug was used chronically. Overall, when substantial event-dependent exposure discontinuation occurs, a unidirectional case–crossover, in which all exposure assessment occurs before an outcome (as was implemented in the simulation study) may be a more appropriate design choice.

While the case–crossover design was not affected by object drug discontinuation, it produced biased results in the presence of a time trend in exposure prevalence. Interestingly, the most biased case–crossover estimates were produced when some patients persisted on the precipitant drug (i.e., were exposed until censored). Some prior research has suggested that studying persistent exposures with the case–crossover design may lead to upward bias, as the only pattern that such exposure may contribute to the analysis is exposed during the hazard period and unexposed during the referent period.¹³ Our results were consistent with these prior findings. At the same time, we observed no bias when the duration of the precipitant drug therapy was 90 days for all exposed patients. Although 90-day exposure is not necessarily transient and a substantial number of patients with 90-day treatments may be censored while exposed, as long as the utilization of a drug is not changing on the population level, there will be similar rates of treatment initiation and discontinuation throughout the study period, and the case–crossover design will yield unbiased estimates.

In the setting where patients both persisted on the precipitant drug therapy and discontinued object drug following an outcome, both designs yielded biased estimates. A cohort design may be more appropriate in such settings.

Our results should be interpreted in the context of the simulation design. First, we generated scenarios under controlled conditions. We assumed no exposure or outcome misclassification, no time-varying or cumulative effects, and no effect modification by other factors. Exposure misclassification, in particular, is likely in studies of drug–drug interactions based on claims data since information on whether patients used the drugs as prescribed or recommended is missing and some patients may be instructed by their healthcare provider to stop or modify object drug therapy while taking the precipitant, especially, if the therapy with the precipitant drug is short. Self-controlled designs may be particularly sensitive to exposure misclassification.

Next, in our simulated data, patients could have had multiple events (up to three) that were assumed to be independent. While usually only the first outcome is included in case–crossover analyses, the self-controlled case series design typically includes all outcomes, but requires them to be independent in order to meet the assumptions of the Poisson model.¹⁴ Very often, however, occurrence of an event increases the probability of subsequent events, which violates the assumption of independence. In such cases, analyses of first outcomes have been recommended.¹ Our analyses of first outcomes produced similar results for most scenarios; however, there was a small downward bias when self-controlled case series analyses were limited to first outcomes only and the precipitant drug exposure prevalence was increasing either due to increasing utilization in simulated cohorts or due to patients’ persistence on therapy. We are not aware of this bias having been documented before, and the possibility that it is an artifact of our simulation setup cannot be completely ruled out. Further research is needed to investigate this phenomenon in more detail.

Further, in each scenario, we focused on a violation of a specific assumption, whether it was confounding, event-dependent censoring, or a trend in exposure prevalence in population. However, in real world, many of these conditions may be present simultaneously and in more complex forms. In addition, we generated an interaction on the multiplicative scale and assessed it using a multiplicative model. It is possible that some clinically relevant drug–drug interactions operate on the additive scale without manifesting as a multiplicative interaction. Although we tried to cover many scenarios, our results are only relevant to the scenarios we evaluated, to our simulation design, and to the selected parameter values.

Design variants have been proposed for situations in which a violation of a specific assumption is expected. For example, the case–time–control design attempts to account for time trends in exposure,¹⁵ while a particular implementation has been proposed as a method to control for time-varying confounding within the case–crossover design.¹⁶ A bidirectional case–crossover that includes referent windows after the outcome may not incur exposure-trend bias, but will require the assumption of no event-dependent censoring or change in exposure. Similarly, variants of the self-controlled case series method have been developed to deal with event-dependent change in exposure, event-dependent censoring, or simultaneous violation of both which may occur frequently in drug–drug interaction studies nested within person–time exposed to one of the drugs when the outcome affects subsequent use of both medications.⁹ We did not evaluate the ability of the proposed approaches to correct the biases within our scenarios.

Finally, in our investigation we focused on the approach that evaluates the effect of exposure to the precipitant drug (the drug that causes the drug interaction) on the outcome within person–time exposed to the object drug (the drug that is affected by the interaction). This approach has been recommended in the pharmacoepidemiology literature and is particularly well suited for pharmacoepidemiologic screening for drug–drug interactions in electronic healthcare data because it inherently controls for confounding or bias associated with the object drug (which can be quite strong) and allows simultaneous evaluation of multiple precipitants.^7,8,10 However, other approaches have been proposed and may be more appropriate, particularly, when a precipitant drug has a direct effect on the outcome or when an interaction needs to be assessed on the additive scale.^10,17

In summary, choosing a self-controlled design for a drug–drug interaction study requires careful consideration of the question of interest and all possible sources of bias. In settings with independent (or rare) outcomes and no substantial event-dependent drug discontinuation (<50%), the self-controlled case series may be preferred to the case–crossover design. With heavy event-dependent drug discontinuation, however, the case–crossover design may be a better choice provided there are no time-related trends in precipitant drug exposure, including due to persistent therapy. In settings with both substantial event-dependent object drug discontinuation and persistent precipitant drug therapy, both designs were associated with considerable bias.