Counterpoint: The Treatment Decision Design

Abstract

The comparative new-user design is a principled approach to learning about the relative risks and benefits of starting different treatments in patients who have no history of use of the treatments being studied. Vandenbroucke and Pearce (Am J Epidemiol. 2015;182(10):826–833) discuss some problems inherent in incident exposure designs and argue that epidemiology may be harmed by a rigid requirement that follow-up can only begin at first exposure. In the present counterpoint article, a range of problems in pharmacoepidemiology that do not necessarily require that observation begin at first exposure are discussed. For example, among patients who are past or current users of a medication, we might want to know whether treatment should be augmented, switched, restarted, or discontinued. To answer these questions, a generalization of the new-user design, the treatment decision design, which identifies cohorts anchored at times when treatment decisions are being made, such as the evaluation of laboratory parameters, is discussed. The design aims to provide estimates that are directly relevant to physicians and patients, helping them to better understand the risks and benefits of the different treatment choices that they are considering.

Editor's note: Counterpoints to this article appear on pages 826 and 834, and a response appears on page 846.

The benefits of studying new medication users have been recognized in pharmacoepidemiology for many years (1–3). With the so-called “new-user design,” patients are followed from the date the medication is started, allowing for the full time course of risk to be characterized. The new-user design makes it possible to study early effects of medications, such as hypersensitivity reactions, that would not be seen among a sample of prevalent users. It also can be used to study long-term, cumulative exposure outcomes, providing that time-varying confounding and selection bias are appropriately addressed (4). By using a comparative new-user design, which compares new users of different medications that have common indications, one can also minimize many common sources of confounding bias (5), such as confounding by indication (6) and confounding by frailty and functional status (7, 8). Remaining differences between the groups can be reduced by restriction and adjustment for baseline covariates (9, 10). The new-user design also enforces appropriate temporal ordering of confounders, treatment, and outcome, protecting the researcher against accidental adjustment for variables affected by treatment (11). In some early examples of studies using a comparative new-user design, Ray et al. (12) studied the risk of hip fracture among new users of statins versus other lipid-lowering agents, and Wang et al. (13) studied mortality risk in new users of atypical versus conventional antipsychotics in the elderly. In recent years, studies like these have proliferated, and the perceived robustness of the comparative new-user design has led to suggestions that it could be productively used for semiautomated medical product safety surveillance (9).

The increasing formulation of research questions in epidemiology in terms of potential outcomes highlights a further advantage of the comparative new-user design: its similarity to a parallel group, randomized controlled trial (14). Typical intention-to-treat analyses of both randomized controlled trials and comparative new-user designs are explicitly attempting to estimate a similar parameter: the effect of initiating different treatments on subsequent outcome risk, a parameter with clear clinical and policy relevance. In a randomized controlled trial, an intention-to-treat analysis assesses the effect of assignment to use a therapy, which is generally highly correlated with the actual use of a therapy. In the new-user cohort design, a similar analysis assesses the effect of initiating therapy.

In this issue of the American Journal of Epidemiology, Vandenbroucke and Pearce (15) discuss some problems inherent in incident exposure designs, including the need to discard many patients who may be long-term users of the medication. They suggest that the field of epidemiology may be harmed by the increasingly stated belief that studies must follow only those patients who are observed from the start of exposure. In this paper, we describe a range of problems in pharmacoepidemiology that do not necessarily require observation to begin at the first exposure and also are not answered by comparing outcome risks among patients initiating different treatments. We then discuss a generalization of the new-user design that can answer these questions.

THE TREATMENT DECISION DESIGN

As it is commonly applied, the new-user design tells us about the relative risks and benefits of starting (and potentially remaining on) different treatments in patients who have no history of use of any of the treatments under study. However, physicians and patients are confronted with a much broader set of decisions about treatments that must be made. For example, among patients who are past or current users of a medication, they would like to know whether treatment should be augmented, switched, restarted, or discontinued, or, for medications that are dosed, they would like to know whether doses of a current treatment should be increased, decreased, or held. For medications that are used intermittently or as needed over the course of a lifetime, such as nonsteroidal antiinflammatory medications or antibiotics, physicians, patients, and regulators may be interested in the safety and effectiveness of a course of therapy in older adults who may have previously used the medications under study. Assuming that it is not possible to enroll children and follow them over their life course, this effect could not be estimated if we limited ourselves to studying only new users of medications.

To learn about the risks and benefits of these treatment decisions, we could do randomized experiments in which we would identify eligible patients with varying treatment histories and then subject the clinical decision of interest to randomization, assigning each patient to one of K well-defined interventions. If patients are adherent to assigned treatment, we can estimate features of the distributions of the counterfactuals Y(k), k = 1,  …, K and contrasts between the counterfactual distributions, such as the difference in risk between 2 different assigned treatments, E[Y(k)] − E[Y(k′)], k ≠ k′.

However, when such experiments cannot be done, because of cost, ethics, or other reasons, we need to determine how best to estimate a similar parameter or causal effect using observational data. To do this, we first need to identify, as closely as possible, the instances in which the treatment decisions are being made among eligible patients. Because the range of clinical options may depend on a patient's treatment history (e.g., one would not reinstitute a treatment that has failed in the past), some amount of treatment history will be needed to identify eligible patients. If the interventions corresponding to the different treatment decisions are well defined and can be clearly identified in the data, we can obtain estimates of the causal contrasts described previously, provided the other assumptions necessary for causal inference hold, including positivity, conditional exchangeability, and no interference (16, 17).

In addition to providing meaningful and easily interpretable effect estimates, a treatment decision design shares another advantage with the classic new-user design. By anchoring our study on the time when a treatment decision is made, beginning follow-up immediately following the treatment decision and defining potential confounders by using data prior to the index date, one achieves correct temporal ordering of confounders, treatment, and outcome and therefore prevents inadvertent adjustment for variables affected by the treatment decision. Appropriately applied, the design also avoids the inclusion of immortal person-time, a common source of bias in nonexperimental studies (18, 19).

However, a treatment decision design possesses some limitations not shared by the more restrictive new-user design. First, the treatment choice may be confounded by a patient's past use of the treatment that may be unobserved by the researcher. For example, if a patient experienced an allergic reaction to penicillin as a child, he or she would be channeled away from penicillin later in life. In a comparative safety study of antibiotics in older adults, this channeling would potentially confound study results if patients with allergies were at increased risk of the study outcome. Second, if the full treatment history is not observed, it will not be possible to study long-term cumulative effects without further assumptions. An additional complication of a treatment decision design is the need to identify in data times when treatment decisions are made among eligible patients. In many cases, this will require clinical and other subject matter knowledge about how and when clinical decisions are made.

Below, some actual and hypothetical examples of treatment decision designs that do not necessarily involve new medication users are considered. These include a discussion on how one identifies patients in whom treatment decisions are being made and some advantages and limitations of these particular approaches.

Example 1: Assessing the effects of a dose or treatment change following a laboratory test

Physicians often order laboratory tests for the explicit purpose of making decisions about treatment. For patients on medications in which doses are often adjusted, physicians may be interested in the safest effective dose for a given patient at a particular point in time. Dose-ranging trials can be conducted to help answer these questions. Such trials involve assigning a randomly selected dose to an eligible patient and then observing what happens. We can potentially mimic these experiments in observational data by identifying patients receiving laboratory testing for the purpose of making decisions about dosing.

For example, intravenous iron is provided to hemodialysis patients to replenish iron lost through the dialysis procedure and other causes of bleeding. Iron treatment decisions are largely driven by laboratory measures of iron storage and availability. By identifying the times when these laboratory values are received by physicians and examining the treatment decisions that occur subsequently, we can mimic an iron dose-ranging study conducted to assess short- to intermediate-term effects of iron treatment decisions on outcomes of interest (Figure 1) (20, 21). This would require that the researcher has available all pretreatment variables that influence the treatment decision and are independent predictors of the outcome. With data on only what treatments were administered, the observational study would differ structurally from the trial. To determine the treatment decisions, the researcher would need to observe the treatment provided over several dialysis sessions. The study would therefore have to require patient survival and continued use of dialysis through some exposure assessment period, potentially leading to selection bias (22).

Figure 1.

Design schematic of a nonexperimental iron dose-ranging study.

Laboratory values may also be used to identify times when physicians make decisions about the initiation and intensification of treatment for chronic disease. For example, if one would like to examine the effects of intensification of oral antidiabetic therapy, it would be reasonable to construct a treatment decision cohort anchored on the date of a hemoglobin A1c (HbA1c) test, a laboratory measure used to assess adequacy of glucose control. With access to electronic health records data, it may be possible to directly identify the treatment decision that was made following the evaluation of the laboratory value. However, if only health-care utilization data are available, pharmacy refill data would need to be used as a proxy for the actual treatment decisions (Figure 2). Nonadherence to the new prescription could lead to misclassification of the treatment decision.

Figure 2.

Design schematic of a nonexperimental study of oral antidiabetic treatment intensification. HbA1c, hemoglobin A1c.

Example 2: Using restriction to identify patients for whom treatment decisions are being made—the effect of perioperative use of statins

There has been speculation that perioperative use of statins may reduce the risk of acute kidney injury among patients undergoing cardiac surgery. One study attempted to assess the effects of statins on postsurgical acute kidney injury by examining outcomes among patients with a history of use of statins before surgery versus those without a recent history of use (23). Prevalent user studies such as this suffer from several limitations. First, confounding control is not straightforward. Any presurgery variables that we might want to include in a regression model, for example, low-density lipoprotein levels, may have been affected by past statin use, and therefore their adjustment may block any effect of statin exposure that they mediate. Second, selection effects may induce spurious associations between statin use and unmeasured risk factors for the outcome. For example, if statins affect the need for surgery, conditioning on surgery creates associations between statin exposure and other variables related to the need for surgery or decision to undergo surgery. Third, prevalent users will be enriched with long-term adherers who may be in better overall health (24, 25). Finally, the state of being a current statin user does not correspond to any particular intervention; physicians cannot assign patients to this status. This is essentially a problem of a poorly defined intervention (26). If we were to observe a decreased risk among patients on a statin, we could start a patient on a statin, hoping that the patient would have a risk similar to our prevalent users. That intervention, however, does not align with the actual exposure that was studied.

If we consider the problem from the perspective of a treatment decision design, many of these problems are diminished. It is clear that physicians face 2 basic decisions related to the use of statins before cardiac surgery. 1) Should I start a patient on a statin who is not currently taking a statin? 2) Should my patient who is a currently taking a statin stop treatment before surgery or remain on treatment? Even with only prescription refill data, we could reasonably address the first question by using sample restriction (Figure 3). We would first identify patients undergoing planned surgery and then restrict the study to patients without a recent history of use of statins up until a few weeks (or days) before surgery. Among these patients, we could compare outcomes among those who fill a prescription for a statin before surgery with those who do not (27). The exposure would be much closer to the actual intervention that could be prescribed by the physician. However, there are secondary issues related to the intervention that are not well defined, such as when to start the statin (1 week vs. 1 day before surgery) and the potency of the prescribed statin. These represent different dimensions of treatment and, in an observational study, we could either redefine the exposure to capture these different dimensions or assume “treatment version irrelevance,” that is, that the different forms of the intervention are equivalent (28). Also, because we would be using prescription claims as proxies for the treatment decision, there again may be some misclassification of the actual treatment decision. With only prescription refill data, we would not be able to identify patients in whom treatment was discontinued, so it would not be possible to assess the effect of stopping statins among patients who are current users.

Figure 3.

Design schematic of a nonexperimental study of perioperative statin use.

Example 3: An observational analogue of a randomized withdrawal trial—diuretic use among patients starting dialysis

Diuretics are effective and widely used medications for hypertension and volume control. Because urine output often decreases rapidly following dialysis initiation, patients starting dialysis often have diuretic therapy withdrawn. However, patients, particularly those that continue to suffer from volume overload, might benefit from continued use of diuretic therapy while on dialysis (29, 30). One could assess the benefits of continuing diuretics versus stopping after the start of dialysis by using a randomized withdrawal trial. In such a trial, patients who are users of diuretics at the time of dialysis initiation would be randomly assigned to either continue or stop treatment. This trial could potentially be mimicked in nonexperimental data by comparing outcomes among patients who stop treatment with those who continue after the start of dialysis. If the existence of pharmacy claims were used to determine treatment continuation, one would not know precisely when the decision to stop treatment was made and would therefore need to assess exposure in some period following the start of dialysis (Figure 4). Like the nonexperimental iron dose-ranging study described previously, this observational study would need to require that patients survive some period of time after the start of dialysis so that the treatment decision could be ascertained. Some unmeasured confounding would be expected here, as sicker patients may be more likely to discontinue treatment.

Figure 4.

Design schematic of a nonexperimental analogue of a randomized withdrawal study of diuretics among patients starting hemodialysis.

DISCUSSION

I agree with Vandenbroucke and Pearce (15) that it is not always necessary to study patients from the start of exposure. However, in pharmacoepidemiology, it is valuable to begin follow-up on patients at times when treatment decisions are made. A treatment decision design generalizes the new-user design and allows the researcher to address a broader range of questions under additional assumptions. The design aims to provide estimates of the effects of different clinical decisions among patients who may not necessarily be new users of medications. We have considered some potential applications of this design that illustrate the use of the approach but also reveal some implementation challenges.

For researchers using such designs, the first challenge is to identify in the data instances in which the treatment decisions are being made. For decisions that follow laboratory testing or other clinical measures, this will often be straightforward. When one is comparing 2 or more active treatments that have been recently started (or restarted), it may be reasonable to assume that the treatment decision occurred prior to the start of therapy. For example, if one wanted to compare the safety of different classes of antibiotics in patients with atrial fibrillation, one could anchor the start of follow-up on the date the antibiotic prescriptions were initially filled (assuming the treatment decision occurred just before the fill date). However, for treatment decisions that involve “no treatment” as an option and are based on patient-reported symptoms, such as insomnia, depression, or pain, researchers will have to infer times at which the decision was likely to have been made. It might be reasonable to anchor the study on the date of a physician office visit in which certain diagnosis codes were or were not reported, effectively establishing an index date when a treatment indication was initially reported.

For some studies, it may also be challenging to determine the actual treatment decision that was made. In such cases, it will need to be inferred from proxy information such as pharmacy claims, which are evaluated during an exposure assessment period. To avoid bias from immortal person-time, follow-up might need to start after the exposure assessment period ends. Longer exposure assessment periods will increase the risk of selection bias; shorter exposure assessment periods may lead to greater misclassification of the treatment. Sensitivity analysis could be used to explore this trade-off. It is important to note that, when the treatment decision is inferred from the actual treatment taken, one is not attempting to study the effect of a treatment decision, for example, prescribing a patient a particular medication, but rather the effect of the treatment itself, at least initially. Given the high rate of primary nonadherence to many prescription medications (31), the difference between these 2 treatment effects may often be substantial.

If a treatment decision design is not explicitly focusing on new users or patients for whom the relevant medical history is available, researchers will have to be aware of potential confounding by previous use of the treatments under study. It also must be recognized that such designs may not be optimal for examining long-term or cumulative exposure effects.

It is also worth noting that treatment decision designs have been used in many previous studies. For example, a common study design involves examining the effects of medication use after hospitalization (32). Although these studies may include patients who used these medications prior to the hospitalization, it may be reasonable to assume that physicians and patients make explicit decisions about which medications to continue or initiate after discharge.

The treatment decision design attempts to create cohorts anchored on times when treatment decisions are being made. The design generalizes the new-user design and expands the scope of problems that can be considered by researchers, and therefore partly addresses one of the limitations cited by Vandenbroucke and Pearce (15). Yet assessing the long-term effects of treatment is no easier with such an approach and remains a challenging problem. Despite some limitations and practical challenges, the treatment decision design may help to improve the validity estimates by avoiding many common sources of bias. The design also aims to provide estimates that are directly relevant to physicians and patients, helping them to better understand the risks and benefits of the different treatment choices that they are considering.