Threats to Validity of Nonrandomized Studies of Postdiagnosis Exposures on Cancer Recurrence and Survival

Abstract

Studies of the effects of exposures after cancer diagnosis on cancer recurrence and survival can provide important information to the growing group of cancer survivors. Observational studies that address this issue generally fall into one of two categories: 1) those using health plan automated data that contain “continuous” information on exposures, such as studies that use pharmacy records; and 2) survey or interview studies that collect information directly from patients once or periodically postdiagnosis. Reverse causation, confounding, selection bias, and information bias are common in observational studies of cancer outcomes in relation to exposures after cancer diagnosis. We describe these biases, focusing on sources of bias specific to these types of studies, and we discuss approaches for reducing them. Attention to known challenges in epidemiologic research is critical for the validity of studies of postdiagnosis exposures and cancer outcomes.

Observational studies of postdiagnosis exposures and risk of cancer recurrence or mortality pose design, data collection, and analytic challenges. These studies generally fall into two categories: 1) studies using health plan automated data with “continuous” exposure information (eg, studies in which medication dispensings are available for each calendar day in a pharmacy database) (1,2); and 2) survey or interview studies that collect information directly from patients once or periodically (eg, every 12 months) postdiagnosis (3–7). Both types of study are susceptible to problems of reverse causation, confounding, selection bias, and information bias. Here, we illustrate how these biases can affect studies of postcancer exposure and outcomes and suggest ways of reducing their magnitude. Although we primarily discuss cohort studies, most of the principles also apply to case–control studies. We do not review all forms of each bias comprehensively, but rather highlight those that are particularly common in or challenging for studies of postdiagnosis exposures and risk of cancer outcomes, such as recurrence or cancer-related mortality (Table 1). Similarly, we do not comprehensively outline all approaches to mitigating these forms of bias, but rather focus on relatively straight-forward and practical approaches to improving the validity of these studies.

Table 1.

Common biases in studies of postdiagnosis exposures and outcomes among persons with a history of cancer

Type of bias	Cause of bias	Approaches to reducing bias
Reverse causation (sometimes called protopathic bias)	Undiagnosed occurrence of the outcome leads to changes in the exposure or measurement of the exposure	Restrict to persons recurrence-free at exposure (Design)
		Lag exposure (Analysis)
Confounding
Confounding by disease progression	Undetected progression or recurrence alters subsequent exposure, causing bias in studies of cancer-related mortality	Restrict to persons recurrence-free at exposure (Design)
		Use marginal structural models (Analysis)
		Lag exposure (Analysis)
Confounding by precancer exposures	Exposure before cancer diagnosis affects risk of recurrence/mortality and is also associated with exposure after cancer	Restrict to persons unexposed before cancer diagnosis (Design)
		Adjust for precancer exposures (Analysis)
Selection bias
Self-selection (sometimes called volunteer bias, nonresponse bias)	People who choose to participate in a study may have different risk factors and a different risk of the outcome than those who do not.	Select a study population that is a random sample of target population (Design)
		Weight individuals based on probability of responding (Analysis)
Failure to account for left truncation (sometimes called length bias, survival bias)	Only those persons who remain event-free until a later time point are included in the study, but they contribute person-time starting earlier.	Observe all participants starting at cancer diagnosis (Design)
		Participants enter into the analysis when they begin being followed for the outcome and not before (Analysis)
Differential loss to follow-up	Being censored is an effect of both the exposure and the outcome	Ascertain the presence or absence of the outcome in a way that is unlikely to be affected by whether the outcome has occurred (Design)
		Design surveys, study visits, etc. to maximize response rate (Design)
		Inverse probability of treatment weighting using variables related to loss-to-follow-up (Analysis)
Information bias
Failure to detect outcomes similarly across exposure groups (sometimes called detection bias)	Event of interest is more likely to be identified in certain subjects than it is in others	Select endpoint for which ascertainment is complete and date determination is relatively precise (Design)
		Conduct study within a clinical trial or cohort study with routine follow-up (Design)
Infrequent exposure assessment based on patient report	Changes in exposure status are not captured if a patient reports his or her exposure only once or infrequently and the exposure is time-varying	Ascertain exposure using automated pharmacy records (Design)
		Ask participants about their exposures at multiple, frequent intervals (Design)
Records in automated databases or patient charts may not reflect true exposure	Some exposures are unlikely to appear in medical records or automated databases	Do not use medical records or automated databases to assess exposures likely to be missing or differentially recorded (Design)
	Databases and charts may include medications prescribed or dispensed but not necessarily taken by the patient	In studies of exposure/outcome relationships with a relatively long induction period, require more than one diagnostic code, prescription, etc. before classifying a person as exposed (Analysis)
Immortal time bias (sometimes called time-dependent bias, survivor treatment selection bias)	Misclassification of unexposed person-time as exposed based on exposure in the future	Classify person-time as unexposed until the subject meets the exposure defining criteria (Analysis)
		Start time zero for both exposed and unexposed persons after a fixed exposure assessment period (Analysis)

Reverse Causation

Reverse causation occurs when occurrence of the outcome of interest leads to changes in the exposure or measurement of the exposure. A common form of this is protopathic bias in pharmacoepidemiologic studies (8) (ie, patients start or stop a medication because of symptoms of the disease under study). In studies of cancer outcomes, disease progression can affect exposures of interest, such as medication use, behaviors, and healthcare utilization. For example, if an undetected cancer recurrence causes symptoms prompting the use of a certain medication, that drug may appear to be associated with an increased risk of recurrence when, in fact, medication initiation was a consequence of recurrence. Perhaps an undiagnosed recurrence caused pain, leading patients to use opioids. Use of these drugs would appear to be associated with a subsequent increase in the risk of recurrence even if no true association existed. Conversely, certain lifestyle exposures, such as exercise, may become less common in persons who are ill with a recurrence even before the diagnosis of that recurrence. If so, exercise would appear to be associated with a reduced risk of recurrence (9). Approaches to minimizing this bias include 1) restricting the study population to persons believed to be recurrence-free at the time of exposure initiation and 2) not classifying a person as having been “exposed” until a certain amount of time has passed after the onset of exposure (ie, “lagging” exposure) (10) for long enough that any undiagnosed recurrence would be likely to have become apparent.

In their study of statin use and colorectal cancer recurrence and survival, Ng et al. (11) used restriction. The authors excluded people who experienced a recurrence before completing the questionnaire assessing statin use. Ahern et al. used the second approach: they lagged their exposure (statin use) by 1 year, a plausible interval during which most or all of then-occult recurrences of breast cancer (the condition they were investigating) would have become clinically evident (12). This is conceptually identical to excluding exposures from a risk set if they occur only within a certain period of time before the time of the event that defines the risk set. One challenge of this approach is that it requires accurate assumptions about the latent period for recurrent cancer (10), as well as accurate assumptions about the interval needed for the exposure to influence the likelihood of recurrence.

Confounding

Confounding can occur when the exposure and the outcome share a common cause (13). Studies of cancer outcomes in relation to postdiagnosis exposures are particularly susceptible to confounding by disease progression and confounding by prediagnosis exposures.

Confounding by Disease Progression or Recurrence

In studies of cancer mortality, disease progression or recurrence may act as a confounder. Recurrence not only increases risk of death but might also lead to changes in exposure status (as described above). Both of the approaches discussed in the section on “Reverse Causation” can mitigate confounding due to recurrence in studies of cancer mortality. In addition, controlling for time-dependent confounding by recurrence using marginal structural models (14,15) may also help. Holmes et al. employed this approach to control for confounding by changes in breast cancer status when studying aspirin use and survival after breast cancer (5). They used marginal structural Cox models to control for cancer progression as a time-varying confounder. Because exposures before and after recurrence are combined, this approach can be useful in analyses that do not seek to distinguish between factors that predispose to recurrence and those that adversely influence mortality risk after recurrence. To implement this approach and restriction (described in “Reverse Causation”), investigators must have complete ascertainment of cancer recurrences and second primary cancers. Although this is potentially possible with chart review and reliable data from surveys or interviews, it is more challenging with automated data alone. However, new algorithms may make it possible to accurately identify recurrences using automated data (16,17). Even so, to the extent that undiagnosed recurrences produce symptoms that lead to exposure, residual confounding may remain in an analysis of cancer-related mortality.

Confounding by Precancer Exposures

In some studies of the effects of exposure after incident cancer diagnosis, there is the potential for confounding by exposure(s) before diagnosis. For example, use of unopposed estrogen hormone therapy (ET) before endometrial cancer diagnosis likely confounds the relation between ET after diagnosis and the risk of disease-specific mortality: women who receive ET typically develop a less aggressive form of endometrial cancer (and may therefore be less likely to experience a recurrence) than those who do not (18,19), and women who receive ET before their cancer diagnosis are more likely to receive it after diagnosis. When confounding by precancer exposures is possible, investigators should measure and consider controlling for exposure before diagnosis. For example, O’Meara et al. evaluated hormone replacement therapy before breast cancer as a potential confounder of the relation between hormone replacement therapy use after breast cancer and risk of recurrence (2). Restriction to control confounding is another possible approach. For example, in their analysis of postdiagnosis statin use and breast cancer outcomes, Kwan et al. excluded women who had used statins before cancer diagnosis to reduce the potential for confounding by precancer exposure to these drugs (20). However, such a broad exclusion could potentially influence generalizability.

Precancer exposures can also pose problems when computing exposure duration. If the scientific question of interest centers on medication use postcancer exposure, then it is important to ensure that precancer medication use is not included in calculation of the duration of the exposure. For example, O’Meara et al. computed duration of use based only on medications dispensed after diagnosis and before recurrence (2).

Selection Bias

Structurally, selection bias is a result of sampling subjects conditional on common effects of the exposure and outcome (or their causes) (13). Selection biases are “distortions that result from procedures used to select subjects and from factors that influence study participation” (21). Selection bias can arise from the initial selection of study participants or from differential loss to follow-up over the course of study.

Self-Selection

The decision about whether or not to participate in a study is usually not random. When participation is related to the exposure and the outcome, bias may result. People who participate in a study may be at either higher or lower risk of the outcome of interest than those who do not participate, depending on the study. For example, because of treatment side effects, people who receive chemotherapy may be less likely to participate in a study of recurrence, and they have a lower risk of recurrence than persons with a comparable stage at diagnosis who did not receive chemotherapy. In the same study, participation may be related to the exposure of interest or one of its precursors. For example, people who use a particular medication may be less likely than nonusers to participate in a study. If a medication user (who is, in general, less likely to participate) volunteers to be in the study, he is also less likely to have had chemotherapy (ie, another factor that predisposes a person not to volunteer). Even if no true association exists between medication use and recurrence risk, an association will be induced; among study volunteers, medication users will be less likely to have received chemotherapy compared with nonusers than in the general population. Medication users would falsely appear to have an increased risk of recurrence. Although this scenario may not be common, for completeness it is important to understand how studying a nonrandom subset of the population of interest can cause bias.

Observational studies of postcancer exposures and outcomes can minimize the bias due to self-selection (ie, nonresponse or volunteer bias) by studying a random sample of the target population rather than clinic-based populations (22) or persons who self-select to enroll in a study. For example, selection bias is less likely to occur in population-based cohort studies that do not require participant consent for inclusion, such as studies using existing data collected by registries or health plans. When direct patient contact or consent is required, selection bias may occur. Methods for achieving high responses are described elsewhere (23); however, research focused specifically on response rates in cancer survivors is limited (24). When possible, comparing characteristics of nonresponders with those of responders (based on existing external data, such as from tumor registries or automated data) may help investigators gauge the potential for self-selection bias. Weighting individuals by the inverse of their probability of responding to a survey is another method to mitigate selection bias due to nonresponse (25,26) and is discussed in the context of selection bias due to loss to follow-up below.

Failure to Account for Left Truncation

Left truncation occurs when, because of study design, persons with an event before a certain point in time are not observed (27). This point in time could, by design, be the same for all subjects, such as in a study that recruits cancer survivors 5 years after diagnosis, or it could be different for each subject. Bias can occur in a cohort study if person-time is counted before investigators actually begin observing outcomes. For example, a study examining risk factors for recurrence might send out a questionnaire to recruit patients with a history of cancer and enroll a cohort to follow prospectively for recurrence. Respondents could have vastly different lengths of time since they were diagnosed. If the analyst includes all survey responders in the analysis from the time of cancer diagnosis—even people who do not come under observation for the outcome until several years after diagnosis—those observed to experience early events would be compared with a biased sample in which those persons who survived longer would be preferentially included. In other words, bias occurs when people must survive long enough to come under observation at a later point in time to contribute any person-time (28). To avoid bias, all participants should enter into the analysis when they begin being followed for the outcome and not before (28,29). For example, in a study of survival in relation to nonsteroidal anti-inflammatory drug use after breast cancer, Blair et al. considered breast cancer patients to be at risk starting at the time of exposure assessment, not at cancer diagnosis, because women had to survive to be surveyed (3).

When left truncation is present—even when it is handled properly to avoid bias—there are still important implications for generalizability that should be noted. In studying survivors recruited 5 years after diagnosis, events that occur within 5 years of diagnosis cannot be observed. Such studies may not be able to provide information on the relation between exposures and outcomes of cancer shortly after the primary diagnosis (6). For example, if medication use influences the likelihood of early recurrences only, studies in which people enter the analysis several years after diagnosis will not detect the association.

Differential Loss to Follow-Up

In longitudinal studies, participants may withdraw from the study, choose not to attend subsequent study visits, stop filling out follow-up surveys, move out of the area or the health plan of interest, experience an event that makes them no longer at risk for the outcome of interest, or die. Loss to follow-up can result in selection bias in cohort studies when being censored is an effect of both the exposure and the outcome (13). Under these circumstances, bias results from informative censoring (ie, censoring times that are associated with unobserved event times). Participants are selectively lost from the study in a way that is related to both their risk of the outcome and their exposure. Suppose breast cancer survivors who experience a recurrence are more likely than those without a recurrence to drop out of a longitudinal survey study before their recurrence is diagnosed or recorded. Suppose that people with diabetes are also more likely to drop out than those without diabetes because of complications of their disease. A woman with a recurrence who does not drop out is less likely to have diabetes (another potential cause of drop-out) than a person without a recurrence. This would result in a false association between diabetes and a reduced risk of recurrence.

The approaches described above under self-selection bias may also help reduce bias due to loss to follow-up. One way to reduce this bias in cohort studies is to ascertain the presence or absence of the outcome in a way that is less likely affected by patient drop-out (eg, following patients through means that do not involve patient response, such as chart review). If patient response is required, it is helpful to make surveys easy to complete or provide modest incentives so that patients are more likely to participate. Spacing surveys or study visits close together may also help because censoring may be more likely to occur if there are long gaps between follow-up times when the outcome is ascertained. When data on variables that cause loss to follow-up are available, each individual’s probability of being lost can be estimated, and inverse probability weighting methods can be used to diminish selection bias (13). Hernán et al. describe the assumptions and limitations of this approach, as well as alternatives, in more detail (13).

The inverse probability weighting approach is not necessarily recommended when the cause of informative censoring is a competing risk (ie, an event that prevents the outcome of interest from occurring (13). Death due to other causes is a competing risk in studies of cancer-specific mortality (30). Making an inference about the cause-specific hazard (ie, the risk of recurrence or cancer- specific mortality in the presence of competing risks) is appropriate (31). However, it is not possible to estimate what the risk of recurrence or death from cancer would be in a population not subject to competing risks without making strong, unverifiable assumptions (32). The analysis of data in the presence of competing risks is discussed in detail elsewhere (31).

Information Bias

Errors in measuring or classifying variables of interest can lead to bias. Information bias can arise through numerous mechanisms (such as recall bias) described elsewhere (23) that may be a concern in nonexperimental studies and can have a range of effects on study results. Below we discuss several forms of outcome and exposure misclassification that are particularly problematic in studies of cancer outcomes in relation to postdiagnosis exposures.

Errors in Measuring Outcomes

Failure to Detect Outcomes Comparably Across Exposure Groups.

Detection bias occurs when an event is more likely to be identified in certain subjects than it is in others. In most of the world, one of the advantages of studying mortality as an endpoint is complete ascertainment and relatively precise determination of date of occurrence. Other outcomes, such as recurrence, may be missed altogether. In some situations, the event of interest might be observed earlier (or later) in exposed subjects; even if no true association between the exposure and recurrence exists, the exposure might appear to increase the risk of recurrence simply because it is associated with increased surveillance. For instance, a person taking a medication that requires routine medical visits may be more likely to see his or her medical provider and have an asymptomatic (or only mildly symptomatic) recurrence diagnosed, compared with a person who is not using medications and therefore not coming in as often for medical visits. In a cohort study, this would influence the observed rate in a person-time analysis because in the exposed subject a shorter amount of time would have accrued before the event was detected than would have for an unexposed person. In a case–control study, unexposed people with recurrence may be less likely to be identified as case subjects during a certain period of follow-up and thus more likely to be falsely classified as control subjects. This misclassification would lead to a falsely elevated exposure prevalence among case subjects.

As described above, studies of recurrence are more likely to be affected by detection bias than are mortality studies. Assessing outcomes similarly in exposed and unexposed individuals is important. Because detection of recurrence requires a medical visit, it is difficult to ensure in observational studies that all participants are being assessed with the same frequency and to the same extent. One way to reduce detection bias is to conduct an observational study within a clinical trial or cohort study that has defined surveillance protocols. For example, Ng et al. conducted their cohort study of statin use and colorectal cancer recurrence (11) within the CALGB 89803 trial, which had standardized follow-up (33).

Errors in Measuring and Classifying Exposures

Infrequent Exposure Assessment Based on Patient Report.

In studies that assess exposure only once, persons who become exposed later will be misclassified because they will continue to be considered unexposed. In a prospective study that is able to adjust for time, this kind of misclassification is likely to be nondifferential. Exposure misclassification that is nondifferential with respect to the outcome will usually attenuate the relative risk estimate. Differential misclassification—where errors in measuring exposure are related to the outcome—can cause bias in either direction.

Using automated pharmacy records to collect and update exposure information is one solution. For example, O’Meara et al. identified women with a breast cancer diagnosis and used their pharmacy records to classify them as hormone replacement therapy users once they filled prescriptions for hormone replacement therapy (2). Limitations of this approach are described in the next section. Surveying participants about their exposures at multiple, frequent intervals can also reduce misclassification due to changes in exposure. For example, a study of aspirin use and survival after breast cancer diagnosis surveyed participants every 2 years and updated their exposure at each survey (5). However, updating exposure information based on self-report can lead to differential misclassification if people with the outcome of interest are less likely or unable to provide new information. Participants who die before their next scheduled survey cannot report updated exposure information.

Exposure misclassification for subjects alive and participating in a 2-year follow-up survey could also occur if, for example, subjects were considered exposed as of the date of the follow-up survey when, in fact, they became exposed earlier in the 2-year window. Or, persons who were exposed for a period of time between the two surveys could be misclassified as unexposed if they were not using the medication on the dates of the follow-up surveys. Surveys that ask about dates and duration of use between surveys may avoid this problem if there is little recall bias. However, exposure misclassification could still occur because people who die between surveys will not be able to update their exposure information; bias is possible because only people surviving long enough to complete the survey could provide information about new exposures since the last survey.

Records in Automated Databases or Patient Charts May Not Reflect True Exposure.

Failure to properly measure exposure can also occur in studies that collect data from chart review or automated pharmacy records. Certain exposures, such as over-the counter medications and vitamins, are rarely captured in automated pharmacy records and may be recorded differentially in the chart (eg, over-the-counter medications may be recorded only among chronic users) and may be related to how often the patient receives medical care. In general, if it is suspected that exposure ascertainment from a particular source is likely to be misclassified, that source should not be used.

Even exposures that seem like they should be completely ascertained by automated databases may not be. For example, health plans or integrated delivery systems may not identify prescriptions dispensed at noncontract pharmacies or in situations where the transaction does not result in an insurance claim. Conversely, a person who appears to be exposed based on claims data may not truly be exposed. For example, a patient may be dispensed a medication but not take all or even any of it. When it is suspected that the medication’s influence is long term in nature, one such way to minimize this misclassification is to require more than one dispensing of the medication within a certain length of time. For example, requiring two dispensings within a 6-month period offers some assurance that a patient is taking the medication (1,2). However, this approach has drawbacks: some people dispensed only one medication were truly exposed for a relatively short length of time but would be classified as unexposed, so this method cannot be applied for acute exposures.

Immortal Time Bias.

Immortal time bias can occur if unexposed person-time is improperly classified. Suissa describes immortal time as “a span of time in the observation or follow-up period of a cohort during which the outcome under study could not have occurred. It usually occurs with the passing of time before a subject initiates a given exposure. While a subject is not truly immortal during this time span, the subject had to remain alive until the start of exposure to be classified as exposed”(34). In cohort studies that use pharmacy records, immortal time bias can occur if persons dispensed medications are classified as users before the medication is dispensed: for example, a person with a first dispensing 1 year after diagnosis is classified as a user from the time of diagnosis forward. Suissa proposes two approaches to dealing with immortal time in cohorts that follow subjects after an event such as cancer diagnosis and that rely on dispensed medications to ascertain exposure. One approach is to classify person-time as unexposed until the subject meets the exposure defining criteria (ie, treating exposure as time varying). For example, in a study of breast cancer recurrence in relation to hormone therapy use, O’Meara et al. classified women as nonusers until the second dispensing for the medication of interest, at which time they began contributing person-time as a user (2). The other approach is to start time zero for both exposed and unexposed persons after a fixed exposure assessment period. For example, a study of cancer recurrence could wait until the 1-year survey to assess if exposure had occurred and begin follow-up on everyone at that time. Using this delayed-entry approach is similar to a solution used to account for left-truncated data; however, because it results from misclassification of exposure during part of follow-up, immortal time bias is distinct from the bias resulting from a failure to account for left truncation (28).

Summary

Reverse causation, confounding, selection bias, and information bias can lead to spurious findings in studies of cancer recurrence and/or mortality in relation to postdiagnosis exposures. Most forms of bias that are present in other observational studies can also affect studies of postcancer exposures and outcomes, but it is important to recognize the specific forms and mechanisms that are particular threats and to be sure to address these in the study design and analysis. Despite the challenges, observational research on cancer outcomes is critical for understanding the effectiveness and safety of exposures that may affect the health of the rapidly growing population of cancer survivors.

Funding

This work was supported by the National Cancer Institute of the National Institutes of Health (grant R01CA120562 to DMB). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

The study sponsor had no role in the writing of or decision to submit the review for publication.

We would like to thanks Dr Susan Shortreed, Dr Michael Jackson, and Dr Rebecca Hubbard for comments on an earlier version of this manuscript.