Where to look for the most frequent biases?

Abstract

Study quality depends on a number of factors, one of them being internal validity. Such validity can be affected by random and systematic error, the latter also known as bias. Both make it more difficult to assess a correct frequency or the true relationship between exposure and outcome. Where random error can be addressed by increasing the sample size, a systematic error in the design, the conduct or the reporting of a study is more problematic. In this article, we will focus on bias, discuss different types of selection bias (sampling bias, confounding by indication, incidence‐prevalence bias, attrition bias, collider stratification bias and publication bias) and information bias (recall bias, interviewer bias, observer bias and lead‐time bias), indicate the type of studies where they most frequently occur and provide suggestions for their prevention.

SUMMARY AT A GLANCE

This article discusses the different types of selection bias and information bias that are found in studies, indicate the type of studies where they most frequently occur and proposes suggestions for their prevention.

1.

SUMMARY AT A GLANCE.

This article discusses the different types of selection bias and information bias that are found in studies, indicate the type of studies where they most frequently occur and proposes suggestions for their prevention.

2.

Where external validity represents the degree to which results of a study may apply to populations or groups that did not participate in the study,1 internal validity is the degree to which a study is free from error. Internal validity is imperative if one wishes to obtain valid results from a clinical study. Such internal validity may be affected by two types of errors; random and systematic errors. Random error occurs by chance and may affect the precision of both the exposure and outcome measurements. A high degree of random error may lead to a poor precision, which in turn may complicate the assessment of the frequency of an exposure or an outcome or the evaluation of the true relationship between the exposure and outcome. The solution, however, is relatively simple; by either increasing the sample size or reducing the measurement error, one may decrease random error. On the other hand, systematic error, also known as bias, in the design, the conduct or the reporting of a study is more problematic. A study may be biased depending on how the study subjects were selected, or how study variables were measured.2 As a consequence, the true frequency of an exposure or an outcome or the true relationship between exposure and outcome may be either under‐ or overestimated, leading to flawed study results. Unfortunately, bias cannot be reduced by increasing the sample size, simply because the error is systematic. Usually, bias can also not be adjusted for. This is in contrast to confounding. Confounding—sometimes referred to as confounding bias—distorts the association between an exposure and an outcome due to the association of the exposure with one or more other factors that influence the occurrence of the outcome. If these factors are known and measured, the real effect of the exposure on the outcome can be obtained by adjustment for these confounding factors.

There are many types of bias,3 but in this article, we will focus on two main categories; selection bias and information bias. We will discuss different types of bias within these categories, indicate in which study designs they are more likely to occur and how to prevent them (Table 1).

Table 1.

Types of bias related to the study designs where they are most frequently occurring

Type of bias	Study designs most at risk
Selection bias
Sampling bias	All study designs (cross‐sectional studies and cohort studies) not using representative samples of the source population, especially those with low response rates (frequently occurring in surveys)
Confounding by indication	Non‐randomized intervention studies (cohort studies, case‐control studies, cross‐sectional studies)
Incidence‐prevalence bias	Study designs using prevalent patients (cross‐sectional studies, cohort studies not using incident patients, case‐control studies)
Attrition bias	Longitudinal studies (randomized controlled trials [RCTs], prospective cohort studies)
Collider stratification bias	Studies (especially cohort studies) investigating groups of patients selected on the basis of a collider (restriction) or adjusting for a collider
Publication bias	All study designs
Information bias
Recall bias	Study designs that use self‐reporting (case‐control studies and retrospective cohort studies)
Interviewer bias	All study designs making use of interviews, especially those where the interviewer has information on the outcome status of the respondent (unblinded case–control studies or retrospective cohort studies)
Observer bias	Study designs using measurements that are prone to subjectivity, especially those where the observer has information on the exposure status of the patient (unblinded RCTs or observational studies)
Lead‐time bias	Cohort studies comparing survival times between screened subjects and those diagnosed on the basis of symptoms; studies comparing survival between patients in different stages of disease

3. SELECTION BIAS

Selection bias stems from errors in the selection procedure of study participants, and from factors affecting study participation. Selection bias implies that the relationship between exposure and outcome may differ in those who participate in the study and those who do not. As this relationship is typically unknown in non‐participants, selection bias can usually not be observed, but only hypothesized.

3.1. Sampling bias

In clinical research, the population of interest (ie, the source population) is often very large making it necessary for researchers to study a sample. This sample needs to be representative of this source population in order for study results to correctly reflect any frequencies or exposure‐outcome relationships in that source population. Whenever possible, sampling will take place based on available information on potential respondents. For example, in studies assessing CKD prevalence in the general population, census data or electoral rolls form suitable sampling frames to identify eligible subjects, and can be used to obtain random samples reflecting the age and sex distribution of the general population. Unfortunately, many studies fail to report sampling frames and sampling designs.4 This leaves readers in the dark regarding the potential of sampling bias in the reported CKD prevalence. Even if a sample was drawn correctly, a second source of sampling bias may be introduced due to non‐response during subject inclusion. In a study reporting on the methodology of European studies investigating CKD prevalence, the response rate was reported in 65% of the studies, ranging from 10% to 87%.4 As non‐response may be caused by selective participation of subjects (either the healthier or the sicker ones) whom may differ from the general population, this forms a potentially important source of bias in the assessment of the true CKD prevalence (Table 1). Similar processes take place in surveys when asking the opinion of groups of people. Sampling and response bias may distort the assessed frequencies in different directions, leading to an over‐ or underestimation of the prevalence. Nevertheless, sampling bias can be limited by using state‐of‐the art techniques to obtain a random sample. In case of non‐response bias, there is no agreed‐upon minimum acceptable response rate. Attempts to increase low response rates include sending automated reminders to those invited to participate, but even then, response rates may remain low and bias may be substantial. This form of bias typically arises in cross‐sectional studies and cohort studies.

3.2. Confounding by indication

By some epidemiologists, this type of selection bias is considered as confounding caused by unmeasured factors. It occurs in observational studies comparing the effect of interventions on an outcome, for instance, two different treatments in relation to mortality (Table 1). This bias is caused by the presence of an indication for at least one of the interventions and leads to a distortion that modifies the association between the exposure (the intervention) and the outcome.5 When studying the effectiveness of treatments, one wishes to measure the effect of the treatment alone, and exclude any effects caused by other factors such as confounders. The best way to achieve this is to randomly allocate the treatments to the study subjects, as it happens in randomized controlled trials (RCTs).6, 7, 8 Randomization increases the likelihood of both known and unknown confounders being balanced across treatment groups, resulting in groups where any remaining differences are due to chance. If all confounders would be known and measured in an observational study, their effects could also be adjusted for during statistical analysis, provided that the study size is large enough. In observational studies, however, treatment is not allocated at random, but by the physician. This physician will always take into account the patient's prognosis and prescribe the treatment that he or she believes is best for—and therefore indicated in—that particular patient. This indication will likely also be determined by confounders that are not measured in the study. Patients receiving, for example, haemodialysis will therefore be different from those receiving peritoneal dialysis. This selection bias in observational studies that is due to unmeasured confounders and induced by physicians is called confounding by indication, and cannot be adjusted for in the statistical analysis.9

3.3. Incidence‐prevalence bias

Incidence‐prevalence bias results from the inclusion of prevalent cases in a study. It is most common in cross‐sectional studies, but can also be found in cohort studies (Table 1). The problem with this type of bias is that the exposure‐outcome relationship tends to be investigated in prevalent patients in whom there is an overrepresentation of those who have lived the longest. Incidence‐prevalence bias is therefore also known as survivor bias. Figure 1 shows a simplified hypothetical example of an incident and a prevalent cohort of patients diagnosed with severe emphysema. As is common knowledge, continued smoking after diagnosis will increase patient mortality risk. This is clearly shown in the figure, as patients who continue smoking (dark bars) tend to live shorter than those who quit smoking (light bars). In the incident cohort, observation started at diagnosis of severe emphysema. The mortality in those who kept smoking was 10 deaths in 14 patient‐years, and in those who quit smoking 10 deaths in 26.5 patient‐years. The mortality rate ratio (MRR) in the incident cohort was therefore (10/14)/(10/26.5) = 1.89. In the prevalent cohort, observation in the same patients was started 1 year after diagnosis. As a result, only 14 patients were left from the original incident cohort, as in the meantime many patients who continued smoking had died. The mortality in the prevalent cohort was six deaths in 5 patient‐years in continued smokers and eight deaths in 17 patient‐years in those who quitted, respectively, leading to a biased MRR of (6/5)/(8/17) = 2.55, substantially higher than that derived from the incident cohort. Usually, the only way to prevent this type of bias is through limiting the inclusion to incident cases.

Figure 1.

Incidence‐prevalence bias in assessing the mortality in patients diagnosed with severe emphysema in cohorts of incident and prevalent patients. The dark bars represent those who continued smoking after diagnosis and the light bars represent those who quit smoking after diagnosis. In the incident cohort, the observation period starts at diagnosis, whereas in the prevalent cohort it starts 1 year after diagnosis. D denotes death

3.4. Attrition bias

Attrition means the loss of participants during a study due to withdrawals, dropouts or protocol deviations. It occurs in longitudinal studies such as prospective cohort studies or RCTs (Table 1). The problem arises when participants are lost, and it is unknown if they continued or discontinued the intervention. More often than not, data on outcomes after their drop‐out are missing. The risk of bias is created if people who leave the study are systematically different from those who continue, which is often the case. Imagine a hypothetical study on the effect of diet on the risk of hypertension. Those who have more problems with compliance to the dietary regimen may be more likely to leave the study, leaving the more health‐conscious patients in the study, which may lead to a potential overestimation of the effect of diet on hypertension. Attrition almost always happens to some extent10 and, if substantial, it represents a threat to correctly measuring the effectiveness of an intervention in the usual “real world” circumstances. Some authors have stated that a loss to follow‐up of 5% or less is unlikely to introduce bias, whereas a loss of 20% should raise concern.11 Others stress that it should also be taken into consideration to what extent those who leave the study are different from those who continue.12 In any case, prevention of attrition wherever possible is of utmost importance.

3.5. Collider stratification bias

A collider is a variable that is directly affected by two or more other variables.1 Selecting on a collider (in the study design) or controlling for a collider (in data analysis) may induce collider stratification bias in the association between an exposure and an outcome. This may be illustrated by the following example. When we investigate the relationship between obesity and mortality in the general population, we will find that obesity is harmful—it increases the risk of death. In patients with end‐stage kidney disease (ESKD), however, a number of studies have shown an obesity paradox meaning that obesity is seemingly protective—it reduces the risk of death. Many explanations have been provided for this phenomenon that occurs not only in ESKD patients, but also in individuals with other chronic conditions such as cardiovascular disease.13 In the directed acyclic graph (DAG) shown in Figure 2, the solid arrows represent the causal relations between obesity, ESKD, mortality, measured confounders and unmeasured risk factors. This DAG also shows that ESKD is a collider as arrow heads from both obesity and unmeasured risk factors point in the direction of ESKD. Restriction of the statistical analysis to a group of patients, for example, ESKD patients, can be regarded as controlling for a collider in the study design and thereby introduce a non‐causal association, in this case between unmeasured risk factors for ESKD and obesity. As a consequence, the true association between obesity and mortality will be distorted due to the introduction of confounding by the unmeasured risk factors (should they have been measured it would have been possible to adjust for them). Collider stratification bias may therefore be added to the list of potential explanations of the obesity paradox in dialysis patients. Collider stratification bias typically occurs in cohort studies. This bias may also add to incidence‐prevalence bias as by selecting prevalent patients one restricts on the collider of having survived. This type of bias can be prevented by not controlling for a collider, for example, by not restricting the study to patients possessing the collider characteristic, and by not adjusting for the collider in the data analysis.

Figure 2.

Directed acyclic graph showing how the obesity paradox in patients with end‐stage kidney disease (ESKD) can possibly be explained by collider stratification bias. Measured confounders affecting mortality may include covariates like age and sex. Unmeasured risk factors may include risk factors that can be considered as a common cause for ESKD and mortality (eg, genetic or lifestyle factors) and that often go unmeasured. Restriction to ESKD patients induces collider stratification bias (ESKD being the collider) by introducing a non‐causal association between obesity and the unmeasured risk factors. This non‐causal pathway distorts the obesity‐mortality relationship by introducing confounding by the unmeasured risk factors and may be responsible for the seemingly protective effect of obesity in ESKD

3.6. Publication bias

Most types of bias are the consequence of flaws in study design or study conduct. This does not apply to publication bias. This bias has been described as ’any tendency on the parts of investigators or editors to fail to publish results on the basis of the direction or strength of the study findings’.14 As a consequence, published papers may not constitute a representative sample of all studies performed on the subject,15, 16 and this impacts the ability to accurately combine and describe the evidence in a given area of research.17 Publication bias originates during the entire selection process; from generating study results, choosing which of them to include in a manuscript, whether or not to submit the manuscript, to eventual acceptance for publication. On the one hand, it is due to self‐limitation on the side of the researcher, who is unwilling to submit inconclusive or negative results for publication, as they expect the manuscript will not be accepted.15, 18 On the other hand, it is due to the acceptance policy of journals where reviewers and editors tend to select manuscripts with positive study results. Therefore, not all studies relevant to a certain subject and using sound research methodology, are accepted for publication. As a consequence, meta‐analyses may give a distorted view of exposure‐outcome relationships, overestimating treatment effects and risk factor associations, which in turn may lead to inappropriate decisions in patient management or health policy.18, 19 Fortunately, unlike other types of bias, publication bias need not only be inferred, but can sometimes be detected by using funnel plots. In these plots, the effect of a given treatment from each trial is plotted against some measure of its size, such as the precision, the SE, or the overall sample size.15 Treatment effects in smaller studies are expected to have a larger variability than those in larger studies. If there is no indication of publication bias, these plots are indeed shaped as a funnel. Figure 3 shows an example of a somewhat asymmetric funnel plot. The emptier lower right side of the funnel may indicate that small studies with negative results are missing, and therefore some degree of publication bias may exist. Publication bias may affect all study designs (Table 1). Publication bias can be prevented, for example by obligating investigators to report all clinical trial results—including the negative ones—directly into a clinical trial registry, and by journals increasing the chances of publication for studies with null results.

Figure 3.

Example of a funnel plot. The precision of each study is plotted against its effect estimate. Larger dots represent larger studies. The vertical line is drawn through the overall pooled estimate of effect to detect symmetry or asymmetry. In this plot the right lower side seems emptier which indicates that small studies are missing pointing to some degree of publication bias

4. INFORMATION BIAS

Information bias is caused by errors in the measurement, collection, or interpretation of the exposure, of the disease, or of both. In studies using categories for exposure and outcome, the possibility exists that individuals may be placed in the incorrect category. As a consequence, exposed subjects are misclassified as non‐exposed or vice‐versa, and diseased subjects are misclassified as non‐diseased or vice versa. This misclassification can be non‐differential or differential. Exposure misclassification is non‐differential if it is unrelated to the presence or occurrence of disease. If it is related, it is differential. Similarly, misclassification of disease is non‐differential if it is unrelated to exposure; otherwise, it is differential. Table 2 shows the effects of non‐differential and differential misclassification in a hypothetical case‐control study investigating the association between the intake of high‐caloric beverages and cerebrovascular accidents. In the case where cases and controls were classified correctly, the true odds‐ratio (OR) is: (250/450)/(100/900) = 5.0 indicating that subjects with a high intake of high‐caloric beverages are five times as likely to experience a cerebrovascular accident. If both the cases and the controls were 20% misclassified with respect to the intake of high‐and low‐caloric beverages, the OR decreases to (290/410)/(260/740) = 2.0, resulting in a 60% decrease in effect. This is typical for non‐differential misclassification of dichotomous exposures: it will always bias an effect towards the null. In other words, non‐differential misclassification dilutes the effect. In contrast, if only in the controls 50% of the subjects with high intake were to underestimate their intake—a case of differential misclassification—the OR changes to (250/450)/(50/950) = 10.6, leading to a sharp increase. In contrast to non‐differential misclassification, differential misclassification may either increase or decrease the true effect.

Table 2.

Non‐differential and differential misclassification in a hypothetical case‐control study investigating the effect of the intake of high‐caloric beverages on the occurrence of cerebrovascular accidents (CVAs)

Correct classification	CVA cases	Controls	Odds ratio
High intake	250	100	5.0
Low intake	450	900	Reference

Non‐differential misclassification (in cases and controls: 20% was considered to have high intake instead of low intake and 20% was considered to have low intake instead of high intake)a	CVA cases	Controls	Odds ratio
High intake	290	260	2.0
Low intake	410	740	Reference

Differential misclassification (in controls: 50% was considered to have low intake instead of high intake)b	CVA cases	Controls	Odds ratio
High intake	250	50	10.6
Low intake	450	950	Reference

^a CVA cases: 20% of 450 with low intake (=90) were misclassified as high intake, whereas 20% of 250 with high intake (=50) were misclassified as low intake. This results in 250 + 90‐50 = 290 with high intake. In addition, 20% of 250 with high intake (=50) were misclassified as low intake, whereas 20% of 450 with low intake (=90) were misclassified as high intake. This results in 450 + 50−90 = 410 with low intake. CVA controls: 20% of 900 with low intake (=180) were misclassified as high intake, whereas 20% of 100 with high intake (=20) were misclassified as low intake. This results in 900−180 + 20 = 740 with low intake. In addition, 20% of 100 with high intake (=20) were misclassified as low intake, whereas 20% of 900 with low intake (=180) were misclassified as high intake. This results in 100−20 + 180 = 260 with high intake.

^b CVA controls: 50% of 100 with high intake (=50) were misclassified as low intake. This results in 100−50 = 50 with high intake and 900 + 50 = 950 with low intake.

4.1. Recall bias

This type of information bias is common in studies using self‐reporting, such as case‐control studies, or retrospective cohort studies where subjects are asked to provide information on exposure only after the disease has or has not occurred (Table 1). Recall bias is caused by differences in accuracy or completeness of recall to memory of past events or experiences.20 Recall bias may lead to misclassification of exposure. A classical example is that of maternal recall bias. In the study of potential causes of birth defects, mothers of babies with birth defects tend to search their memories more thoroughly than mothers with healthy babies. In nephrology, this bias may occur in patients with glomerulonephritis who are asked about possible toxic exposures and may remember better than those without glomerulonephritis. Another example concerns that of a case‐control study investigating the risk factors associated with acute pyelonephritis. While considering if genetic factors may be involved, investigators may ask participants if their mother also suffered from frequent urinary tract infections. Cases may be more likely than controls to have discussed this with their mother. Therefore, they may more often respond that this was the case. Recall bias can be prevented by making use of hospital records to verify individuals' recall or by the use of controls with another disease who may have searched their memory in a similarly thorough way.

4.2. Interviewer bias

Interviewer bias has been defined as the systematic error due to interviewer's (sub)conscious gathering of selective data, or their influencing of subject response.1 The interviewer's expectations or opinions may interfere with their objectivity, causing them to ask leading questions, whereas respondents may reply in a manner that reflects what he or she believes the interviewer wants to hear (social desirability bias). Interviewer bias may increase recall bias when physicians or investigators question cases more intensively on exposures that are known to be associated with the disease (exposure suspicion bias).21 Interviewer bias can occur in all studies that make use of interviews (Table 1). It can be reduced by standardization of the interview and—for example in case‐control studies and retrospective cohort studies—by blinding the interviewer to the outcome status of the respondent.

4.3. Observer bias

The dictionary of epidemiology describes observer bias as ‘a systematic difference between a true value and the value actually observed due to observer variation’.1 When judgements are subjective there is a much greater potential for variability between observers than in case of objective data.22 For example, blood pressure readings using analogue sphygmomanometers were notorious for observer variability. Doctors tended to round down blood pressure readings in patients they knew were on antihypertensive medication and to round them up in patients who were not. If such differences in judgement are systematic—for example, because they result from particular preconceptions on the side of the observer—they may induce bias. Observer bias may not only play a role in the reporting, evaluation and processing of data but also in the statistical analysis. It may form a problem in both RCTs and observational studies (Table 1). If it affects outcome assessment, treatment effects may be substantially under‐ or overestimated. For this reason, clinicians, patients and analysts should preferably be blinded to the treatment allocation.

4.4. Lead‐time bias

Finally, a type of information bias that is not due to misclassification is lead‐time bias. Lead‐time bias is defined as an apparent increase in survival resulting from disease detection at an early stage. In reality, there is no actual effect on survival, just a longer period with the diagnosis. Figure 4 shows that, if screened, patients can be diagnosed with disease at an earlier stage than the point in time when symptoms occur. The difference in time between screening and the occurrence of symptoms is called lead time. Lead‐time bias causes problems when the survival of patients with a particular disease is being compared between regions with and without screening programmes. In nephrology, it has become widely known as a problem of cohort studies comparing policies of early and late start of dialysis. When patients start dialysis at a higher eGFR without severe uraemic signs and symptoms, and their survival is compared to those starting dialysis at a lower eGFR with severe uraemic signs and symptoms, then this comparison may suffer from lead‐time bias favouring those with an early start.23, 24, 25 Some studies have attempted to take lead‐time bias into account by estimating the date at which patients had a predefined level of eGFR before dialysis start and then counting survival time in both early and late starters from that date onwards.25

Figure 4.

Lead‐time bias. Often diseases are diagnosed at the onset of symptoms; sometimes they are diagnosed earlier at screening before causing any symptoms. Should this patient have been screened and diagnosed with the disease in 2014 when still asymptomatic, whereas otherwise he would be diagnosed in 2017 at the onset of symptoms, his survival would have appeared to be 5 years instead of 2 years. The difference of 3 years is called ‘lead‐time’

In summary, this paper discusses a number of biases in relation to study designs where they are most frequently occurring. Once it has occurred the problem of bias can usually not be solved. As a researcher, it is therefore important to do everything possible to prevent bias in the study design, the study conduct and in the reporting of the study results. However, no study is completely free from bias. Therefore, any source of bias that is considered potentially important in the interpretation of the study findings should be mentioned in the discussion. Some study designs are particularly vulnerable to specific types of bias. Also reviewers and readers should realize that for them too it is crucial to look for biases, in general but also specifically the ones frequently occurring in the study design at hand. This is necessary to assess whether the study findings may have been affected by the way the study was designed or carried out, and to subsequently judge to which degree the findings really reflect the truth.

5. CONFLICT OF INTEREST

We have no conflict of interest to report.

Jager KJ, Tripepi G, Chesnaye NC, Dekker FW, Zoccali C, Stel VS. Where to look for the most frequent biases? Nephrology. 2020;25:435–441. 10.1111/nep.13706