Yland et al. Respond to “Heuristics and Wish Bias”

We thank Dr. Hamra for a thoughtful commentary (1) on our article (2). Dr. Hamra eloquently explains the origins of the heuristic that nondifferential misclassification results in bias toward the null and theorizes about why it remains so popular among researchers. He points out that we have known for 20 or 30 years that there are exceptions to this heuristic. In this, we completely agree. The fallibility of the heuristic has long been the elephant in the room.

In considering why this heuristic remains popular, Dr. Hamra describes the phenomenon of confirmation bias: investigators’ “tendency to cast study findings into a preferred image” (1, p. 1497). We agree that confirmation bias likely plays a role alongside incentives for “positive” findings to get published. By framing study findings into a preferred image, investigators inadvertently cast a shadow over other potentially relevant biases. Use of some version of the phrase “misclassification was probably nondifferential, and therefore any bias would be toward the null” has become nearly ubiquitous in the Discussion sections of epidemiologic articles. By focusing on this issue, investigators appear to be transparent in disclosing a bias while drawing the reader’s attention away from biases that are not “guaranteed” to bias one’s results toward the null. In many cases, this illusion (whether intentional or not) may facilitate publication and thereby contribute to overall distortion of the evidence base. Taken together, this framing implicitly endorses a value system that weighs overestimates or false-positive associations as uniformly more dangerous than underestimates or false-negative associations, without due consideration of who bears what costs from the resulting distortions and whether this unequal burden maps to public health values and priorities.

To mitigate this, we argued for the use of quantitative bias analysis in our article (2). Unfortunately, many investigators who include quantitative bias analyses focus solely on nondifferential misclassification. This virtually guarantees a point estimate that is shifted away from the null to the exclusion of other biases. A better approach would be to consider the combined role of multiple biases acting at once.

Finally, we contend that the goal of etiological epidemiology is to identify a valid and precise estimate of the effect of an exposure on an outcome. Dr. Hamra argues that “claiming that nondifferential misclassification of a binary exposure or outcome biases results towards the null is generally fair” (1, p. 1497). While this may be true in general, we must not forget that what matters for inference is the magnitude of the bias. There is probably some nondifferential misclassification in everything we measure. In some instances, this will have virtually no impact on inference. In others, bias due to nondifferential misclassification could modify our inferences, changing the course of clinical care or policy-making.