Incorrect statistical reasoning in Guyll et al. leads to biased claims about strength of forensic evidence

Guyll et al. (1) make an error in statistical reasoning that could lead judges and jurors in criminal trials to grossly misinterpret forensic evidence. Their error leads to highly inflated claims about the probability that a cartridge case from a crime scene was fired from a reference firearm, typically a firearm found in the possession of a defendant. These claims are being presented in at least one ongoing homicide case (2). We wrote a paper to address this error, summarized here. Some text below is excerpted from (3).

Guyll et al. aim to estimate the (posterior) odds that a crime scene cartridge case was fired from a reference firearm (“same source”) or not (“different source”) given a firearms examiner’s decision.

They represent the posterior odds as the product of

• (i)an assumed prior odds of same versus different source, and
• (ii)a likelihood ratio (LR) of the examiner’s conclusion given same versus different source.

[1, equations 4 and 6, p. 3]. They refer to the prior and posterior odds as “pretest” and “posttest” odds, respectively.

Next, Guyll et al. state that “a trier of fact can directly take a decision’s LR as the posttest odds of the evidence being same-source if pretest odds equal 1, a situation corresponding to being initially unbiased and withholding all judgment as to a comparison’s ground-truth status prior to hearing the forensic decision” [1, pp. 3–4] We italicized the erroneous claim, i.e., that prior odds of 1 corresponds to being initially unbiased. In fact, this prior is almost invariably biased against the defendant.

By definition, prior odds of 1 represent the a priori belief that “ground truth” is equally likely to be same or different source. Same source means that a single firearm—the reference firearm—fired the cartridge case, while different source means that the cartridge case was fired by any other firearm. If more than 2 possible firearms could have fired the crime scene cartridge case, then allocating 50% prior probability to the reference firearm and the remaining 50% to be divided across all other compatible (i.e., same caliber, etc.) firearms biases the analysis toward the reference firearm. This type of error is called equiprobability bias (4–6).

As an analogy, consider the probability that you have the same birthday as George Washington. This probability is about 1/365 = 0.3%, but the (erroneous) logic of Guyll et al. implies that an unbiased observer should believe the two scenarios “same birthday” and “different birthday” are equally likely.

With more reasonable prior odds, the posterior probability decreases. For illustration, suppose there exist n candidate firearms, each with prior probability 1/n of having fired the cartridge case. As n increases, the posterior probability of same source decreases from 99.4% (n = 2) to 64.2% (n = 100) to 15.1% (n = 1,000) to 1.7% (n = 10,000). See Fig. 1.

Fig. 1.

Reproduced from ref. 3: The above plot shows how posterior probabilities (vertical axis) are highly dependent on assumptions about the prior (horizontal axis). The horizontal axis represents the number n of guns in a simplified scenario with n possible guns and where the prior probability of same source ground truth is assumed to be 1/n. We are not suggesting to base the prior odds solely on the number of possible firearms, rather using it to illustrate sensitivity of the posterior to the choice of prior.

In summary, setting the prior odds to 1 is almost invariably biased against the defendant and can lead to inflated conclusions about the strength of forensic evidence. We agree with refs. 7–10 who argue that forensic experts should not make assertions about prior or posterior odds.