Reply to Rosenblum et al. and Carriquiry and Ommen: Understanding probative value, triers-of-fact, and usefulness of validity study results

We thank Rosenblum et al. (1) and Carriquiry and Ommen (2) for their interest in our article (3) regarding the validity of cartridge-case comparisons. They make three assertions: that we misrepresent the technique's probative value, that setting trier-of-fact pretest odds = 1 is improper, and that our results may not speak to the validity of the forensic technique. We appreciate the opportunity to clarify these misunderstandings, which we address in turn.

Rosenblum et al.'s belief that we misrepresent the technique’s probative value stems from their viewing trier-of-fact posttest beliefs as the measure of probative value (or, in their words, “strength of forensic evidence”). In fact, the likelihood ratio is the true measure of probative value. Guyll et al. correctly highlight the likelihood ratio as the empirical probative value measure that determines how much change is justified in trier-of-fact beliefs from any particular pretest value to a consequent posttest value. Likewise, the actual transcript (4) of the relevant testimony cited by Rosenblum et al. reveals an accurate portrayal of the technique’s probative value as quantified by the likelihood ratio by showing how much change it would justify in the trier-of-fact beliefs from an initial pretest value to a consequent posttest value. Nowhere do Guyll et al. or the testimony improperly represent posttest beliefs as the measure of probative value.

Both Rosenblum et al. and Carriquiry and Ommen disagree with our assertion that unbiased trier-of-fact pretest odds = 1. We argue that there is no other choice. We base our argument on the foremost requirement that constrains trier-of-fact decision-making; specifically that it must be influenced by nothing except evidence properly adduced at court (5–7). Because decision-making must be influenced by nothing but evidence, evidence of equivalent probative value must equivalently influence decisions. A thought experiment shows that satisfying this requirement necessitates pretest odds = 1. Imagine two cases. In both cases, a trier-of-fact must decide between the same two hypotheses; a hypothesis that favors the prosecution and its alternative that favors the defense. In case A, evidence is presented that favors the prosecution hypothesis, and the probative value of that evidence is quantified by a likelihood ratio value of LR′. In case B, evidence is presented that favors the defense hypothesis, and the probative value of that evidence is equivalent to that of the evidence in case A (albeit in favor of the alternative hypothesis) and thus quantified by a likelihood ratio value of 1/LR′. Because ideal trier-of-fact decision-making must be influenced by nothing but the evidence presented, the trier-of-fact’s belief in the prosecution hypothesis in case A must equal the trier-of-fact’s belief in the defense hypothesis in case B, requiring that the posttest odds be reciprocals of each other. This equality can be expressed in terms of pretest odds and the probative value of the evidence in each case, specifically

$$pretest odds\times L{R}^{\prime }=\frac{1}{pretest odds \times \frac{1}{L{R}^{\prime }}} .$$

This equality necessitates pretest odds = 1.

Counterarguments by both Rosenblum et al. and Carriquiry and Ommen violate the constraint that decision-making must be influenced by nothing but evidence. They propose that before hearing evidence the trier-of-fact should set their pretest odds by assuming something like a process of random selection from a population (e.g., guns in society, people on an island). This is not permissible because the assumption is not evidence. (Indeed, the assumed facts might never become evidence and might not even be true). Because pretest odds influence all subsequent posttest odds, to permit such a nonevidentiary influence on pretest odds is to permit a nonevidentiary influence on all subsequent decision-making through the consequent posttest odds. In this way, the nonevidentiary material improperly acquires influence that is reserved for evidence alone. It will improperly supplement evidence that weighs in favor of one hypothesis and improperly counteract evidence that weighs in favor of the alternative hypothesis.

To justify pretest odds less than one Carriquiry and Ommen also cite the presumption of innocence but commit a common error in doing so. The presumption of innocence is routinely misunderstood to mean that triers-of-fact should initially view the defendant as innocent. In truth, the term “presumption of innocence” is a misnomer and means no such thing. Legal scholars, evidence manuals, and precedent setting legal decisions are clear that the presumption of innocence means nothing more than that the prosecution bears the burden of proof (e.g., refs. 5–11). These sources are clear that the presumption of innocence is not evidence and that it may not influence decision-making as if it were evidence. That is, it may neither supplement evidence indicating innocence nor counteract evidence indicating guilt.

We agree with Rosenblum et al.‘s and Carriquiry and Ommen‘s arguments regarding pretest odds for other situations they present, such as forensic experts, diagnosticians, or estimating random event probabilities. These situations do not entail the same constraint that binds the trier-of-fact.

Finally, Carriquiry and Ommen correctly note that we neglected to discuss the limitations of using a volunteer sample, an oversight we are happy to address. They seem to imply that this limitation could have been avoided by randomly selecting individuals for invitation to participate. However, because only volunteers will ultimately participate, a self-selected sample results nonetheless. Carriquiry and Ommen suggest that volunteers might have been motivated to bias error rates downward below what occurs in fieldwork. However, study examiners knew their conclusions were anonymous and would affect no actual case, whereas in fieldwork, examiners' conclusions are not anonymous, are often peer-reviewed, and directly impact real cases. Hence, examiner motivations could work to make error rates lower in the field. It is also plausible that examiners are motivated to demonstrate excellence in their professional tasks in all contexts, resulting in no bias. Ultimately, we perceive no compelling reason to doubt the President’s Council of Advisors on Science and Technology who view the results from appropriately designed studies such as ours as indicating the technique's validity in general (12). That said, we invite other researchers to improve upon our method and contribute to the body of empirical work.