An inconvenient truth: More rigorous and ecologically valid research is needed to properly understand cognitive bias in forensic decisions

Cognitive bias is an umbrella term used to explain subjective perceptions of people, objects and/or events that deviate from a normative framework [1]. Biases are normally created through an interaction of heuristics (cognitive short-cuts [2]; with world experiences [3]. Assessing the prevalence, impact and type of cognitive bias on decision making in different parts of the justice system should be prioritised to ensure continued trust in the system [4].

1. Juror decision making and their perceptions of forensic evidence

Due to the weight of responsibility placed on jurors, any factors which impact their decisions are, understandably, of particular interest. For example, our understanding of the impact of racial biases [5], rape myths [6], and gender biases relating to defendants, victims and expert witnesses [7,8] on jurors have been markedly improved by a wealth of rigorous and experimentally valid research. More recently, Curley and colleagues have studied the processes through which jurors form their judgements and make their final verdict [[6], [9]]. Forensic psychology, legal research and now decision science research are making joint headway in understanding the impact of biases on the decisions made at each stage of the legal process.

Jurors place different weight on different types of evidence presented to them. Forensic evidence (evidence obtained using scientific methods such as fingerprints, blood splatter, bite mark or DNA analysis) is typically weighted strongly by juries [10]. Lynch [10] suggests that DNA evidence is presently viewed as the current ‘gold standard’ in relation to the evidence presented in court. Briody [11] determined that the likelihood of a conviction after DNA evidence was presented to a jury was much higher than without DNA evidence. The convicted criminals themselves put a strong emphasis on DNA evidence when considering the dangers of committing a crime. Prainsack and Kitzberger [12] interviewed criminals to determine their views on DNA and other forensic evidence. These criminals believed that it was impossible not to leave a DNA fingerprint at a crime scene, and that DNA evidence is both impenetrable and intimidating due to the scientific rigour applied by an expert when interpreting the evidence. Furthermore, evidence provided by a forensic science laboratory, despite being circumstantial, is seen as strong, important and influential by legal laypersons when making decisions.

A recent review by Eldridge [13] discusses some of the difficulties in presenting forensic evidence by an expert to a layperson jury. For example, the review highlighted that jurors see the term “match” as indicating a strong association between the suspect and the forensic evidence left at a crime scene, whereas forensic examiners perceive the term match as a weaker association than terms like “individualisation” and “identification”. Eldridge [13] also argues that jurors may prefer numerical testimony but may not fully understand said testimony; visual aids that help jurors understand probability may help here. Further, communication errors may not be confined to juries and the adversarial system, as Canela et al. [14] found that legal experts, in the inquisitorial system, self-reported that they had a relatively poor understanding of medical evidence, and that there was no consensus on how legal experts should evaluate medical and forensic reports. Furthermore, legal decision making research has highlighted that forensic laypeople struggle with scientific and mathematical terms, making it difficult for expert witnesses to communicate their evidence effectively to said individuals. To mirror the point of Eldridge [13], future research (particularly from cognitive psychologists and decision scientists) is needed to further understand how jurors process forensic evidence, and how their comprehension can be aided.

2. Previous research on cognitive bias in forensic science

The United Kingdom Home Office, Association of Police and Crime Commissioners, National Police Chief’s Council and the House of Lords all agree that the fair administration of justice relies upon public trust in the criminal justice system [[15], [16]]. This public trust depends on proper interpretation of forensic evidence by examiners, and how said information is communicated to jurors [17]. Therefore, to ensure that the public has trust in the legal system, research from decision scientists is needed to ensure how forensic scientists and other legal actors (such as police and jurors) are making decisions.

Dror and Colleagues took pioneering steps in the field of forensic science to identify and highlight the stability of the effects of bias across different forensic domains [17,18]. Their research highlights that task-irrelevant contextual information can influence (or bias) the judgments of forensic scientists when they are presented with ambiguous information [17,18]. Bias produced from task-irrelevant contextual information in forensic scientists (aptly named contextual bias) may therefore impede the legal systems ability to deliver justice in society.

Dror [19] also discussed ‘bias cascades’ and a ‘bias snowball’, where bias from one piece of an investigation can accumulate, potentially increasing the strength of the bias as each element of an investigation interacts. Real-life cases (e.g. Madrid bombing) have indicated that bias in forensic decision making may not be confined to the researcher’s lab and may lead to miscarriages of justice. Once again highlighting the importance that can be placed on investigations into contextual bias and the potential effects that it can have on the decision making of forensic scientists.

3. An inconvenient truth: more rigorous and ecologically valid research is needed

Bias, from a layperson’s perspective, is a word synonymous with error. The concept of anyone involved in the criminal justice system making biased judgements – especially forensic scientists– is an anathema. In the papers discussed above, it is assumed that cognitive biases in forensic decision making reduce the accuracy of the decisions. Indeed, very few studies have gone as far as to show, through basic principles utilised in social science research (such as: 1) experimentally rigorous and ecologically valid methodologies; and, 2) generalisable samples), that bias is associated with a decrease in accuracy. There is every possibility then that cognitive biases may actually improve accuracy in certain decision scenarios (see Refs. [20,[21], [22]]. For instance, Stevenage and Bennett [23] found that task-irrelevant contextual information (DNA match, DNA unclear, DNA does not match) influenced (or biased) the decisions of students when analysing fingerprint evidence in both positive and negative ways. Further, the knowledge of a DNA match increased accuracy when the fingerprints being analysed were a match, but decreased accuracy on non-matching trials. These findings highlight that the current discussion surrounding task-irrelevant contextual information and bias in forensic science is currently being conducted in a reductionist manner, bias is a complex phenomenon and should be treated, and studied, in such a way. In summary, the relationship that bias has with accuracy in forensic science is difficult to assess because of the methodological limitations in most studies and a lack of external validation of cognitive biases (e.g., biases not being studied in ecologically valid settings).

Cooper and Meterko [24] provide a recent systematic review which outlines the susceptibility of forensic scientists to ‘confirmation bias’. In their review, they highlight that many studies that investigated cognitive bias used small samples and/or non-practitioners (e.g., students of various abilities). Of the 21 studies they reviewed that used practitioners, 13 had a sample size of less than 25 practitioners. The participant number mean of 36.85 (across all the studies in the systematic review) was mainly due to a single experiment with a large sample of 192 practitioners [25,26]. None of the 29 reviewed studies provided information about randomisation of trials.

In addition, many of the reviewed studies did not address differences between experimental and control groups, and this may have confounded results. For example, Kukucka and Kassin [27] do not report the age, gender and/or any other demographic information about their participants; except that they were undergraduate students in psychology. This failure to report demographic materials makes it difficult for researchers to generalise the results of their studies. Cooper and Meterko [24] also identified that previous researchers consistently failed to blind participants to the purposes of their research.

Further, several results sections did not include basic statistical information such as effect size, measures of variability and/or an inferential test statistic; some, worryingly, even conducted inappropriate statistical tests [24]. Finally, most of the previous research on bias in forensic scientists has failed to generalise to the real-world working practices of forensic scientists, and has therefore been unsuccessful in demonstrating how bias influences the operational accuracy of forensic scientists [28]. While an increase in interest in the impact of bias on forensic decision making is welcome, more experimental rigour and ecological validity is required to draw generalisable conclusions.

4. A collaboration between decision science and forensic science is needed

Decision scientists (experts in decision making from disciplines such as cognitive psychology, mathematics, philosophy and/or economics; [29]) have developed methodological tools to conduct research that can provide valid answers to questions regarding cognitive bias in forensic decision making. Decision scientists have the expertise to conduct carefully controlled experimental studies that make use of well-established theoretical foundations in their field. For example, decision scientists have identified various types of biases and fallacies, which allow decision scientists to investigate the influence that biases may, or may not, have on forensic decisions in a more nuanced manner. They can also rely on results from different research programmes (e.g. bounded rationality; heuristics and biases, normative theories), and are well versed in sampling, experimental design and statistical analyses.

In summary, all of the factors mentioned above, would allow decision scientists to test more rigorously the accuracy of forensic decisions when task-irrelevant contextual information is taken into account. They have already effectively investigated decision making in other disciplines such as medicine, law and finance [2,30,31]. However, where decision scientists need guidance from forensic scientists is on the selection of stimulus material, contextual (both task-relevant and -irrelevant) information and experimental manipulations [32]. Together, decision scientists and forensic scientists can provide ecologically valid scientific data that can reveal the true impact of task-irrelevant contextual information on forensic decision making.

CRediT authorship contribution statement

Lee J. Curley: Conceptualization, Writing - original draft, Writing - review & editing. James Munro: Conceptualization, Writing - original draft, Writing - review & editing. Martin Lages: Writing - review & editing, Writing - original draft, Conceptualization.

Declaration of competing interest

None of the authors report any conflict of interest.