Abstract
Historically, forensic science results have been admitted in court, with minimal scrutiny regarding their scientific validity. However, following the National Academy of Sciences (NAS, 2009) report, the forensic community has undergone a significant transformation. This shift has demonstrated that forensic scientists and laboratories want to ensure the scientific rigor and quality of their results, but that they are often uncertain where to begin when addressing concerns about error and bias. In response to these challenges, the Department of Forensic Sciences in Costa Rica designed and began a pilot program within the Questioned Documents Section of the laboratory. This program incorporates various existing research-based tools, including Linear Sequential Unmasking-Expanded, Blind Verifications, case managers, and other important mitigation strategies to enhance the reliability of and reduce subjectivity in forensic evaluations. This article discusses the journey from initial planning through to implementation and the impact of the strategies that were adopted. The article describes how the Department systematically addressed key barriers to implementation and maintenance after implementation, providing a model to other laboratories for prioritizing resource allocation. This successful pilot program demonstrates that there are feasible and effective changes that can mitigate bias, and this article presents evidence that existing recommendations in the literature can be used within laboratory systems to reduce error and bias in practice.
Keywords: Bias, Human cognition, Linear sequential unmasking, Case manager, Blind verifications
Although pattern-matching disciplines in forensic science (e.g., fingerprint examination, handwriting analysis) have been practiced and admitted as evidence in court for over a century, they have faced significant scrutiny in recent times, in particular since the report by the National Academy of Sciences (NAS) in 2009 [1]. There were two main issues highlighted in the NAS report. First, pattern-matching disciplines have a “dearth of peer-reviewed published studies” (p. 8) and, thus, there are questions about the scientific underpinnings and foundational validity of many disciplines. Second, compounding the issue of uncertain error rates, these disciplines are susceptible to cognitive bias effects due to their reliance on people to make judgments about evidence without sufficient scientific safeguards to protect against bias and error.
Since 2009 NAS report, there has been a substantial increase in new research efforts seeking to build out the scientific foundation of various forensic disciplines, but that topic is beyond the scope of this article. We focus here on the potential for bias in forensic disciplines and practical solutions to the problems that arise when bias effects go unchecked in forensic practice. It has been a challenge to convince the forensic community that cognitive bias is a legitimate problem that is worthy of their time and resources, with many experts believing that they themselves are not vulnerable to bias in their casework [2]. These beliefs have contributed to the lack of progress in addressing the issue of bias in forensic practice and have been discussed in other important works (referred to as fallacies in Ref. [3], such as the fallacy of “Expert Immunity”, the “Blind Spot” fallacy, and the “Illusion of Control” fallacy). So, even though the forensic community has acknowledged other problems for decades, such as those resulting from physical contamination of evidence, less attention has been paid to the equally crucial issue of cognitive contamination in forensic sciences.
Notably, the end-users of forensic results (e.g., judges, jurors, police, lawyers) typically lack the scientific knowledge needed to critique forensic methods and tend to have pre-existing beliefs about the reliability and accuracy of forensic results based on misleading and inaccurate sources (e.g., entertainment and news media). So, forensic results can sometimes appear very reliable and objective to end-users in the criminal legal system [4]. Yet, any discipline that relies on people to make key judgments and decisions will involve some level of subjectivity. Forensic experts play a pivotal role in criminal investigations and trials and the accuracy of the information presented in their reports and testimony will depend on how the individuals conducting the analysis approached those subjective judgments and what checks, if any, were in place to manage biases and erroneous outcomes [[5], [6], [7]]. Because forensic examiners are responsible for examining and interpreting the data from crime scenes, as well as communicating their findings to end users, all of these stages in the forensic examination process will be vulnerable to any cognitive shortcomings to which the examiner is susceptible.
The importance of considering cognitive factors in forensic science has been further solidified internationally by reports, standards, and policies created by the Scottish Fingerprint Inquiry, the NIST Fingerprint Expert Group, the United Kingdom Forensic Science Regulator, and the United States National Commission on Forensic Science [8]. Additionally, high-profile reports, such as those from the President's Council of Advisors on Science and Technology in 2016 [9], have highlighted specific concerns that arise from a reliance on human examiners to make critical decisions in forensic disciplines when there are insufficient safeguards against the potential for cognitive bias built into the laboratory and criminal legal systems [1,[9], [10], [11]]. Hence, it seems there is a growing understanding of the importance of these issues in the legal and forensic community.
Still, the role of human and cognitive factors is often overlooked or considered low priority in forensic practice [1,4,7]. That said, the human factors issue that has received perhaps the most attention in the forensic community is how task-irrelevant information can inappropriately influence forensic judgments, resulting in cognitive bias effects [12], largely due to the focus on this issue in the 2009 NAS report [1]. As a result of more research demonstrating the effect of task-irrelevant information on forensic results, researchers and practitioners alike are becoming more open to idea that human factors knowledge and research can improve the integrity of forensic science and be used to encourage evidence-based, effective legal and policy reforms [4,12].
1. Cognitive bias in forensic examination
Cognitive biases are decision-making shortcuts that occur automatically whenever people are faced with a situation where they lack sufficient data to make a truly informed decision, lack the time and resources to review the necessary data, or both. These situations are often called “uncertain” or “ambiguous” in psychological literature. The technical definition for cognitive biases is decision patterns that occur when people's “preexisting beliefs, expectations, motives, and the situational context may influence their collection, perception, or interpretation of information, or their resulting judgments, decisions, or confidence” [5,7]. There are different types of cognitive bias. For instance, confirmation bias describes the tendency for people to seek out information that supports their initial position or idea, or their pre-existing beliefs, and ignore contradictory information that is just as valid. Also sometimes called “tunnel vision”, confirmation bias can play a contributing role in convicting innocent people by changing how ambiguous evidence is perceived and interpreted by forensic examiners or police investigators.
While these mental shortcuts are undoubtedly useful in a wide variety of settings, they rely on learned patterns that will not necessarily be informed by the relevant, case-specific data, so these shortcuts will not result in correct outcomes in every case, which is why cognitive bias is one cause of errors in forensic judgments. A well-known example of this is the FBI's misidentification of Brandon Mayfield's fingerprint in connection with the 2004 Madrid train bombing investigation. Several latent print examiners were assigned to verify the identification made by the primary examiner, who was a highly respected supervisor with many years of experience. Because these verifiers knew that the initial conclusion made by their esteemed colleague, their unconscious assumption was that “identification” was likely the correct conclusion [11].
Thus, permitting cognitive bias effects to occur in forensic science contexts allows a certain amount of error into the criminal legal systems that use the forensic results to convict individuals. The Innocence Project has highlighted the role of forensic sciences in wrongful convictions, showing that invalidated, misapplied, or misleading forensic results were a contributing factor in 53 % of the wrongful convictions in their database of exonerations [13]. By addressing the human factors issues in forensic science, laboratories and practicing examiners are also working to reduce the prevalence of injustices that can result from the use of misleading or inaccurate forensic results.
One goal of the current project was to acknowledge the existence and the impact of cognitive bias effects on competent expert forensic scientists. This can be difficult because there are many myths about cognitive bias harbored by those in the forensic community and legal system [[2], [3], [4],12]. Here, we summarize six fallacies [3], which are common misconceptions held by examiners and people responsible for laboratory management and provide some reasons that they are inaccurate.
-
i.
Ethical Issues - “Only bad people are biased”: Cognitive bias is not the same as corruption, misconduct, or dishonesty. It is not an ethical issue; it is merely a normal decision-making process that has limitations which must be considered in situations where accuracy is important.
-
ii.
Bad Apples - “Only people who are not good at their job are biased”: Cognitive biases do not occur due to a lack of skill or incompetence. Cognitive biases are normal, efficient, helpful decision strategies that occur outside a person's awareness.
-
iii.
Expert Immunity - “I am an expert, and I have many years of experience, so I am not susceptible to bias”: Expertise is not a cure for bias. In fact, experience and practice may mean that experts rely on automatic decision processes more often, because they face these types of decisions so frequently.
-
iv.
Technological Protection - “Introducing more algorithms, AI, and technology will solve the subjectivity issue in forensic examination”: The use of artificial intelligence, technology, objective measures and instruments, and automation can reduce bias, but these systems are still built, programmed, operated, and interpreted by humans, so they will not eliminate bias effects from forensic science.
-
v
Blind Spot - “I know that bias is an issue, but I am not one of the examiners who is vulnerable to it”: People often show a “bias blind spot”—people can be quite willing to admit bias is an issue generally, and even somewhat an issue in their particular forensic domain, but they are much less willing to acknowledge that they themselves are susceptible to bias effects in their own casework [2].
-
vi
Illusion of Control - “Now that I know cognitive bias is an issue, I'll just be mindful of that during my analyses so that does not happen”: Willpower and awareness alone cannot overcome bias effects. Bias occurs automatically without people consciously experiencing it and people cannot prevent decision processes that they do not know are happening. Instead, there needs to be systems built around examiners to catch instances where biasing information may be playing an inappropriate role in their examinations.
Although the field has tended to focus on the biasing effect of task-irrelevant contextual information, there are actually a wide variety of sources of bias. In 2020, Dror published [3] a summary of eight sources of bias that each have unique and compounding effects on expert decisions. These are.
-
1.
The Data: The evidence obtained in connection with the crime event can contain biasing elements and evoke emotions that can influence decisions.
-
2.
Reference Materials: The materials gathered to compare to the data and infer something about its source can affect forensic examiner's conclusions. In particular, comparing the data and reference materials side-by-side can lead to confirmation bias effects because the examiner will tend to emphasize what is similar between the data and the reference materials.
-
3.
Contextual Information: Case information that is available to the forensic examiner that has varying levels of relevance to their primary task can impact their collection, perception, and interpretation of the evidence and, thus, the forensic results.
-
4.
Base Rates: Personal experience with cases with difference features and what types of outcomes commonly occurred in those cases will affect forensic examiners approach to examinations and their conclusions.
-
5.
Organizational Factors: Aspects of the system within which examiners operate that can influence an examiner's approach to examinations and their conclusions, such as which side of the adversarial legal system they are retained by, backlogs in the laboratory, and the local laboratory culture and policies.
-
6.
Education and Training: The quality of their education and training, as well as the views, beliefs, and culture of those responsible for providing training, will affect forensic examiners' decisions.
-
7.
Personal Factors: A person's traits, attitudes, and motivations can impact decision-making and bias (e.g., someone who has a high tolerance for risk taking will approach forensic examinations differently than someone who is risk averse).
-
8.
Human and Cognitive Factors: Cognitive and social dynamics inherent to all individuals' decision-making, such as the use of automatic decision processes like cognitive bias when rushed, uncertain, or unmotivated.
Ultimately, people's interpretation of the world is shaped by a mix of actual facts, observations, feelings about and interpretations of the situation they are in, as well as their individual traits, knowledge, beliefs, and desires. By considering what psychological research has already established about each of these eight types of influence, laboratory managers can capitalize on cognitive strengths and protect against cognitive weaknesses, which will improve the accuracy and reliability of results produced by their agency [3,6,11].
Other researchers have discussed specific examples of types of potentially biasing information commonly available to forensic examiners as they complete casework [11,14]. For instance, examiners frequently communicate directly with the police investigators and examiners will also exchange information with one another, either informally or through “peer review" processes intended to validate conclusions. Additionally, police and lawyers may request re-examinations if test results do not align with their expectations, other evidence, or other case information. This is not something that examiners have control over—even if they do not wish to receive this information, they themselves cannot control what they receive alongside a request for analysis, and many potential biasing pieces of information can be subtle [11,12].
In light of this important research at the intersection between human factors and forensic science, the Department of Forensic Sciences in Costa Rica decided to undertake a project that would address the issue of cognitive bias in a very practical way by providing a model for implementation to the forensic community. This project presented many challenges to the Department because any proposal needed to practical and feasible to ensure they would enhance the work coming out of the laboratory rather than compound existing issues with limited resources. Despite the large body of literature on the topic of cognitive bias, though, there were very few examples of forensic laboratories or even individual examiners who had implemented these research-based procedural changes designed to reduce cognitive bias. Often the literature suggested vague approaches to forensic examination that were very theoretical (e.g. Ref. [8]), but did not propose concrete guidelines or policies that would lay the foundation in a laboratory to achieve these goals.
Of course, there are some important differences between forensic domains that have important implications for cognitive bias mitigation and what information is and is not task relevant. Some disciplines require the examiner to play a greater role in deciding how to collect, sample, and interpret the data. Other disciplines limit the role of the examiner to comparisons of already collected and processed evidence. Some disciplines also rely on more objective data than others (e.g., DNA analysis [15]), which means they are less vulnerable to bias than other disciplines that allow the examiner more discretion or primarily rely on the examiner's visual perception skills. But more objectivity does not prevent cognitive biases—there is a still a lot of decision-making that is left to the examiner, even in supposedly “objective” disciplines like DNA [16].
After considering these elements of the various disciplines currently operating at the Department of Forensic Science, we decided that the Questioned Documents Section was best for pilot testing research-based mitigation strategies in the Department. The examination of questioned documents, a pattern-matching discipline, relies heavily on examiners’ observations, judgments, and subjective interpretations, so this field is likely to be more vulnerable to cognitive bias effects than some others without adequate mitigation. Furthermore, like other disciplines where the task is to determine whether two samples came from the same source (e.g., two fingerprints were made by the same finger, whether two bullets were fired from the same gun), forensic document examiners are also tasked with linking a piece of evidence connected with a crime (e.g., a signature) to a particular suspect—the examiner compares the visual data and determines whether the handwriting samples are “sufficiently similar” to conclude that they originated from the same source [8]. In addition to the goal of mitigating bias effects, we hoped to use this project to raise awareness about the existence of bias and provide a guide for how to plan and implement feasible solutions to control the potential impact of bias in forensic laboratories.
2. Method
The Department of Forensic Science took an approach to implementing bias mitigation strategies that follow the recommendations made in Itiel Dror's paper on different sources of bias [3], Itiel Dror and Jeff Kukucka's (2021) article describing LSU-E [10], and the published LSU-E worksheet and training materials created by Adele Quigley-McBride and colleagues [11]. These articles were used to identify potential sources of bias that our examiners could expect to encounter and be influenced by, and then compile a variety of specific countermeasures to address bias arising from those sources.
Initially, it was clear that the first step to effectively addressing bias was training the examiners. We organized for presentations to raise awareness about the nature and existence of bias, as well as educating examiners that bias was not something that could be addressed by sheer willpower and mental effort. The organization's psychology office collaborated with the project team in these first general training sessions.
Next, specialized training was provided exclusively to the Questioned Documents Section regarding cognitive factors in forensic decision-making. Specifically, they learned more about the definition of cognitive bias, classification of sources of cognitive bias in forensic sciences, how bias affects decisions in forensic examination contexts, and how information management protocols, such as LSU-E, work [10,11].
Before implementing any countermeasures in practice, the team created flowcharts of the different processes for Questioned Documents (sometimes called “process maps” in the literature). The flowcharts were developed during focused sessions between the Questioned Documents group and the project team. These flowcharts permitted the team to identify steps in their process that were likely to be more vulnerable to biasing information and what types of contextual information examiners could expect to receive at these stages of the process.
To classify the types of sources of cognitive contamination, Dror's taxonomy of eight sources of bias was used [3]. Once a source of bias was identified and classified, the risk was assessed using the methodology proposed by Dror and Pierce [17], who suggested a risk matrix that evaluates the risk of bias in relation to the severity of its effects. This approach was used to prioritize what interventions or procedures would be most important for minimizing or mitigating cognitive bias. For each source, Questioned Documents examiners were also asked to provide their expert opinion on the classification of the type of source and the risk associated with each source.
Subsequently, a brainstorming session was conducted, in which the Questioned Document examiners were asked to propose possible interventions that could mitigate the potential sources of cognitive bias. Based on these data, Pareto diagrams [18] were created to determine which actions for mitigating the bias that have the greatest impact on the different sources of bias identified in the flowcharts. Our goal in taking these initial steps was to permit a more streamlined, targeted approach to reducing or documenting the potential effect of contextual information when the time came to optimize the use of laboratory resources.
Finally, before executing the bias mitigation plan, a feasibility analysis was conducted for the various interventions or procedures proposed to mitigate the effect of contextual information. This involved considering the required investment (i.e., financial, personnel, time), the anticipated benefits, and whether the necessary resources were available.
3. Results
Thirty-three potential sources of bias were identified during the working sessions between the project team and the Question Document examiners. The most common type of bias identified was contextual information. For each source, the team from the Questioned Documents section were asked to provide their expert opinion about how that source of bias should be classified and the level of risk associated with each source. The final results from this exercise are shown in Table 1 below. Also, refer to Fig. 1 for a graph of the number of sources associated each type of bias in Dror's taxonomy [3].
Table 1.
Relationship between the type and source of cognitive bias, the associated risk, and possible actions to mitigate any biasing effects.
| DESCRIPTION OF BIAS | CLASSIFICATION | RISK LEVEL | RECOMMENDED MITIGATION |
|---|---|---|---|
| 1. Communication between the person responsible for receiving and the person in charge of delivering the case. | Contextual Information | Low | TRAINING, COMMUNICATION POLICY |
| 2. Communication between judicial authority and section personnel to resolve rejection. | Contextual Information | Low | TRAINING, COMMUNICATION POLICY |
| 3. Communication between judicial authority and section personnel to clarify aspects of request. | Contextual Information | Low | TRAINING, COMMUNICATION POLICY |
| 4. Record of priorities (vulnerable population, detained person, legal deadline and urgent other). | Organizational Factors | Medium | CASE MANAGER |
| 5. Telephone calls from parties, inquiring about case status and judicial authority for urgent case handling. | Contextual Information | High | CASE MANAGER |
| 6. Transfer of information between the person responsible for receiving and the person responsible for preliminary analysis. | Contextual Information | Low | TRAINING, COMMUNICATION POLICY |
| 7. Communication between judicial authority and the person responsible for preliminary analysis. | Contextual Information | Low | CASE MANAGER |
| 8. Information recorded in the local LIMS and available to the analyst. | Contextual Information | High | CASE MANAGER |
| 9. Transfer of information between the person responsible for preliminary analysis and the person responsible for case analysis. | Contextual Information | Medium | CASE MANAGER |
| 10. Case assignment for re-analysis at the request of judicial authority. | Organizational Factors | High | LSU-E, TRIAGE |
| 11. Re-analysis of cases due to additional comparison elements (continuation of the expertise). | Organizational Factors | High | TRIAGE |
| 12. The person responsible for taking the writing sample may be the same person who performs the analysis. | Contextual Information | Very High | ADDITIONAL SITUATION-SPECIFIC MEASURES ∗ |
| 13. The person responsible for analysis participates as a witness during the writing sample collection. | Contextual Information | Medium | ADDITIONAL SITUATION-SPECIFIC MEASURES ∗ |
| 14. Personal information (photograph and personal data) in the writing sample form. | Reference Material | High | CASE MANAGER, LSU-E |
| 15. Transfer of information between the person responsible for preliminary analysis and the person responsible for analysis. | Contextual Information | Medium | CASE MANAGER, TRAINING |
| 16. Review of all case information (including irrelevant information) before performing the analysis. | Contextual Information | Very High | LSU-E |
| 17. Irrelevant information for analysis available in questioned elements. | Data (questioned material) | High | CASE MANAGER |
| 18. No real sequential analysis of manuscripts (questioned first, then comparison). | Organizational Factors | High | CASE MANAGER, LSU-E |
| 19. Peer consultation to issue a criterion (in cases of complex signatures). | Education and Training | High | COMMUNICATION POLICY, TRAINING |
| 20. Presence of parties during the analysis or review of evidence. | Contextual Information | High | COMMUNICATION POLICY, TRAINING |
| 21. Communications when court summons are presented. | Contextual Information | High | COMMUNICATION POLICY, TRAINING |
| 22. Direct communication between judicial authority or parties and the person responsible for analysis. | Contextual Information | High | COMMUNICATION POLICY, TRAINING |
| 23. No double-blind testing program. | Organizational Factors | Medium | BLIND TEST SAMPLES |
| 24. Insufficient training and long periods without specific training in handwriting analysis. | Education and Training | High | TRAINING |
| 25. Experience in past cases, such as “Rena Were" or bank withdrawal vouchers. | Base Rate | High | BLIND TEST SAMPLES |
| 26. Transfer of evidence involves interaction between the analyst and the verifier. | Contextual Information | High | BLIND VERIFICATION |
| 27. The verifier knows the range of results in advance, overlooking other possibilities. | Contextual Information | High | BLIND VERIFICATION |
| 28. The verifier knows in advance who is responsible for the case. | Contextual Information | High | BLIND VERIFICATION |
| 29. Resolution of differences between the results of the analyst and the verifier. | Organizational Factors | High | BLIND VERIFICATION |
| 30. Single-person criterion of the technical leader to resolve differences in the result verification process. | Contextual Information | High | BLIND VERIFICATION |
| 31. Resolution of differences between the analyst and the peer reviewer (critical aspects). | Organizational Factors | High | COMMUNICATION POLICY |
| 32. Single-person criterion of the technical leader to resolve differences in the peer review process (critical aspects). | Organizational Factors | Medium | COMMUNICATION POLICY |
| 33. Resolution of differences between the analyst and the section head (critical aspects). | Contextual Information | High | COMMUNICATION POLICY |
Note. ∗Additional measures outside the scope of the seven major actions to mitigate cognitive bias: Relocating the site where writing samples are collected to a different area outside of the primary section. Furthermore, the individual responsible for collecting the writing samples must not be the person assigned to work on the case.
Fig. 1.
Classification of the 33 potential sources of bias according to Dror's taxonomy [3].
Next, the 33 potential sources of cognitive bias were assessed in a risk matrix, which involved considering the risk of bias alongside the anticipated gravity of its effects to determine which sources of bias were higher priority for the minimization and mitigation of bias. We produced Fig. 2 with these data, which shows that there were two potential sources of cognitive bias that were classified as “very high risk” and 20 potential sources of bias classified as “high risk”. The other sources of bias identified were considered “medium risk” or lower.
Fig. 2.
Classification of the 33 potential sources of bias according to their perceived level of risk.
The brainstorming session where the project team and the Questioned Documents examiners worked on potential methods or actions to mitigate the identified cognitive biases resulted in over 45 ideas, which were later grouped into the following categories.
-
1.
Use of a Case Manager
-
2.
Use of Linear Sequential Unmasking-Expanded (LSU-E)
-
3.
Triage
-
4.
Use of Blind Verification Procedures
-
5.
Create and Use Policies for Managing Cognitive Bias
-
6.
Implementing Blind Proficiency Testing
-
7.
Training
-
8.
Additional Situation-Specific Measures
Table 2 shows the number of potential sources of cognitive bias within each risk level and action category for mitigating bias. This shows the anticipated impact of each action category. For instance, for the action category “Case Manager," it is shown that this action will impact a total of nine potential sources of cognitive bias, of which one is classified as low risk, three as medium risk, and five as high risk. Thus, “Case Manager” is likely to have a larger impact on more types of bias at different risk levels than “LSU-E”, which the examiner expect will help with four different sources of bias, but all of these sources of bias are “high” or “very high” risk.
Table 2.
Number of potential sources of cognitive bias within each risk level and mitigation action category.
| Action Category for Mitigation | Total | Low | Medium | High | Very High |
|---|---|---|---|---|---|
| 1. Case Manager | 9 | 1 | 3 | 5 | 0 |
| 2. Linear Sequential Unmasking – Expanded (LSU-E) | 4 | 0 | 0 | 3 | 1 |
| 3. Triage | 2 | 0 | 0 | 2 | 0 |
| 4. Blind Verification | 5 | 0 | 0 | 5 | 0 |
| 5. Policies for Cognitive Bias Management | 11 | 4 | 1 | 6 | 0 |
| 6. Blind Proficiency Testing | 2 | 0 | 1 | 1 | 0 |
| 7. Training | 10 | 4 | 1 | 5 | 0 |
| 8. Other Targeted Measures | 2 | 0 | 1 | 0 | 1 |
Using Table 2, the Pareto chart shown in Fig. 3 was constructed. This analysis identifies which categories of actions to mitigate bias are likely to have greatest impact on the 33 potential sources of cognitive bias. Fig. 3 shows that by applying the action categories labeled as cognitive bias management policies, training, case manager, and blind verification, we could address approximately 80 % of the potential sources of cognitive bias identified during the assessment of the Questioned Documents section of the laboratory. Thus, focusing resources on these categories was likely to result in the most value and benefit.
Fig. 3.
Pareto diagram assessing the categories of actions to mitigate bias and source of bias.
For each mitigation category, we determined the relevant parts of the existing Questioned Document section procedures that needed improvement, why the improvement was necessary, the benefits of the improved procedure, what needed to be done to implement the procedure, and the resources needed to successfully implement this mitigation strategy. Here, we describe our approach to each mitigation category and the results of those efforts.
-
1.
Case Manager
To implement this form of mitigation, we needed to create a new role for someone to manage information and oversee cases in a way that minimizes examiners exposure to irrelevant, potentially biasing contextual information. This is a strategy that has been recommended by researchers in the relevant literature [10,11]. Though there would be costs associated with creating a new case manager role in the laboratory, this mitigation strategy was a potential solution to nine of the 33 potential sources of cognitive bias, which were categorized as medium or high risk.
To implement of this mitigation strategy, the case manager uses the Linear Sequential Unmasking Expanded (LSU-E) worksheet to regulate the flow of information provided to examiners. The case manager becomes the point of contact between any judicial authorities and the forensic experts assigned to work on the evidence. Their primary responsibility is to sort through the case information, ask any questions or request any missing information from authorities, and determine what should be passed to the examiner to review and what should be withheld, at least initially. They also determine the feasibility of the analysis requested by the judicial authority and help guide the authority in framing their requests based on the evidence available and their goal, improving the efficiency of requests from authorities. To ensure their success, case managers need to be trained to detect possible sources of cognitive bias before the case moves on to the analysis phase.
The impact of introducing a case manager role proved to be significant in this pilot program. Creating this formal role in the laboratory addressed nine potential sources of bias, with a focus on medium to high-risk sources. By acting as an intermediary between judicial authorities and forensic experts, the case manager was able to control whether and when irrelevant case information was provided to analysts, thereby reducing the risk of bias and saving the primary examiner time. Because the case manager reviewed the requests for analysis initially, the experts received limited background information on the circumstances of the case and communicated with the judicial authorities less often.
-
2.
Linear Sequential Unmasking-Expanded (LSU-E)
LSU-E is a procedure used to manage the flow of information to examiners to help them consider the information associated with a particular case and how that information might influence their examination of the evidence and, ultimately, their conclusion about the evidence. The potential impact of information and whether that impact is likely to be appropriate or not is evaluated using three guiding principles when using LSU-E: biasing power, subjectivity, and irrelevance [10,11]. This evaluation is then used to determine when information should be released to examiners. For example, when following LSU-E, information should be revealed to examiners in a sequential manner—from the most relevant information (i.e., the crime evidence) through to the information that is least relevant (e.g., tangentially relevant or irrelevant case information that is not necessary to perform the examination requested). In addition, the examiner should keep records of when they received information, as well as how they think it affected their examination and conclusion, if at all.
Information associated with cases submitted to the Questioned Document section were evaluated using these guiding principles and a plan was made for the controlled release of case information to the examiner by the case manager. This was done using a modified version of the LSU-E worksheet created by Quigley-McBride and colleagues [11]. This mitigation strategy addressed four of the cognitive bias sources identified by the project team in collaboration with the Questioned Documents section. Although this strategy was linked with only a few sources of bias, these sources of bias were categorized with high or very high risk, suggesting that the potential impact on cases was serious.
We reviewed the current system prior to implementation any mitigation strategies, and we found that there was no method or procedure available to sequentially expose information to the Questioned Document examiners. Instead, the system was designed to show all case information to the expert, so they could gain access to all of the available information even before beginning their examination. Thus, this mitigation strategy represented a significant change from the existing protocols in the Questioned Documents section of the laboratory.
This approach directly mitigated four sources of bias classified by the project team and the Questioned Documents Section as high or very high risk. LSU-E encouraged examiners to evaluate each piece of evidence independently and consider the impact that extraneous information might have on their judgments about the evidence, which helped to maintain the integrity of the forensic examination through information management and transparent documentation. In particular, we found that the cognitive biases resulting from access to the Reference Material (the suspect's reference samples) are effectively reduced using LSU-E—ensuring that the evidence associated with the crime is reviewed first prior to examining any samples from the suspect [6,11].
-
3.
Blind Verifications
Blind verifications are an independent check on the initial examiner's decision—the verifier is blind to the conclusion that initial examiner made, or is not informed of the initial examiner's conclusion, so they are not anchored to the initial examiner's opinion or expecting a particular outcome. If the verifier knows what their similarly-qualified colleague already concluded about the evidence, they will have a tendency to search for evidence that confirms the initial result and ignore contradictory information without realizing it—one way confirmation bias impacts forensic results. This can even occur in situations where only one type of conclusion is verified in a laboratory—often same source or identification judgments. Even if the verifier is not explicitly told what the initial examiner concluded, they can surmise based on the protocols in the lab. Thus, verification decisions made under blind conditions provide independent assurance that a particular conclusion from an examiner is unlikely to be an error or inappropriate based on the data available.
To implement this strategy, we amended the existing verification process. The new verification process was arranged by the case manager because the verifier is shielded from information that is not necessary for the execution of the verification—including the original examiner's opinion. If any discrepancies are detected, the Case Manager is responsible for convening both parties to resolve the issue. If the verifier reaches a different conclusion to the initial examiner, the technical leader intervenes and makes the final decision. By ensuring that verifiers were unaware of the initial results or the identities of the examiners, our team addressed five sources of bias. We found that implementing blind verifications ruled out alternative explanations for the verifiers agreement, which made the results more reliable. Importantly, we also found that it was very feasible to implement in our laboratory context.
-
4.
Policies for Managing Cognitive Bias
It was important to create information management policies that, if adhered to, to control the flow of irrelevant and potentially biasing information from police, stakeholders, other forensic examiners to examiners. So, initially it was determined that oral or written statements of irrelevant information should not be shared with the examiner, if possible. The objective of these policies was to ensure that the case information made available to the examiners did not create expectations or beliefs that could impact the examiners' interpretations of the evidence and data, their process of analyzing the evidence, or how they interpreted and reported the results. These policies were designed to clarify acceptable and unacceptable communication practices to minimize the examiners’ exposure to irrelevant information and maintain their independence.
Examples of information identified by the Questioned Document examiners and the project team as problematic types of irrelevant contextual information include.
-
-
whether the suspect confessed;
-
-
whether the suspect was identified by witnesses and other analyses;
-
-
whether the suspect has a criminal record; and,
-
-
the suspect's name and sociodemographic characteristics.
We proposed the creation of a document which outlined new protocols for, and expectations regarding, access to case information, interactions with the various end-users, other examiners, and requesters, as well as considering the different ways that they might come to communicate directly or indirectly with the examiner. The new policies specified that requests for analysis or other queries should be addressed to the Section Chief or the technical leader, not the examiner's assignment to the case. Communication between experts from different forensic areas (e.g., examiners brought in to perform a peer review, to transfer the evidence, or informal exposure to case information and evidence through shared workspace) should not occur. If any of these events could not be avoided and occurred anyway, this was documented in the case record.
Although this strategy is similar to mitigation strategies 1 and 2, the creation of policies highlights and standardizes the laboratory's formal intentions to engage in information management practices. Given that much of the information that needs to be managed in casework has the potential to bias examiners, the best way to align with these policies or achieve these goals may be to involve another examiner or a case manager (as we decided to do in this case). Although the application of information management frameworks cannot turn a subjective judgment or conclusions into an objective one, these strategies limit the potential risk of bias in a uniform way that requires the least amount of time and resources on the part of the laboratory and the examiners.
Developing and training examiners to follow policies designed to manage cognitive bias effects, and enforcing those policies, limited access to unnecessary or potentially biasing information in the examiners’ case flow. This strategy impacted a wide range of sources of bias. We found that emphasizing clear communication protocols, peer review, verification, conflict resolution protocols, and reducing examiners exposure to task-irrelevant information in the policies was key to the success of this pilot program.
-
5.
Blind Proficiency Testing
Creating a Blind Proficiency Testing plan was chosen as a bias mitigation strategy because it would allow us to evaluate policies and error rates within the laboratory by sending mock cases (that mimic routine cases but for which ground truth is known) through the typical channels that an ordinary request would. Thus, examiners’ method conformance and performance could be evaluated internally [19]. nThese tests were proposed to supplement other, inter-laboratory tests as they serve a different purpose—blind proficiency tests can help monitor the risk of bias, evaluate examiner competency, and test the validity of the methods use in the lab. They can also reveal areas for improvement in the laboratory environment and culture, or regarding the internal policies used by the laboratory.
Blind proficiency tests are superior other kinds of tests for other reasons too. First, when people know they are being tested, they tend to behave in ways that they think that they are supposed to act rather than how they ordinarily would. So, blind proficiency tests, theoretically, should yield results that are more similar to the examiners' true behavior in real cases [20]. Second, other kinds of competency testing plans assess an individual's competency in one or more typical tasks, rather than the entire quality management system and process in a laboratory. Blind proficiency tests shed light on the entire process, from evidence submission to verdict, and can reveal strengths and weaknesses at each step.
For the current project, though, this particular strategy was not implemented due to the significant financial costs involved in creating realistic test cases and submission materials. Costa Rica has a single forensic laboratory that serves the entire population and, despite the potential benefits of implementing blind proficiency testing, we were not able to justify the resource investment that would have been required. Note that the Houston Forensic Science Center has published an article on their experience implementing a testing program like this that was helpful for evaluating its feasibility for the current project [21].
Although this bias mitigation strategy shows great promise for monitoring the impact of bias and testing the efficacy of the Quality Control system as a whole, its implementation was not feasible financially for our Department. That said, we recognize the many benefits that blind proficiency testing offers for laboratories and agencies that have the resources to undertake such a system. Recent analysis of proficiency testing suggests that blind testing identifies issues in examiner's workflow and the laboratory system, and can reduce error rates by as much as 46 % [21]. Moreover, both the American Society of Crime Laboratory Directors (ASCLD) and Laboratory Accreditation Board (LAB) recommended blind proficiency testing in the 2009 National Academy of Sciences (NAS) report as they are a better test of examiners' accuracy.
-
6.
Training
Education and training play an important role in addressing bias and understanding the impact of cognitive bias effects. Examiner cooperation is crucial to successfully mitigating bias. Thus, providing examiners with ongoing education opportunities that can give them with the knowledge, skills, and tools to help them implement the mitigation strategies proposed here was important for the success if the pilot program. At a minimum, if examiners understand what cognitive biases are and how they can arise, they can appreciate the importance of policy changes and proposed changes to their protocols and environment.
The training provided to examiners facilitated the implementation of the strategies identified here, and sought to engage examiners in their maintenance and improvement over time. We delivered training sessions that covered the concept of cognitive bias, the effects of cognitive biases, and how to address cognitive bias in casework. Continuing education on cognitive bias and its impact on forensic analysis feasible with even very few resources. Ongoing training helped the forensic practitioners in our Department recognize and address biases, and help examiners contribute to a more reliable forensic examination system. It is important for forensic scientists to acknowledge that bias exists and identify where biases arise and affect interpretation, even if awareness and training is not sufficient to prevent bias. To demonstrate, studies show that, untrained examiners were 26 % less likely to see bias as a problem in the forensic sciences, 54 % less likely to see bias as a problem in their own domain, and 35 % less likely to see their own judgments as vulnerable to bias [2].
3.1. Implementing mitigation strategies for questioned documents
By implementing policies designed to manage cognitive bias and other improvements, documents, and forms, the Quality Management Systems were modified at the Section and Department level. Specifically, two Section procedures, two general procedures, the Department Quality Manual, one record, and two new forms were created. Additionally, the symbol Ⓢ was incorporated into each document in the Quality Management System to identify elements related to actions addressing bias.
To date, the implementation of these actions to mitigate cognitive bias has led to Questioned Documents identifying a total of 24 cases with a very high potential for cognitive bias effects, for which 38 actions were taken to mitigate the effects of cognitive bias for those specific cases. In addition, as a result of these new tools, strategies, and policies, the Questioned Documents Section has carried out 201 blind verifications, which led to 24 non-critical disputes, four critical disputes, and two interventions by the technical manager to resolve the critical disputes for which the initial and verifying examiner could not come to an agreement. Finally, as part of the new system, the case manager has been responsible for more than 161 clarifications and queries that were sent to the legal authorities without involving the examiners assigned to work on the case, thereby minimizing the flow of irrelevant information to which the forensic experts are exposed.
4. Discussion
The results from the pilot program within the Questioned Documents Section of the Department of Forensic Sciences in Costa Rica provide significant insights into the nature and impact of cognitive bias sources and the effectiveness of various mitigation strategies. The study identified 33 potential sources of cognitive bias, with contextual information identified as the leading source of potential bias. These sources of bias were also classified according to their risk levels and mitigation strategies were identified for each that the project team and Questioned Document examiners thought could mitigate those risks. In this article, we have provided a comprehensive overview of the approach, the challenges faced throughout this process, how we determined which approaches were and were not feasible, and which changes and strategies were implemented to address the sources of bias identified.
4.1. Identifying and classifying sources of cognitive bias
To aid other laboratories in the implementation of similar bias mitigation programs, the current paper provides detailed information about the steps taken by the project team at the Department of Forensic Science in Costa Rica to create new information management and bias mitigation procedures. Our hope was to create a reproducible model that other laboratories interested in bias mitigation could adopt, aiming to reduce barriers to implementation. We describe here how the project team first worked directly with the Questioned Documents Section to determine what research-based best practices should be considered. These were evaluated for their priority in light of the current systems within which the examiners were working and the level of risk posed by different sources of bias. Our approach also considered which strategies and tools were feasible given the resources available to the Department.
By engaging in some of this foundational work before implementing any changes, we were able to determine which interventions would have the largest effect. “Effect” was considered both in terms of the number of systems and cases likely to be affected, but also the severity of any potential errors and biases that could arise. We hope that this can provide guidance for how to determine what the highest priority issues are so that resources can be targeted most efficiently during implementation.
Another key element of our approach was organizing the mitigation strategies into broader categories based on the goal of that proposed strategy. This provided a simple framework for evaluating our mitigation efforts, rather than a piecemeal approach for which the value of the changes made might be unclear. We focused evaluations on the ability of each strategy to address the high-priority, high-risk sources of bias identified by the team and the Questioned Documents group. We recommend a similar approach in other laboratories implementing bias mitigation strategies to facilitate ongoing evaluations of new practices.
4.2. Quality assurance
The mitigation of cognitive bias is a task for the Quality Control and Assurance (QA/QC) systems in laboratories. Of course, there is a need for rigorous quality assurance and control measures in forensic science even without considering the issue of cognitive bias, but without the input of the QA/QC personnel, it is not possible to regulate and maintain interventions designed to mitigate bias. Both International Organization for Standardization (ISO) 17,025 and ISO 17020 emphasize the importance of impartiality and a risk-based approach to QA/QC tasks and management and highlights the central role that people play in forensic examination, even when advanced quantification methods and instrumentation are employed [16].
The revised ISO 17025 standard further emphasizes the call for impartiality in forensic testing, mirroring the impartiality requirements seen in ISO 17020. We ensured that the interventions implemented by the current aligned with the QA/QC requirements of ISO 17025 and ISO 17020. This consistent message in two internationally recognized forensic standards indicates that there is a universal call for laboratories to implement protocols and tools that can increase objectivity and impartiality. This is true of all forensic sciences, not just the pattern matching techniques (e.g., handwriting, fingerprints) typically critiqued for the amount of subjectivity permitted by their methods and procedures [1,9].
4.3. Evaluating the effectiveness of mitigation measures
Table 2 and Fig. 3 illustrate the impact of each bias mitigation category. Policies for managing cognitive bias, training, introducing a case manager, and blind verification were identified as the most effective measures, collectively addressing approximately 80 % of the potential sources of bias. The Pareto analysis allowed us to weigh the importance of these strategies in managing bias and decreasing subjectivity in forensic evaluations prior to implementation. We recommend that laboratories seeking to implement bias mitigation strategies undergo this type of analysis prior to determining what to implement to save resources. A detailed evaluation prior to implementation will also reduce the likelihood that some of the strategies need to be discontinued or altered after implementation due to issues that could have been predicted. We will continuously monitor and evaluate the impact and efficacy of these bias mitigation strategies by conducting periodic reviews of the new systems and how the examiners are managing the new tools and policies.
5. Conclusions
The results of this pilot program highlight the critical need for a multifaceted, strategic approach to managing cognitive bias in forensic evaluations. By identifying potential sources of bias, assessing their risk levels, and implementing targeted mitigation strategies, the Department of Forensic Sciences in Costa Rica was able to make significant steps toward enhancing the reliability and objectivity of forensic analyses in their Department. The Department has been pleased with both the feasibility and efficacy of introducing Case Managers, LSU-E information management protocols, and Blind Verifications. The success of these strategies in our Department demonstrates that strategic interventions can reduce the effect of cognitive bias. Ongoing training and policy development will further support the goal of minimizing bias. While some measures, such as Blind Proficiency Testing, were not implemented due to resource constraints, the overall approach taken by the Department provides a robust framework for improving forensic evaluation practices.
Future efforts should focus on refining these strategies and exploring additional mitigation techniques that have different resource demands. Laboratories implementing bias mitigation techniques should continually assess the impact of the implemented strategies to ensure that the benefits of these interventions are maintained and continuously improved in light of new research findings and international forensic standards. We wish to emphasize that mitigating cognitive bias in forensic science is an ongoing commitment rather than a one-time effort. Cognitive bias effects pose a significant threat to the integrity of forensic evaluations, so it is imperative for laboratories to create systems that support examiners in their goal of basing their conclusions solely on the evidence and information that is relevant to the examination they have been asked to perform.
It is essential to balance the implementation of bias reduction strategies with practical considerations. For instance, overly stringent measures can complicate workflows and potentially undermine good existing practices. The goal should be to minimize the risk of contextual bias without introducing inefficiencies that could delay forensic processes or confuse examiners. Strategies to reduce bias should be implemented only after a detailed cost/benefit analysis has been conducted and the impact on workflow has been carefully considered. As emphasized by Dror [1], addressing contextual bias effectively requires a thoughtful approach that considers risks and implements proportionate and appropriate measures—we believe that we have achieved that with the current project and encourage other laboratories to use our model to start this conversation with their forensic community.
CRediT authorship contribution statement
Carolina Rojas Alfaro: Writing – original draft, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Diego Ureña Mora: Writing – review & editing, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Mauricio Chacón Hernández: Writing – review & editing, Supervision, Resources, Project administration. Adele Quigley-McBride: Writing – review & editing, Methodology, Conceptualization.
Declaration of competing interest
None.
Acknowledgments
The authors express gratitude to the Government of the United States through the Bureau of International Narcotics and Law Enforcement Affairs (INL) and the International Criminal Investigative Training Assistance Program (ICITAP) for their valuable help with the study tour of various leading laboratories in the United States, which served as a guide for the implementation of the plan in our country.
References
- 1.National Research Council . National Academies Press; 2009. Strengthening Forensic Science in the United States: A Path Forward. [Google Scholar]
- 2.Kukucka J., Kassin S.M., Zapf P.A., Dror I.E. Cognitive bias and blindness: a global survey of forensic science examiners. Journal of Applied Research in Memory and Cognition. 2017;6(4):452–459. doi: 10.1016/j.jarmac.2017.09.001. [DOI] [Google Scholar]
- 3.Dror I.E. Cognitive and human factors in expert decision making: six fallacies and the eight sources of bias. Anal. Chem. 2020;92(12):7998–8004. doi: 10.1021/acs.analchem.0c00704. [DOI] [PubMed] [Google Scholar]
- 4.Spellman B.A., Quigley-McBride A. Diritto & Questioni Pubbliche; 2023. Reasoning about Forensic Science Evidence. Legal Reasoning and Cognitive Science: Topics and Perspectives. [Google Scholar]
- 5.Spellman B.A., Eldridge H., Bieber P. Challenges to reasoning in forensic science decisions. Forensic Sci. Int.: Synergy. 2022;4 doi: 10.1016/j.fsisyn.2022.100274. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Mattijssen E.J., Kerkhoff W., Berger C.E., Dror I.E., Stoel R.D. Implementing context information management in forensic casework: minimizing contextual bias in firearms examination. Sci. Justice. 2016;56(2):113–122. doi: 10.1016/j.scijus.2015.11.004. [DOI] [PubMed] [Google Scholar]
- 7.Kassin S.M., Dror I.E., Kukucka J. The forensic confirmation bias: problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition. 2013;2(1):42–52. doi: 10.1016/j.jarmac.2013.01.001. [DOI] [Google Scholar]
- 8.Stoel R.D., Dror I.E., Miller L.S. Bias among forensic document examiners: still a need for procedural changes. Aust. J. Forensic Sci. 2014;46(1):91–97. doi: 10.1080/00450618.2013.797026. [DOI] [Google Scholar]
- 9.President’s Council of Advisors on Science and Technology . Executive Office of the President of the United States; Washington DC, USA: 2016. Report to the President Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods.https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf Retrieved from. [Google Scholar]
- 10.Dror I.E., Kukucka I.E. Linear sequential unmasking–expanded (LSU-E): a general approach for improving decision making as well as minimizing noise and bias. Forensic Sci. Int.: Synergy. 2021;3 doi: 10.1016/j.fsisyn.2021.100161. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Quigley-McBride A., Dror I.E., Roy T., Garrett B.L., Kukucka J. A practical tool for information management in forensic decisions: using Linear Sequential Unmasking-Expanded (LSU-E) in casework. Forensic Sci. Int.: Synergy. 2022;4 doi: 10.1016/j.fsisyn.2022.10021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kukucka J., Dror I.E. In: The Oxford Handbook of Psychology and Law. DeMatteo D., Scherr K.C., editors. Oxford University Press; 2023. Human factors in forensic science: psychological causes of bias and error; pp. 621–642. [DOI] [Google Scholar]
- 13.Innocence Project Explore the numbers: innocence project's impact. 2024. https://innocenceproject.org/exonerations-data/
- 14.Saks M.J., Risinger D.M., Rosenthal R., Thompson W.C. Context effects in forensic science: a review and application of the science of science to crime laboratory practice in the United States. Sci. Justice. 2003;43(2):77–90. doi: 10.1016/S1355-0306(03)71747-X. [DOI] [PubMed] [Google Scholar]
- 15.Jeanguenat A.M., Budowle B., Dror I.E. Strengthening forensic DNA decision making through a better understanding of the influence of cognitive bias. Sci. Justice. 2017;57(6):415–420. doi: 10.1016/j.scijus.2017.07.005. [DOI] [PubMed] [Google Scholar]
- 16.Dror I.E., Hampikian G. Subjectivity and bias in forensic DNA mixture interpretation. Sci. Justice. 2011;51(4):204–208. doi: 10.1021/acs.analchem.0c00704. [DOI] [PubMed] [Google Scholar]
- 17.Dror I.E., Pierce M.L. ISO standards addressing issues of bias and impartiality in forensic work. J. Forensic Sci. 2020;65(3):800–808. doi: 10.1111/1556-4029.14265. [DOI] [PubMed] [Google Scholar]
- 18.Alfaro C.R., Madrigal G.B., Hernández M.C. Improving forensic processes performance: a Lean Six Sigma approach. Forensic Sci. Int.: Synergy. 2020;2:90–94. doi: 10.1016/j.fsisyn.2020.02.0019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Swofford H., Lund S., Iyer H., Butler J., Soons J., Thompson R., Desiderio V., Jones J.P., Ramotowski R. Inconclusive decisions and error rates in forensic science. Forensic science international. Synergy. 2024;8 doi: 10.1016/j.fsisyn.2024.100472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Mejia R., Cuellar M., Salyards J. Implementing blind proficiency testing in forensic laboratories: motivation, obstacles, and recommendations. Forensic science international. Synergy. 2020;2:293–298. doi: 10.1016/j.fsisyn.2020.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Hundl C., Neuman M., Rairden A., Rearden P., Stout P. Implementation of a blind quality control program in a forensic laboratory. J. Forensic Sci. 2020;65(3):815–822. doi: 10.1111/1556-4029.14259. [DOI] [PMC free article] [PubMed] [Google Scholar]



