Vacuous standards – Subversion of the OSAC standards-development process

Abstract

In the context of development of standards for forensic science, particularly standards initially developed by the U.S. Organization of Scientific Area Committees for Forensic Science (OSAC), this perspective paper raises concern about the publication of vacuous standards. Vacuous standards generally state few requirements; the requirements they do state are often vague; compliance with their stated requirements can be achieved with little effort – the bar is set very low; and compliance with their stated requirements would not be sufficient to lead to scientifically valid results. This perspective paper proposes a number of requirements that we believe would be essential in order for a standard on validation of forensic-science methods to be fit for purpose.

1. Perspective

In interpreting Federal Rule of Evidence (FRE) 702, the U.S. Supreme Court in Daubert [1] advised that trial judges should act as gatekeepers to “ensure that any and all scientific testimony or evidence admitted is not only relevant, but reliable” (p. 589). The Court stated that “In a case involving scientific evidence, evidentiary reliability will be based upon scientific validity” (n. 9, emphasis in original); hence references to “reliability” in the legal texts quoted in the present paper should be interpreted as meaning “scientific validity”. The Court identified “the existence and maintenance of standards controlling [a] technique’s operation” (p. 594) as one of several indicia of scientific validity.

The National Research Council of the U.S. National Academy of Sciences’ 2009 report on forensic science [2] identified lack of standardization as a problem, and recommended (in its Recommendation 1) that an independent body be established that would, among other things:

• a.Establish and enforce best practices for forensic science professionals and laboratories;
• b.Establish standards for the mandatory accreditation of forensic science laboratories and the mandatory certification of forensic scientists.

In response, the U.S. Organization of Scientific Area Committees for Forensic Science (OSAC) was established [3] with the more limited mission of “strengthen[ing] the nation’s use of forensic science by facilitating the development of scientifically sound forensic science standards, and by promoting the adoption of those standards by the forensic science community” ([4] §1.1).

The Daubert ruling also identified “appropriate validation” ([1] p. 590) as an indicium of scientific validity: “a key question to be answered in determining whether a theory or technique is scientific knowledge that will assist the trier of fact will be whether it can be (and has been) tested …. ‘[T]he statements constituting a scientific explanation must be capable of empirical test’” ([1] p. 593). The National Research Council report was concerned that “Much forensic evidence … is introduced in criminal trials without any meaningful scientific validation” ([2] pp. 107–108), and the 2016 report by the U.S. President’s Council of Advisors on Science and Technology (PCAST) emphasized that “For forensic feature-comparison methods, establishing foundational validity based on empirical evidence is … a sine qua non. Nothing can substitute for it” ([5] p. 6). In its Recommendation 1, the PCAST report stated that “It is important that scientific evaluations … be conducted, on an ongoing basis, to assess the foundational validity of current and newly developed forensic feature-comparison technologies.”

The purpose of OSAC is clearly to improve the scientific validity of forensic practice, and we fully support this goal. OSAC has had some success, but we are concerned that there are instances in which its standards-development process is being subverted. We are concerned that some standards initially developed by OSAC are detrimental to the goal of improving the scientific validity of forensic practice. These standards are vacuous:

• •They usually state few requirements.
• •Their stated requirements are often vague.
• •Compliance with their stated requirements can be achieved with little effort – the bar is set very low.
• •Compliance with their stated requirements would not be sufficient to lead to scientifically valid results.

These vacuous standards do not reflect an increased consciousness in the forensic science community that forensic practice should have rigorous scientific foundations. They appear to be designed to allow laboratories and practitioners to continue with existing poor practice, and if challenged to be able to respond that they are following established standards.1 Even if this is not the intention, the result will still be the same. If standards allow the continuation of poor practice rather than require good practice, then “the existence and maintenance of standards” ([1] p. 594) is not an indicium of scientific validity. There is a danger, however, that a court may not look further than the fact that a standard exists, and be misled into believing that conformity to a vacuous standard is indicative of scientific validity, even though it is not. For this reason, it would be better to have no standard than to have a vacuous standard.

Two examples of vacuous standards are ANSI/ASB 030 “Standard for a quality assurance program in bloodstain pattern analysis” [10] and ANSI/ASB 072 “Standard for the validation of procedures in bloodstain pattern analysis” [11]. These standards were initially developed by OSAC, then further developed and published by the American Academy of Forensic Sciences Standards Board (ASB). ASB’s mission is “to provide accessible highest quality science based consensus forensic standards”.2 ASB was established in 2015, approximately one-to-two years after OSAC. ASB’s website states that it “will work closely with the Organization of Scientific Area Committees for Forensic Science (OSAC) and its subcommittees which are dedicated to creating a national registry of forensic standards”.3 The two example standards that we discuss are not intended to be a sample representative of OSAC’s or ASB’s output. The fact that they relate to bloodstain pattern analysis (BPA) is incidental – since method validation is black-box testing, methods across all branches of forensic science are in-principle validatable, and methods across all branches of forensic science should be validated, no branch of forensic science should be exempt. The choice of ANSI/ASB 030 and ANSI/ASB 072 is due to the fact that the present perspective paper is a revised and expanded version of responses to requests for comment on whether those standards should be added to the OSAC Registry of Standards.

ANSI/ASB 030 represents the ultimate in vacuous standards. The only requirements it contains are requirements for forensic science providers to have a series of written procedures. It leaves the content of those written procedures almost entirely to the discretion of each individual forensic science provider.

ANSI/ASB 072 is a particularly egregious example of a vacuous standard because it purports to be a standard on method validation,4 and thus appears to be designed to support an argument that testimony based on a method that has been validated using this standard should be admitted because the “appropriate validation” indicium of scientific validity has been satisfied.

The purpose of method validation is to empirically demonstrate the degree to which a method is fit for its intended purpose. OSAC’s guidance on developing validation standards states that “Methods shall be evaluated to determine whether they work as intended and are fit for purpose” ([12] §10).

ANSI/ASB 072’s stated requirements are extremely vague. The standard leaves so much of the validation procedures and the choice of performance characteristics to assess to each individual forensic science provider that there is a danger that lack of consistency will undermine trust in validated methods. BPA includes a wide range of methods addressing different questions, for example: What was the angle of impact of the blood that caused the stain? From what location did the blood originate? What mechanism caused the observed bloodstain pattern, e.g., blood dripping from a wound, castoff from a knife stab, contact of clothing with a bloody surface? ANSI/ASB 072 does not focus on how to validate specific methods within BPA. It does not contain any requirements detailing what must be done so that the validations conducted would themselves be fit for purpose. It does not contain any requirements that would result in validation reports that would provide the information necessary to determine “whether the reasoning or methodology underlying the testimony is scientifically valid and … whether that reasoning or methodology properly can be applied to the facts in issue” (Daubert [1] pp. 592–593), or whether “(c) the testimony is the product of reliable principles and methods; and (d) the expert has reliably applied the principles and methods to the facts of the case” (FRE 702, as amended 2011).5

For a validation standard in forensic science to be fit for purpose, i.e., ultimately to assist a court to decide whether it can trust results generated by a validated method, we propose that the standard should include the requirements given in the numbered list below. The list is not intended to be exhaustive, but to be indicative of requirements that we believe are essential. In compiling the list, we drew on the PCAST report [5], on the England & Wales Forensic Science Regulator’s guidance on validation [15], and on the Australia & New Zealand National Institute of Forensic Science’s guideline on empirical study design [16]. A validation standard specific to a particular branch of forensic science should be more concrete because it should describe the requirements in the context of methods implemented to address specific questions in that particular branch of forensic science.

In the list of requirements below, each item in the first level of numbering states a requirement. The items in the second level are informational.

• 1.A method that is used to produce results that will (potentially) be proffered as evidence in court shall be validated prior to use.
• 2.
  The method as a whole shall be validated.

  • 2.1.
    The method could include multiple components (e.g., data, devices, procedures, processes, statistical models) that are used to derive the results. Validation of component parts alone is not sufficient.
• 3.
  The method shall be validated using test data that reflect the conditions of the particular case under consideration.

  • 3.1.
    This could be achieved via anticipatory validation: The method is validated using data that reflect anticipated casework conditions. For each new case an assessment is made as to whether its conditions are sufficiently similar to the conditions of an existing validation study, and, if so, the appropriate validation report is selected.
  • 3.2.
    Alternatively, it could be achieved via case-by-case validation: If the conditions of a new case are not sufficiently similar to the conditions of an existing validation study, a new validation study is conducted using data that reflect the conditions of the new case.
• 4.
  The validation study shall use a sufficient number of test trials to support the anticipated or ultimately claimed level of performance of the method.

  • 4.1.
    For example, assuming a method that produces either positive or negative results, if out of 1000 truly negative test trials it produces 100 false positives, it would be reasonable to claim that the false-positive rate is about 10%, but it would not be reasonable to make the same claim based on 1 false positive from a total of 10 truly negative test trials. Mutatis mutandis for the false-negative rate.6
• 5.If the results depend on the individual practitioner implementing the method, the implementation of the method by the practitioner shall be validated.
• 6.If the results do not depend on the individual practitioner implementing the method as long as the practitioner follows a prescribed protocol, the ability of the practitioner to follow the protocol shall be assessed via competency testing.
• 7.The validation shall be constructed in such a way that the practitioner implementing the method cannot know and (other than by implementing the method) cannot infer the truth-value for each test trial, nor the proportions of different truth-values within the test set.
• 8.The acceptance criteria for the validation study shall be pre-defined, and should not be known by the practitioner implementing the method.

ANSI/ASB 072 does not include any of these requirements,7 hence we argue that it is not fit for purpose.

Our concerns are not specific to BPA, but with the precedent that the publication of ANSI/ASB 030 and ANSI/ASB 072 set for the development of standards across all branches of forensic science, and particularly for the development of method-validation standards. All forensic-science standards developed by OSAC, ASB, or other organizations, should be carefully assessed with respect to whether they are fit for purpose.

OSAC is currently implementing a number of reforms of its structure and processes. The outcome of the reforms is branded “OSAC 2.0”. We are hopeful that these reforms will prevent future subversion of the OSAC standards-development process.

Disclaimer

All opinions expressed in the present paper are those of the authors as individuals. Unless explicitly stated otherwise, nothing in the present paper should be construed as representing the policies or positions of any organizations with which the authors are associated.

CRediT authorship contribution statement

Geoffrey Stewart Morrison: Conceptualization, Writing - original draft, Writing - review & editing. Cedric Neumann: Conceptualization, Writing - review & editing. Patrick Henry Geoghegan: Conceptualization, Writing - review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.