Collaborative versus traditional method validation approach: Discussion and business case

Abstract

For accredited crime laboratories and other Forensic Science Service Providers (FSSPs) performing a method validation can be a time consuming and laborious process, particularly when performed independently by an individual FSSP. In this proposed collaborative method validation model, FSSPs performing the same task using the same technology are encouraged to work together cooperatively to permit standardization and sharing of common methodology to increase efficiency for conducting validations and implementation. FSSPs following applicable standards that are early to validate a method incorporating a new technology, platform, kit, or reagents are encouraged to publish their work in a recognized peer reviewed journal. Publication of validation data provides communication of technological improvements and allows reviews by others that supports the establishment of validity. It also permits other FSSPs to conduct a much more abbreviated method validation, a verification, if they adhere strictly to the method parameters provided in the publication by the original FSSP. By completing this verification, the second FSSP has reviewed and accepts the original published data and findings, thereby eliminating significant method development work. Utilization of published validation data increases efficiency through shared experiences and provides a cross check of original validity to benchmarks established by the originating FSSP. Utilization of the same method and same parameter set enables direct cross comparison of data and ongoing improvements. A business case will be provided to demonstrate the cost savings of the collaborative validation model using salary, sample and opportunity cost bases.

1. Background

The legal system requires use of scientific methods that are broadly accepted in the scientific community, applying a Frye or Daubert standard to ensure that methods and the results they produce are reliable [1]. FSSPs are essentially applied scientists, applying validated methods to new samples. While the circumstances of a crime are unique, samples typically occur in a normal range. Testing must be applied to produce sound results which will guide the finder of fact, the judge or jury, to enable them to properly evaluate the weight given to the evidence.

FSSPs do not create instrumentation, but rather work with existing manufacturers, particularly those who have supplied instruments that purport to have application to forensic samples. In some applications, reagents are also supplied that are specific to the forensic purpose and may include appropriate quality control materials. This increases the value proposition to FSSPs as they save time in preparing chemicals and developing their own quality assurance. Vendors have a vested interest in ensuring their instruments and reagents are fit for purpose, so they can be investigated, adopted and used by FSSPs. In many instances, however, vendors do not possess the same level of forensic experience as that found in FSSPs and therefore additional work is frequently needed to develop the specific methods for application to forensic samples.

Technology is ever increasing in capability, complexity, sensitivity and cost. FSSPs are in constant pursuit of best practices, seeking to maximize the value of their evidence. Improvements can take place in terms of speed, sensitivity, specificity, ease of use and a number of other parameters which make the technology a potential improvement to existing forensic methods. FSSPs face difficult decisions in which new technologies to invest the time and energy to investigate, let alone purchase and implement. The primary goal of FSSPs is to work cases submitted; however, casework is not being completed with those resources that are applied to researching new methods or instrumentation [2]. Everything that is not casework, comes at the expense of casework. Therefore, FSSPs must choose very carefully and frugally where to best invest precious resources.

Validation of methods is an essential component of accreditation. All methods must be fit for purpose, scientifically adding evidential value to the evidence found at a scene while conserving sample for future analyses. Validation must be completed prior to use of the method on evidence submitted to the FSSP. Depending on the FSSP’s scope of accreditation, there may be additional steps required by the accreditation body prior to implementation of the method. FSSPs must meet or exceed the requirements included in accreditation standards, or they will not be in compliance and issued a nonconformity when audited. Any finding of nonconformance must be resolved in a timely manner to maintain accreditation.

2. Proposal

FSSPs spend a great amount of time and resources performing method validation. Currently, each FSSP tailors a validation to their own needs, frequently modifying parameters and changing procedures prior to completing the validation. The result is 409 US FSSPs [3] each performing similar techniques with minor differences. This is a tremendous waste of resources in redundancy, but also is missing the opportunity to combine talents and share best practices among FSSPs. Coordination of efforts enabled through publication and dissemination of model validations has the potential to reduce the overall burden of validation in the field of forensic science and streamline audit processes to the most updated standards (e.g. QAS audits), while simultaneously recognizing originating FSSPs, elevating scientific methods, promoting best practices and increasing information sharing.

Collaboration of FSSPs conducting method validation will save significant effort and permit more efficient sharing of best practices. FSSPs performing method validation on new techniques who share and publish their work enable other FSSPs to significantly reduce or eliminate the time required to develop specific techniques and parameters. FSSPs who choose to adopt exact instrumentation, procedures, reagents and parameters of the originating FSSP are able to move directly toward verification, thereby streamlining their implementation of new technology and improvements dramatically.

Originating FSSPs are encouraged to plan method validations with the goal to share their data, via publication, from the onset. This includes both method development information and their organization’s validation data. Well designed, robust method validation protocols that incorporate relevant published standards (e.g. published by standards development organizations; OSAC and SWGDAM) should be used. This strategy will ensure that all FSSPs are risen to the highest standard in one fell swoop, meeting or exceeding standards for accreditation and best practices.

FSSPs have a number of common attributes which lend themselves to the collaborative validation model proposed. While there are many differences between jurisdictions, FSSPs examine common evidence types using similar technologies and methods. The vast majority of FSSPs are governmental [3], and therefore lack competitive forces to inhibit sharing of procedures and best practices. Data sets as well as samples can be shared, which reduces the number of samples necessary to assess instrument and method performance. Many FSSPs are small in size and have limitations on the time and resources that can be applied to method validation. With the ability to share expertise from larger entities forging the way, smaller FSSPs are enabled by reducing the activation energy to acquire and implement technology, provided they adhere to the same set of parameters and written method. The concept of validation by one FSSP and subsequent verification by other FSSPs is supported in requirements used as the basis for accreditation (e.g., ISO/IEC 17025) and is thereby acceptable practice [4].

Journals supporting the publication of forensic validations should be approached to assist with this effort. Forensic Science International: Synergy and Forensic Science International: Reports are amenable to this initiative, as well as providing open access format to ensure broad dissemination of documentation. Granting organizations are an avenue to cover open access fees. Through the publication process, method validations will be available for the forensic community to adopt and emulate. Publications can be listed and linked by forensic organizations to assist in disseminating method validations, as well as contact information for individuals who can serve as resources.

An added benefit to mirroring a validation completed by a previous FSSP is that there is data available for comparison. If a FSSP independently conducts its own unique validation, there is no benchmark to ensure that results are optimized. Emulation of a previous validation provides an inter-FSSP study which adds to the total body of knowledge using specific methods and parameters, which supports all FSSPs using that technology. FSSPs following the published validation are strongly encouraged to join a working group to share results and monitor parameters to optimize their direct cross comparability with other FSSPs. In addition to the original method validation, results could be published regarding method performance and process improvements over time. Publication, presentation, and various other options of sharing are available including direct collaboration.

Collaboration need not be limited to other FSSPs. Many educational institutions have forensic programs, many offering graduate degrees. With the thesis requirement to complete a graduate program, what better way to conduct relevant research than to work on a method validation? Accreditation requirements restrict access to forensic casework to those individuals completing specialized training and competency testing, thereby precluding students from actively working on cases or case samples. Validation studies require FSSPs to use samples that mimic evidence, thereby creating the ideal sample set for student experience in generating data for evaluation and perfection of protocols, as well as conducting components of the validation. This level of practical experience will be of great value to the student and their future employability, which will be further accentuated by publication of the validation data. FSSPs will also benefit by engaging students already knowledgeable in new technology application. This model is currently employed by the New York State Police Crime Laboratory System with both the University at Albany State University of New York and The University of Illinois at Chicago.

A number of vendors and contractors provide a professional validation service. As these scientists performing validations see data and obtain experience from multiple sites, they bring that experience to each new FSSP they work with, effectively eliminating unnecessary method modification by transporting a refined method between FSSPs. In the time they spend on site they provide targeted participation for the FSSP scientist. Once the validation is complete, a consistent training package is provided to meet training and competency requirements. Unfortunately, the cost for such services is frequently not included in the instrument and reagent costs and is therefore a limiting factor for many FSSPs to employ such specialists. An opportunity exists here to leverage private and governmental resources to facilitate new technology en masse to FSSPs in a more cost-effective manner than the current ad hoc approach.

3. Method validation

Validation is the provision of objective evidence that the method performance is adequate for intended use and meets specified requirements [4]. Validation of a forensic method demonstrates that the results produced by the method are reliable and fit for purpose, supporting admissibility in the legal system. In this collaborative method validation proposal, the latest applicable standards published by standards developing organizations are incorporated into the validation procedure, thereby increasing the value proposition for the new FSSB to adopt previously validated techniques. Strengths and limitations of the technique are identified, and parameters for data interpretation reporting of results are established, therefore risk is decreased for FSSPs considering technology upgrades.

Validation of methods for forensic applications can be broken into three phases. All three can be conducted by a single organization or by different organizations, however all must be completed to support a method as being fit for purpose. Phase One (Developmental Validation) is typically performed at a very high level, often with general procedures and proof of concept. It is frequently conducted by research scientists (e.g., DNA loci can be used to individualize people, chromatography can be used to separate components in a mixture, the mass spectrum of a compound is unique if measured with sufficient sensitivity) and often migrated from non-forensic applications. Publication of this material in a peer reviewed journal is common. The FBI Quality Assurance Standards refers to this phase as Developmental Validation and requires publication [5].

Phase Two (Method Development) begins with the transition to the FSSP seeking to evaluate and implement the technology. There is frequently a significant gap to fill between the high-level Phase One developmental validation and the detailed method that will be used daily on casework evidence. Phase Two of method validation often stretches out for many months as the nuances of the technique are discovered specific to the forensic application. The process of method development works toward a procedure which provides results that answers the scientific question posed, in the forensic enterprise typically related to identification and comparison of evidentiary materials. Once the technology has been determined to perform as needed, parameters set, and documented procedures (instructions) have been finalized, only then can the third phase of validation be performed to demonstrate it is fit for purpose in the hands of the organization’s staff. Variation is introduced into specific parameters and procedures when written by different FSSPs. This variation introduced by a lack of collaboration not only limits sharing of resources, it limits sharing of data to identify and develop best practices.

The Organization of Scientific Area Committees (OSAC) for Forensic Science has identified that method development components [6], as applicable, typically include, but may not be limited to:

• •purpose of the method
• •expected outcome (e.g., reduced noise in an audio signal or concurrent amplification of DNA loci)
• •pertinent literature references
• •suitable sample type(s)
• •necessary instruments, software, and other equipment
• •expected or acceptable operating parameters for equipment and instruments
• •metrological traceability, if applicable
• •calibration model and range, if applicable
• •expected limit of detection (LOD), if applicable
• •expected limit of quantification (LOQ), if applicable
• •
  specific steps on how to perform the method

  • o
    sampling protocol, if applicable
  • o
    required reagents (including volumes and formulations), if applicable
  • o
    required consumables, if applicable
• •
  steps to minimize or mitigate cognitive bias, if any

  • o
    description of the information that will be available at the time that the method is performed
• •steps to minimize or mitigate potential contamination
• •
  interpretation of the results

  • o
    the statistical model to be used, if applicable
  • o
    evaluation of the variation in the relative frequency of characteristics across different reference populations, if available
• •health and safety concerns, if applicable

Phase Three (Validation) is performed using the parameters set and the documented procedure established in phase Two Method Development. The appropriate sample number and type are used to establish the capability of the technique in the hands of the organization’s staff. Samples akin to those seen in actual casework are required, with sufficient replicate data to demonstrate accuracy (precision and bias). In this collaborative model, validations are designed from the onset to have data sets available for publication, as well as sharing of data, parameters and procedures for FSSPs considering adopting the validation. Considerations include use of data that is anonymized (e.g. DNA and latent prints from FSSP employees and their families) to protect individuals from identification.

The OSAC for Forensic Science has identified Validation components [6], as applicable, typically include, but may not be limited to:

• •
  robustness of criteria established for assessing the suitability of evidence for analysis:

  • o
    repeatability (consistency of results obtained by a single person using the same instrument or process)
  • o
    reproducibility (consistency of results obtained across staff within a forensic service provider or by different forensic service providers)
• •
  method performance:

  • o
    sensitivity:

    • ⁃
      true positive probability
    • ⁃
      limit of detection (LOD)
    • ⁃
      limit of quantitation (LOQ) [if reported]
  • o
    specificity

    • ⁃
      true negative probability
  • o
    quantification of bias
  • o
    quantification of precision

    • ⁃
      repeatability
    • ⁃
      reproducibility
  • o
    evaluation/estimation of measurement uncertainty [in quantitation]
• •
  robustness of data interpretation steps:

  • o
    repeatability
  • o
    reproducibility

Validation of non-instrumental, comparison-based methods is more challenging due to the increased samples necessary to adequately evaluate the impact of individual knowledge, skills and aptitude on method performance and robustness.

The evaluation and summary of the validation data determines necessary on-going quality control/quality assurance as well as limitations to report statements and testimony. The FBI QAS refers to Phase Two and Three as Internal Validation [5], as these are typically combined and performed by one FSSP or crime laboratory system.

4. Verification

Verification is the provision of objective evidence that the method performs at the same stated level of performance as that in the validation, as originally performed by the primary FSSP. The components of Phase Three Validation above would be reviewed to identify those that are susceptible to variation when the method is used by a different FSSP. Depending on the method, verification may need to cover all aspects of the Phase Three Validation but can generally do so with a smaller sample size. Time is significantly reduced for adopter FSSPs, as Phase Two is eliminated and Phase Three can be simplified with acceptance of the data from the collaborating primary validating FSSP.

An example of the verification process is the New York State Police (NYSP) Crime Laboratory System’s installation of new GC-MS instruments. The main laboratory, known as the Forensic Investigation Center (FIC), conducted phases two and three of validation to demonstrate the new instruments were fit for purpose, replacing much older, outdated instruments. The manufacturer provided parameters for drug identification, however the laboratory needed to develop a specific procedure and demonstrate it was fit for purpose in the hands of their staff. While procedures were in place from previous instruments, the differences between instrument models were sufficient that methods from the previous model needed to be tested and updated. As a result, a new validation was conducted. Many replicates were run to establish sensitivity, repeatability and reproducibility and demonstrate the method was robust. Completion of Phase Two (Method Development) and Phase Three (Validation) at the FIC took the equivalent of over six months of a senior scientist’s time.

The NYSP Crime Laboratory System includes three satellite FSSPs which utilize the same methods established by the FIC. With one centralized set of QA documents, methods and parameters, all could be shared to enable a Verification at the other locations. A targeted set of the same samples could be replicated in each of the satellite FSSPs with their identical instruments, demonstrating the technique was valid in their hands. This verification was completed within two months, with data produced consistent with that of the original studies performed by the FIC.

The NYSP Crime Laboratory Drug Section validation of their new GC-MS instruments serves as an example of the collaborative validation model, with a notable exception that the sharing of methods and data occurred within the same agency. The proposed model is for independent FSSPs to conduct the same level of cooperation, with the second FSSP mirroring the same acceptance of method and data, the only difference being differing jurisdictions. Therein lies the beauty of science; it knows no bounds between jurisdictional distinctions made by man. The model is transportable; again, provided the same instrument, method and parameters are applied.

5. Modification

A central premise of the collaborative validation proposal is that the adopting FSSP will make no modification to the method as validated. However, the second FSSP may desire to modify the method, either prior to or after initial validation completion. A modified method would be any deviation from the stated platform and parameters, and therefore require additional study. Each modification is typically unique; therefore, each FSSP must assess how different the change is from the original validation. Modifications return the validation to Phase Two Method Development. The more different, the more work required to demonstrate that the original method development and internal validation data are still applicable. The additional validation required will be directly proportional to the magnitude of the difference between the original and the modified method.

An example of a modification will be provided for Drug Chemistry, using a GC-MS to identify controlled dangerous substances. A FSSP wishes to use a different manufacturer to utilize a generic versus the instrument manufacturer brand column of the same specification. While the FSSP might hope that the same specifications for a column would provide the same results, appropriate samples must be run to provide objective data to demonstrate their current method is supported, or perhaps a minor method change would be required to enact the change. Either way, data is necessary to support a modification for the different manufacturer’s column.

Another example of a modification would be a FSSP changing the quality of a reagent, moving between reagent grade, laboratory grade and technical grade, reagent grade being the most pure and technical grade being the least pure. With a change in reagent grade there would be a difference in the level of impurities, which may or may not have a significant impact on the data. As in the column manufacturer change example, appropriate samples would need to be run to demonstrate the level of impact of the reagent change, along with appropriate method modifications to address wording related to this reagent.

An example of a modification in keeping with the Drug Chemistry Section GC-MS column example, would be a column length change. Depending on the impact the different column made on data for the specific application, the impact may be none or significant. The retention time of appropriate drugs across the spectrum of drugs tested by the method would need to be checked. Each FSSP would need to evaluate the significance of the impact of this modification to the method and design a Phase Three Validation appropriate to fit the need.

6. Accreditation

Accreditation is a third-party attestation that conveys formal demonstration of the FSSP’s competence to carry out the services listed on the scope of accreditation [7]. This includes the required competence to conduct method validation, any or all phases, and method verification. As mentioned in the Validation Section, validation can be broken into three phases. All three can be conducted by a single organization or all three can be conducted by different organizations but all three must be completed to support a method as being fit for purpose.

If the accredited FSSP will be using an external service provider to perform aspects of the validation, the accredited FSSP will be required to conform to requirements for selection of this vendor and to have a copy of those records. As noted, Phase One proof of concept developmental validation is typically published. This article promotes the original FSSP’s publication of Phase Two and Phase Three, which in turn can be emulated by subsequent FSSPs.

The following is an excerpt from the FBI QAS for Accreditation, used to illustrate requirements for developmental and internal validation for casework DNA laboratories:

Accreditation – FBI QAS (DNA Casework) [5].

• •Effective July 2020
• •
  8.2.1 Developmental Validation:

  • •
    Characterization of genetic marker
  • •
    Species specificity
  • •
    Sensitivity studies
  • •
    Stability studies
  • •
    Case-type samples
  • •
    Population studies
  • •
    Mixture studies
  • •
    Precision and accuracy studies
  • •
    PCR-based studies to include reaction conditions and assessment of differential and preferential amplification
• •
  Developmental Validation (continued):

  • •
    PCR-based studies to include:

    • •
      Reaction conditions
    • •
      Assessment of differential and preferential amplification
    • •
      Effects of multiplexing
    • •
      Assessment of appropriate controls
    • •
      Product detection studies
• •Are peer-reviewed publication(s) of the underlying scientific principle(s) of a method available?
• •
  Except as provided in Standard 8.3.1.1 [multi-laboratory systems], have internal validation of all manual and robotic methods been conducted by each laboratory?

  • •
    a. Were the appropriate sample number and type to demonstrate the reliability and potential limitations of the method used?

Internal Validation (8.3.1).

• •
  Have internal validation studies included, as applicable:

  • •
    1. Known and non-probative evidence samples or mock evidence samples? Yes No N/A
  • •
    2. Precision and Accuracy studies? Yes No N/A
  • •
    3. Sensitivity and stochastic studies? Yes No N/A
  • •
    4. Mixture studies? Yes No N/A
  • •
    5. Contamination assessment studies? Yes No N/A

Multi-laboratory Systems (8.3.1.1).

• •8.3.1.1 Internal validation data may be shared by all locations in a multi-laboratory system.
• •
  8.3.1.1 For multi-laboratory systems:

  • •
    a. Are the summaries of all shared validation data available at each site?
  • •
    b. Has each laboratory in a multi-laboratory system completed, documented, and maintained applicable site-specific studies:

    • •
      1. Precision studies? Yes No N/A
    • •
      2. Sensitivity studies? Yes No N/A
    • •
      3. Contamination assessment studies? Yes No N/A

The following is an excerpt of ISOIEC 17025:2017 Validation Section [4]:

7.2.2Validation of methods

7.2.2.1 The laboratory shall validate non-standard methods, laboratory-developed methods and standard methods used outside their intended scope or otherwise modified. The validation shall be as extensive as is necessary to meet the needs of the given application or field of application.

NOTE 1 Validation can include procedures for sampling, handling and transportation of test or calibration items.

NOTE 2 The techniques used for method validation can be one of, or a combination of, the following:

• a)calibration or evaluation of bias and precision using reference standards or reference materials;
• b)systematic assessment of the factors influencing the result;
• c)testing method robustness through variation of controlled parameters, such as incubator temperature, volume dispensed;
• d)comparison of results achieved with other validated methods;
• e)interlaboratory comparisons;
• f)evaluation of measurement uncertainty of the results based on an understanding of the theoretical principles of the method and practical experience of the performance of the sampling or test method.

7.2.2.2 When changes are made to a validated method, the influence of such changes shall be determined and where they are found to affect the original validation, a new method validation shall be performed.

7.2.2.3 The performance characteristics of validated methods, as assessed for the intended use, shall be relevant to the customers’ needs and consistent with specified requirements.

NOTE Performance characteristics can include, but are not limited to, measurement range, accuracy, measurement uncertainty of the results, limit of detection, limit of quantification, selectivity of the method, linearity, repeatability or reproducibility, robustness against external influences or cross-sensitivity against interference from the matrix of the sample or test object, and bias.

7.2.2.4 The laboratory shall retain the following records of validation:

• •a) the validation procedure used;
• •b) specification of the requirements;
• •c) determination of the performance characteristics of the method;
• •d) results obtained;
• •e) a statement on the validity of the method, detailing its fitness for the intended use.

In this model, a FSSP must keep documentation which demonstrates the acceptance of all parameters and data in addition to the FSSP’s own validation data. This may be as simple as a technically responsible individual conducting a documented review of the data from the original FSSP conducting the validation, and an acknowledgement that the data is sound and is accepted to act as a foundation for the second FSSP to base their validation studies upon. A copy of the original data, review and acknowledgement would serve as a record to demonstrate compliance with accreditation requirements.

7. Business case

A business case will be presented in three formats to compare and contrast the traditional to the proposed collaborative approach to conducting method validation. Elimination of the method development component provides for a more concise Phase Three validation study, resulting in potential cost savings. Costs from each approach will be used to demonstrate management choices for optimal allocation of resources, be it for casework or validation, which will permit improvements to the casework it impacts. It can be argued that time must be taken away from chopping wood to sharpen one’s axe, or better, to bring a new chainsaw. Hence, the discussion is not whether to use technology to improve forensic methods, but rather what is the best mechanism to conduct method validation to make improvements more efficiently and effectively?

A modest assumption obtained through the authors’ personal experience as noted in the Drug Chemistry GC-MS upgrade above and also an informal interview of several FSSPs provided that a senior analyst working full time would take six months to complete a validation, including both Phase Two Method Development and Phase Three Validation. With collaborative validation eliminating the Phase Two Method Development, and use of the same approach for Phase Three Validation and verification to be mirrored in both traditional and collaborative approaches: all concurred that two months would be a reasonable time for completion, and that a less experienced scientist could complete the work. Three models will look at the impact to salary savings, sample cost and impact to casework productivity.

For the first model, a comparison will be provided between the traditional validation model taking 6 months of senior analyst time versus the collaborative model consuming 2 months of experienced analyst time for completion (see Table 1). Generalized salaries are provided based on New York State salary scales, including a very general estimated fringe benefit. Assuming $90,000 annual salary with 25% fringe benefit for additional personnel related costs such as unemployment insurance, healthcare benefits, etc., 6 month of traditional validation costs approximately $56,250.

Table 1.

Cost savings: salary cost basis.

Validation Type	Analyst Type	Salary	Fringe (25%)	Project Time (years)	Total Labor Cost
Traditional	Senior	$90,000	$22,500	0.5	$56,250
Collaboration	Experienced	$70,000	$17,500	0.17	$14,583
Delta		$20,000	$5,000	0.33	$41,667

Using the collaborative model of validation where a previous FSSP has completed Phase Two and Phase Three permitting a verification, 2 months of experienced analyst time is estimated for completion. Assuming an annual salary of $70,000 per year with 25% fringe benefits, the estimated cost for 2 months of collaborative validation is $14,583. This equates to a $41,667 savings based on salary costs alone. As a percentage savings, the collaborative approach saves 385%, or provides a savings of $3.85 per $1 spent.

Using a second model based on sample costs, similar saving can be demonstrated. Based on the authors’ experience and an informal polling of experienced FSSPs, the sample number of 100 was selected for the traditional Phase Three internal validation versus 20 samples for the collaborative model verification. While 20 and 100 samples are somewhat arbitrary selections based on experience, there are some references providing guidance on sample replicates in method validation. After running 5–10 replicate samples for a particular experiment, John Butler of NIST concluded there are diminishing returns to adding additional results [8]. The 1996 National Research Council report The Evaluation of Forensic DNA Evidence requires at least five observations of an allele when establishing a minimum allele frequency [9], and the National DNA Index System (NDIS) expert system validation requirements involve the observation of at least five challenge events for each issue such as stutter, spikes, etc. [10]. This provides some basis for selecting 20 and 100 samples respectively, to account for testing different analysts, instruments and parameters, as well as adding control samples.

To calculate the potential cost savings between the traditional and collaborative validation approach, cost per sample used was $389.50 per DNA sample provided from the Wickenheiser study [11]. Under the sample cost basis model, traditional validation costs $38,950 while the collaborative model costs $7,790, representing a $31,160 savings (see Table 2). As a percentage savings, the collaborative approach saves 500%, or provides a savings of $5.00 per $1 spent.

Table 2.

Cost savings: sample cost basis.

Validation Type	Sample Number	Cost per sample	Total Cost/sample
Traditional	100	$389.50	$38,950.00
Collaboration	20	$389.50	$7,790.00
Delta	80	$389.50	$31,160.00

Utilizing a third model, the opportunity cost of conducting method validation versus forensic casework will contrast traditional versus collaborative method validation. There is some rationale favoring the opportunity cost model for calculating savings, as employers retain staff because the value of the service they provide is greater than their cost, hence supporting their employment by adding value. Using the 6 months traditional method validation time versus the 2-month collaborative verification time estimate, a delta of 4 months occurs between the two validation approaches. Rather than look at the cost savings based on examiner salary or sample costs, the value of the activity of the forensic analyst will be examined in terms of the cost savings of lost casework in society crime costs.

Maximization of the value of evidence has a time component. Evidence is perishable. A result today is worth more to investigations than a result in a month. Not only does evidence quality itself decay over time, but so too does its worth to an investigation, which is spending resources on other avenues of investigation to help solve a case while awaiting forensic results. Furthermore, recidivist offenders also have greater time to perpetrate additional offenses on new victims.

The “Forensic Fire-station” model articulates this loss of value in a visual manner, as investigative resources are burning while a backlog of “casework fires” exists at the FSSP [12]. Imagine calling a fire-station to inform them a house is on fire, only to be told there is a backlog of fires and the firefighters will be there in three days. Criminal investigative resources are front end loaded, with greater resources being spent to solve a crime closer to when the crime was committed. Solving crimes more quickly adds value by enabling resources to be focused on the best technology, rather than spend time in wait while resources are consumed using less optimal and less objective methods.

To illustrate the opportunity cost, or what alternative more valuable activity the asset would be producing, the lost opportunity is examined. As the primary outputs of a FSSP are casework, reports and testimony; and reports and testimony are predicated on the casework analysis being done, less cases equals less crimes solved. This is particularly true when the cases are no-suspect cases where the forensic result provides investigative aid. Therefore, the opportunity cost of conducting additional work on method validation that could be shortened will be used to illustrate the cost of removal of those resources from the FSSPs’ primary mission of casework. Shorter method validation means more casework accomplished and more crime solved.

Using forensic DNA analyst output obtained in the Wickenheiser study [11], the average analyst produces approximately 96 DNA cases annually. Therefore, in the 4-months of analyst time saved by using collaborative versus traditional method validation, analysis of 32 cases is the opportunity cost of this choice (see Table 3).

Table 3.

Cost savings: opportunity cost.

Validation Type	Project Time (years)	Case/year output	Total case output
Traditional	0.5	96	48
Collaboration	0.17	96	16
Delta	0.33		32

With 32 analyzed cases foregone by selecting the traditional model for method validation versus the collaborative model, next the cost of this choice must be calculated. Between July 25, 2018 and January 31, 2020, New York State Police Biological Sciences Section analyzed 886 sexual assault cases, producing 256 CODIS eligible profiles, which equates to obtaining a CODIS profile in 28.9% of cases analyzed. As of May 01, 2020, those 256 profiles generated 105 informative hits, where a new offender, arrested individual or another crime scene were matched to provide an investigative lead. This computes to a 41.0% hit rate for CODIS profile entries and an overall 11.8% hit rate on the total cases analyzed.

While these numbers represent a snapshot in time and the number of hits will increase as additional offenders and arrestees are added to the database, it provides a reasonable estimate of the likelihood of providing valuable investigative information on the 32 cases that would be foregone to conduct traditional versus collaborative validation. Using the NYSP Biological Sciences Section experience, analyzing 32 cases with a 28.9% CODIS profile success rate would generate approximately 9 CODIS profiles. With a 41.0% hit rate for CODIS profiles, approximately 3.7 hits would occur. While a fraction of hits would not occur in life, rather than round up to 4 hits, the number 3.7 hits will be used to calculate the opportunity cost of forgoing analysis on 32 cases.

An extensive study conducted by the U.S. Department of Justice Bureau of Justice Statistics estimated the cost of a single instance of sexual assault at over $87,000 in 1995 dollars [13]. This study focused only on the victims’ costs without also estimating the cost to society. Therefore, costs to the criminal justice system and other social costs associated with the fear of crime, and private security expenditures are not included in this figure. Another study estimated the cost of each sexual assault ranged from $111,238 with the potential to save 7 additional crimes with each database hit [11]. Most recently, estimates for victim related cost rise to $435,419 when including a more comprehensive estimate of damage [14]. This calculation does not account for the potential for a CODIS hit to not only solve crime, but to prevent future offenses. The mean number of sexual assaults per offender is 7.1 per year over a 7-year span of an offender’s active career [15]. Combining with the crime prevention opportunity, the match of each sexual assault has the potential to avert 26.22 sexual assaults that would have been committed [14].

Total opportunity cost of foregoing the analysis of 32 cases ($435,419 × 3.7) = $1,611,050. The return on investment of using one analyst to conduct casework versus conduct method validation is $1,611,050/$41,667 labor costs = 3866% or $38.66 for every dollar spent on labor. The opportunity cost demonstrates that collaborative method validation is extremely well supported to reduce analyst time and effort.

If, however the recidivist factor of 26.22 is added to account for the potential of each CODIS hit to prevent future crime, the potential cost savings increases to ($435,419 × 3.7 X 26.22) = $42.2 Million. The return on investment is $42.2 Million/$41,667 labor costs = $1,013 for every dollar spent on labor. The prevention of future crime provides compelling rationale for minimizing method validation time in favor or conducting forensic analysis.

A more conservative estimate of the per-victim cost and total lifetime economic burden of rape is $122,461 [16]. Using this lower estimate which does not include the higher cost location of Los Angeles and the “willingness to pay” factor utilized in the Wang and Wein study, the opportunity cost of foregoing the analysis of 32 sexual assault cases is ($122,461 × 3.7) = $453,105. With the lower cost estimate, the return on investment of using one analyst to conduct casework versus conduct method validation is ($453,105/$41,667 labor costs) = 1087% or $10.87 for every dollar spent on labor. If the 26.22 recidivist factor is added, the potential savings increases to ($122,461 × 3.7 x 26.22) = $11.8 Million. The return on investment is $11.8 Million/$41,667 labor costs = $285 for every dollar spent on labor. Therefore, depending on the cost of crime model utilized, the savings of crime per victim ranges from $10.87 to $38.66 per one dollar expended, and jumps to a range of $285 to $1,013 per dollar when recidivism is considered.

In approximately 8 out of 10 sexual assault cases, the assailant is someone known to the crime survivor [17]. While these sexual assault cases may not be stranger rapes per se, if a CODIS profile is obtained and entered into the databank, the profile may link to cases where the same assailant was not known to that survivor, thereby assisting in solving other sexual assault cases. Hence, there is value in preventing the cost of future sexual assaults by placing profiles in NDIS, regardless of whether the assailant is known or a stranger to the survivor.

Besides demonstrating the value of conducting collaborative versus traditional method validation, this dramatic savings demonstrates the need to allocate resources expeditiously to conduct all cases as soon as they are committed, versus cases sitting in backlog waiting for analysis to commence. “Through accurately forecasting demand and establishing supply just above that demand, prompt response times are assured to reap the maximum benefits of forensic technology. No longer would there be a backlog of fires at the forensic fire station [12].”

8. Collaboration

By not making initial changes when a method is adopted, original implementation is streamlined. When improvements are seen as experience is gained, this can be shared with the group of colleague FSSPs using the same parameter set to encourage adoption of best practices. FSSPs are encouraged to collaborate with the primary FSSP as joint experience is gathered, and publish subsequent validation data to benefit the entire forensic community. Establishing and coordinating efforts through working groups enables the experience of one FSSP to become the experience of all FSSPs. This concept of “FSSPs helping FSSPs” supports the entirety of the forensic enterprise, rather than developing experience and expertise in silos.

Forensic organizations, that include, but are not limited to, the American Society of Crime Laboratory Directors (ASCLD), the American Academy of Forensic Sciences (AAFS), the Society of Forensic Toxicologists (SOFT), the Association of Firearm and Tool Mark Examiners (AFTE), the International Association for Identification (IAI) and the National Association of Medical Examiners (NAME), foster and facilitate collaboration, including sharing best practices, administrative policies, training and research [18]. The OSAC Implementation Plan includes the use of scientific peer mentorship via OSAC member external engagement to facilitate adoption of forensic standards including those related to method validation [19].

9. Result/conclusion

FSSPs early to conduct and publish method validation data not only supports establishment of scientific validity, but permits leveraging of the FSSPs’ procedures, parameters and data to assist other FSSPs to streamline their method validation by eliminating the method development step and reducing internal validation to a verification. FSSPs joining in a collaborative method validation benefit by following a well-established process, improving their success while saving resources. The larger forensic scientific community benefits from technique standardization, following well designed and well documented method validation procedures which are designed to meet the latest standards from the outset. FSSPs and their jurisdictions benefit by more rapid implementation, with the capability to share data between FSSPs, examine trends and track performance. Motivation to adopt this model is supported by a business case demonstrating significant time and cost savings, particularly when looking at the opportunity costs of foregone casework. Everything which is not casework takes away from casework, hence more effectively implementing new technology permits FSSBs to make best use of their resources and focus on their core mission of casework. Collaborative method validation is a valid means to meet accreditation requirements, capable of bringing new technology to forensic cases in a more transparent and efficient manner.

CRediT authorship contribution statement

Ray Wickenheiser: Conceptualization, Data curation, Formal analysis, Visualization, Writing - original draft, Writing - review & editing. Laurel Farrell: Conceptualization, Writing - review & editing.