Quality issue management and disclosure in forensic science: A survey of practice and perceptions

Abstract

Addressing calls for transparency regarding errors and limitations in forensic processes is an ongoing concern for the forensic science service provider community and the stakeholders it serves worldwide. Foundational to this goal is developing a consistent approach to the identification of issues that have, or could have, an impact on the quality and reliability of forensic results. A standardized approach to the classification of quality issues detected within forensic agency management systems may be the strategic key to supporting consistent identification and disclosure, along with enhancing a positive quality culture throughout forensic service providers and building understanding of “error” in forensic science with end users of forensic information. A survey of international forensic science service providers was conducted to gain deeper insights into current systems of issue identification, classification, management, and disclosure along with perceptions on quality issues, their use and communication by forensic agency staff. The survey results demonstrate that development of a standardized approach would be of significant value to the forensic science community and its stakeholders, with potential benefit not only to improved communication and use of quality issue data but also in advancing a positive culture of quality and credibility in forensic service provision to support justice outcomes.

Highlights.

• Data on quality issues can support benchmarking and understandings of “error” in forensic science.

• Survey results highlight current challenges to the management and disclosure of quality issues.

• A lack of standardized classification of issues makes comparison and benchmarking challenging.

• A negative quality culture in an agency impedes efforts to use quality issue data effectively.

• Standardized classification will support transparency, consistency, and positive quality culture.

1. INTRODUCTION

Errors affecting the reliability or validity of forensic results can have catastrophic consequences for the criminal justice system, victims and suspects of crime, and the wider community. Cases in both the media and literature demonstrate that the nature of such errors is wide and varied, from matters of scientific validity of methodologies [1], to errors influenced by human factors such as bias and competency [2, 3], to how results and information are communicated effectively to criminal justice decision‐makers [4]. Whilst calls for transparency with regards to error and limitations in forensic science have been prominent for decades [5, 6] the development of consistent practice in the communication and disclosure of issues affecting the validity of forensic results and, by extension, a collective understanding of “error” in forensic science and the impact on outcomes of the forensic process is a matter of ongoing discussion and research within the field [7, 8, 9].

It has been suggested that part of the solution may lie in the quality management systems already embedded in forensic science service provider agencies worldwide where the recording of issues detected within the forensic testing process is undertaken as standard practice [8]. The information held within these agencies represents a rich data source on the types of issues that are detected, their impact on the forensic process (both potential and actual) and the effectiveness of actions taken to correct the root cause of the problem [8]. Such events are known as “quality issues,” being issues detected within the quality management system that may have actual or potential impacts on the agency being able to meet its objectives (e.g., accurate results, timely delivery of service, etc.). A robust system of identification, classification, management, analysis, and correction of quality issues within an organization is key to organizational success, as demonstrated in examples from other high‐reliability fields where a culture of low tolerance for error and a reluctance to simplify cause analysis is essential to mitigating risks associated with increasing system complexity [10, 11].

While forensic science service provider agencies all over the globe record and retain this information within their quality management systems, examples of agencies publicly sharing this information are extremely rare [8]. A significant contributing factor to this situation may be the reluctance to share information on actual and perceived “errors” within the forensic agency and the negative attention this brings on individual analysts, agencies, and even entire disciplines. However, a compounding factor to this reluctance is the lack of standardized practice or language between forensic facilities around what is considered a “quality issue” and how it is classified, recorded, and managed [8, 12]. Without consistency in this area, comparison of data between agencies is extremely challenging and may invite unfair assessments about the quality (or perceived lack thereof) and associated reliability of results from one forensic service provider to another. Developing an accepted, standardized approach to the identification, classification, and recording of quality issues in forensic science may be the necessary foundation to support disclosure efforts for the purposes of accreditation, oversight, continuous improvement, and regulation [8]. Furthermore, standardized categorization and benchmarking of quality issue data may have significant benefits for forensic science service providers in recognizing both outlier issues requiring rectification within their own systems and the identification and monitoring of trends across providers that may indicate systemic issues to be addressed through collaborative research and improvement efforts [13, 14].

A deep understanding of current quality management practice and systems for the identification and management of quality issues in forensic science, along with the factors influencing variations between service providers, is the basis for the development of a practical system of standardization. Following the insights gained from a survey of forensic science service provider agencies in Australia and New Zealand to investigate systems for the identification, classification, management, disclosure and use of data related to quality issues detected in that region [12], a further survey of experts from international (predominantly US‐based) forensic science service provider agencies was conducted. The international survey also aimed to understand how quality issues were perceived within the agencies and the factors influencing this that may be relevant to the development and application of a standardized classification system. The resulting conclusions from both surveys will be used to inform the development of a classification tool for quality issues in forensic science, to be tested on real‐world examples for applicability and usability.

2. SURVEY METHODOLOGY AND PARTICIPANT DEMOGRAPHICS

The survey was designed and conducted using the online tool Qualtrics (https://www.qualtrics.com) for ease of use and submission by participants. Target participants for the survey were forensic quality practitioners or forensic practitioners involved in the management of quality issues, such as forensic laboratory directors, technical leaders, or discipline managers. An open invitation to participate in the survey was circulated to key target groups through conference presentations (American Society of Crime Laboratory Directors (ASCLD), Association of Forensic Quality Assurance Managers (AFQAM), Florida International University Global Forensic and Justice Center and International Association of Forensic Sciences (IAFS)) and expert group forums (AFQAM Forums, ASCLD Forensic Research Committee) along with direct word‐of‐mouth invitations to forensic agencies and experts made by the research team during visits and meetings held as part of the research project. It should be noted that the primary professional organization networks utilized in the recruitment process, AFQAM and ASCLD, are both based in the USA, and although both organizations have global membership the representation of member countries is heavily US‐based. Fifty‐eight participants completed the survey, with 15 being excluded from the data analysis due to non‐completion of the survey. In total, 43 valid completed surveys were included in the analysis. Given that this was a voluntary, opt‐in survey targeted at practitioners working operationally in forensic quality issue management, the demographic information provided by the participants completing the survey indicated that the targeted recruitment was successful. More than 95% of responses came from individuals in a Quality Manager, Quality Director, or Quality Practitioner role (41 of 43 responses). The remaining responses were received from participants identifying themselves as a forensic practitioner/analyst (one response) or a Forensic Laboratory Director (one response). All respondents indicated that they held more than 2 years' experience in forensic science, including 79% who indicated that they held more than 10 years' experience.

Participants were able to access the online survey tool via QR code or link, where participant information was provided in order to enable the participants to provide informed consent before continuing. The survey allowed participants to complete it anonymously if desired, including withholding details of the country where the agency is located. As anticipated by the membership make‐up of the professional networks that were used for survey recruitment, the location of the majority of participants was self‐identified as the United States (33 responses), with only three participants located outside of the USA and seven participants preferring not to disclose their location. The participants represented a diverse range of jurisdictions serviced, as shown in Figure 1, with seven respondents indicating their agency services more than one type of jurisdiction. It is acknowledged by the research team that the sample size and US‐dominant cohort of respondents could influence the results obtained.

FIGURE 1.

Types of jurisdictions serviced by surveyed participant's agencies (n = 43).

The survey consisted of multiple sections with certain questions being displayed based on responses to previous questions. The survey was set up to allow participants to return to and amend previous results as well as save and finalize their responses at a later date.

The survey questions were divided into sections, reflected in the headings presented in the results below. Questions were designed as either requiring selection from a set of provided options (nominal scale) or as free‐text fields to permit unstructured data collection. Extra space was provided for the majority of option questions to allow additional free‐text comments to be made. All questions were set as forced responses.

The majority of the survey tool and the survey questions were directly taken from the previous survey conducted by the research team of Australian and New Zealand forensic agencies [12] to facilitate direct comparison of responses between the two survey participant groups. Additional sections were added to the survey tool in this instance to investigate particular areas of interest stemming from the outcomes of the Australian and New Zealand community responses.

A copy of the survey tool questions used for this study is provided in the Appendix S1.

3. SURVEY RESULTS AND DISCUSSION

3.1. Accreditation and certification of forensic disciplines

Only two of the 43 respondents (both from facilities external to the USA) indicated that no accreditation or certification was held for the disciplines offered by the agency. Of the disciplines represented by the other 41 respondents, the vast majority had an accreditation/certification rate of more than 90% (Figure 2 and Table 1). ISO/IEC 17025 was the predominant accreditation standard (>90% accreditation across illicit drugs, forensic biology/DNA, ballistics and firearms, fingerprints, toxicology, trace evidence, toolmarks, document examination, and fire debris analysis), with FBI QAS accreditation the next most adopted (>85%), applicable to forensic biology/DNA only. Other standards noted include ISO/IEC 17020 (33% of anthropology agencies, 15% of crime scene, 6% toolmarks, and 3% ballistics and firearms) and ISO 15189 (3.5% of toxicology agencies).

FIGURE 2.

Accreditation/certification status of disciplines for participants (n total = 41) with accredited agencies, across all standards.

TABLE 1.

Percentage accreditation/certification status of disciplines for participants (n total = 41) with accredited agencies, across all standards.

	Total	Accredited/certified	Percent
Illicit drugs	37	35	95
Biology/DNA	34	34	100
Ballistics and firearms	30	30	100
Fingerprints	30	30	100
Toxicology	29	29	100
Crime scene attendance	26	25	96
Trace evidence	17	16	94
Toolmarks	17	17	100
Digital forensics	15	13	87
Blood pattern analysis	12	9	75
Document examination	9	9	100
Fire debris	5	5	100
Footwear impressions	4	3	75
Facial identification	3	1	33
Pathology	3	2	67
Anthropology	3	2	67
Entomology	2	1	50
Property and evidence	1	1	100

Thirteen participants indicated "other certification" was applicable to a variety of disciplines offered, including standards or requirements authored by professional bodies such as the American Board of Forensic Toxicology (ABFT), National Integrated Ballistic Information Network (NIBIN) and the National Association of Medical Examiners (NAME), as well as state‐based certification/accreditation requirements.

The responses provide insights into current trends being observed across accredited forensic science facilities in the Global North. Although ISO/IEC 17025 remains the predominant standard, the adoption of additional layers of certification and/or competency standards is becoming more prevalent. The forensic science community, in recent years, has recognized the need for industry‐ and discipline‐specific standards to supplement the broader laboratory‐centric scope of ISO/IEC 17025 and to address longstanding issues with training and competency assurance, validation of scientific methods, and transparency of processes [15, 16, 17]. Although this survey represents a small sub‐sample of forensic facilities, the findings in Figure 2 demonstrate that efforts from the industry to develop and apply additional layers of certification and assurance are being observed. With the full release in June 2025 of the new Forensic sciences standard ISO 21043 it will be of interest to observe how this notable advance in industry‐led standardization will be received and adopted worldwide.

3.2. Identification and recording of “quality issues”

All survey respondents indicated that their agency has a documented process for the management of quality issues, which is not unexpected given that this is an accreditation requirement for ISO/IEC 17025, ISO/IEC 17020, and ISO 15189. Eighty‐eight percent (88%) of respondents also noted that their documented process includes specifics on the types of issues that should be reported and/or recorded. Participants were provided with a list of commonly observed “issues” in a forensic quality management system and surveyed on which options would generally be considered a “quality issue” to be raised and managed (Table 2).

TABLE 2.

Survey responses to “Which of the following types of issues would generally be considered as a “quality issue” to be managed/logged?”

Description of issue	Number of affirmative responses from participants (n = 43)
A documented procedure was not followed	43 (100%)
A reported/disclosed RESULT requires correction	42 (98%)
Failure/investigation of an EXTERNAL Quality Assurance Program (QAP) result	42 (98%)
Non‐conformance identified at audit (internal or external)	42 (98%)
Results WERE invalidated/unable to be reported	40 (93%)
Failure/investigation of an INTERNAL Quality Assurance Program (QAP) result	34 (79%)
Customer/client complaints	34 (79%)
Critical equipment failure	32 (74%)
Staff competency issue identified as part of performance review	32 (74%)
Results COULD have been invalidated/delayed	31 (72%)
Customer/client feedback (non‐complaint or compliment)	31 (72%)
The content of a disclosed report, other than the result, requires correction	28 (65%)
Court testimony feedback	27 (63%)
Recommendations identified at audit (internal or external)	22 (51%)
Non‐critical equipment failure	13 (30%)
Customer/client compliments	11 (26%)
Results WERE delayed	5 (12%)
Other	7 (16%)

Free‐text responses provided for those participants selecting “Other” included:

• Chain of custody/sample integrity issues

• Quality control failure

• Risk management issues

• Batch sample inconsistencies

Very strong agreement (>90%) was obtained across all responses for several scenarios. Of the scenarios receiving <50% of the responses, it is notable that delayed results were considered as quality issues for only 12% of participants. When the same question was asked in the survey of Australian and New Zealand forensic agencies, this same scenario was also markedly less selected than the other options [12]. The reasons for not considering a delayed result as a quality issue may be related to the definition of whether the delay constitutes a non‐conformance with an existing requirement or policy, such as a service level agreement for turnaround of results. However, the lack of timely provision of forensic testing results and the push to address backlogs of unexamined cases remain an ongoing concern for several jurisdictions, as evidenced in the literature [18, 19, 20, 21], making this a significant quality issue for the industry to address. Where a delay in results (or a backlog) may not be recorded in the agency's “quality issue” management process, it is incumbent on the facility to monitor this indicator of performance quality elsewhere, given the impact that these issues have in the wider criminal justice system, including investigative, judicial, and intelligence applications reliant on timely delivery of forensic information.

Following identification of a quality issue, it must be logged, classified and managed, the extent of which will generally be based on agency‐specific risk tolerances and management system processes. Issues with a higher level of consequence or risk of recurrence may be managed formally via a corrective action process, whereas lower severity or isolated events may be recorded as a note on the applicable batch or case record as a means of notification that the issue has occurred. When a corrective action process is applied, a formal record of the issue is used to document the issue, the subsequent investigation, and any actions taken (generally known as a “Corrective Action Request” (CAR), or a similarly named tool). Where appropriate, issue investigation includes root cause analysis (RCA) in order to devise the most effective corrective actions to prevent recurrence of the underlying problem [22]. For the issues described in Table 1 that survey participants indicated would be logged/recorded as “quality issues” within their management system, further information was sought as to how these issues may be logged or managed. A list of common approaches was provided for participants to select from (CAR including RCA; CAR not including RCA; note or comments on affected batch; note or comments in case file), along with “Other” to capture alternative approaches. Participants were able to select as many options as applicable to each scenario, with results provided in Figures 3, 4, 5, 6, 7, 8.

FIGURE 3.

Methods of recording issues related to reported results that participants indicated may be used in their agency.

FIGURE 4.

Methods of recording issues related to QAPs that participants indicated may be used in their agency.

FIGURE 5.

Methods of recording issues related to equipment that participants indicated may be used in their agency.

FIGURE 6.

Methods of recording issues related to audits and procedures that participants indicated may be used in their agency.

FIGURE 7.

Methods of recording issues related to competency that participants indicated may be used in their agency.

FIGURE 8.

Methods of recording issues related to customer feedback that participants indicated may be used in their agency.

3.2.1. Issues with reported results

Reported results constitute the end “product” of the forensic science process that is subsequently fed back into the criminal justice system for incorporation into investigations, judicial proceedings, and intelligence models as appropriate. Accuracy, timely delivery, and effective communication of reported results are particularly relevant in the context of contemporary crime disruption and prevention, where investigative leads need to be acted on rapidly to disrupt crime or prevent repeat offending. The survey results shown in Figure 3 indicate that over 75% of respondents would manage incidents of incorrect or invalid results as a CAR with associated RCA, whereas other issues related to the reporting of results may be recorded and managed less formally.

3.2.2. Issues with Quality Assurance Programs (QAPs)

Forensic science involves the examination of unknown, historical traces, introducing an inherent uncertainty into any form of subsequent analysis or interpretation [23, 24]. Therefore, it is acknowledged that assurance of the accuracy or reliability of forensic casework results cannot be measured against a known ground truth of the trace. In the absence of this assurance, the proxy solution from a forensic quality management perspective is the validation and ongoing verification of the testing system and methodology using tools and mechanisms with known ground truth, including QAPs such as proficiency tests. Where a participant returns a result deviating from the known ground truth, this represents an opportunity to investigate the root cause of the deviation and potentially identify system components in need of improvement. More than 90% of the survey participants who would consider failures/investigations of an external QAP as a quality issue indicated this would be managed as a CAR with associated RCA, the predominant method for this type of scenario, with failures or investigations related to internal QAPs showing a wider range of approaches toward management and recording (Figure 4).

3.2.3. Issues with equipment

The significance of the suitability, performance, and reliability of equipment used in the analysis of forensic samples is evident through the requirements detailed in ISO/IEC 17025 [25]. In the contemporary forensic environment, bulk processing of test samples and high‐throughput analysis models mean that single failures in critical equipment can impact large numbers of samples and cases at once, and unchecked systemic equipment issues can be catastrophic for forensic service providers. The divide between the significance of critical and non‐critical equipment failures was evident in the survey results (Figure 5), with critical equipment failures being nearly twice more likely to be recorded and managed using CAR and associated RCA than non‐critical equipment failures.

3.2.4. Issues from audits and with procedures

A robust audit program is one of the leading components of a competent quality management system, providing regular and structured analysis of system components and outputs and their efficacy in meeting system objectives. The audit process provides a formal mechanism for the identification of non‐conformances against system requirements and, subsequently, an obligation for noted non‐conformances to be investigated and corrected at a root cause level. The survey responses align with this requirement, with more than 95% of respondents indicating that a non‐conformance identified at audit would be managed with CAR and associated RCA, the highest result of any scenario surveyed (Figure 6). Recommendations made at the audit may include auditor suggestions for preventive actions or improvement opportunities, not associated with a non‐conformance and this is reflected in the lower use of RCA observed in the survey results for this scenario.

Documentation of the quality management system sets the foundation for a consistent approach and, therefore, deviations from the documented system are a commonly recorded quality issue among forensic science service providers as seen in Table 1, and in the results observed in the Australian and New Zealand survey [12]. The high rate of recording deviations from documented procedures across almost all methods shown in Figure 6 demonstrates the importance survey participants place on addressing deviations in documented processes, which underpin the forensic quality management system.

3.2.5. Issues with competency

The expertise and competency of all staff involved in the delivery of forensic services are of paramount importance to the assurance of a quality product, and recent studies have demonstrated challenges in the assessment of particular areas of competency necessary for forensic analysts [16, 26]. Competency issues can have a direct impact on the reliability of results and often require significant resources to correct. This survey included two scenarios where potential issues of competency could be identified: performance review and court testimony feedback. “Competency” was not defined for participants in this survey, and, therefore, selections may be subjective based on the interpretation of the term by individual participants. Issues identified as part of the performance review were strongly rated as being managed via the CAR and RCA process, whereas court testimony feedback was more evenly rated across both the CAR processes and other approaches which included evaluation forms, personnel files, monitoring by the Quality Manager, and Performance Improvement Plans (if required) (Figure 7). Furthermore, court testimony feedback is not necessarily associated with a quality issue in need of correction and may be wholly complimentary in nature, which may explain the lower levels of recording indicated by the survey responses.

3.2.6. Issues with customer feedback

The ISO Principles of quality management state that “the primary focus of quality management is to meet customer requirements and to strive to exceed customer expectations” [27]. In the forensic science service provider context, the definition of “customer” may be significantly broader than just the purchaser of the forensic service and can also encompass all facets of the criminal justice system utilizing or receiving the service's outputs including the courts, interjurisdictional law enforcement, and even the wider public. Customer feedback provides the facility with an opportunity to monitor performance against "customer" expectations and calibrate improvements to the service with consideration of these expectations. Several survey respondents indicated that instances of customer feedback would be recorded in separate systems to other quality issues, although complaints from customers were more likely to be managed through a formal CAR and associated RCA process (Figure 8).

3.3. Types of quality issues most commonly seen

The results presented in the previous section related to how quality issues might be recorded and managed, limited to a predetermined list of example scenarios presented to survey participants to choose from. To gain a deeper understanding of the most common types of issues observed in contemporary forensic science quality systems, the survey participants were provided with an opportunity to provide examples of the most common types of quality issues being reported within their agency. Unstructured text responses were received from all 43 participants, and the information provided was sorted into groups based on coding of the themes appearing in the responses. The frequency of each theme was tallied to establish the most common types of issues seen across all 43 responses (Table 3).

TABLE 3.

Frequency of themes identified in responses (n = 43) to the question “What are the most common types of quality issues you see being reported in your agency?”

Major theme	Sub theme	Frequency
Manual processing issues	Procedure not followed	29
	Data entry e.g. transcription error/data not recorded	13
	Lab process e.g. pipetting error, sample mix‐up	13
	Interpretation/calculation issue	3
	Re‐testing required	1
Exhibit issues	Chain of custody not maintained	16
	Discrepancies with item & paperwork (pre‐lab issue)	2
Laboratory processing issues	Contamination	16
	Equipment issue e.g. malfunction or calibration	16
	Quality Control failure	11
Reporting issues	Incorrect information in reports	9
	Report delivery issue	2
	Testimony issues	1
Quality assurance issues	Issues in a proficiency test	11
	Unauthorized staff performing procedure	2
	Lack of validation	2

The themes and sub‐themes identified in Table 3 touch on the characteristics of quality issues commonly identified in the forensic science process. When the frequency of occurrence of these issues is reviewed, it becomes simpler to identify areas of common concern for facilities. It should be noted that this data does not reflect the levels of risk associated with the various sub‐themes, as that evaluation would be specific to each quality issue within a facility. However, given the clear major sub‐themes that are identifiable in this survey data, it is feasible that if there were a mechanism to also incorporate risk classification for each incident alongside the categorization of a sub‐theme, together with a sufficient sample size of the dataset, a meaningful analysis of areas of high risk in forensic quality across facilities could be developed.

3.4. Terminology around quality issues

It could be argued that the vocabulary of quality is well‐established, well‐used, and commonplace amidst organizational quality management systems. However, in literature regarding the challenges of communicating issues identified in systems of forensic science service provision, the lack of standardized language has been cited as a key barrier [8, 9, 28]. The common usage of a term, even one appearing as a specific term within a defined scope or discipline (such as quality management) does not necessarily indicate that the term's meaning is consistently applied and/or understood, including the scenario where the intended meaning from the communicator differs from the end interpretation by the recipient [29, 30].

Just under half of the survey respondents (21 of 43) indicated that their agency has a documented glossary of terms related to quality issues, with 16 of the remaining 22 noting that the policies and procedures used within their agency include terms related to quality issues (such as “error,” non‐conformance,” and/or “contamination”).

Survey participants were asked to provide their own definition for commonly used terms in forensic quality management (“non‐conformance,” “quality issue,” “error,” “contamination,” and “fail”). The unstructured responses for each of the terms were reviewed to identify common associations in the definitions, and the frequency of occurrence of these paraphrased associations across responses was tallied. The findings demonstrate the strong concordance between understanding and definition of some terms (“non‐concordance”) and the variation in meanings embedded into others (Table 4).

TABLE 4.

Combined paraphrased participant definitions of terms commonly associated with quality issues in forensic science and % of responses where the definition was observed (from 43 respondents, with some respondents providing more than one definition per term).

Term	Common definition(s) identified and frequency (%) the definition was observed in responses
“Non‐concordance”	Does not conform with documented policy/procedure/standard/requirement (81%)
	Deviation (unapproved) from documented policy/procedure/standard (14%)
	A condition potentially detrimental to quality (7%)
	Unexpected result (2%)
“Quality issue”	Something that calls (or could call) the quality of the work into question (28%)
	Anything affecting the quality management system (12%)
	A non‐conformance in testing that requires documentation, action and resolution (7%)
	Something contravenes a set of prescribed activities (7%)
	Anything that could negatively impact results being released to the customer (5%)
	Anything that affects the work product (5%)
	An issue lower than a non‐conformance but could be (low probability of re‐occurrence) (5%)
	An issue involving equipment, report writing or case review (5%)
	Used interchangeably with non‐conformance (5%)
	Something that needs further evaluation/RCA before deciding if CA is necessary (2%)
	Term not defined in quality management system (2%)
	A non‐conformance or potential non‐conformance, area for improvement or unexpected outcome in testing or QC (2%)
	Work that followed all procedures but ended with an unexpected outcome (2%)
	An event that raises to the level of quality awareness. May or may not be a non‐conformance but should be documented (2%)
	A non‐conformance being investigated by the Quality Division (2%)
	Includes staff not following protocols and policies, when staff don't recognize a QA has failed and still use the instrument etc. (2%)
	Term not applicable (2%)
“Error”	A mistake (44%)
	Difference from the true value/ground truth (18.5%)
	Term not used (9%)
	Incorrect result or action (7%)
	A failure (4.5%)
	The probability of receiving an incorrect answer/result when applying a testing method correctly (4.5%)
	Similar, but more specific than, a quality issue (2%)
	Unexpected result that was incorrect after evaluation (2%)
	Term not defined in QMS (2%)
	Processes and procedures that are in need of improvement (2%)
	An incorrect step, note or result in documentation (2%)
	Not right or incorrect, misleading (2%)
	Not following a SOP (2%)
	Error in report/records (2%)
“Contamination”	Unintentional addition/transfer and detection of one material to another (32%)
	Presence of something foreign or unexpected detected in the original material (30%)
	Unknown material introduced to test item during collection, inventory and testing that affects quality of results (9%)
	An unwanted substance in a blank (7%)
	Change to the evidence that hinders the ability to test it (4.5%)
	Something present in a sample that may or may not invalidate results (4.5%)
	Event of an agency member leaving some of them on evidence (DNA, latents etc.) or cross contamination between items (2%)
	A result of a sample or testing concern, specific to an item or sample (2%)
	Term not defined in QMS (2%)
	Possible exposure of samples and evidence to any and all additional sources of DNA (2%)
	Sample integrity compromised (2%)
	Term not applicable (2%)
“Fail”	Not meeting a required level of performance on a test (30%)
	Related to a QAP (technical errors are unsatisfactory) or a QC/performance check (18.5%)
	More specific to an instrument or equipment (9%)
	Not meeting a quantifiable requirement (9%)
	Term not defined in QMS (4.5%)
	Intended results of the test were not achieved (4.5%)
	Term not used (4.5%)
	Incorrect result in a known sample (4.5%)
	Term not applicable (4.5%)
	Failure to do something (2%)
	Incorrect individualization or classification (2%)
	Sample falls outside policy guidelines (2%)
	Something not working as expected (2%)
	Error or inaccuracy in a procedure (2%)
	Not standard (2%)
	Complete shutdown of a system (2%)

The highest level of consensus between respondents (81% from 43 responses), and the only term with a definition of above 50% consensus, was obtained for the term “non‐concordance” (“Does not conform with documented policy/procedure/standard/requirement”) with limited alternative definitions observed (three alternative definitions identified from all responses). All of the other terms surveyed returned a range of responses that could not be combined into less than 10 common definitions.

The next highest level of consensus on a term (44%) was obtained for the term “error” being defined as “a mistake.” This meaning, along with several of the alternative definitions observed from other respondents, aligns closely with the Collins Dictionary definitions for “error” [31] and so, whilst it would appear from the survey results that the definition of the term “error” has a level of shared understanding, there are subtle variations in meaning dependent on context. The connotations of the word “mistake” imply an unintended, even uncontrolled, deviation from that which is accurate, suggesting a lack of certainty and unreliability. In a forensic science context, these are unacceptable traits. Even if the term is used in the context of quality management and correction, for example, in the documentation of a quality issue disclosed within a case, the association of the word “error” with a forensic process, regardless of the outcome or impact on the case at hand, may impart an unavoidable bias with the message recipient based on the underlying subtext of unreliability, making it challenging for the communicator to effectively frame the nature of the finding and its true relevance to the quality of the result being communicated. It is worth noting that the second highest consensus definition for this term was “A difference from the true value/ground truth” (18.5%), also aligning with the dictionary definition [31] and one that would be more commonly associated with a scientific application of the term “error”. Further investigation as to the reasons for the selection of either term by participants may provide insight as to whether this is related to the nature of forensics being a field of science situated within a non‐scientific system (criminal justice) and the inherent challenges faced by scientists in spanning this divide [32].

3.5. What is quality issue data used for by the forensic science agency?

Whilst the identification, raising, and recording of quality issues is a requirement for forensic science facilities, and disclosure of these issues is often mandated by regulatory or legal obligations, the effort required to collect this data implies that utilizing it for the purposes of improvement is a foregone conclusion. However, how it is utilized and to what extent may be facility‐dependent and so survey participants were asked to select from a list of options (multiple selections allowed) on how the data collected on quality issues is used within their agency (Figure 9). Other than the formal reporting mechanisms indicated in Figure 9, the two respondents selecting “Other” detailed that data would also be used for continuous improvement and for emphasizing transparency to customers, the judicial system, and accrediting bodies.

FIGURE 9.

Survey responses to: “What do you do with the data collected on quality issues in your agency?”

This question was followed up in the survey by asking respondents for their opinion on whether they felt as though enough was being done with the data collected on quality issues (Figure 10), with just over half (55.8% of 43 responses) indicating they agreed that enough was done with the data on quality issues. Participants answering “Unsure” or “No” (n = 19) were invited to provide more details about their response, with three key themes identified in the responses: challenges to identifying trends, impediments to actions stemming from quality issue data, and opportunities for more that can be done. A selection of the answers provided, grouped by theme, is presented in Table 5.

FIGURE 10.

Survey responses (n = 43) to “In your opinion, do you feel like enough is being done with the data collected on quality issues?”

TABLE 5.

Additional details about response provided by participants who answered “No” or “Unsure” to “In your opinion, do you feel like enough is being done with the data collected on quality issues?” (selected responses, grouped by theme identified).

Theme	Example responses
Challenges to identifying trends	1. “Each corrective action is handled individually – there is no review for trends, no meaningful management review for similar incidents or similar causes that could point to larger, system level preventative actions” 2. “I conduct anecdotal trend analysis but would like to do more of that to see if certain root causes are more common than others and can be addressed in a more holistic way” 3. “The current issue I'm facing is that I can use my data to look for trends, but I as the user have to initiate that query. What I hope to do, but am struggling with how to‐ is how can the data inform ME of the trends instead of me having to go searching for the trends?” 4. “It can be difficult to conduct trend analysis without categorizing the various types of events that occur” 5. “We currently do not have a good system in place to conduct trend analysis” 6. “Our nonconforming work process begins with evaluation of the nonconforming work, determining if it can be handled as a correction, or as a preventative action or corrective action. If we determine corrective action we come up with a plan and then review how the plan worked downstream. We then review the corrective actions at the annual management review. We are a small lab ([redacted]‐member team) so I think it's easier to determine if the corrective action plan has assisted in reducing the risk of additional nonconforming work”
Impediments to actions stemming from quality issue data	7. “I feel like we are collecting a lot of data about the issues that happen and doing a good job of correcting them. But we have brainstorming sessions to try to prevent things or have sections get ahead of identified risks but it is hard to get buy‐in to “do more work” when they are already behind in the work they have so, if there isn't an issue yet, then they don't want to implement ideas to prevent issues that haven't “happened” yet” 8. “Laboratory leadership finds QA burdensome and something done to check boxes for accreditation rather than a tool to improve. Trends are analyzed by the QM but no action is typically taken” 9. “I am unaware of what the managers do with the information that is recorded. I am not privy to this information” 10. “Reports that affect quality matters should be acted on as soon as possible to ensure continuity of work, immediate correction to avoid further damages. This will require other parties take responsibility especially in financial requirements. The responsible persons may take time or not do it entirely hence greatly contributing [sic] correction of quality issues in the laboratory” 11. “Quality Manager wants to do more, but there is interference from Management. Conflicting interests: QM focused on quality improvement vs. Management focused on reputation and resources” 12. “Management may also focus on individuals as opposed to process/SOP inefficiencies, workload and mental health of staff, and appropriate training for staff. Quality Manager has responsibilities and no authority”
Opportunities for more that can be done	13. “The QA department is currently working with management and supervisors to improve and expand the use of the data collected by QA. Atypical events are recorded in the “Incident Log” by anyone involved. These are now reviewed weekly by supervisors who decide if they need to be escalated to a corrective action” 14. “Good tracking of quality issues assists the agency in training and competency shortages. It also could indicate burnout with individual examiners. A lengthy trend analysis of the overall unit(s) could also indicate the level or difficulty of stress associated with those types of examinations” 15. “We collect the data and report it out. We have not really used the data to remove someone from employment, even when it is warranted. Usually we just move them around to another discipline. We have used the data to tighten up policies and training programs, but it mostly feels like we are spinning our wheels and always having the same issues” 16. “Analyzing data collected on quality issues is very important for developing preventative measures and continuous improvement, and there is always room for more in depth analysis. On the other hand, focusing too much on quality issues may lead to overreactions and unnecessary scrutiny of what is often unavoidable human error. Furthermore, mass reporting of these issues, or not distinguishing the small mistakes from larger quality issues while reporting to external stakeholders (in the name of transparency) may lead to low morale for the workforce (i.e., employees may feel that they are being defined more by their errors than they are by their good work)”

Trend analysis of quality issues is a critical tool in both the management of corrective actions and in the management review process, and effective trend analysis of issues detected and root causes is a prominent feature of high‐reliability fields such as aviation and medicine [10]. The responses obtained in the survey highlight several challenges to effective trend analysis, ranging from systems incompatible with trend analysis (responses 1 and 5) through to the challenge of having too much data and no way to mine it efficiently (response 3).

Among the responses relating to impediments to actions stemming from quality issue data, a lack of buy‐in from management and/or staff affecting the ability to action outcomes from quality issue data appeared multiple times. Notions of quality being “burdensome” (response 8) and being a task that is not prioritized (response 7) or actively impeded (response 11), by management were noted, along with there being a lack of authority delegated to the Quality Manager in order to effect change (responses 9 and 12).

Opportunities that were noted as “in progress” or as potential avenues to enhance the use of quality issue data included collaboration with leadership to develop more integrated systems of review (response 13) and utilizing trend analysis to identify wellness stressors in analysts (response 14). It was also raised that, in the pursuit of transparency in reporting quality issues, the ongoing effects of reporting on staff morale need to be considered where there may be little differentiation between events designated as “minor” versus “major” (response 16).

3.6. Perceptions on the raising and disclosure of quality issues

Survey participants were asked to rate statements on comfort levels around raising and disclosure of quality issues using a 5‐point Likert scale. For the statement “In your opinion, how comfortable are staff with raising quality issues?” (i.e., raising issues internally within the agency), more than 80% responded as “Comfortable” or “Very comfortable” (Figure 11), whereas when asked “In your opinion, how comfortable are staff with disclosing quality issues?” (i.e., disclosing issues externally to the agency) the spread of responses migrated to considerably lower across the scale (Figure 12).

FIGURE 11.

Plot showing spread of responses (n = 43) to the Likert scale question “In your opinion, how comfortable are staff with raising quality issues?” (Scale: Center of plot = 0 responses with outer ring = 20 responses).

FIGURE 12.

Plot showing spread of responses (n = 43) to the Likert scale question “In your opinion, how comfortable are staff with disclosing quality issues?” (Scale: Center of plot = 0 responses with outer ring = 20 responses).

When invited to provide an opinion on the factors influencing the rating given in Figure 12, those who had responded as either “Very comfortable or “Somewhat comfortable” stated reasons such as mandatory reporting requirements to State and accreditation bodies (e.g., “Disclosure to ANAB and the Commission is mandatory for significant events.”) and the familiarity of staff with the disclosure process influenced their selection (e.g., “…we've been accredited for 12 years, it's par for the course.”). More frequently cited in the responses were comments around having a positive quality culture/management culture around quality issues (e.g., “Quality culture from management. Understanding that fixing quality issues makes us better.”, “We have tried to build and foster a culture of transparency.” and “…I as the Quality Manager worked to change the culture of QA to empower employees to ask questions of the quality office, submit solutions to potential problems, and finally to ensure that quality issues were handled with no bias and were not accusatory.”).

At the other end of the scale, “Somewhat uncomfortable” and “Very uncomfortable”, a fear of reputational damage or having to defend a quality issue was noted in multiple responses (e.g., “Fear of being asked about it in court”, “Concern of discredit during testimony” and “Disclosing quality issues to external stakeholders is intimidating to staff due to the stigma associated with it. Staff do not want to be seen as incompetent or negligent or “not smart”; they worry how is how they will be seen during those discussions with external stakeholders.”). Just as a positive workplace/quality culture influenced a favorable association with disclosure of quality issues, multiple responses cited that a negative culture, unsurprisingly, had the opposite effect (e.g., “Punitive action against the errant officers.”, “People get fired for what some people in the company think come from quality issues being identified…”, “Treatment of the QA issue differs depending on supervisor and person who makes the mistake if a human error.” and “Non‐conformance and corrective action used to be considered somewhat disciplinary, informally, 15 years ago. Some older employees still maintain that attitude and some managers…have used CAPA as a threat to get employees to follow procedures…”).

Reasons provided for equivocal responses (“Neither comfortable nor uncomfortable”) ranged from factors such as consistency of practice (e.g., “No uniformity to how [redacted] state handles this.”, “It depends on the issue, how many cases involved, over what time period.” and “Some report because it is the right thing to do while [others] do not report as they feel it is not their responsibility.”), to reluctant acceptance (e.g., “Staff know that is it the right thing to do, but may be worried about the consequences.”, “Staff aren’t necessarily comfortable with that but it's not their choice.” and “They most likely feel this way because it is uncomfortable and can raise questions, but it must be done to provide transparency.”), to a lack of exposure to quality issue disclosures (“It doesn't usually come up with testimony so they have no reason to really disclose anything.”).

4. CONCLUSIONS

The ongoing commitment from the global forensic community to improving standards of quality, validity, and professionalism is evident throughout the literature and initiatives emerging from within the industry, along with academia and professional bodies [24, 33, 34, 35, 36, 37, 38, 39, 40]. The results of this survey and the previous survey of Australia and New Zealand [12] indicate that the high level of accreditation and certification for forensic science service provider facilities in the Global North provides a strong foundation on which to build this improvement. However, there are challenges to be overcome, both in the development of robust forensic science globally [37] and in the collective identification, acknowledgment, and effective communication of limitations within that science, including quality issues detected within forensic science service provider systems that may affect the accuracy or reliability of results.

Leveraging operational data on issues detected within systems to enhance practice and drive improvement and innovation is a hallmark of high‐reliability organizations (HROs) [41]. The results of this survey demonstrate that the well‐established quality management frameworks operating throughout forensic science facilities capture a vast amount of data on those actual and potential issues identified in the forensic process, data which may be used to target areas of scientific risk, human factors, and limitations impacting the reliability of results and their communication with end users. The findings also confirm the nuanced variations in practice between facilities in the systems for identification, classification, recording, management, and disclosure of quality issues, a subject anecdotally recognized but not previously confirmed through published data. These variations are indicative of the challenges of sharing data on quality issues between facilities, ranging from technical challenges (such as differences in which issues are classified as quality issues and the varying mechanisms of recording them) to cultural challenges (such as buy‐in from management and reluctance from staff to raise or disclose quality issues).

Overcoming the technical challenge of a lack of standardized classification and language around quality issues in forensic science is the focus of the wider research project for which this survey was conducted, and in conducting this survey the participants were also asked for their opinions on what would be the most valuable features of a standardized approach to the classification of quality issues in forensic science. Among the highest‐ranking themes observed from the responses were:

• To drive consistency in what is and what isn't a non‐conformance or corrective action.

• To promote consistent practice and to aid comparison between agencies.

• To aid trend analysis and lead data‐driven decision making.

Building consistent practice in this area benefits individual agencies in streamlining processes through objective and value‐adding decision making, along with cross‐agency benefits, such as the ability to benchmark operational quality metrics to drive improvements and inform resource allocation in business planning and strategic management [42].

However, the benefits noted above go deeper than just operational business metrics. Developing an effective standardized approach to the identification, recording, classification, and management of quality issue information is crucial to driving a positive quality culture in forensic science agencies and throughout the field. The survey respondents recognized the importance of this in the other highest‐ranking features of a standardized approach:

• To improve external understandings of quality issues.

• To build understanding among staff and destigmatize quality issues.

The significance of organizational culture in modern quality management systems cannot be understated. A systematic review of factors affecting patient safety incident reporting in healthcare by Archer and colleagues [43] identified that the two main facilitating factors to engagement with incident reporting were processes and systems of reporting (the features of the system that enable reporting, such as reducing the complexity of the process) and organizational factors (values, beliefs and policies around incident reporting). A transparent, objective, and consistent approach to the management of quality issues is foundational to the development of a “just culture” that encourages reporting of issues and eschews blame [44], and without an integrated culture of reliability, HROs would be considered as just “well‐run” entities [41]. To genuinely achieve the demonstrable characteristics of an authentic quality culture in forensic science, such as transparency, accountability, a focus on scientific best practice, and continuous improvement, the forensic science service must emulate these qualities throughout all facets of its structure, governance, and leadership [45].

The relationship between a positive quality culture and a robust and transparent system of quality issue identification, management, and communication within a forensic science agency maybe somewhat “the chicken or the egg?” – which comes first? We suggest that both components are symbiotic. To successfully support efforts to improve the foundational principles of forensic science and enable transparent and credible communication of results with end users, quality management must progress beyond a compliance mindset and toward the development and integration of an authentic culture of quality throughout all aspects of forensic science service provision. Standardization of practice around the communication and management of quality issues in forensic science will underpin a “just culture” focused on identification and improvement within and across the field. The respondents of this survey demonstrated that this goal is both needed and valued by forensic science service providers, regardless of the current status of quality culture or practice maturity.

Vazire and Holcombe [46] note that a scientific community that encourages transparency while discouraging criticism is not one that prioritizes self‐correction. It is critical that the forensic science community lead the efforts to improve transparency in its practices, including the identification and correction of issues detected, and overcome concerns regarding public scrutiny in order to maintain and build credibility. Utilizing the findings gathered from this study, a tool for the standardized classification of quality issues in forensic science will be developed to be tested by forensic agencies for applicability and usability on real‐world examples. This will provide the foundation for further research into the communication and disclosure of issues detected in forensic science to support greater transparency, inform research and development opportunities to build credibility, strengthen forensic science practices, and enhance understandings of “error” in forensic science with end users toward better justice outcomes.

CONFLICT OF INTEREST STATEMENT

The authors have no conflicts of interest to declare.

Supporting information

Appendix S1

Heavey AL, Houck MM, Turbett GR, Lewis SW. Quality issue management and disclosure in forensic science: A survey of practice and perceptions. J Forensic Sci. 2025;70:1866–1881. 10.1111/1556-4029.70121

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.