Investigation of the long-term yield of auditing for conformity with the ISO 15189:2012 quality standard in a hospital pathology laboratory

Abstract

Introduction

Pathology laboratories are increasingly seeking accreditation to quality standards to assure Quality of Service (QoS). However, there is little data available regarding the value of this in laboratories with well-established Quality Management Systems (QMS). Moreover, critics of accreditation claim it redirects resources toward trivial issues. Our objective was to investigate the value of auditing for conformity with the ISO 15189:2012 standard in such laboratories.

Design

and Methods: In total, 483 Audit-Identified Non-Conformities (AINCs) were documented within our department since transitioning to an ISO 15189:2012 compliant QMS. The potential consequences of these were assessed by three clinical laboratorians who assigned them into categories based on their likely impact. These were: Unlikely (no clear consequences); Possible (potential for poor QoS/harm); and Probable (Likely to cause poor QoS/harm). Additionally, total numbers/severity of Real-Time Non-Conformities (RTNCs) detected outside of auditing were examined to provide additional insight into the effects of accreditation on QoS.

Results

According to majority decision: 395 (81.8%) of AINCs were classified Unlikely, 88 (18.2%) were Possible, and none were Probable. The relative proportion of Unlikely AINCs also rose over time. Total numbers and severity of RTNCs dropped in the short-term following transition to an ISO 15189:2012 QMS, but steadily rose thereafter.

Conclusions

Our data suggest auditing for conformity with ISO 15189:2012 standards may be effective in attaining accreditation, but may have diminishing returns in the long-term once the QMS is established, unless there is continual improvement in the audit process to promote better use of resources.

1. Introduction

Laboratory testing is integral to modern medical practice and often provides clinical decision makers with crucial information regarding the diagnosis and management of disease [1,2]. Therefore, access to rapid and reliable service from pathology laboratories is essential for clinicians to provide patients with appropriate, high-quality care. In recent decades an increasing number of pathology laboratories have sought to assure high Quality of Service (QoS) by means of accreditation to ISO (International Organisation of Standardisation) standards [3].

The ISO is an international organisation composed of representatives from many national organisations, whose purpose it is to set standards to assure QoS across a wide range of industries. The ISO presently holds pathology laboratories accountable to the ISO 15189:2012 standard; a specialised set of requirements developed to provide a standardised quality management framework which takes into account the specific nature of the work undertaken in a medical laboratory setting [4]. A recent overview from the College of American Pathologists states that the purpose of the ISO 15189 standard is to “create systems that are as failure resistant as possible”, “catch mistakes before they become a problem” and “reduce errors by getting things right the first time”. They additionally state that accredited laboratories should “identify opportunities for improvement at all times” and “involve and empower their staff by involving them in the solving of problems and the implementation of solutions” [5].

Bodies that deliver accreditation on behalf of the ISO determine whether laboratories meet the necessary criteria for accreditation by means of auditing for conformity with the relevant standard. This auditing process consists of regular internal audits for conformity with each area of the standard, as well as inspections by an independent external auditor representing the accrediting body at least once every 18 months [6]. Logistical constraints preclude an exhaustive scrutiny for conformity with standards. Consequently, such audits use a sampling process whereby various aspects of the laboratory are selected randomly for inspection.

Accreditation to the ISO 15189:2012 standard is widely considered to be an assurance that a clinical laboratory meets the necessary requirements to provide a level of QoS conducive to high-quality patient care [5,[7], [8], [9]]. However, critics of the accreditation process argue that strict compliance with such standards mandates considerable wasteful activity; redirecting important human, monetary and technical resources toward unimportant work, while doing little to improve QoS [10,11]. Furthermore, even in the absence of supportive outcome measures, if one were to concede that the quality framework prescribed by the ISO 15189 standard can indeed improve QoS in medical laboratories, the fact remains that there is little evidence presently available regarding the benefits of continued auditing for compliance with ISO standards after the necessary systems have already been put in place to satisfy the requirements for accreditation.

In the present study we sought to investigate the value of continued auditing for compliance with the ISO 15189:2012 standard as a means of assuring long-term QoS and mitigating patient harm in a clinical laboratory whose-QMS has become established over a 6 year period of accreditation to ISO 15189:2012.

2. Methods

A total of 483 non-conformities with the ISO 15189:2012 standard were documented from audits undertaken in the Department of Clinical Chemistry and Diagnostic Endocrinology of the Mater Misericordiae University Hospital (MMUH) in Dublin (Co. Dublin, Ireland) between the years 2012 and 2018. This period represents the years during which the laboratory has been accredited to the ISO 15189:2012 standard for which complete records of non-conformities were available. These Audit Identified Non-Conformities (AINCs) arose from a number of internal audits, as well as external audits undertaken by Ireland's ISO accreditation body, INAB (Irish National Accreditation Board).

Inspectors for INAB categorise non-conformities identified under different severity ratings. Non-conformities can be classed as either Minor or Major [12]. A Minor non-compliance is defined by INAB as a “failure to comply with the scheme requirements”; while a Major non-compliance is defined as “failure to comply with the scheme requirements to the extent that could compromise the confidence that is placed in the accredited activity”. A third categorization, Observation, is sometimes used to highlight an issue of potential concern that is not specifically in violation of any aspect of the ISO standard [12].

For the present study, the severity of each of the 483 documented AINCs was re-evaluated according to a set of criteria modified from a similar study recently reported by Wilson and colleagues at the Royal Victoria Hospital in Belfast (Co. Antrim, United Kingdom) [11]. These criteria were constructed to reflect the likely impact of the documented events on QoS, rather than simply their deviation from the ISO 15189:2012 standard. The categories of severity used were as follows: Unlikely (The occurrence had no clear adverse consequences and was documented purely due to non-conformity with the ISO 15189:2012 standard); Possible (Such an occurrence has the potential to cause poor QoS or harm to patients/staff, and has a reasonable likelihood of doing so at some point); and Probable (The occurrence has either lead to poor QoS/harm, or such an occurrence would be considered likely to cause poor QoS/harm). The nature of each of the individual AINC was re-evaluated separately by three clinical laboratorians with training and experience in QMSs and auditing; two of whom are based in our department and one of whom works across a network of Irish hospitals. These individuals used their own knowledge and judgement to assign each AINC to one of the above categories. Each individual was blinded as to the decisions of the other two while evaluating the AINCs. Where disagreements arose regarding which category a non-conformity should be reassigned to, a two out of three majority decision was accepted as the result.

Additionally, the total numbers and severity of real-time non-conformities (RTNCs) (non-conformities that were not identified by the audit process, but that arose spontaneously and were detected by other means, such as: error detection systems, staff vigilance, and customer complaints) documented in our department between the years 2012 and 2018. These were gathered in an effort to provide additional information regarding any effects of ISO 15189:2012 compliance on QoS. These RTNCs were grouped into severity levels based upon departmental guidelines. The classifications used for RTNCs were as follows: Observation (A finding warranting clarification/investigation to improve QMS), Minor (A minor deficiency or lapse in discipline), Serious (An incident that had the potential to have a negative clinical impact or cause a serious technical error), Critical (An incident that would have had clinical impact on a patient, but was caught prior to having done so), and Major (An incident in which there was clinical impact on a patient).

3. Results

Of the 483 AINCs reassessed during our study, the majority decision of the assessors was that 395 non-conformities (81.8%) were classified “Unlikely”, 88 (18.2%) were classified “Possible”, and none were considered “Probable”. These findings are broken down by year in Fig. 1. A disproportionately low number of AINCs were reported in 2012 as this data was partial, and represented the very beginning of the switchover process. A breakdown of the post accreditation data from 2014-2018 showed that more than half of AINCs from this period related to document control issues (e.g. a controlled document being passed its scheduled review date) (Table 1). The total numbers and assigned severity ratings of all RTNCs documented between 2012 and 2018 are shown in Fig. 2A, and are displayed graphically in Fig. 2B.

Fig. 1.

Breakdown of audit-identified non-conformities and their estimated impacts on QoS by year. The total numbers of audit-identified non-conformities and the estimated likelihood of their having a detrimental effect on QoS are broken down according to year.

Table 1.

MMUH audit identified non-conformities for years 2014–2018 broken down according to section of the ISO 15189:2012 standard and by assessed severity.

Section of ISO 15189:2012 standard	Total AINCs by section	Total AINCs ranked “Probable”
4.3 Document control	145 (57.8%)	7 of 145
5.3 Laboratory equipment reagents and consumables	24 (9.6%)	4 of 24
5.2 Accommodation and environmental conditions	17 (6.8%)	4 of 17
5.4 Pre-examination processes	15 (6.0%)	7 of 17
5.5 Examination processes	14 (5.6%)	3 of 14
5.1 Personnel	12 (4.8%)	1 of 12
4.1 Organization and management responsibility	4 (1.6%)	0 of 4
5.6 Ensuring quality of examination results	4 (1.6%)	1 of 4
5.8 Reporting of results	4 (1.6%)	4 of 4
5.9 Release of results	4 (1.6%)	0 of 4
5.10 Laboratory information management	3 (1.2%)	2 of 3
4.2 Quality management system	1 (0.4%)	0 of 1
4.5 Examination by referral laboratories	1 (0.4%)	1 of 1
4.13 Control of records	1 (0.4%)	0 of 1
4.14 Evaluations and audits	1 (0.4%)	0 of 1
4.15 Management review	1 (0.4%)	0 of 1
Total	251	34 of 251

Fig. 2.

Numbers and severity ratings of real-time non-conformities by year. The total numbers and assigned severity ratings of all real-time non-conformities documented between 2012 and 2018 were gathered. Raw numbers are shown in table format (A) and are additionally displayed graphically (B) to illustrate observed trends. Severity ratings are based upon criteria set by our hospital's pathology department. Observation = A finding warranting clarification/investigation to improve the quality management system; Minor = A minor deficiency or lapse in discipline; Serious = An incident had the potential to have a negative clinical impact or cause a serious technical error; Critical = An incident that would have had clinical impact on a patient, but was stopped prior to having done so; Major = An incident in which there was clinical impact on a patient.

4. Discussion

The methodology of the present study was adapted from that of a recent meta-audit undertaken by Wilson et al. (2016) [11]. In their study, two qualified scientists assessed the impact on QoS of non-conformities reported from a selection of internal and external audits performed between the years 2000 and 2013 in a public health laboratory accredited to the ISO/IEC (International Electrochemical Commission) 17025 standard (2000 and 2005, at the time of the original audits); standards applied to laboratories that carry out testing and calibration [11,13]. For their evaluations, the two scientists discussed and agreed upon a classification for each of the AINCs based on their ability to affect QoS. This differed from our own improved methodology, where three laboratorians assessed each AINC separately, blinded to one another's decisions, and a majority decision was accepted where disputes arose. This approach reduced the potential for bias from individual assessors, as the assessors could not influence one another's decisions.

Wilson et al. (2016) reported that the vast majority of non-conformities identified through auditing were deemed to be of little importance, and were unlikely to have any negative impact on QoS [11]. Despite the differences in methods, our results corroborate these previous findings, where the majority of AINCs examined were classed as Unlikely to have adverse consequence. It could be inferred unfairly from these aggregate findings that auditing for compliance with ISO 15189:2012 is an inefficient and ineffective means for clinical laboratories to enhance their QoS to a clinically relevant level. However when AINCs were examined for each year it is clear that these were greatest from 2013, during a period when our laboratory was initially transitioning its QMS in accordance with the ISO 15189:2012 standard. During this initial phase, many AINCs were uncovered and permanently addressed by implementation of new procedures or documentation. For example, many pieces of equipment formerly had no unique asset number assigned. Once a unique asset number was given to each of these pieces of equipment, these non-conformities would not recur during the lifetime of that piece of equipment or its use in the department. With benefits primarily realised at the outset, but diminishing over time, our data shows that there is room for considerable improvement in the auditing process as required for compliance with ISO 15189:2012, and laboratories should periodically evaluate the benefit of the audit process and implement appropriate changes to maximise its effectiveness and efficiency.

The decrease in AINCs from 2014 could potentially be taken an indicator of quality improvement (i.e. that the new QMS was causing an overall decrease in the occurrence of AINCs). However, this is clearly open to interpretation and could be a consequence of many different factors including differences in the levels of experience and aptitude of the staff performing audits and reporting non-conformities over the years. For example, in the early phases of our accreditation to ISO 15189:2012, the majority of audits were carried out by an appointed quality officer. Since then, auditing has largely been delegated to other members of staff who have received basic audit training, but whose primary functions in the laboratory lie outside of such quality assurance roles. Moreover, as years go by, staff may succumb to “audit fatigue” and become complacent in their investigations, which could lead to artificially decreased total numbers and relative importance of AINCs found.

It should be noted that while total numbers of AINCs have decreased every year since our accreditation that, on average, the proportion of those found that were deemed unlikely to have any meaningful effect on QoS has also increased over time. This also potentially indicates that continued auditing for conformity to the same standards yields diminishing returns on QoS following the initial perceived benefits observed after setting up of the standard's framework as the QMS matures and reaches a new “steady-state” of conformity with its new standard. A breakdown of the AINCs found during the post-accreditation period from 2014 – 2018 (shown in Table 1) showed that more than half of the AINCs documented from audits during this time period related to unimportant document control issues. Particularly common were issues such as a controlled document being passed its date for review, or for a certain document to have not yet been distributed to or acknowledged by all members of the department. This finding is mirrored by the previous work reported by Wilson et al. (2016) who also noted that a high percentage of laboratory AINCs were concerned with document control and textual amendments [11].

A downward shift in the total numbers in RTNCs of all severity ratings was observed following the initial phase of implementing an ISO 15189:2012 based QMS (Fig. 2A and B). This trend however appeared to reverse in the subsequent years, with total numbers of RTNCs tending to increase as time went on (Fig. 2A and B) . Interestingly, as the total numbers of RTNCs increased over time, this was associated with a decline in the relative proportion of RTNCs classified Minor in severity, and an increase in those classified as Serious in severity (Fig. 2A & B). Total numbers of Observations, as well as Critical and Major RTNCs did not fluctuate significantly after an initial decline in 2012; with the exception of an isolated increase in Critical RTNCs in 2015 which, with further analysis, was traced to a collection of related pre-analytical incidents reported at that time.

It is difficult to draw any definitive conclusions regarding any effects that the actual original implementation of the ISO 15189:2012 framework in our laboratory between 2012 and 2013 may have had on overall QoS from this study. The observed shifts in both absolute numbers and relative proportions of Unlikely and Possible rated AINCs may be interpreted in a number of ways. Similarly, it is not possible to determine whether the patterns observed in RTNCs reported were attributed to changes in the actual occurrences of such non-conformities, or whether increases in RTNCs were due to improvements in our ability to detect and document them. Moreover, the nature of auditing by sampling randomly selected facets of a laboratory for conformity means that only a small fraction of potential problems in a laboratory may be highlighted by the process; thus serious issues may go unnoticed, simply as they were not included in the scope of the audits. Measuring appropriate quality indicators prior to and following accreditation could provide an insight into whether such laboratories achieve an improvement in their QoS by adopting and complying with a quality standard. Such an approach has been used in previous studies examining the effects of laboratory accreditation to ISO 15189:2012 and other international quality standards.

A number of medical centres in sub-Saharan Africa have demonstrated considerable improvements in several quality indicators following the WHO-AFRO's (World Health Organisation – Region Office for Africa) implementation of an laboratory accreditation program based upon the ISO 15189 standard [3,[14], [15], [16], [17], [18], [19], [20]]. For example, the pathology laboratory of the Bugando Medical Centre (Mwanza, Tanzania) demonstrated marked improvements in several quality indicators following implementation of the ISO 15189:2012 standard, including improvements in EQA (external quality assurance) performance, as well as decreases in the number of customer complaints, rejected samples and blood culture contaminations [20].

It is of importance to note that prior to their implementation of quality standards, that QoS in many of these African laboratories was so poor that clinicians would often forego laboratory testing, instead making the majority of their decisions based purely on clinical findings and judgements [3,20]. The improvements in quality indicators observed in the Bugando Medical Centre's pathology laboratory were the result of 24 months of intensive efforts, including substantial systemic and procedural alterations in order to bring the laboratory to the standard required to attain accreditation to ISO 15189:2012 [20]. With this in mind, it should be noted that the results obtained from such studies conducted in these resource limited settings may not necessarily reflect the effects of implementing a quality standard in an established laboratory in the developed world. A comparable analogy would be that an unwell patient will likely benefit from a therapeutic intervention, while a healthy patient generally will not; and in some cases may even suffer mal-effects. Relating back to accreditation, this implies that adopting a quality standard such as ISO 15189:2012 will likely help poorly performing laboratories reach a minimal acceptable standard, but may not eventuate similar improvements in laboratories already performing at or above this level.

A 2004 study conducted in the cervicovaginal screening section at the National University Hospital in Singapore demonstrated minimal, and largely statistically insignificant improvements in quality following accreditation to quality standards set by the College of American Pathology (2000) [21]. Similarly, a more recent study undertaken in the histopathology department of Our Lady of Lourdes Hospital in Drogheda (Co. Louth, Ireland) showed no statistically significant differences in sample turn-around times or in EQA performance before and after their accreditation to ISO 15189:2012 standard, and through focus group studies determined that staff found “paper trails” and “inspections” to be negative aspects of the accreditation process; although in the same study staff surveys showed that laboratory personnel viewed the accreditation as a positive action overall [22].

Even if one were to concede that accreditation to a quality standard such as ISO 15189 improves QoS initially, we contend there is very little evidence regarding the value of continued auditing for conformity once the necessary quality framework has already been put in place. Our results suggest that continued auditing for compliance with the ISO 15189:2012 standard, as it presently exists, is not an efficient or effective way of highlighting such issues, and scarcely justifies the amount of resources that are allocated to carrying out such audits. This is particularly so if resources are taken from other core laboratory duties involving monitoring and controlling analytical performance, where deficiencies in procedure and process are more likely to have clinical impact.

It should be noted that neither this investigation, nor any similar studies which throw doubt on the efficacy of auditing for conformity with a quality standard such as ISO 15189:2012, can actually conclusively disprove that doing so sustains good QoS. We must also recognise the possibility that in lieu of the rigor that is prompted by the requirements of accreditation, that some laboratories may indeed encounter lapses in quality. Indeed, simply having such a structure in place may foster a culture of vigilance and conscientiousness amongst staff, which in turn begets improvement in QoS. However, while such benefits of compliance with ISO 15189:2012 remain unproven, it is only prudent to consider whether it is indeed worth the considerable time and cost that goes into ensuring that a centre maintains its accredited status.

To conclude, the high proportion of non-compliances deemed to be unlikely to have any important effect on QoS indicates that continued auditing for conformity with ISO 15189:2012 standards, using a static audit process, may be an inefficient and largely unavailing approach to further improving the QoS in clinical laboratories which have developed a mature QMS. Accordingly, there is room for improvement in the audit process required for compliance with ISO 15189, to promote more efficient and effective use of time and resources to better deliver and sustain a high QoS. There is also need to define better indicators of QoS which could allow the value of compliance with such quality standards to be more meaningfully assessed to enable delivery of a high QoS conducive to the provision of excellent patient care.

Funding

The authors received no financial support with respect to this article.

Ethical approval

Not required.

CRediT authorship contribution statement

Alastair David Green: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing - original draft, Writing - review & editing. Lucille Kavanagh-Wright: Formal analysis, Investigation, Methodology. Graham Robert Lee: Conceptualization, Formal analysis, Investigation, Methodology, Writing - review & editing.

Declaration of competing interest

The authors declare no potential conflicts of interest with respect to this article.