Design of external quality assessment schemes and definition of the roles of their providers in future epidemics

Abstract

During an epidemic, individual test results form the basis of epidemiological indicators such as case numbers or incidence. Therefore, the accuracy of measures derived from these indicators depends on the reliability of individual results. In the COVID-19 pandemic, monitoring and evaluating the performance of the unprecedented number of testing facilities in operation, and novel testing systems in use, was urgently needed. External quality assessment (EQA) schemes are unique sources of data reporting on testing performance, and their providers are recognised contacts and support for test facilities (for technical–analytical topics) and health authorities (for planning the monitoring of infection diagnostics). To identify information provided by SARS-CoV-2 genome detection EQA schemes that is relevant for public health microbiology, we reviewed the current literature published in PubMed between January, 2020, and July, 2022. We derived recommendations for EQA providers and their schemes for best practices to monitor pathogen-detection performance in future epidemics. We also showed laboratories, test facilities, and health authorities the information and benefits they can derive from EQA data, and from the non-EQA services of their providers.

Introduction

The COVID-19 pandemic, caused by SARS-CoV-2, severely affected the world and its economic, health-care, and social systems. On March 11, 2020, the SARS-CoV-2 outbreak was declared a pandemic by WHO, and one of the key messages from the WHO Director-General was to increase testing frequency (“test, test, test”¹) to identify and isolate infected individuals.² This call was extensively followed and resulted in more than 15 billion SARS-CoV-2 PCR tests done by June, 2022, worldwide.³ Laboratory-developed SARS-CoV-2 tests were established in early 2020, and the first commercial test systems became available soon after.⁴ Medical laboratories increased their testing capacities, and new test facilities were dedicated exclusively to SARS-CoV-2 testing. After 3 years of the pandemic, an unprecedented number of test facilities are still in operation, with many different test systems—a situation previously unknown for other pathogen diagnostics. Given this situation, there continues to be an imperative to monitor and assess the quality of the test facilities and test systems, and to give support for quality improvement.

Defining external quality assessment (EQA)

EQA is a procedure for interlaboratory comparison throughout all disciplines in laboratory analysis, in which the analytical performance of participants is evaluated with predetermined criteria. EQA schemes usually consist of several individual ring-test rounds per year, with the number of samples in individual rounds varying depending on the provider. The relevant International Organisation for Standardisation (ISO) 17043 standard generally uses the term proficiency test for interlaboratory comparison, and the term EQA is more commonly used in medicine and medical research.⁵ All test facilities enrolled in an EQA scheme receive sample panels with the same known, but undisclosed, characteristics; thus, they have the same initial conditions for analysis. Participants establish the samples' properties or measure concentrations of target analytes, and submit their quantitative, semiquantitative, or qualitative results to the EQA provider. All individual results are assessed according to specific criteria, and are compared with an assigned target and the results of other laboratories. The participants receive confidential feedback on their proficiency and a summary report comparing the results of each peer group, highlighting overall areas for improvement where identified, and describing the specifics of each round. There are major differences in national legislation regarding the obligation to participate in the round robin test, official monitoring of laboratory performance, and the consequences of failing the review. Therefore, the applicable legal provisions state whether the laboratory can, or must, implement the identified need for action.⁶ The benefits for laboratories participating in EQAs are the confidential evaluation of the analytical performance of their methods by a competent, independent third party, and the opportunity to compare their results with those obtained by other laboratories, assays, or procedures. In this way, the potential for improvement can first be identified, and opportunities for improvement, as compared with other participants, can then be considered. There are prerequisites for the use of samples for EQAs. First, they should be homogenous and stable up to the date when they are analysed and the results are returned, so that all participants have the same basis for analysis. Second, the samples should present clinically relevant challenges as required by the international standard ISO 15189.⁷ Finally, the samples should be commutable, so that they are suitable for obtaining comparable results from different test systems.8, 9, 10

In addition to providing services to participating laboratories, EQA schemes and their aggregated results provide important information about the performance of all included assays and testing facilities in the field.¹¹ EQA data and the decisive role of their providers is especially important during a pandemic.12, 13

The need for monitoring testing performance during the COVID-19 pandemic

With many unprecedented features, the COVID-19 pandemic highlighted the importance of third-party monitoring of assay and testing facility performance. Although emergency regulations allowed the unrestricted use of laboratory-developed tests, and numerous commercial test systems were brought onto the market, little objective information about test performance was—and still is—available. Furthermore, numerous new manufacturers and distributors of in vitro diagnostics emerged. Operators of the many new test facilities might have had reduced demands on staff qualifications and competence. Due to the use of non-trivial laboratory methods and the need to deliver results with medical and epidemiological relevance, information on the performance of the new facilities and tests was urgently needed throughout the pandemic. This need was not expected before the COVID-19 pandemic, and the limited availability of already-validated tests was considered a major challenge for pandemic preparedness.¹⁴ During the pandemic, it was the unique role of EQA providers to report as an independent third party on the performance of all participating test facilities and assays enrolled in their schemes, both to give confidential individual feedback to participants, and as an anonymised, aggregated summary of all participants' analytical performance to public health officials. However, even before the COVID-19 pandemic, EQA schemes have proven to be an excellent tool for post-market surveillance of assays by monitoring their reliability with randomly selected EQA samples.¹⁵

Microbiology laboratories are the primary barrier against the risks posed by communicable diseases. Before the COVID-19 pandemic, strategies were developed to detect communicable diseases and antimicrobial resistance, assess risks, and monitor public health through reliable and comparable microbiological data that are shared and used in a timely manner.¹⁶ After the start of the COVID-19 pandemic, guidance and recommendations for performance expectations and the use of certified in vitro diagnostics (eg, under emergency-use authorisation and for in-house tests) were issued from different parts of the world and from countries with different income levels. Prominent examples are the recommendation concerning the acceptable and desirable limit of assay detection,¹⁷ and the recommendations for national SARS-CoV-2 testing strategies and diagnostic capacities (panel 1 ) by WHO,¹⁸ the United States Food and Drug Administration (FDA) Policy for Coronavirus Disease-2019 Tests During the Public Health Emergency,¹⁹ the European Centre for Disease Prevention and Control Testing strategies for SARS-CoV-2,²⁰ the US Centers for Disease Control and Prevention testing overview,²¹ the UK Medicines & Healthcare products Regulatory Agency Guidance “How tests and testing kits for coronavirus (COVID-19) work”,²² the African Society for Laboratory Medicine Guidance on Quality Assurance for COVID-19 Molecular Laboratory Testing,²³ the Global Fund Interim Quality Assurance Requirements for the Procurement of COVID-19 Diagnostic Products,²⁴ and the Guidance for In-house Test Development for Molecular Detection of SARS-CoV-2 by the Foundation for Innovative New Diagnostics and the African Society for Laboratory Medicine.²⁵ Additionally, a technical guide for COVID-19 testing has been published by the International Organisation for Standardisation with the aim of standardising pre-examination, examination, and post-examination processes for the detection of SARS-CoV-2 using nucleic acid amplification.²⁶ Thus, there are detailed specifications on testing with all its characteristics and challenges. Although many of these guidelines and recommendations refer to EQA, there is no guidance on how to design and implement such schemes in case of a public health emergency caused by an infectious disease.

Panel 1. Areas and topics* as aggregated from publications on external quality assessment results for SARS-CoV-2 genome detection *Numbers in parentheses refer to the information criteria in the publications listed in table 1.

General information

• •Types and numbers of enrolled assays (1)
• •Counts (2)
• •Categories of participant test facilities (3)
• •Study time (4)

Performance indicators

• •Rates of false negative and false positive results and their relation to virus or RNA load in samples (5)
• •Analytical sensitivity (6)
• •Interassay variability (7)
• •Intratype variability of results (8)
• •Indications of specificity (11)
• •Linearity (12)
• •Repeatability (13)
• •Verification of manufacturers' specifications on the limit of detection (14)
• •Performance under extraordinary conditions, such as analysis of sample pools (15)

Assay specifications

• •Proportion of test systems meeting specific recommendations (9)
• •Those reporting on human housekeeping genes (16)

Sample specifications

• •Specifications of sample materials including virus or RNA load (10)
• •Source of the samples (ie, clinical, virus culture, or RNA; 17)
• •Information on the presence of human housekeeping genes (18)
• •Carrier matrix in samples (19)
• •Physical properties of samples on arrival in the laboratory (20)
• •Information on homogeneity and stability of samples (21)
• •Other
• •Any special features of the scheme or additional noticeable information gained (22)

We reviewed the current literature on published properties and characteristics of completed SARS-CoV-2 genome detection EQA schemes and rounds, and evaluated the extra-EQA services supplied by EQA providers. Our aim was to provide recommendations for the rapid establishment of EQA schemes in future epidemics—or pandemics—that best monitor and report on the performance of epidemiologically relevant assays and test facilities, and for the provision of appropriate extra-EQA services.

SARS-CoV-2 genome detection EQA schemes

Literature on EQAs are generally rare. We searched PubMed using the terms “EQA” or “external quality assessment” or “proficiency testing” or “PT” and “SARS-CoV-2” or “COVID-19” for full-text articles published between January, 2020, and July, 2022 and identified 17 publications on EQA of SARS-CoV-2 (appendix p 1).27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43 Each publication was evaluated for general information on the EQA scheme, performance indicators of participant assays and laboratories, specifications and features of assays, specifications of samples used, and any additional noticeable information provided (panel 1). From this aggregated information we derived the subjects and sorted them into two categories. Epidemiologically relevant subjects contained information relevant for the quality of public health microbiology; reports on an EQA scheme or round were not acceptable unless this information was provided. Subjects were considered epidemiologically conditionally or imperceptibly relevant if collecting and sharing this information might have enhanced the epidemiological relevance of the report, or was of minor relevance from an epidemiological point of view for the quality of public health microbiology (table 1 ). Allocation to these categories corresponds with the personal opinions of the authors.

Table 1.

Criteria for classifying information as epidemiologically relevant, or conditionally or imperceptibly relevant, and estimated effort required for EQA providers to report missing information

	Reason for assignment to category	Estimated effort required to report information
Relevant
Types and numbers of registered assays (1)	These are key data of EQA; all components of the analytical procedure (devices and reagents used for sample preparation [eg, extraction], amplification, and detection) should be recorded.	If only reagents, but not their different batches, are considered in an EQA scheme, the effort required to adapt the EQA provider's software and the processes of EQA rounds is presumably low.
Counts of participant laboratories enrolled (2)	These are key data of EQA.	NA
Categories of participant laboratories (3)	Performance of test facilities in individual categories is particularly relevant in times of a pandemic, when new test facilities are set up specifically for pathogen detection. Any identified need for action can then be directed selectively to members of a category. Due to country-specific classifications of categories, it might be difficult for international schemes to evaluate various categories of laboratories.	Minimal effort is required if individual laboratory category is included in the participant's data with the EQA provider; extension of dataset is required if not included. If the policy of an EQA provider is to not evaluate results according to laboratory categories, this should be reconsidered in times of a pandemic to identify or rule out weak points in individual categories.
Study time (4)	Retrospective assignment of EQA results to individual phases of the pandemic and the then prevailing pathogen variants is particularly important if the pathogen changes its characteristics rapidly.	Low effort required for reporting such missing information, since the provider knows about it anyway.
Rates of false-positive and false-negative results (5)	This is a key output of EQA.	NA
Analytical sensitivity (6)	This is a key output of EQA.	NA
Interassay variability of results (7)	This is particularly important when values are obtained by different assays, and their results are used for medical and epidemiological decisions.	Minimal effort required if such values are already included in an EQA scheme; otherwise, a minor adjustment of the scheme and software would be necessary.
Intratype variability of results (8)	This is particularly important to evaluate the susceptibility of a test system to user or environmental influences; inexperienced personnel should preferably use assays with low intratype variability.	Minimal effort required if such quantitative results are already included in an EQA scheme; otherwise, a minor adjustment of the scheme and software would be necessary.
Proportion of test systems compliant with specific recommendations (9)	Verification of assay compliance with recommendations is a main task of EQAs during pandemics.	Verification through samples with particular specifications does not require any technical effort in the software or database; otherwise, minor technical adjustments are required.
Concentration of virus or RNA (10)	Information on sample characteristics is as relevant as presenting this information in a commonly used unit.	Additional effort for sample characterisation might be required if new determination methods must be applied.
Conditionally or imperceptibly relevant
Analytical specificity (11); linearity of quantitative results (12); repeatability or intraassay variability of results (13); and verification of manufacturers' specifications on limit of detection (14)	This information provides deeper insight into the performance of test systems and thus about the reliability of their quantitative and qualitative results. EQA schemes can only give an indication of these performance criteria (11–14), and it is up to the laboratory to verify the manufacturer's specifications, evaluate the results, and keep records. The focus of EQA schemes is on the educational effect for test facilities that are not fully familiar with the verification test procedure. Regarding the detection limit, the manufacturers should agree on the same test methods for determining it, and on the same units when specifying it.	This information can only be obtained by appropriate design of an EQA round and suitable samples; no special technical requirements.
Pooling (15)	If pools of samples are analysed, the loss of relative sensitivity by dilution through pooling procedures should be made evident.	Evaluation of pooling effects by EQA requires collaboration of test facilities; no adaption of software or database structure is required.
Proportion of test systems including human housekeeping genes (16)	Test facilities that analyse samples self-collected by individuals or by unexperienced personnel should use assays with sampling controls.	Minimal effort required if such values are already included in an EQA scheme, otherwise a minor adjustment of the scheme and software would be necessary.
Sample origin (17)	Differences in the sample properties from virus cultures or clinical samples are of interest to EQA providers and the scientific community.	This information is known to the EQA provider anyway and can therefore be provided easily.
Presence of housekeeping genes in samples (18); carrier matrix used in samples (19); physical properties of samples on shipment (20)	This information should be provided with sample specification data.	If required, such information might be collected during the production or testing of sample materials.
Stability and homogeneity of sample materials (21)	The provision of stable and homogeneous sample material is a basic requirement in EQA and needs no further mention once a round has been completed.	NA
Not assigned
Remarkable information (22)	Any special features of the scheme or additional noticeable information gained.	NA

EQA=external quality assessment. NA=not applicable.

Relevance of information for public health

Among the 17 publications reviewed, three reported on international EQA schemes or rounds,32, 33, 40 one on a binational (Australia and New Zealand) EQA scheme or round,³⁸ and 13 on national EQA schemes or rounds (Austria,28, 29, 30, 31, 34 China,35, 39, 43 India,³⁷ Japan,27, 36 and South Korea).41, 42 None of the publications32, 33, 38, 40 on international schemes reported any difficulties shipping EQA sample panels to any countries with import restrictions in place. The number of test facilities enrolled per round was between 32 and 953, and individual rounds consisted of between two and 12 samples. The study time across all 17 publications was between February, 2020, and early 2022 (not specified in more detail). A non-peer-reviewed report on an international EQA scheme was included because of its reporting of so-called best practice features of EQA schemes during pandemics⁴⁴ (appendix p 1).

Information that can be read from EQA data and the relevance of this information to public health microbiology is presented in table 1. The presence of each individual criterion (1–22) in the respective publications is shown in table 2 . Each publication contained data and information; however, none contained all ten criteria identified as epidemiologically relevant—most contained between six and nine criteria. Types and quantities of registered assays were reported by all, but only three reported that they registered batch numbers of reagents and included them in the evaluation of results. Interlaboratory comparisons in the post-market surveillance of medical in-vitro diagnostics are important, and should go down to the level of individual batches.⁹ Counts of test facilities enrolled were also reported by all publications, but categories of participating test facilities (eg, hygiene and virology institutes, pharmacies, COVID-19 test facilities, and physicians' private or hospital laboratories) were only reported by nine publications. If the policy of an EQA provider is to not evaluate results according to laboratory categories, this policy should be reconsidered in a pandemic to identify weak points in individual categories and remedy them in a targeted manner.

Table 2.

Publications on EQA results for SARS-CoV-2 genome detection categorised according to epidemiological relevance of criteria 1–22

	Relevant										Conditionally or imperceptibly relevant											Not assigned	Total (n)*
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22
Peer reviewed
Asai at al (2022)27	x	x	x	x	x	x	..	..	..	x	..	..	..	..	..	..	x	x	x	x	x	x	7
Buchta et al (2020)28	x	x	..	..	x	x	x	x	..	x†	x	..	x	..	..	..	x	x	x	x	x	x	7
Buchta et al (2021)29	x	x	..	x	x	x	x	x	..	x	x	x	..	..	..	x	x	x	x	x	x	x	8
Buchta et al (2021)30	x	x	..	x	x	x	x	x	x	x†	x	..	x	..	..	x	x	x	x	x	x	x	9
Buchta et al (2022)31	x	x	..	x	x	x	x	x	..	x	x	..	x	x	x	x	x	x	x	x	x	x	8
Edson et al (2020)32	x	x	x	x	x	x	..	..	x	x	x	..	..	..	..	..	x	x	x	x	..	..	8
Fischer et al (2020)33	x	x	x	x	x	x	..	..	..	x	..	..	..	..	..	..	x	x	x	..	..	..	7
Görzer et al (2020)34	x	x	..	x	x	x	..	..	..	x†	..	..	..	..	..	..	x	..	x	x	x	x	6
Guo et al (2022)35	x	x	..	x	x	x	x	x	..	x	..	..	..	..	..	..	x	..	..	x	x	x	8
Ishii et al (2020)36	x	x	x	x	x	x	x	x	..	x	..	..	..	..	..	..	x	..	x	x	..	x	9
Kaur et al (2022)37	x‡	x	x	x	x	x	..	..	..	x†	..	..	..	..	..	x	x	x	..	..	..	x	7
Lau et al (2022)38	x	x	x	x	x	x	..	..	..	x	..	..	..	..	..	..	x	..	..	x	x	x	7
Li et al (2022)39	x	x	x	..	x	x	..	..	..	x	..	..	..	..	..	..	x	..	x	x	..	..	6
Matheeussen et al (2020)40	x	x	..	x	x	x	..	..	..	x	x	x	..	..	..	..	x	..	..	x	..	x	6
Park et al (2022)41	x	x	x	x	x	x	x	..	..	x†	..	..	x	..	..	x	x	x	x	x	x	x	8
Sung et al (2020)42	x	x	x	x	x	x	x	x	..	x	x	..	..	..	..	..	x	..	x	x	x	x	9
Wang et al (2021)43	x	x	..	..	x	x	x	..	..	x	x	..	..	..	..	..	x	..	..	x	..	x	6
Published by the EQA provider
Zeichhardt et al (2020)44	x	x	..	x	x	x	x	x	..	x	x	x	..	..	..	..	x	..	x	x	..	x	8
Total (n)§	17	17	9	14	17	17	9	7	2	17	8	2	4	1	1	5	17	9	12	15	10	14

Cross indicates criteria is present in publication. EQA=external quality assessment.

^*Publications reporting information categorised as relevant.

^†Concentration of pathogen not specified in common units.

^‡Only in-house assays were used.

^§Peer-reviewed publications.

14 publications reported the study time. The results of an EQA round should be assignable to individual phases of the pandemic and the prevailing pathogen variants at that time. This assignment is particularly important when the pathogen and its properties change rapidly, as was seen with SARS-CoV-2. The rates of false-positive and false-negative results, and their relation to virus or RNA load in samples, was also reported by each publication. Interassay variability was reported by nine publications and the intratype variability was reported by seven publications. This information shows the differences in the—supposedly interchangeable—values that are achieved with different or uncalibrated test systems, and that are mistaken as quantitative laboratory results. The compliance of assays with specific recommendations, such as the WHO recommendation concerning the limit of detection¹⁷ and the International Federation of Clinical Chemistry and Laboratory Medicine's recommendation to use at least two target genes for the detection of SARS-CoV-2 RNA,⁴⁵ was reported by two publications. Data, the rates of false-positive and false-negative results, and their relation to virus or RNA load in samples, are of great importance for public health microbiology, as they enable an estimate of the number of unreported infections. It can also be estimated whether the results are due to a general weakness of a test, or to problems specific to detecting mutated pathogens. Concentrations of genome equivalents in the samples used for EQA are also reported by each publication, but only 12 reported the concentration in a common metric unit (eg, copies per mL); five publications stated the concentration indirectly as Cq or Ct values, making it impossible to compare. Specifying sample characteristics is as essential as presenting this information in a comparable manner. Missing information from the category relevant for public health microbiology could easily be provided by minor adjustments to the summary report, minor adaptions of the EQA scheme or software used, or by use of appropriate samples (table 2).

Characteristics of sample materials regarding their origin, presence of housekeeping genes, matrix used as carrier (eg, buffer solution, virus transport medium, or sodium chloride), and physical conditions of the samples on arrival at the test facility (eg, liquid at ambient temperature, cooled, frozen, or lyophilised), are of conditional or imperceptible epidemiological relevance. However, these characteristics are of great interest to other EQA providers and the scientific community; while sample origin was reported by all 17 publications, the presence of housekeeping genes, the matrix used, and the physical conditions of the samples were reported by nine, 12, and 15 publications, respectively. No information was published on the rationale behind the decision to ship samples under each condition (ambient, refrigerated, frozen, or lyophilised). In six of the EQA rounds or schemes referenced, samples were sent liquid at ambient temperature; in another six samples were sent frozen on dry ice; one as cooled or refrigerated; one was referred to as distributed in cold-chain (but not whether frozen or refrigerated); and two (only one from the group of peer-reviewed publications) used lyophilised material. The EQA providers would have taken their technical equipment, transport costs, and experience with comparable pathogens into account when deciding the state and conditions in which samples were to be shipped. One publication reported on pooling of samples, and five reported on the proportion of assays that included human housekeeping genes. Both data can have epidemiological relevance if these methods are widely used. For the sake of clarity, information on sample stability and homogeneity was given in eight publications.

14 publications reported information and findings from their schemes (panel 2 ). Eight publications reported on analytical specificity, two on linearity, four on repeatability of the results, and one on the verification of manufacturers' specifications on the limit of detection. EQA schemes can only give an indication of these performance criteria in individual test systems, and it is up to the laboratory to verify the manufacturers' specifications, evaluate the results, and keep records. However, it can be helpful, especially for testing facilities that are not familiar with assay verification procedures, for EQA providers to design their schemes accordingly and support them in verifying the performance of their test systems.

Panel 2. Noticeable information reported by EQA schemes.

• •Inclusion of all laboratories nationwide in the EQA round.41, 42 Full coverage of all testing facilities in a country or region makes data on their performance more reliable.
• •Provide a rapid report after submission of results to give feedback before the final completion of the assessment.⁴⁰
• •Dependence of performance on laboratory category.27, 36
• •Unequal performance of different batches of the same reagent, and relation of performance to extraction method;36, 43 however, one other publication found no such relation.³⁷
• •Loss of relative sensitivity of the assays when analysing pooled samples.³¹
• •Largely meeting, and in some cases exceeding, the sensitivity specifications given by the manufacturer.³¹
• •Improvement of accuracy between first and third rounds of the scheme.³⁸
• •Substantial interlaboratory variance in reported Cq (Ct) values, making a quantitative estimate of genome concentration unreliable and inappropriate for its use to guide clinical decisions such as releasing patients from isolation.28, 29, 30, 31, 34, 35, 36, 41, 42, 43 It was affirmed that the Cq (Ct) value in SARS-CoV-2 PCR is firstly mistaken for a metric result that meets quality requirements for quantitative laboratory values, and secondly mistaken for a harmonised value (eg, independent of the test system). In fact, the Cq (Ct) value is neither, but was used for a long time for medical and epidemiological decisions.46, 47

Limitations of EQA data

There are limitations concerning the interpretation of EQA results. Although EQA performance is objective, it is only one of several quality indicators of a testing facility or assay. Furthermore, results from EQA summary reports only refer to participating laboratories, so their general performance cannot be applied to regional laboratories. For example, the total quantity of tests performed in a region cannot be extrapolated from the percentage of correct or incorrect EQA test results. Data from this form of EQA relate exclusively to analytical performance and do not provide any information about pre-analytical procedures, although this part of SARS-CoV-2 detection has a major effect on the reliability of results. Finally, it must be trusted that each participant analysed the samples themselves, using the specified method.

Extra-EQA services of EQA providers

The services of EQA providers cover more than their naming suggests. In addition to organising and supervising interlaboratory comparisons, EQA providers are a contact for medical and technical inquiries, and serve as a network centre that links test facilities, experts, and public health authorities. Extra-EQA services of EQA providers were reported to have positively influenced the quality performance of laboratories, at least in immunohaematology.⁶ Both EQA and extra-EQA services give their providers a unique position and relevance to the laboratory analysis community. Their services are especially needed during pandemics, when knowledge about the pathogen and pathogen diagnostics is initially low and then grows rapidly, and public health measures and recommendations for diagnostics are constantly adapting based on key epidemiological data. National EQA providers have a clear advantage here, as they already cooperate with local experts and are in contact with national health authorities and can therefore quickly respond to changing epidemiological, virological, or regulatory situations.

EQA providers as a competent contact for general information and support for laboratory analysis

The importance of EQA providers as a contact point for a wide variety of analytical inquiries from participants is difficult to document and even more difficult to measure. We report unpublished data from a 2022 survey by the European Organization for External Quality Assurance Providers in Laboratory Medicine (EQALM) of its member organisations on the services of individual EQA providers. 35 of 38 responders regularly received inquiries about non-EQA issues, regardless of SARS-CoV-2, and they all reported to be prepared for such inquiries and had sufficient competent staff to process them (Buchta C, unpublished).

Examples of extra-EQA services relevant to public health microbiology

At the onset of a pandemic, EQA providers can give guidance to participants to assess their competence in using their routine tests. In an EQA scheme of April, 2020, all four SARS-CoV-2-positive samples—which were from a 10-fold dilution series—were quantified by digital droplet PCR, covering a linear concentration range between approximately 360 000 and 380 copies per mL of viral RNA.⁴⁴ With this approach, it was possible to anchor measured Cq (Ct) values with defined viral loads. In the same round of this EQA scheme, an interim evaluation was published that revealed the target values of three of the seven samples in the panel. This evaluation allowed participating laboratories to review their applied tests in terms of sensitivity and specificity, and enabled them to improve their test methods in the event of incorrect measurements, at short notice before the official deadline of the EQA programme.

Another example for an extra-EQA service of EQA providers is the provision of two national reference materials with assigned viral RNA loads of 10⁷ copies per mL and 10⁶ copies per mL.⁴⁸ Following this study, the use of reference materials for the quantification of SARS-CoV-2 RNA in patient samples might contribute to the standardisation of results obtained by different test systems for the detection of SARS-CoV-2 RNA.

In the context of EQAs, a positive effect on the harmonisation of results for SARS-CoV-2 quantification was also shown in practical implementation when Ct values were converted into standardised units.⁴⁹

EQA scheme providers should be aware that they can play a central role in the public perception of diagnostic performance in a region, especially in a pandemic. In this context, it has proven beneficial for EQA providers to collaborate with scientific societies and health authorities. Examples include statements on the significance of EQA results48, 50 and clarifications on making clinical decisions based on quantitative anchoring in reference samples, considering viral load rather than Cq (Ct) values.51, 52

Raising awareness

The UN notes the need to raise awareness; promote the exchange of information, scientific knowledge and best practices; provide quality education; and support advocacy programmes on epidemics at the local, national, regional, and global levels, as effective measures to prevent and respond to epidemics.⁵³ To highlight the need for prevention of and preparedness for epidemics, Dec 27 was declared the International Day of Epidemic Preparedness by the United Nations General Assembly.⁵⁴ As a contribution to this awareness, we have evaluated the support of EQA schemes and their providers in addressing the global health crisis caused by SARS-CoV-2, to derive suggestions for coordinated and effective actions in future epidemics.

Pandemic preparedness to raise pandemic readiness

Considerations on the design of EQA schemes in pandemics

The appropriate design of an EQA is based on a well considered definition of sample specifications, and their adaptation to the changing pathogen properties throughout the pandemic. Sample specifications define the expected significance of the results. An EQA panel should therefore contain samples that challenge the performance of test systems, such as pathogen concentrations around the recommended limit of detection. Individual samples can then be indicated as either core (ie, participants are expected to report correctly) or educational (ie, participants can report results; these samples are primarily used to provide additional information on assays and procedures), and results from participants are expected accordingly. Using only unequivocally positive or clearly negative samples is unhelpful and does not provide clinically relevant challenges.⁷

From SARS-CoV-2 we learned that mutations and variants pose additional challenges for test systems.55, 56 Therefore, monitoring the performance of test systems in terms of specificity is necessary, and this requires adapting samples to the changing properties of the pathogen over the course of the pandemic. Flexibility regarding the frequency of EQA should enable early reactions to developments in the pandemic, and should give facilities that are introducing new test systems the opportunity to participate in EQAs at short notice.

To save resources, some facilities tested pools of multiple samples during the COVID-19 pandemic, and only analysed individual members if a pool tested positive. A mathematical model for the extent of the loss of sensitivity has been published.57, 58 To investigate the performance of test systems when analysing pooled samples, the design of a special EQA round is required, as is the willingness of participants to engage in such an imitated pooling procedure where the participants have to dilute the EQA samples before analysis. The expected loss of relative sensitivity was evident in the EQA round that reported on nine false negative results of 30 (30%) when samples were analysed in pools with a size of between five and ten participants, compared with two of 30 (7%) when the same samples were analysed individually.³¹

Testing for housekeeping genes in a specimen can be used to verify proper sampling and sample preparation, and could be of interest to monitoring test facilities that are analysing samples taken by individuals themselves, or by inexperienced personnel.⁵⁹ It is advisable to also provide samples with and without human genes for such test facilities. These facilities will then evaluate the correct differentiation of samples, with the result either not detectable (ie, housekeeping gene positive and pathogen RNA not detected means correct sampling) or not evaluable (ie, housekeeping gene not detected and pathogen RNA not detected indicates poor quality of sampling).

Legal and regulatory precautions

To prepare for future pandemics, EQA providers can proactively address two related issues that have sparked debate during the SARS-CoV-2 pandemic. The first is the cost of EQAs, and the second is the obligation to participate in, and the minimum number of times per time period they should participate in, EQAs. The information gained through monitoring analytical performance far outweighs the financial expenditure of EQAs, especially with non-profit organisations, and would justify the assumption of costs by the public sector. Unless already regulated by law, participation in pathogen-specific EQAs should be made mandatory for all testing facilities, and their participation should be verified. A strict obligation for each test system used by test facilities would also prevent the use of test systems not monitored by EQA for routine analysis. Strictly obligatory (and in return, free) EQAs could be agreed on to prepare for future pandemics. In addition, it can be considered an extra-EQA service that the participants are offered a helpful review of their preventive measures after a failed EQA.

Cooperation of EQA providers in a network

EQA providers are mostly members of professional EQA networks. From April, 2020, 15 representatives from EQALM member organisations regularly met to exchange information on the implementation of SARS-CoV-2 EQAs. Communication on this platform was helpful for all involved and contributed substantially to the successful establishment of several EQA schemes for SARS-CoV-2 diagnostics. In 2022, this informal group was replaced by the EQALM Working Group Virology, which aims to promote and coordinate the pandemic preparedness of EQA providers, and to increase the efficiency of their schemes and services in future pandemics.⁶⁰ Information exchange in such an expert group will result in more efficient designs of EQAs, more targeted selection of sample materials, and in avoidance of pitfalls. EQA providers should be encouraged to support and participate in such concerted campaigns and groups.

Limitations of our considerations

We acknowledge that our work is limited by the referencing of reports predominantly from high-income countries, and that we discussed pathogen detection by nucleic acid amplification testing as the only laboratory method. We identified only one report from a non-high-income country, India, which is currently described a lower-middle-income country by the World Bank.⁶¹ Regarding our exclusive focus on pathogen detection by PCR, our findings and recommendations also apply to other direct methods (eg, antigen detection) and indirect methods (eg, antibody determination).62, 63, 64

Recommendations

In summary, we make recommendations for EQA providers, their schemes for infection diagnostics, and their non-EQA services in future pandemics (panel 3 ). We hope that we will not need them for a long time.

Panel 3. Recommendations to external quality assessment (EQA) providers for future epidemics.

• (1)Seek early arrangements with public health authorities so that in the case of an outbreak, epidemic, or pandemic:
  • •
    All test facilities, ideally with each of their individual test systems, are obliged to participate in EQA
  • •
    Test facility participation is verified
  • •
    In return, participating in EQA should be free of charge for test facilities participating in public health-relevant analytics
  • •
    Preventative actions after a failure in EQA are reviewed by experts
• (2)Provide EQA schemes early. EQA should be available as soon as testing begins
• (3)Be flexible in designing and adapting EQA schemes so that they best accompany the epidemic and the participating laboratories and test facilities; done in coordination with public health authorities
• (4)Prepare schemes and reports to regularly report on:
  • •
    Types and numbers of registered assays
  • •
    Counts and categories of test facilities enrolled
  • •
    Study time
  • •
    Rates of false-positive and false-negative results, and analytical sensitivity of assays
  • •
    Interassays and intratype variability
  • •
    If applicable, proportion of test systems compliant with relevant recommendations
  • •
    Sample specification in a commonly used unit
  • •
    Reporting on these categories will support participants, public health authorities, other EQA providers, and the scientific community
• (5)Make the summary report available shortly after the end of a round, or give participants immediate feedback on their results
• (6)Immediately report suspicious or alarming findings to health authorities
• (7)Take the role as a contact for non-EQA inquiries and a network partner seriously:
  • •
    Use the central position to share up-to-date information with participants
  • •
    Support participants standardising their assays
• (8)Support concerted campaigns and expert information exchange on EQA through participation

Declaration of interests

CB is chairman of the executive board of the European Organisation for External Quality Assurance Providers in Laboratory Medicine 2020–23. HZ was majority owner of Gesellschaft für Biotechnologische Diagnostik (until November, 2022) and is owner and managing director of Institut für Qualitätssicherung in der Virusdiagnostik. AG is president of the Austrian Association for Quality Assurance and Standardization of Medical and Diagnostic Tests. All other authors declare no competing interests.