The importance and value of EQA for diagnostic genetic laboratories

Abstract

Quality control in a laboratory setting requires the establishment of effective training, standard operating procedures, internal quality control, validation of tests and external quality assessment (EQA). A structured quality management system subject to regular internal and external audits will minimise the error rate. EQA, therefore, gives assurance, both to patients and referring clinicians, that the diagnostic laboratory is competent to produce results that are reliable and accurate. EQA is educational and aims to improve and validate the overall quality of genetic service to the user. Regular EQA assessment compares laboratory performance against set standards and also allows comparison between laboratories. Sometimes EQA can also help to define good standards (best practice), although this does depend on the type of EQA test. EQA interprets best practice standards (=quality) into a numerical score (=quantity). While international bodies or professional organisations set these standards, EQA is able to assess whether these standards are met, with any omissions resulting in a reduction in the total score. Although EQA has an educational role rather than a punitive role, critical errors affecting clinical management will result in a laboratory receiving a poor performance categorisation. Accurate analysis and interpretation are essential quality parameters that require extensive knowledge of the aetiology of genetic abnormalities/disease and risk factors. Training of staff in interpretation of the results together with a comprehensive means of reporting normal and abnormal genetic results underpins the diagnostic service to the patient. Poor-performing laboratories are, therefore, encouraged to review their internal processes. EQA schemes that have been established for many years have seen improvements in the analytical and reporting content over time, thereby improving the quality of diagnostic service available to patients.

Introduction

Analysis of the genome currently incorporates three different disciplines: cytogenetics, molecular genetics and biochemical genetics. Results may be given for a specific gene or at a whole genome level. In addition, genetic conditions may arise from multiple genetic abnormalities of different aetiologies resulting in different clinical case scenarios. Therefore, to establish the correct result, a genetic test may stand alone or be used in conjunction with another genetic analysis technique such as fluorescent in situ hybridisation (FISH), multiple ligation-dependent probe amplification (MLPA), microarrays (arrayCGH), quantitative fluorescent PCR (QF-PCR), Southern blotting, etc.

Accurate results are essential in genetic testing. There may be only a single opportunity for testing and it is important to minimise repeat sampling in order to avoid the hazards and stress of unnecessary invasive sampling procedures and to provide timely results. Some patients have no clinical phenotype or symptoms, and with regard to prenatal and pre-implantation genetic diagnosis tests, diagnosis may be undertaken prior to any clinical phenotype being evident. Results often involve the patient or their family having to make important lifetime decisions. Consequently, it is essential that any diagnostic genetic laboratory has in place reliable procedures underpinned by a robust internal quality management system to minimise errors and failures and to reassure the patient and the clinician making the referral that acceptable international standards are being met (OECD 2005, 2007).

External quality assessment (EQA) is performed by bodies independent of the participating laboratories. The process usually involves the distribution of samples, materials or images to laboratories for analysis and reporting (prospective EQA), but some EQA schemes provide a retrospective assessment in which a selection of diagnostic cases is submitted from the laboratory (Hastings et al. 2008; Howell and Hastings 2006).

Ongoing commitment to participation in EQA is required for all aspects of the diagnostic service to confirm attainment of satisfactory performance and to ensure that the diagnostic procedures undertaken by a laboratory are reliable and accurate. EQA provides a tangible measure of performance, a means by which technical, analytical and interpretive skill can be measured and benchmarked. Also, by using EQA results as a reference standard, the laboratory is given a means by which it can benchmark its own performance against its peers and validate its internal quality control systems (Dequeker et al. 2001).

After receipt of the EQA results, the laboratory has the opportunity to review its performance and address any deficiencies. If poor performance is identified, internal audit should identify the corrective actions needed to improve the quality of the diagnostic service. EQA results will also form part of the annual management review of the laboratory quality management system. A review of the EQA records and any corrective actions will also be included in the external audits undertaken by inspecting bodies for the purposes of accreditation (ISO 15189 2003; OECD 2005, 2007).

EQA is a valuable educational tool, giving the laboratory an opportunity to review its internal standards and policies and, as EQA submissions are marked against existing guidelines, the participant is given information and advice on best practice. Although EQA providers are generally not responsible for setting best practice, EQA can have an influence on national or international best practice guidelines as a consequence of the variation found in laboratory performance and practices. EQA has also made a valuable contribution in the assessment of the performance of in vitro diagnostic devices (kits) in routine use or laboratory-designed assays used in several diagnostic genetic centres. Examples include the Cystic Fibrosis (CF) European Network EQA scheme finding erroneous results in a commercial CF kit due to inappropriate primers that resulted in a redesign of the kit; and in 2004, the European Molecular Genetics Quality Network (EMQN) EQA scheme for familial breast cancer demonstrated the failure of a primer set in common use in a national laboratory consortium (Elles and Kamarainen 2009; Dequeker, personal communication).

Technical, analytical and interpretive components of a genetic test may all be examined in the course of EQA. The interpretive element to any EQA assessment is encouraged in the ISO standards (ISO 15189 2003; ISO/IEC Guide 43-1 1997) and OECD guidelines (2007). Analysis comprises of establishing the genotype or karyotype. For some tests, a quantitative measurement may be required, for example, a determination of the number of triplet repeats in an allele of the gene causative of Huntington disease in a patient with symptoms of the condition, while for other tests, a quantitative risk calculation is necessary, such as the residual risk of an individual carrying a cystic fibrosis mutation following a test to exclude the most common mutations. In cytogenetics, accurate identification of a chromosome abnormality and its description using correct nomenclature is required. The interpretation is measured by examining and marking the accuracy and clarity of a genetic report in relaying information in an unambiguous fashion to the clinician regarding the clinical significance of the result, including risks and need for follow-up.

Genetic testing laboratories may be part of a department or division that includes, or has close links with, clinical genetic counselling services. In these situations, the laboratory interprets the genetic results in the context of the clinical referral/indication for the test (including any phenotype), any prior risk of an adverse outcome or risk of recurrence. The purpose of the laboratory report is to inform the counselling process by giving specialist information. The interpretative part of the report is also particularly important where referrals are from physicians/clinicians who are not familiar with genetic testing.

In countries where laboratories are not allowed to give any interpretation in their own report, the EQA scheme requires involvement of the clinical geneticist (or haematologist for oncology) to provide the necessary interpretive component in their report submission. This is because an EQA scheme sets out where possible to assess the overall service provided to the patient, including pre-analytical, analytical and post-analytical report processes, and not just the competence of the laboratory. An exception is permitted in some molecular genetics EQA schemes where laboratories can be assigned a more restricted technical role and are permitted to submit an assessment on the basis of genotyping accuracy only. However, only offering a technical EQA does not entirely conform to the OECD and ISO recommendations (OECD 2007; ISO 15189 2003; ISO/IEC Guide 43-1 1997).

External quality assessment

In countries where no national EQA programme exists, participation in a European or an international EQA scheme is appropriate. CAP, Cytogenetics European Quality Assessment (CEQA), CF Network, EMQN, European Research Network for evaluation and improvement of screening, Diagnosis and treatment of Inherited disorders of Metabolism (ERNDIM) and UK NEQAS are examples of such international schemes offering EQA in various aspects of the diagnostic service (Dequeker et al. 2001; Elles and Kamarainen 2009; Fowler et al. 2008; Hastings et al. 2006, 2007, 2008).

EQA scheme participants are identified by a unique code known only to the scheme organiser. Laboratories are requested to remove department and staff identifiers prior to submission. Anonymity ensures that the submission will be assessed fairly. Participation in EQA, though not laboratory performance, is published. The laboratory cannot be identified by any of the public data.

The EQA must, where possible, enable appropriate investigations to be undertaken given the referral indication, for example, screening for mosaicism, or FISH or PCR tests for specific recurrent oncology gene rearrangements. However, as cellular material or genomic DNA may be the analyte, laboratories must agree as a condition of their EQA participation that their analysis is strictly restricted to the clinical question indicated in the particular EQA challenge. Laboratories should not carry out analyses on loci not relevant to the clinical indication forming the EQA challenge.

EQA materials (e.g. DNA) are not certified control material so must not be run by the participating laboratories as controls for their diagnostic tests. EQA sample distribution is specifically exempt under the European Union In Vitro Diagnostic Device Directive (IVDD) provided such samples are not included as a clinical test control by the diagnostic laboratory, as this would then constitute using them as a component of an IVDD and potentially contravene the directive.

EQAs aim to mimic the normal diagnostic processes that occur in the laboratory, so in most instances, patient-derived biological materials or images are distributed together with the clinical referral details. Alternatively, a set of results or a case scenario may be provided for interpretation. The information on the EQA referral form may include mock patient demographics, the indication for testing including signs or symptoms, previous genetic tests/results and/or any relevant patient or family history.

An EQA assessment is designed to enable the laboratory to select its approach and analytical methods to answer the problem that is set. Materials or case scenarios selected are designed to reflect the range of cases referred to a diagnostic laboratory and will include cases with normal results. A complex case would only be distributed where the detection of a rarely seen genetic anomaly is particularly problematic. The genetic test may be required for either a specific genetic condition or gene/target combination, or sample type or whole genome analysis (e.g. arrayCGH, chromosome analysis). Genetic EQA schemes usually address only the more common genetic diseases although there is a need for EQA providers to address the possibility of assessing a wider range of rarer genetic diseases.

EQA assessments usually allow for use of a variety of test methods capable of achieving the correct result, for example, in prenatal rapid aneuploidy testing, MLPA, FISH or QF-PCR techniques could be accepted. Use of an inappropriate test or kit, however, would result in an unsatisfactory submission.

It is difficult for EQA schemes to repeatedly source and/or store sufficient quantities of appropriate biological material (e.g. blood, amniotic fluid) that may require prior culture for the genotypes of genetic conditions that are usually defined as rare. Where the genetic tests involve DNA, a previous diagnostic sample is often the only material source for EQA samples, although cell lines or reference materials that have been formally certified by an external standardisation body may provide an acceptable source of DNA. Sourcing DNA may not be an option for a rare genetic test where the quantity of tissue available is limited. However, EQA providers have overcome this obstacle by providing multiple images of the relevant test results online or via compact discs.

Validation of EQA material, consisting of an independent confirmation of the expected test result by at least two accredited (or reference) laboratories, is undertaken prior to the release to participants. If more than one technique can be used to obtain the result for the EQA sample, then the validation is carried out using all the different platforms/techniques.

Depending on the type of EQA, the EQA provider may request the data to be sent on a template document or, alternatively, allow the laboratory to submit reports in its normal report format. The former option allows essential informative data to be collected while the latter option assesses whether the standard report format used by the participant is suitable as a permanent medical record. Whichever submission option is used, it is important to assess the normal processes and not to ask a laboratory to unnecessarily adapt its procedures for the purpose of participation in an EQA exercise.

Reporting of EQA results

The components of the report will be assessed in the course of an EQA. It is expected that the report be clear and unambiguous with the karyotype/genomic copy number described in text as well as the correct nomenclature (HGVS, http://www.hgvs.org/mutnomen/; ISCN 2009). The International System for Human Cytogenetic Nomenclature (ISCN) and the Human Genome Variation Society (HGVS) mutation nomenclature have been developed to convey the precise nature of a genetic result, respectively, for a cytogenetic or molecular genetic test (HGVS, http://www.hgvs.org/mutnomen/; ISCN 2009). Although this nomenclature may seem confusing to non-geneticists, the correct nomenclature designation is important in order to avoid ambiguity when describing a genetic abnormality, and it, therefore, forms an integral part of any EQA assessment.

An interpretation of the findings should be given clearly indicating the diagnostic and/or prognostic significance of the results, including their relevance to the reason for referral, the patient's age and other clinical factors as appropriate. EQA can also assess whether the laboratory, when interpreting the significance of the results, gave sufficient consideration to the range of different genetic mechanisms and aetiologies that represent abnormal, variant and normal results. The limitations of any analysis should be documented, in particular if further tests may be required to complete or refine the interpretation or risk assessment. The report should also clearly indicate the robustness of the result (e.g. number of abnormal and normal cells, number of informative markers for abnormal QF-PCR results), indicate any specific limitations of the assay and define the tissue or fluid analysed and whether the results were obtained from cells, blastomeres, polar bodies, metaphases, interphase nuclei, RNA or DNA (or a combination of these). Probe sets or kits used to perform the genetic analysis must also be identified and include the name of the manufacturer.

The value of EQA

EQA assesses and surveys whether laboratories are fulfilling defined criteria, whether that is best practice guidelines or international standards. In the course of EQA assessments, several recurrent omissions or errors have been identified that involve either the clerical, technical, analytical or interpretive processes of a laboratory. Clerical errors may involve the wrong sex being stated on a prenatal cytogenetic report, inappropriate computer-generated caveats on reports, wrong patient name transcribed from the referral card and so on. Inaccurate transcription of essential data such as patient identifiers that compromises the integrity and accuracy of the genetic report will result in the submission receiving a lower mark. Such errors point to possible deficiencies in the internal quality management system of the laboratory, as an efficient mechanism that includes checking all the processes, and authorisation of reports will minimise mistakes when reporting diagnostic or EQA results (Hastings and Howell 2009).

Technical processes

Technical quality, for example, cytogenetic banding resolution, or chromatographic separation of organic acids or DNA primers/probes used, or copy number variations, or unclassified variants in the EQA material, is assessed for any adverse or inappropriate technical comments in the laboratory report. EQA assesses that there has been a systematic approach to technical preparation and examples of possible technical errors revealed by EQA include:

• Cross-contamination of preparations or different cultures—a problem that can be avoided by permitting only one sample to be handled at a time in the preparation area at any time and keeping vessels capped at all times (UK Clinical Molecular Genetics Society Best Practice Guidelines—Internal Quality Guidelines, http://www.cmgs.org.uk);

• Incorrect or mislabelling of samples or test results at the hospital or within a laboratory (DNA, culture vessels, slides, etc.) can be avoided by ensuring that information is directly transcribed and witnessed and by handling only a single sample at any time (UK Clinical Molecular Genetics Society Best Practice Guidelines—Internal Quality Guidelines, http://www.cmgs.org.uk). Parallel or duplicate culture systems can also minimise this problem (Hastings et al. 2006);

• Lack of appropriate investigations—supplementary investigations, e.g. Southern blotting or FISH may be required to confirm or exclude a disease/chromosome abnormality. Appraisal of incoming referral forms by a senior member of staff is advisable to clarify any ambiguities and to check no further investigations are required when the first part of the analysis has been completed (Association for Clinical Cytogenetics 2007; Hastings et al. 2006; UK Clinical Molecular Genetics Society Best Practice Guidelines, http://www.cmgs.org.uk).

Analysis

The primary measure of EQA performance is the ability of the participating laboratory to correctly detect the expected result (qualitative) or to achieve a quantitative measure within defined and appropriate limits. For any individual case, the amount of analysis and the preparation quality should comply with national or international guidelines (American College of Medical Genetics 1999; Sciacovelli et al. 2001; McGovern et al. 2003; Association for Clinical Cytogenetics 2007; Hastings et al. 2006; Hastings et al. 2007; EMQN Best Practice Guidelines, http://www.emqn.org; UK Clinical Molecular Genetics Society Best Practice Guidelines, http://www.cmgs.org.uk) given the understanding that guidelines often state the minimum requirement. A laboratory providing a high quality of service will frequently achieve a higher standard than is set out in the guidelines. Every case will be assessed individually by the EQA scheme, and that there may be circumstances in which the level of analysis may need to be greater than the minimum, for example, screening for clones or mosaics when the proportion of abnormal cells is expected to be low.

Occasionally, EQA identifies an incorrect result where the laboratory has not identified the discrepancy between its result and the clinical referral or there is unexplained difference between the current and a previous test (given in EQA), which has not been followed up by further analysis or alternative methods to ascertain the ‘real’ result. Examples include missing a second chromosome abnormality, detection of only one part of a reciprocal translocation or missing a clone because a partial or an incomplete analysis was undertaken. One of the gravest errors a genetic laboratory can make is to diagnose a genetic disorder in a patient who is in fact genotypically normal. An example of this type of error includes misclassifying normal polymorphic variants as abnormalities, an issue that may become more problematic with the introduction of arrayCGH (or whole-genome sequencing) into the diagnostic repertoire where the ‘normal’ variation at this high level of resolution is currently not fully known.

An error rate of 5–20%, dependent upon the degree of difficulty for each EQA, currently exists amongst the four European laboratory schemes (see Annual Reports from CEQA, CF Network, EMQN and ERNDIM). While this may be an overestimation of the true error rates with diagnostic samples, it clearly highlights the fact that a significant number of laboratories are in urgent need of improving their procedures and the quality of their diagnostic analysis.

Interpretation of results

EQA acts as a surveillance mechanism and identifies common errors including failure to follow routine procedures or professional guidelines which results in incorrect interpretation of the results. These have included unawareness of breakpoint heterogeneity, underestimating the level of residual disease in chronic myeloid leukaemia by not analysing neutrophils in follow-up FISH BCR/ABL studies (Reinhold et al. 2003), not taking into account the tumour load of a sample, misinterpreting colocalised FISH signals as fused or diffuse fusions as split signals or misunderstanding of the characteristics of a probe/primer set used in the analysis (Hastings 2009). The skilled geneticist will recognise these problems and use knowledge, published resources (including professional guidelines) and laboratory operating procedures to reach a dependable conclusion. A few examples include:

• The ability to distinguish between normal polymorphic variation and an abnormality;

• Knowing when and how to follow-up abnormal findings with supplementary methods for confirmation;

• Knowledge of genetic syndromes, for example, those having tissue-specific abnormalities such as Pallister–Killian syndrome;

• The ability to distinguish normal background level of breakage and the recurrent chromosome 7 and 14 rearrangements seen in cytogenetic lymphocyte culture;

• Recognising common age-related X chromosome gain or loss as opposed to true sex chromosome mosaicism in adult females;

• Dealing with sex chromosome mosaicism in prenatal samples, including knowing when it is not legitimate to predict a phenotypic outcome based on that of postnatally ascertained patients and recognising maternal cell contamination;

• Awareness of heterogeneity of gene mutations or chromosome breakpoints;

• Taking into account the tumour load of a sample, e.g. a low-level clone in a sample with a small tumour load is significant but may not be if the tumour load is high (depending on clinical indication);

• Misinterpreting the genetic abnormality or the aetiology of the genetic abnormality.

EQA assessment process

An EQA assessment panel includes experts in genetics who assess laboratory performance against pre-determined marking criteria, including a definition of a minimum satisfactory performance. Current professional guidelines, peer-reviewed publications and standard textbooks are used to determine the marking criteria. For each EQA case, the panel of assessors decide in advance the marking criteria for key elements/points expected to be in the report submission and assign a quantitative score for each key point. Omission of these key points results in a deduction from the total score for the EQA case. The same error is only scored once within a specific EQA and less essential issues will receive a comment only. If an incorrect genotype is given, the interpretation for this case cannot be marked.

The EQA assessment of a report includes establishing that the report contains a description of the likely implication for the patient of the detected genotype in establishing or excluding a diagnosis (giving any limits of exclusion), the likely clinical outcome and the implications for blood relatives (see best practice guidelines, e.g. Hastings et al. 2006). For some EQA cases, assessors may expect the report to contain options such as further tests, health surveillance or advice regarding prenatal diagnosis. Reports should be clear, concise and unambiguous so the genetic result is clearly communicated to the recipient. Where the implications of a test result are clinically significant, for example, establishing a diagnosis of a heritable disease, the report should recommend and explain why referral of the patient for genetic counselling is appropriate. It is essential that reports are not interpreted as directive to the patient; this is especially applicable for prenatal results.

Occasionally, a comment may be included on the reporting style and clarity and, where applicable, refer to any best practice guidelines on clinical reporting. This applies where irrelevant information or poor structure to a report makes it potentially misleading or hides the real implication of the result.

EQA and poor performance

One of the important functions of EQA is to be educational and improve the quality of the diagnostic service through peer group review. EQA providers have a duty to protect the public from sub-standard and potentially dangerous clinical practice. Laboratories that have a critical error in genotype or interpretation that would significantly alter patient management will receive a ‘poor performance’ categorisation. Such poor-performing laboratories are requested to put in place corrective measures to overcome the detected deficit. In some instances, the laboratory may be offered assistance or technical advice or required to participate in an additional round of EQA.

If poor EQA performance does not improve, the EQA scheme or an official body to which it is accountable may take additional measures to protect the public. Unfortunately, a minority of laboratories choose to withdraw from the EQA scheme when their laboratory has performed poorly in EQA instead of addressing the internal analytical or interpretation problem. Conversely, EQA providers have also reported that some laboratory directors decide to withdraw from providing individual services when a poor performance occurs and, therefore, the public are protected.