A longitudinal evaluation of performance of automated BCR-ABL1 quantitation using cartridge-based detection system

Abstract

SummaryAn automated cartridge-based detection system (GeneXpert; Cepheid) is being widely adopted in low throughput laboratories for monitoring BCR-ABL1 transcript in chronic myelogenous leukaemia. This Australian study evaluated the longitudinal performance specific characteristics of the automated system.

The automated cartridge-based system was compared prospectively with the manual qRT-PCR-based reference method at SA Pathology, Adelaide, over a period of 2.5 years. A conversion factor determination was followed by four re-validations. Peripheral blood samples (n = 129) with international scale (IS) values within detectable range were selected for assessment. The mean bias, proportion of results within specified fold difference (2-, 3- and 5-fold), the concordance rate of major molecular remission (MMR) and concordance across a range of IS values on paired samples were evaluated.

The initial conversion factor for the automated system was determined as 0.43. Except for the second re-validation, where a negative bias of 1.9-fold was detected, all other biases fell within desirable limits. A cartridge-specific conversion factor and efficiency value was introduced and the conversion factor was confirmed to be stable in subsequent re-validation cycles. Concordance with the reference method/laboratory at >0.1–≤10 IS was 78.2% and at ≤0.001 was 80%, compared to 86.8% in the >0.01–≤0.1 IS range. The overall and MMR concordance were 85.7% and 94% respectively, for samples that fell within ± 5-fold of the reference laboratory value over the entire period of study.

Conversion factor and performance specific characteristics for the automated system were longitudinally stable in the clinically relevant range, following introduction by the manufacturer of lot specific efficiency values.

INTRODUCTION

The BCR-ABL1 fusion transcript characterises the myeloproliferative neoplasm chronic myelogenous leukaemia (CML) and is the basis for diagnosis and monitoring of the disease.¹ Karyotyping, fluorescent in situ hybridisation (FISH) and reverse transcriptase polymerase chain reaction (RT-PCR) are all techniques that are employed to establish a diagnosis of chronic myeloid leukaemia. In about 90–95% of subjects with CML in chronic phase, where t(9;22) is the sole abnormality, monitoring of minimal residual disease with quantitative RT-PCR (qRT-PCR) is standard clinical practice for follow-up.¹

Over the last few years, an automated cartridge-based detection system (GeneXpert; Cepheid, USA) has become available with comparable limits of detection, sensitivity, specificity, and a more rapid turnaround time with a favourable cost analysis.² The automated cartridge-based assay (GeneXpert) performs RNA extraction, cDNA production, nested real-time PCR and signal detection in a single cartridge directly from a whole blood sample. Markedly raised white cell count, delay in testing (>48 h sample processing time) and change in the lot numbers of the cartridges have been recognised as the cause of error for the cartridge-based automated system.³

A longitudinal evaluation of performance specific characteristics of the automated cartridge based system was undertaken at Hunter Area Pathology Service/Pathology North-Hunter, Newcastle, NSW, Australia (HAPS), which was designated the primary laboratory, against a manual technique performed at SA Pathology (Adelaide, South Australia), which was the reference laboratory. The SA Pathology laboratory serves as and is recognised worldwide as a reference centre for BCR-ABL1 testing in CML.^4,5 This study was specifically undertaken to evaluate independently the validity and longitudinal stability of the conversion factor as well as other performance characteristics of the automated cartridge-based system. The specific characteristics evaluated included the mean bias, proportion of results within specified fold difference (2-, 3- and 5-fold), the concordance rate of major molecular remission (MMR) as well as concordance across a range of IS (international scale) values on paired samples using the automated cartridge system compared with the reference manual qRT-PCR.

DESIGN AND METHODS

Laboratories, subjects and validations

The automated cartridge-based qRT-PCR on GeneXpert was performed at HAPS and validation by the reference manual qRT-PCR was undertaken at SA Pathology. The study cohort was chosen from a total of 275 peripheral blood samples from subjects with CML who were tested consecutively between January 2011 and June 2013 for either routine diagnosis or monitoring. Out of these, 129 paired samples were chosen as they had both whole blood and Trizol aliquots available. All samples were processed within 24 h of receipt at HAPS laboratory and Trizol was used to stabilise the RNA prior to being sent to the reference laboratory. Four samples were excluded (inadequate), so 125 paired samples were tested across both laboratories. The fold-change and concordance was evaluated on 109 samples, as samples where the transcript was either undetectable or >10% IS were excluded for this comparison.

The conversion factor determination was completed in May 2011 and subsequent revalidation cycles were planned prospectively at approximately 6 monthly intervals or with lot number change for automated cartridges, whichever occurred earlier (between July 2011 and June 2013).

Testing BCR-ABL1 by GeneXpert automated cartridge technique and manual reference qRT-PCR method

GeneXpert cartridges and reagent solutions were purchased from the manufacturer (Cepheid, USA). The assays were run, according to the manufacturer's protocol, on the GeneXpert IV DX system for the entire period of the study. The Trizol sample was analysed by manual duplicate qRT-PCR analysis at the reference laboratory as previously reported.⁵

Establishment, stability of conversion factor and concordance analysis

The difference between the manual reference method and the automated system was plotted against the mean of the methods for bias plot (Bland and Altman). The 95% limits of agreement was estimated by mean difference ± 1.96 standard deviations (SDs) of difference, which provided an interval within which 95% of differences between measurements were expected to lie. The antilog of estimated mean bias between methods was designated as conversion factor.⁶ Only raw values obtained on the automated method were used for measurement of conversion factor in all validation cycles. The manufacturer (Cepheid) provided a lot specific conversion factor as well as specific efficiency (E _ΔCt) values after the second revalidation cycle for the automated cartridge. The manufacturer's conversion factor was compared to the conversion factor obtained by the inter-laboratory revalidation cycles. The values for reporting MMR ≤0.1 IS) were defined as per the International Randomized Study of Interferon and STI571 (IRIS) trial and the subsequent international standardisation efforts.^6–9

Statistics and ethics

The data were analysed and graphed using XLSTAT (Addinsoft, USA) and Analyse-it software (UK), where appropriate. The study was conducted as per the ethical requirements for a quality assurance project at both sites.

RESULTS

Patient bias conversion factor determination and first re-validation

The initial conversion factor obtained after comparison with the reference method was determined to be 0.43 (calculated as antilog of bias). It was observed that 86% of the samples fell within 5-fold of the reference laboratory value when this conversion factor was employed. The first re-validation that followed demonstrated stability of the conversion factor at 0.43 (Table 1). The log bias ranged from −0.23 to 0.08 in the first revalidation and results were obtained on a cartridge with the same lot number as the initial base line conversion factor determination (Fig. 1A).

Table 1. Conversion factor for each cycle and comparison of values pre and post-conversion as 2-, 3- and 5-fold difference from reference laboratory.

Testing cycle/parameter	Calculated conversion factor (manufacturer's values)	2-fold raw (%)	2-fold post-conversion (%)	3-fold raw (%)	3-fold post-conversion (%)	5-fold raw (%)	5-fold post-conversion (%)
Baseline conversion factor determination (n = 21)	0.43 (NA)	7 (33)	14 (67)	11 (52)	17 (81)	81 (86)	18 (86)
1st revalidation cycle (n = 25)	0.43 (NA)	13 (52)	15 (60)	17 (68)	20 (80)	21 (84)	23 (92)
2nd revalidation cycle (n = 22)	0.47 (NA)	15 (68)	13 (59)	18 (82)	19 (86)	22 (100)	21 (95)
3rd revalidation cycle (n = 12)	0.47 (0.47)	3 (25)	11 (92)	8 (67)	11 (92)	12 (100)	11 (92)
4threvalidation cycle (n = 29)	0.47 (0.47)	6 (21)	24 (83)	15 (52)	26 (90)	26 (90)	28 (97)

The manufacturer also provided a lot specific efficiency value for cycles 3 and 4 which was incorporated into the instrument software after July 2012. The conversion factor determined by the manufacturer is provided in brackets for cycles 3 and 4. The % is calculation of the percentage of the reference laboratory's results.

Fig. 1.

Bias plots for initial baseline conversion factor determination (A) and four subsequent revalidation cycles (B–E).

Subsequent revalidations

A total of five cartridge lot number changes occurred in the period of the study. Two cartridge lot number changes occurred between the second and third revalidations in short time frames of within 4 weeks where no separate revalidation could be undertaken. This resulted in a total of four revalidations for the entire study period.

The log bias for revalidation cycles 2 to 4 ranged from −1.9 to −1.0 (Fig. 1B–E). A significant negative deviation from the reference method was seen in cycle 2 where mean bias was −1.9 (Fig. 1C). A change to instrument software utilising lot specific conversion factor and ‘efficiency values’ for cartridges was introduced after the second validation; this value being unavailable to the instrument users prior to this time point (July 2012). The bias was recalculated with raw values in the subsequent cycles 3 and 4. It was also observed that the manufacturer's conversion factor that was subsequently provided matched the conversion factor obtained by our comparison with the reference laboratory.

Proportion within 2-, 3- and 5-fold difference from reference method

The proportion of samples falling within 2-, 3- and 5-fold of the reference laboratory values prior to and after the conversion to the IS for each of the revalidation cycles are shown in the Table 1. Across the four revalidation cycles, 94% of the results fell within 5-fold of the reference lab values (range 92–97%). There were only five samples across the four revalidation cycles that were greater than 5-fold difference from the reference laboratory values.

Concordance for MMR, high and low IS values

An overall concordance of 85.7% (54/63) for MMR was observed across the four revalidation cycles. The overall concordance for any IS value obtained in HAPS, when compared to the reference laboratory, was 76.1% (83/109). Table 2 shows specific grouping of results according to the level of international standard values. A concordance rate of 80% was achieved in IS values ≤0.001, concordance of 78.2% was achieved in values between >0.1 and ≤10% IS whereas 86.8% concordance was achieved in values >0.01 to ≤0.1 IS. A concordance rate of 75.5% (74/98) and 85.7% (54/63) was observed for all values >1 and ≤0.1 IS respectively, the two clinically important decision points (data not shown in table).

Table 2. Concordance at various IS values and for MMR.

Samples as per IS values	HAPS GeneXpert assay automated cartridge based method	SA Pathology manual reference qRT-PCR method	Concordance between the two methods n (%)*
Values >0.1 and ≤10 IS	39	46	36 (78.2)
Values >0.01 and ≤0.1 IS	53	53	46 (86.8)
Values >0.001 and ≤0.01 IS	17	10	8 (80)

^*The % is calculated as the percentage of the reference laboratory's results.

DISCUSSION

The IRIS trial demonstrated that the international scale for reporting of qRT-PCR results for BCR-ABL1 transcripts is key to the optimal management of CML. The baseline value, which represents 100% on IS and the 3-log reduction from standardised baseline, which is fixed at 0.1 IS (as criteria for MMR) are considered the two critical values.⁷ Failure to achieve MMR and more recently ≤10% at 3 months post-initiation of tyrosine kinase inhibitor (TKI) therapy represents a warning response.^10,11 We report the results on a large independent longitudinal evaluation of performance specific characteristics which includes validity and longitudinal stability of conversion factor, proportion of results within 2-, 3- or 5-fold difference, MMR concordance and within a range of IS values of the automated cartridge system for quantitation of BCR-ABL1 across two laboratories in Australia.

Several previously published studies provide variable information regarding some features of the automated cartridge-based system but none have comprehensively evaluated the specific performance characteristics of the automated cartridge-based system. During the process of the initial standardisation of the automated cartridge-based technique, Winn-Deen et al. showed good concordance in spite of a high coefficient of variation (CV) in samples with qRT-PCR results <0.01 IS.³ In a study by Jobbagy et al., the cartridge based system was able to detect BCR-ABL1 reliably and correlated well with the reference assay.¹² They determined that the CVs were around 40% for IS values in range of 0.1 and this was comparable to the qRT-PCR technique employed in their experiments. Cayuela et al. assessed inter-laboratory variability between the two methods in which they observed a 90% MMR concordance with the concordance dropping to between 60–70% at deeper levels of molecular response.² In the recent study by O’Dwyer et al., the performance of the GeneXpert system in a cohort of subjects on nilotinib showed a progressive decline in correlation between the methods, as the log IS level declined.^13,14 An international collaborative effort from laboratories using the manual reference method described three groups of laboratories in comparison with the Adelaide reference laboratory with 91% (group 1), 74% (group 2) and 60% (group 3) MMR concordance rates.⁶ In this study, it was also observed that 95% of the results fell within 5-fold of the reference laboratory's value for the group 1 laboratories.

The initial conversion factor established for the HAPS facility on the cartridge-based system was 0.43 and the results for first validation cycle followed the recommendations of previous standardisation efforts.⁶ The mean bias was within acceptable limits except for the second revalidation cycle. A significant negative deviation of the mean bias from the reference method (−1.9) was noted during this second revalidation cycle resulting in lower than expected results on the automated system. Discussion with the manufacturer confirmed that there had been a change in lot numbers of the cartridges just prior to this revalidation. The manufacturer, at that point in time, had also independently observed a drift in their results leading to introduction of a lot specific conversion and efficiency factor to reduce lot-to-lot variability of cartridges (personal communication). Cartridges that included software-incorporating lot-specific ‘efficiency values’ were subsequently introduced and revalidation in cycle 3 as well as cycle 4, returned results of mean bias within desirable range. In this study, revalidations were planned to coincide with lot number changes for the automated cartridges or every 6 months, whichever occurred earlier. This enabled the detection of the negative bias in the second revalidation cycle.

It was observed that across the four revalidation cycles 92–97% of values were within 5-fold of reference laboratory value after conversion, which is comparable to the results of group 1 (best performing) laboratories. The automated cartridge system showed an overall MMR (≤0.1 IS) concordance of 85.7%, which was close to group 1 (best performing) laboratories in the multicentre international study.⁶ An 86.8% concordance was observed for values >0.01 to ≤0.1 IS which was also close to best performing (group 1). An 80% concordance was achieved in values between >0.001 and ≤ 0.01 IS performing between group 1 and group 2 laboratories from the international standardisation effort.⁶ Between values of >0.1 and ≤10 IS, the concordance dropped to 78.2% with a performance that is closer to group 2 laboratories in the international study.⁶ The discordance between laboratories seen in up to 22% (the range being 14–22% as shown in Table 2), particularly seen at higher IS values, needs to be considered when evaluating samples for BCR-ABL1 transcript levels at 3 months or levels that determine time to achieve t^1/2 from baseline, both of which are now considered important for long-term prognosis based on emerging data.^10,11,15,16 This has direct clinical relevance where consecutive samples in a given patient are measured in different laboratories or using different techniques at ‘clinically decisive’ transcript levels. For example, achievement of a 1% and subsequent 0.1% is important and measuring the levels by two methods may give results with several-fold difference. In the international study comparing 38 laboratories that used a variation of manual RT-PCR technique, the MMR concordance ranged from 60% to 91%.⁶ In our study a 75.5% and 85.7% concordance rate was observed at 1% and 0.1% IS and this reflects the need for serial monitoring in any given analytical system.

Our study did not specifically evaluate the clinical risk scores, treatment regimens and specific clinical outcomes with the PCR results. We also did not evaluate values ≤0.001 (close to MR4.5 value) given the very few samples in this range (n = 6) from the initial cohort. As the values approach 0.0032 IS or less, the limits of detection by any current methodology are potentially reached. At very low levels of minimal residual disease (MRD), stochastic effects become more important and imprecision increases in quantitative assays.¹⁷ Replicate assays may improve sensitivity but this is difficult to achieve in the diagnostic setting where the starting material is usually limited and costs prohibitive. Moreover, the actual rate of degradation may vary between whole blood and Trizol samples, thus introducing a pre-analytical variable that is difficult to control for and may well influence the discrepancies noted at this low level of detection.

Apart from inbuilt housekeeping genes and manufacturer's controls, independent control samples are not commercially available for quality assurance of the automated cartridge system. Most external quality assurance programs use Trizol samples, not whole blood, and have been developed to be suitable for manual qRT-PCR systems.^18–20 The manufacturer of the automated system now reports standardisation of every lot of cartridge with a World Health Organization (WHO) BCR-ABL1 standard and provides an IS conversion factor as well as efficiency factor in order to eliminate the source of variation as observed in the second validation cycle in this study. This has ensured the overall stability of the conversion factor. The recent development of synthetic armoured RNA quant (ARQ) analytical reference panels calibrated to WHO primary standards, for generating four level curves, is also likely to achieve greater standardisation.²⁰ At this stage, we have continued to perform validations against the reference laboratory at 6 monthly intervals or at every change in lot number to continue monitoring the performance of the automated system. This also provides an opportunity to observe concordance at high or low IS values which are emerging as critical values for management of CML given that greater numbers of low throughput laboratories, particularly in the Asia-Pacific region, are now adopting the automated cartridge-based system.²¹ A collated international experience in the form of a prospective study would be useful to determine how intensively such systems need to be monitored and revalidated.

Conflicts of interest and sources of funding: The authors state that there are no conflicts of interest to disclose.