Analysis of “Capillary and Venous Blood Glucose Accuracy in Blood Glucose Meters Versus Reference Standards: The Impact of Study Design on Accuracy Evaluations”

Abstract

The article by Macleod and coauthors outlines an accuracy study of two blood glucose monitoring systems (BGMSs) against glucose-oxidase- and hexokinase-based assays showing how the reference/comparison method and inappropriate sample comparisons can affect accuracy conclusions. The dangers of independent institutions producing inappropriate conclusions when the methodology used for product regulatory or registration accuracy requirements is not according to best practice are stressed. The authors highlight several important aspects of a multistep accuracy evaluation protocol. However, it is essential that anyone undertaking or reporting BGMS accuracy studies should have a clear understanding of each and every protocol point, best practice, and how each can influence accuracy conclusions. Claims against regulatory accuracy criteria should be made only if the detailed specified protocol design and analysis is followed.

The article by Macleod and coauthors¹ highlights problems that can be encountered in evaluations of blood glucose meter system (BGMS) accuracy when inappropriate protocols and different reference/comparison methods are used. An ISO 15197:2013/2015 user trial was undertaken with a comparison of blood sample results generated on two BGMSs against a glucose-oxidase-based YSI 2300 reference analyzer and hexokinase-based methods on laboratory Siemens ADVIA 2400 and Abbott Architect C16000 analyzers. Illustrations are provided showing how the reference/comparison method used, and inappropriate sample comparisons can both significantly affect accuracy claims. The authors indicate anecdotally that accuracy evaluations, producing inappropriate accuracy conclusions when methodology used is not according to best practice, are being carried out and that as part of addressing in-vitro diagnostic regulation, notified bodies will conduct their own accuracy assessments of product performance where no requirements on test methodology are provided. This could lead to independent testing institutions developing their own test methods different from best practice resulting in increased variation in blood glucose accuracy measurement. Device assessments of accuracy review as per country specific requirements for product registration could be similarly affected.

A range of articles outline how to assess and present accuracy^2-4 and best practice quality guidelines or recommendations for undertaking and reporting appropriate evaluations of BGMS accuracy are available from several sources.^5,6 Determining accuracy is however a complex process dependent on correct procedural design. Several standards/guidelines^7,8 outline requirements for BGMSs used by lay users performing self-monitoring of blood glucose. Post launch accuracy investigations include the Diabetes Technology Society (DTS) Surveillance program.⁹ Each describes detailed protocols for the assessment of accuracy as well as accuracy criteria. Although each has its own, separate opinion regarding the requirements,¹⁰ valid compliance statements with minimum accuracy criteria should be made only if the specified protocol is used and best practice understood.

Macleod et al indicate that a thorough understanding of reference instrument performance, including imprecision, calibration, and traceability at the time of the evaluation is critical to understanding any potential impact on BGMS bias. This is essential as the ISO 15197:2013/2015 standard⁷ (although system accuracy and user performance sections differ) and FDA guidance on OTC BGMs⁸ allow the use of reference/comparison methods different from the manufacturer’s stated reference instrument that was used to calibrate BGMSs against. It is however important to understand that many other aspects of best practice protocols can also influence accuracy evaluation data.

Although the correct evaluation of BGMS accuracy is complex, many publications have outlined guidelines to ensure this is done correctly. It is essential that deviation from protocol guidelines, inappropriate performance, or systematic bias in the reference/comparison method is not mistaken for inaccuracy in the investigated BGMSs.¹¹ When interpreting system accuracy data, one should be aware that the use of different methods/analyzers and deviations from best practice specific protocols have an influence on the results obtained and hence compliance with accuracy criteria. This becomes increasingly important as minimum accuracy acceptance criteria become tighter.

In addition to highlighting potential issues with the reference/comparison method and correct sample comparison the Macleod et al article incorporates many protocol details important in ensuring valid conclusions. Irrespective of if ISO 15197:2013/2015 or FDA guidance is used, it is essential that all aspects of the specified protocol are followed. Over 20 individual protocol points that must be addressed have been outlined previously and best practice identified.⁶ These are grouped below into categories based on (1) the general protocol, (2) samples and subjects, (3) reference/comparison method, and (4) data analysis, assessment, and presentation. It is essential any accuracy evaluation study or publication addresses and clearly documents each issue.

General Protocol

The type of evaluation and the specific standard/guidelines followed determines the protocol used. This will include for example details of if it is an analytical accuracy or user study, the scope, test conditions such as temperature, humidity, and location, procedural details, inclusion/exclusion based on the system’s instructions for use (IFU) or standard/guideline criteria, and who performs the tests and how. Details of the meters and strips used should also be provided including how they were obtained and stored, the number of different strip lots used together with details of their batch numbers, expiry dates and when the evaluation was performed, and the specific enzyme the strip chemistry is based on.

Accuracy evaluation studies or publications should ensure the following in relation to the general protocol:

• - Only appropriate, approved, and full protocols are used to assess BGMS accuracy against specific standard/regulatory criteria.

• - There is a clear unambiguous statement that all aspects of the protocol have been followed, supplemented with sufficient individual protocol point details to enable a full understanding and valid interpretation of results and conclusions. Accuracy publications^12-14 in relation to ISO 15197:2013/2015 illustrate appropriate inclusion.

• - All aspects of the BGMS IFU are understood and considered in relation to the standard’s protocol guidelines. These include claims, appropriate samples, operation, and inclusion/exclusion criteria.

• - Best practice aspects of all standards/guidelines are considered.

Samples and Subjects

Details of the samples and subjects to be used in specific standards/guidelines vary. The number of subjects and samples, handling of samples, the spread and distribution of sample glucose concentrations needed, and the use of specimens in which glucose concentration is altered all vary and should be appropriate when accuracy claims are made against specific criteria. A BGMS’s IFU will state the intended use of fresh capillary samples with any alternative sample type mentioned if appropriate. Any suitable anticoagulant/preservative will also be stated. As shown by Macleod et al, a sample type, such as venous blood, not claimed as suitable for use in the IFU should not be used for accuracy determination.

Accuracy evaluation studies or publications should ensure the following in relation to samples and subjects:

• - Fresh capillary blood samples are compared on both the BGMS and the reference/comparison method in line with standard/guidelines, that is, comparing like with like. Comparison of capillary against venous samples is not appropriate and introduces both potential physiological and analytical differences.

• - An appropriate number of samples, providing the specified number of data points, are taken from the number and type of individuals identified in the standard/guidelines. Samples satisfy the glucose concentration spread requirements outlined.

• - Use of altered specimens is as detailed in the specific standard/guidelines.

• - Possible effects due to glycolysis or sample handling are appropriately eliminated or minimized.

Reference/Comparison Method

As demonstrated by Macleod et al the reference/comparison method used in accuracy studies can affect a system’s apparent compliance with standard/guideline criteria. This warrants discussion and understanding of which reference/comparison methods should be used, their performance, acceptance or exclusion of results, and differences between methods.

The use of an appropriate well-managed reference/comparison methods remains of paramount importance in performing accuracy evaluations correctly.¹⁵ Its role becomes even more important with increasingly strict performance requirements for BGMSs. Reference/comparison analyzer errors and systematic measurement differences (biases) between methods that previously may have been considered acceptable may now be consequential and particularly important for comparison against analytical accuracy criteria which are more stringent than clinical accuracy requirements.

Which Reference/Comparison Method

Previous best practice⁶ and Macleod et al¹ required BGMS results to be compared against results generated by the reference/comparison method specified by the manufacturers. ISO 15197:2013/2015 (system accuracy section 6.3.2) however states “A reference measurement procedure that conforms to the traceability requirements of ISO 17511 shall be used to assign the glucose reference values.” FDA guidance quotes “comparator method refers to a laboratory-based glucose measurement method that has been well-validated for precision and accuracy and that is traceable to a higher order reference material and/or method.”

Ideally, a “gold standard” analytical method such as isotope dilution mass spectrometry should be used in accuracy studies or traceability to it shown. However the method is lengthy, is expensive, and has a lower sample throughput than glucose-oxidase- or hexokinase-based laboratory analyzers¹⁶ against which currently available BGMSs are calibrated.^10,17 In recent accuracy evaluations against ISO 15197:2013/2015 criteria 68% of publications used a glucose-oxidase-based comparison method with 61% using a YSI, 42% of studies used a hexokinase-based method with 30% performed on a Cobas instrument. Both methods were used for comparison measurements in 11% of studies. Due to the impact on accuracy evaluations, harmonization of comparison methods and definition of specific requirements should be established in calibration of BGMSs and performance analyses.¹⁷

Performance of the Reference/Comparison Method

Macleod et al correctly indicate it is critical to understand performance of the reference analyzer relating to imprecision, calibration, and traceability such as to National Institute of Standards and Technology (NIST) standards. Performance and traceability of the reference/comparison method should be determined and reported when subject samples were analyzed. Standards/guidelines however differ on statements regarding what is appropriate performance. ISO 15197:2013/2015 and the CLSI guideline¹⁸ describe procedures to verify trueness and precision of the reference method, for example, by using certified reference material.⁵

Reference/comparison method analyzers should be calibrated according to manufacturer’s instructions, and best practice would suggest to ensure traceability a minimum of four NIST glucose standard reference samples spanning the measurement range of the BGMS should be tested and bias estimated with each batch of study samples.⁹ Theoretical mathematical recalibration and adjustment of the comparison method results according to certified NIST targets have been reported.^19,20 While recalibration cannot eliminate imprecision, it could harmonize different laboratory methods with respect to bias.²⁰

ISO 15197:2013/2015 states that any reference method chosen should have adequate performance criteria, and information obtained from the manufacturer in relation to compliance with traceability requirements of ISO 17511 may be used to make this determination. It also states that the trueness and precision of the reference IVD medical device shall be verified during the performance evaluation, although concrete requirements such as allowed deviations are not included. FDA requirements only state that the reference method has to be “well-validated for precision and accuracy.” The CLSI¹⁸ and DTS⁹ guidelines require the reference/comparison method to fulfil specific performance criteria in that bias should not exceed 2.2%, analytical imprecision 2.9%, and total error 6.9%. This should be considered best practice and incorporated and reported in accuracy evaluations and publications. Quite correctly it has been suggested that in consideration of the tight accuracy criteria for BGMSs, the standards should pay much more attention to the definition of clear criteria for the reference method and that they should be stricter than those proposed by national directives for routine laboratory systems.⁵

As highlighted in a previous analysis article,²⁰ full details and data on the trueness and imprecision of the reference/comparison method should be included in accuracy studies and publications. The amount of quantitative information provided in recently published ISO 15197:2013/2015 accuracy studies vary from none, to a simple statement of imprecision CV% at a single glucose concentration, to detailed information on bias and imprecision at each of a range of glucose concentrations spanning the clinical analytical range. Jendrike et al¹⁴ indicate that traceability to NIST SRM 965b was confirmed during the study and that bias and imprecision were ⩽1.0% and ⩽1.3%, respectively, for YSI 2300 STAT Plus and ⩽0.6% and ⩽1.0%, respectively, for Cobas Integra 400 Plus when using manufacturer-provided control materials.

The Macleod et al article states calibration, maintenance and use of the hospital comparison analyzers were per individual hospital protocol and traceability and performance requirements were maintained throughout the study. In addition, trueness and precision of the YSI 2300 glucose analyzer were verified during the test process by daily quality control measurements which followed LifeScan internal standard operating procedures. Although mean bias is shown for different methods further details and values relating to reference/comparison method performance, traceability, calibration, and quality control would have been useful. This is particularly relevant to the YSI comparison method, which although in widespread use, is complex and variable in terms of reported maintenance²¹ calibration and checks,^22,23 traceability,²⁴ target and acceptance values, imprecision, and the exact procedure used for sample analysis.^15,21,25

Acceptance or Exclusion of Reference/Comparison Method Results

Specific requirements on acceptance or exclusion of results vary with individual standards/guidelines. FDA guidelines indicate duplicate measurements with the reference method are optional. ISO 15197:2013/2015 however states at least duplicate reference measurements are mandatory and measurements before and after the BGMS measurements are required to ensure glucose concentration stability. Data exclusion criteria and drift analysis dictate that if samples measurements differ from each other by more than 4 mg/dl for values ⩽100 mg/dl, or by more than 4% for values >100 mg/dl, then the data cannot be used. CLSI guidelines also exclude data/results if there is >4% (4 mg/dl) deviation from certified material. An indication or definition of how close NIST results should be when using specific standards/guidelines would be useful. ISO 15197:2013/2015 excludes samples with interferences such as hematocrit that fall outside the BGMS’s stated range in the IFU. FDA guidance however specifies inclusion of all hematocrit levels with an ideal range stated.

Differences Between Reference/Comparison Methods

In line with previous concerns^15,17,20 Macleod et al highlight recognized differences between reference/comparison methods and demonstrate that results obtained when comparing YSI to other hospital analyzers depend on the glucose level. Systematic bias differences between glucose oxidase and hexokinase reference/comparison methods have been previously described^17,26 with deviations of up to 8% observed, even though traceability of both methods was established.^20,26 Differences in the YSI measured accuracy versus the isotope dilution mass spectrometry reference method have also been reported, as has influence by both hematocrit and plasma water concentration in a glucose concentration dependent manner relative to a hexokinase method. Low hematocrit levels can introduce up to a 15% positive bias and high hematocrit levels up to a –18% bias.²⁷ Differences of approximately 4% between YSI instruments in different locations and selection of YSI analyzers that exhibited the least bias relative to serum target values have been reported.¹⁵

Accuracy evaluation studies or publications should ensure the following in relation to reference/comparison methods:

• - An appropriate reference/comparison method in line with standards/guidelines is employed and details of use, calibration, traceability, quality control, and maintenance provided.

• - The imprecision and bias of the method at the time of analysis is reported and within guidelines.

• - NIST standards are run with each batch of specimens to ensure traceability and bias reported.

• - The enzymes used in the BGMS strips and reference/comparison method, and the manufacturer’s stated comparison method stated.

Data Analysis, Assessment, and Presentation

Different requirements for accuracy data analysis, assessment, minimum acceptance criteria, and presentation are outlined in specific standards/guidelines. These include analytical accuracy criteria, graphical and tabular presentation of accuracy and bias, scatter plot and regression analysis, and clinical accuracy error grid analysis.

Accuracy evaluation studies or publications should ensure the following in relation to data analysis, assessment and presentation of results:

• - Data are recorded, analyzed, and presented as outlined in the specific standards/ guidelines used.

• - Statements on compliance against criteria outlined in standards/guidelines are made only if all aspects of their specific protocol are complied with and reference/comparison performance at the time of analysis was appropriate and acknowledged and all aspects of bias against NIST standards considered.

Possible Differences in Compliance When Using Different Reference/Comparison Methods

Due to differences in bias in glucose oxidase and hexokinase reference/comparison methods, it is acknowledged that although showing a high level of accuracy against ISO 15197:2013/2015, BGMSs calibrated against glucose oxidase or hexokinase would likely show systematically different results.¹⁶ Many commercially available BGMSs have been evaluated using both glucose oxidase and hexokinase reference/comparison methods, one of which would have been the manufacturer’s designated comparison method, and many shown to either meet or not meet analytical accuracy criteria by both methods.^14,17,28 There are however a few examples^13,14,16 where BGMSs satisfied accuracy criteria when using the manufacturers designated comparison method but with use of another reference/comparison method, 2 out of 3 strip lots failed to meet analytical accuracy criteria and were therefore classified as noncompliant despite meeting clinical error grid clinical accuracy criteria.

As stressed by Macleod et al, this reinforces the importance of when interpreting accuracy data, not only should it be ensured that an appropriate study design has been used, but one should also be aware that the use of different reference/comparison methods can influence results.

In conclusion, when BGMS accuracy evaluations are performed and interpreted they should:

1 Ensure and state an appropriate protocol has been used and fully, not partially, complied with and best practice considered

2 Have details of the reference/comparison method used and its performance at the time in terms of imprecision, trueness, bias, and traceability

3 Be aware that the use of different reference/comparison methods can have an influence on results and hence compliance with accuracy criteria

4 Only make claims relating to compliance with minimum accuracy regulatory criteria if the appropriate detailed specified protocol and analysis is followed