Assessing the Quality of Publications Evaluating the Accuracy of Blood Glucose Monitoring Systems

Abstract

Many studies determine the performance of blood glucose monitoring (BG) systems. Correct evaluation is, however, complex, and apparent contradiction of results creates confusion. This study aimed to provide an overview of frequently made errors and to develop easy-to-use checklists to verify the quality of such studies. Building on the work from Mahoney and Ellison and subsequent re-evaluation, study designs of accuracy studies were assessed, and best practice and internationally accepted norms were determined. Key issues were collated, and two simplified checklists were developed: one for the assessment of analytical accuracy studies and a second for guidance with studies assessing the influence of interferences. The checklists have been used in a feasibility study with 20 representative studies selected from a literature search between 2007 and 2012. This check revealed that limitations in the designs and methods of studies assessing the performance of BG systems are common. The use of the accuracy checklist with the 20 representative studies showed that only 20% were in agreement with most of the issues deemed important and that 40% showed clear nonconcordance with ISO 15197. The use of the interference checklist showed that only 50% of the publications were in good agreement with the quality checks. In agreement with previous studies, which concluded many evaluations are performed poorly and present questionable conclusions, the use of these checklists demonstrated that few publications adhered to international guidelines and recommendations. Taking this into consideration, it becomes obvious that the publications must be examined in more detail to establish their quality and the validity of conclusions drawn.

Rationale and Objective

Blood glucose (BG) systems have been widely available for many years and have become the subject of an extensive range of studies evaluating the accuracy of these products. Such studies can involve a single BG system or several. Considering the number of publications comparing the accuracy of several systems, the number of publications has been increasing significantly over the last 5 years. A nondefinitive estimate based on the search undertaken is provided in Figure 1.

FIG. 1.

Number of publications assessing the performance of several blood glucose systems between 2004 and 2011.

Building on the work from Mahoney and Ellison^1,2 and subsequent re-evaluation,³ study designs of accuracy studies were assessed, and best practice and internationally accepted norms were determined. The evaluation of BG system accuracy and the influence of interferences is complex, and testing must be carefully designed and performed.⁴ Clinicians and technicians performing such studies need to be aware of numerous factors in order to obtain data free of protocol-bias or patient-specific bias.⁵ A wide range of variables must be taken into account to ensure any inaccuracy of results is due to the BG system and not due to other factors such as the reference method, variations in the specimens compared, experimental artifact, random patient interferences, or using the meter outside the manufacturer's claims in the instructions for use.

Studies on BG system accuracy and the influence of interferences divide into several categories. Accuracy can be determined under defined conditions in which the study incorporates exclusion criteria determined by limitations quoted by the manufacturer. Such a study can correctly reflect the accuracy possible and claimed by the manufacturer for a BG system. Accuracy studies may, however, be performed under what the investigators consider more “real life” or “routine” conditions. Here, precise conditions are not adhered to or the possibility exists to include data that should be excluded by reference to the manufacturer's limitations and instructions for use. Not adhering to the manufacturer's instructions for use restricts the validity of conclusions about accuracy and the influence of interferences made in such studies.

Studies can also be undertaken to determine influences of potential specific BG system-dependent interferences, such as hematocrit, temperature, reducing substances, etc. Although sharing many common guidelines with accuracy evaluations, to correctly identify interferents and their degree of influence on accuracy, interference studies require specific assessment against ISO 15197,⁶ ISO/DIS 15197,⁷ and CLSI recommendations for interference testing.⁸

Documents that list best practice quality guidelines or recommendations for undertaking and reporting appropriate evaluations of BG system accuracy are available from several sources^1–3 (see Supplementary Table S1; Supplementary Data are available online at www.liebertonline.com/dia). International guidelines reflect procedures designed to produce conclusions with the greatest degree of confidence. ISO 15197 provides such information for BG systems and is the document to which manufacturers are required to adhere in order to demonstrate the accuracy and performance of a commercially available system. Guidelines or recommendations vary in detail and complexity but taken together cover all aspects of accuracy evaluation (see Supplementary Table S2).

Although these protocol guidelines and recommendations are readily available, investigators rarely incorporate consensus standards or quality guidelines into BG accuracy evaluations, and a standardized approach has so far not been widely adopted.⁹ Strict adherence is needed by using an appropriate reference method, defined protocols, and operating BG systems in accordance with manufacturer's instructions for use. BG system accuracy studies also require a protocol that incorporates the use of fresh human whole blood because a standard reference material for whole blood is not available.^1,2,10

Four potential sources of error must be considered in the evaluation of a BG system¹¹:

1. analytical imprecision—controlled by testing product that conforms to specifications (i.e., supplied, handled, and stored according to manufacturer's instructions and so potentially capable of achieving the performance claimed)

2. bias—controlled by testing product that conforms to specifications

3. protocol-specific bias—controlled by adherence to careful study design

4. random patient interferences—controlled by inclusion and exclusion criteria

Although a large number of BG system accuracy studies from the scientific community have been published, the general consensus when compared against recommendations is that most evaluations are performed poorly and present questionable conclusions.^1,2 It seems that publications rarely adhere to all the points outlined in guidelines and that many studies do not follow published recommendations for study design and methodology or do not appear to address many of the variables that can adversely impact validity. To establish if an accuracy study is valid, a wide range of details must be checked, an issue complicated by publication word count restrictions that may necessitate additional protocol information being made available elsewhere.

In 2007, Mahoney and Ellison¹ published a comprehensive evaluation of published BG system accuracy studies (between 2002 and 2006). The study showed that the average BG system accuracy evaluation used less than 50% of the combined CLSI and Standards for the Reporting of Diagnostic Accuracy Studies (STARD) recommendations and that the overall quality of reports was low. Compliance with these recommendations varied widely (median, 53%; range, 21–84%), and only one study out of 52 followed CLSI recommendations for checking reference test results. Fewer than half (42%) contained STARD-recommended statements regarding how and when comparative measurements were performed. The low rate of compliance to recommendations suggests many researchers do not follow published recommendations for study design, methodology, and reporting. All this may have affected negatively quality and conclusions.

The analysis of BG system accuracy publications¹ highlighted particular deficiencies with (a) reference methods, (b) ensuring comparison of appropriate samples and the importance of timings, and (c) the acceptance criteria used. Valuable publications emphasized BG system training with performing a thorough reference method evaluation, tested BG system and reference results in duplicate, and emphasized control of elapsed time and glycolysis.

Moreover, Mahoney and Ellison² proposed a checklist combining key elements from different guidance and recommendations that outlined a standardized approach to BG system evaluations, along with associated references applicable to international standards and consensus recommendations. This provides a basis for protocols that are (a) evidence based, (b) scientifically defensible, and (c) sufficiently descriptive to allow for test and result reproducibility.

Building on the work of Mahoney and Ellison^1,2 and considering the increasing number of publications and the apparent contradiction of results that create confusion and doubts about the reliability of the results of such studies, it is important to provide an assessment overview of study designs, to identify and review frequently made errors, and to develop simple checklists of basic requirements to verify the quality of BG system accuracy performance and comparison studies (either with regard to accuracy or interferences).

Materials and Methods

Materials

Literature outlining guidelines, recommendations, rationale, or best practice for undertaking high-quality BG system evaluations was identified.

A “key words” literature search identified 82 BG system performance articles between 2007 and 2012. For this feasibility study, from these, 20 publications^12–31 (see Supplementary Table S3) were subjectively selected as being representative for checklist analysis. Articles chosen incorporated those from different sites/countries, included different types of publications ranging from full articles to poster abstracts, involved healthcare professional and/or patient operators, and covered different aspects of system performance. Publications were classified as studies primarily dealing with accuracy, interferences, or both, and study designs were evaluated to identify any potential issues of concern using the appropriate proposed checklist. When a single publication addressed both accuracy and interference, both checklists were used.

Methods

International guidelines and recommendations, the standardized approach to assessing BG system performance proposed by Mahoney and Ellison^1,2 and subsequent re-evaluation,³ and other BG evaluation publications were collated. Best practice, study designs, and alignment with internationally accepted norms, practice, and guidelines were determined. This provided a basis for a comprehensive summary of important factors (see Supplementary Table S4). Key issues were collated and used to develop two simplified checklists (A and B [Tables 1 and 2, respectively]) to aid design quality assessment of accuracy and interference publications, respectively.

Table 1.

Checklist A for “Accuracy” Studies: Identifying Weaknesses in Study Designs in Blood Glucose System Accuracy Publications

	Yes	Partial	No
Reference method	Use of reference method specified by the manufacturer. Reference samples analyzed in duplicate and checked for differences. Information given on imprecision, quality assurance, bias, and traceability to higher reference methods.	Reference method appropriate but protocol and performance not fully specified.	Use of a reference method other than specified by the manufacturer. Inappropriate comparison against a hexokinase- or glucose oxidase-based reference.
Comparing “like with like” samples	Comparisons of “like” fresh whole blood specimens. Ideally capillary versus capillary and split sample analysis. Correct collection with minimal delay in analysis and post-collection control of sample handling time. Capillary and venous blood glucose levels should not be assumed to be equivalent.	Appropriate specimens described, but information on collection, protocol, and delays not supplied.	Comparison of capillary against venous samples or other dissimilar correlations.
Number of samples	At least 100 fresh capillary samples and 200 data points.	Minimum of 40 different samples and donors.	Less than 40 samples from different donors.
Spread of glucose concentrations	Sufficient spread of results spanning the analytical range. Percentage of results as in ISO 15197. Capillary samples with very high/low glucose concentrations can be provided by using appropriately modified samples.	Spread of sufficient sample results illustrated diagrammatically but percentage within ranges not provided	No indication of or inappropriate spread of results.
Accuracy criteria	a. A plot of the difference between individual results from meters against the mean of specific reference values plotted as the dependent variable.	Appropriate information only partially supplied.	Information supplied in forms differing to specified acceptance criteria.
	b. Tables of degree of meter results difference compared with the reference method. For reference glucose values (i) <4.2 mmol/L (75 mg/dL) showing the number of meter samples (%) within±0.28 mmol/L (5 mg/dL),±0.56 mmol/L (10 mg/dL), ±0.83 mmol/L (15 mg/dL) of the reference method. For reference glucose values (ii) ≥4.2 mmol/L providing the number of samples (%) within±5%, 10%, 15%, 20% of the reference method.
	c. A summary of results identified as acceptable using current acceptance guidelines.
	d. A clinical accuracy assessment such as by Parkes or consensus error grid analysis.
Number of strip lots	Number of lots used stated. Use of several lots, ideally three different lots, indicates robustness of accuracy.	Use of only one lot of strips. Information of multiple lot use not supplied.	Not applicable.
Full details provided	Sufficient information provided to verify the study was of appropriate design and conclusions were justified and correct. ISO standard 15197 contains a full list of requirements and information. This includes diverse protocol information such as the number of meters, details of lots and reagents, dates of expiry, temperature range, exclusion criteria based on the instructions for use, serial numbers of meters, etc.	A majority of relevant details supplied.	No or few relevant additional details provided.
Independency	Unbiased and independent; conducted at external sites, outpatient clinics or hospital settings; no potential manufacturer bias.	Not applicable.	No clear information if the study was performed independently; no peer review.
Concordance ISO 15197	Full details provided to establish if the study was undertaken appropriately in accordance with ISO standard 15197 or other relevant guidelines and recommendations.	Partial concordance with ISO 15197.	Clear differences in protocol and study design from ISO 15197.

Table 2.

Checklist B for “Interference” Studies: Identifying Weaknesses in Study Designs in Blood Glucose Interference Publications

	Yes	Partial	No
Reference method	Well-characterized reference method, ideally that quoted by the manufacturer. Full details provided. Reference glucose measurements used to control the stability of the sample.	Full details of reference method not supplied.	Use of an inappropriate reference method. Reference method affected by interference.
Use of appropriate samples	Heparinized venous whole blood should be used, ideally fresh from a single donor. Serum or plasma samples are allowed. Effects should be determined at single, specific glucose concentrations. Paired difference testing consists of adding a potentially interfering substance to a sample and evaluating bias relative to a control portion of the same sample. Full details provided.	Incomplete details provided.	Use of an inappropriate sample. No details provided.
Investigation at different glucose concentrations	Interference investigated at a number of glucose concentrations (low, normal, high). Details of concentration and preparation provided. ISO/DIS 15197:20117 recommends two concentrations for general interferences and three for hematocrit studies.	Use of two or inappropriate glucose concentrations.	Investigation at a single glucose concentration.
Number of interference levels tested, rationale, and preparation	Interferent free, therapeutic/normal, and toxic/high levels should be ideally tested at concentrations recommended in guidance documents. The sample matrix should be diluted by no more than 5%. Control pool prepared exactly as the test except interferent is replaced with the same volume of solvent.	Incomplete details provided.	Inappropriate concentrations of interferents tested. Incorrect dilution of sample.
Details of BG meter system performance	Meter imprecision and quality control checks should be provided to help determine the number of replicates and confidence levels in interpreting interference responses and to verify the system is working correctly. Precision should be consistent with manufacturers' performance specifications.	Full details of meter performance not provided.	No details of meter performance provided.
Number of replicates	Assayed at least three times within one analytical run. Number of replicates should be appropriate for the imprecision and claim. Tables of the number of replicates needed to detect interference effects with 95% confidence and power and determined by the within-run imprecision standard deviation can be found in CLSI EP7.8	Incomplete details provided.	No details provided.
Presentation of results	Appropriate table or graph of results with the observed effect on the y-axis and the interferent concentration on the x-axis.	Results explained but not presented appropriately.	No tabular or graphical presentation of results.
Interpretation of results	Use of appropriate, reasoned levels of deviation and statistics to classify difference or effects as interference. Clinical relevance determines whether an analytical effect is considered interference (CLSI EP78).	Use of partially reasoned deviations to determine interference.	Use of arbitrarily defined deviations to determine interference. Difference in results from BG meter system in presence and absence of interferences not used for interpretation.
Full details provided	Sufficient information provided to verify the study was carried out in accordance with CLSI EP78 and ISO/DIS 15197:2011,7 of appropriate design and conclusions justified and correct.	Basic relevant details supplied.	No or few relevant additional details provided.
Independency	Unbiased and independent; no potential manufacturer bias.	Not applicable	No clear information if the study was performed independently; no peer review.
Concordance CLSI EP78 and ISO/DIS 15197:20117	Concordance with guidelines.	Partial concordance with guidelines.	Deviation from guidelines.

BG, blood glucose.

As a feasibility check the checklists were used to examine the 20 representative studies. Fifteen publications, of which four^16,18,21,22 used patients as operators, were examined using the accuracy checklist (10 studies assessing accuracy of BG systems and five studies assessing accuracy of BG systems and influence of interferences), and 10 were examined using the interference checklist (five studies assessing the influence of interferences and five studies assessing accuracy of BG systems and influence of interferences). Although a subjective process, and publications may omit provision of full details, the checklists enabled each important point to be examined and categorized as being in agreement (yes), partial agreement (partial), or not in agreement (no) with guidelines.

Results

Checklists for assessing the quality of BG system accuracy publications

Definitive checking of the validity of an accuracy study can be an extensive process. Many aspects of a BG system performance evaluation need to be considered when assessing the quality of the study design and the validity of findings.

Although differing from international guidelines in terms of layout and order of important features, the integrated information compiled in Supplementary Table S4 provides a comprehensive tool to establish the quality of published BG system evaluations. This list provides details of key issues of BG system evaluation study design to allow comprehensive examination of a publication's agreement with consensus guidelines, the quality of its study design, and the validity of results. It also provides a requirement list for a good accuracy comparison study design. Supplementary Table S4 represents a modified and extended version of the original checklist.² It includes additional data from ISO 15197 and broadens examination to include aspects such as independence/impartiality and the general applicability of findings by observing the number of different batches of strips used.

A general indication of important deficiencies in study design can generally be found from looking at a few key points that are summarized as the two checklists. The two short simplified checklists (A [Table 1] and B [Table 2]) cover the major aspects readers need to consider when examining BG system accuracy and interference evaluation publications. Checklists include examining the details supplied on the reference method, the specimens used for comparison, details of the protocol, and display/acceptance criteria for results.

Evaluation of study designs using the accuracy checklist

Application of the accuracy checklist showed that only 20% (three of 15) of publications were of clear high quality and in agreement with all, or disagreement with only one, of the issues deemed important; 47% (seven of 15) were not in agreement with four or more of the quality checks. Only one publication¹⁵ showed full concordance with ISO 15197. Key areas of nonagreement included more than 47% (seven of 15) using an inappropriate reference method for specific BG systems and 67% (10 of 15) not demonstrating an appropriate spread and range of results, ideally with at least defined percentages within specified concentration ranges. Sixty percent (nine of 15) were considered to not provide full study details. Thirty-three percent (five of 15) did not compare “like with like” samples, and 80% (12 of 15) were considered to use only partially appropriate acceptance criteria. All results of limitations in study designs and their frequency identified using the accuracy checklist are summarized in Supplementary Table S5.

Evaluation of study design using the interference checklist

Use of the interference checklist showed that 50% (five of 10) of publications incorporating interference studies were in good agreement with the quality checks. However, only one publication²⁷ was considered to demonstrate clear concordance with recommendations of CLSI EP7 and ISO/DIS 15197:2011. Key concerns were that 40% (four of 10) inappropriately presented results and 30% (three of 10) presented no information on BG system performance.

Results of 60% (six of 10) studies were considered to partially interpret results appropriately. Only 20% (two of 10) were considered to provide full study details. All results of limitations in study designs and their frequency identified are summarized in Supplementary Table S6.

Discussion

Evaluation of accuracy and performance of BG systems is complex, and performing studies correctly in accordance with guidelines assumes greater importance as accuracy demands increase and tighter standards are introduced. In agreement with previous studies that concluded most evaluations are performed poorly and present questionable conclusions,^1,2 preliminary use of the checklists demonstrated that only few publications adhered to international guidelines and recommendations for appropriate study design or fully address many of the variables that can adversely impact on the validity of conclusions. The easy-to-use checklists help raise awareness of important issues involved, identify limitations in study design, and will aid readers drawing clear and valid conclusions from the increasing number of publications in the BG system area.

Readers of BG system accuracy publications need to be aware of limitations in study design and protocols that can lead to differences in results inappropriately being attributed to BG system inaccuracy. The use of an appropriate reference method remains of paramount importance in ensuring a correct comparison. The reference method chosen for a specific BG system should be the one stated by the manufacturer to avoid for example the negative biases of approximately 3–8% reported between the Yellow Springs Instrument (Yellow Springs, OH) glucose oxidase- and hexokinase-based reference.^21,32 This is commonly not done or acknowledged, as is provision of details of performing reference tests in duplicate on laboratory systems of known total error and traceability. An appropriate spread of results from patient samples should also be demonstrated. Not comparing “like with like” samples and acknowledging potential differences between capillary and venous blood samples and ensuring correct minimal timings between specimen collection and analysis also remain common limitations that could potentially lead to differences of up to 30%. Results also require analysis using appropriate recognized acceptance criteria, statistical methods,³³ and correct presentation.

Conclusions

It must be concluded that it is not sufficient to read just the conclusion of BG system accuracy publications. Each publication must be examined more in detail to establish any variation from recommended study designs and to establish their quality and the validity of conclusions drawn. The use of the checklists proposed in this study provide aids to interpreting studies on the performance of BG systems allowing selection of valid, reliable, transparent, and comparable results and conclusions.

Future studies extending examination to additional publications and adaptation of checklists in light of any modifications to international standards are necessary to confirm the generality and validity of findings.

Acknowledgments

This study was funded by an educational grant from Roche Diabetes Care, Mannheim, Germany.

Author Disclosure Statement

No competing financial interests exist.