A Simulation Study to Assess the Effect of Analytic Error on Neonatal Glucose Measurements Using the Canadian Pediatric Society Position Statement Action Thresholds

Abstract

Background:

The Canadian Pediatric Society (CPS) has endorsed an algorithm for the screening and immediate management of babies at risk of neonatal hypoglycemia that provides time-dependent glucose concentration action thresholds. The objective of this study was to evaluate the impact of glucose analytic error (bias and imprecision) on the misclassification of glucose meter results from a neonatal intensive care unit (NICU) using the CPS guidelines.

Methods:

A simulation dataset of true glucose values (N = 100 000) was derived by finite mixture model analysis of NICU glucose data (N = 23 749). Bias and imprecision were added to create measured glucose values. The percentages of measured glucose values that were misclassified at CPS action thresholds were determined by Monte Carlo simulation.

Results:

Measurement biases ranging from −20 to +20 mg/dL combined with coefficients of variation 0% to 20% were evaluated to predict misclassification rates at 32, 36, and 47 mg/dL. The models demonstrated low risk of false normoglycemia—at 5% CV and +10 mg/dL bias: 0.8% to 5% misclassification at the 32 and 47 mg/dL thresholds due to bias. The models demonstrated risk of false hypoglycemia—at 5% CV and −10 mg/dL bias: 3% to 12.5% misclassification at 32 and 47 mg/dL thresholds due to both bias and imprecision.

Conclusion:

Using CPS action thresholds, the simulation model predicted the proportion of neonates at risk of inappropriate clinical action—both of omission or “failure to treat” and commission or “overtreatment” in response to NICU glucose meter results at specific bias and imprecision values.

Introduction

Measurement of blood glucose in high-risk neonates is ubiquitous across neonatal intensive care units (NICUs) with significant implications for ongoing glucose monitoring and intervention. Untreated neonatal hypoglycemia in at-risk neonates has been associated with unfavorable long-term neurodevelopmental outcomes.^1,2 Neonatal hypoglycemia has also been associated with neuroanatomical changes on neuroimaging.³ While the ongoing debate exists regarding the magnitude of the relationship between hypoglycemia and negative neurodevelopmental outcomes, the current clinical practice heavily relies upon neonatal glucose measurements at clinical decision points for further blood glucose monitoring, second tier hypoglycemia work-up, and thresholds for intervention.

In both Canada and the United States, national pediatric associations have produced and disseminated clinical practice guidelines for the screening of hypoglycemia in the neonatal population.^4,5 In Canada, the Canadian Pediatric Society (CPS) position statement on hypoglycemia screening for at-risk newborns is embedded within standard clinical practice across most if not all NICUs. While timing and treatment thresholds vary slightly between Canadian and U.S. guidelines, the core clinical action points are alike with a dependence on the rapid, precise, and accurate glucose measurement. Both hypoglycemia algorithms indicate precise glucose thresholds with an interthreshold difference of as little as 3.6 mg/dL (0.2 mmol/L). Given the narrow discrepancy, neonatal hypoglycemia algorithms are intrinsically dependent upon robust glucose measurements.

In our Canadian center, the CPS position statement⁴ serves as the basis for routine glucose monitoring of the majority of NICU patients which may be done with glucose meters, blood gas analyzers, or clinical laboratory chemistry analyzer platforms. While each methodology carries its own benefits and drawbacks, glucose meters have become standard of care for initial glucose screening in our center and throughout many centers nationwide. This is primarily because of the small blood volume required for analysis as well as the timeliness of obtaining test results. As the pendulum swings toward point of care testing (POCT) glucose measurement, clinical decision-making for neonates may be compromised by many factors including a lack of awareness of glucose meter analytic performance limitations and an abundance of devices in a rapidly changing market. It is challenging for individual providers to keep up with the current analytical publications related to the reliability and susceptibilities of using POCT devices for clinical decision-making at defined glucose thresholds, such as those provided by the American Academy of Pediatrics and the CPS.

Computer simulation models have served as a tool to interrogate potential analytical error produced from POCT devices and the effect of this error upon clinical-decision making. Recent simulation examples include the evaluation of clinical risk of administering either too much or too little insulin or warfarin based on method bias and imprecision associated with POCT glucose or International Normalized Ratio (INR) devices. ^6-8 Simulation models have also been used to determine the influence of analytical error with cardiac troponin on the misclassification of patients presenting with potential acute coronary syndrome to the emergency department.⁹

In light of well-established clinical practice guidelines for neonatal hypoglycemia screening and the increased reliance upon glucose POCT, it is imperative for providers to be aware of and account for the inherent analytical error that may be introduced. Therefore, the objective of this study was to assess the impact of glucose analytic error (bias and imprecision) through simulation modeling to predict the rates of misclassification of glucose meter results using the CPS guidelines for neonatal hypoglycemia.

Methods

The Royal University Hospital in Saskatoon, Saskatchewan, Canada is a tertiary care pediatric hospital with a level III NICU serving the province. The NICU maintains 42 beds and services approximately 1200 admissions annually.

Patient Data

Twelve consecutive months of glucose meter results (N = 23 749) from the NICU at Royal University Hospital were analyzed. No patient identifiers were extracted. All glucose results were recorded in mmol/L with one decimal place and subsequently converted to mg/dL by multiplying the mmol/L result by 18.016. De-identified glucose data were collected during a quality assurance review of glucose monitoring in the NICU and used in accordance with the University of Saskatchewan Human Research Ethics Board policy for de-identified laboratory data (eg, this study was exempt from the full review process).

Canadian Pediatric Society Position Statement on Hypoglycemia

Blood glucose thresholds for glucose testing were stated in the CPS position statement, initially affirmed in December 2004 and re-affirmed in February 2018. Threshold values for clinical decision-making in this statement fall at 32 mg/dL (1.8 mmol/L), 36 mg/dL (2.0 mmol/L), and 47 mg/dL (2.6 mmol/L).

Finite Mixture Model

The distribution of the neonatal glucose meter results (N = 23 749) was analyzed and found to be represented by three Gaussian populations using finite mixture model (FMM) analysis with Stata LP v15.1 (StataCorp, College Station, TX, United States). The sum of three population components is shown in Equation (1): f(y) = π₁ f₁(y) + π₂ f₂(y) + π₃ f₃(y), where y is the glucose concentration, the parameters π₁, π₂, and π₃ are the component weights and f₁, f₂, and f₃ are the density functions of three components with distinct means (mg/dL) and standard deviations (SD).⁹ The FMM analysis uses an expectation-maximization algorithm for maximum likelihood to estimate the model parameters, as shown in Table 1. Using the three-component distributions, a simulated model NICU dataset (true glucose) of 100 000 data points was created for use in the Monte Carlo simulation (Figure 1).

Table 1.

Component Weights (as a Decimal Fraction) Determined Using Finite Mixture Modeling on the Neonatal Intensive Care Unit Glucose Meter Dataset.

Component weights	Glucose mean (SD), mg/dL
π1 0.807	72 (18)
π2 0.173	111 (34)
π3 0.020	238 (109)

Abbreviation: SD, standard deviation.

Figure 1.

Relative frequency distribution of the original neonatal intensive care unit glucose meter, N = 23 749 (a) and the simulation dataset, N = 100 000, derived from the finite mixture model (b).

Statistical Methods

Monte Carlo simulation and contour plots were generated with Stata LP v15.1 (StataCorp, College Station, TX, United States). Equation (1) describes the addition of error to true glucose values as previously reported by Boyd and Bruns⁶:

$$\begin{array}{l}\mathrm{Glucmeasured}=\mathrm{Gluctrue}+\\ \left(\mathrm{Gluctrue}*\mathrm{CV}*\mathrm{n}\left[0,1\right]\right)+\left(\mathrm{Bias}\right)\end{array}$$ (1)

where Gluc_true is the initial glucose value randomly selected from 100 000 results in the FMM dataset, Gluc_measured is the glucose value for the given bias/imprecision condition, CV is the variation of the assay expressed as fraction, n (0,1) is a random number drawn from a Gaussian distribution with a mean of zero and a SD of 1, and bias is the absolute assay bias (mg/dL). To determine the effect of error (bias and imprecision) at the CPS hypoglycemia thresholds of 32 mg/dL (1.8 mmol/L), 36 mg/dL (2.0 mmol/L), and 47 mg/dL (2.6 mmol/L), the CV values spanned from 0% to 20% in increments of 1%, while the absolute bias spanned from −20 to +20 mg/dL in 1 mg/dL increments. Glucose values (n = 90 000) were randomly sampled with replacement using the Monte Carlo simulation with 10 repetitions using Stata software. The frequency that measured glucose values changed sides of the hypoglycemia thresholds was determined at each combination of CV and bias. The contour plots show the percentage of misclassification for the diagnostic thresholds as a function of bias and imprecision.

Results

The relative frequency distribution of the original NICU glucose meter dataset (panel (a)) and the FMM glucose dataset (panel (b)) are shown in Figure 1. In the original data, 215/23 749 (0.9%) of the results were <32 mg/dL (1.8 mmol/L), 357/23 479 (1.5%) were <36 mg/dL (2.0 mmol/L), and 1594/23 479 (6.8%) were <47 mg/dL (2.6 mmol/L), in accordance with the CPS clinical decision thresholds. A FMM analysis of the original data was constructed for improved representation to study the very small sample sizes in the neonatal hypoglycemic range. The FMM analysis found that the neonatal glucose distribution (Figure 1(a)) can be represented by three Gaussian population components outlined in Table 1. A FMM glucose dataset of 100 000 data points (Figure 1(b)) was created using the weights in Table 1 for use in Monte Carlo simulations. To test if the FMM glucose data represent the original glucose data, the original and the FMM datasets were compared using quantile regression at the 5th, 10th, 25th, 50th, 75th, 90th, and 95th percentiles. No statistically significant differences (P > .05) were detected at the aforementioned percentiles (Table 2).

Table 2.

Quantile Regression Analysis at Percentiles on Glucose Concentrations (mg/dL).

Percentile	NICU dataset (mg/dL)	Simulation dataset (mg/dL)	P-value
5th	45	44	.955
10th	50	51	.606
25th	61	63	.208
50th	76	76	.889
75th	92	92	.499
90th	115	116	.166
95th	139	140	.217

Abbreviation: NICU, neonatal intensive care unit.

The percentage of the FMM dataset that was misclassified at CPS clinical decision thresholds, 32 mg/dL (1.8 mmol/L), 36 mg/dL (2.0 mmol/L), and 47 mg/dL (2.6 mmol/L), was determined using a Monte Carlo simulation model, and contour plots were generated to depict the isocontour lines as a function of method bias and imprecision. Threshold specific contour plots demonstrated how bias and imprecision could cause glucose results from “below the threshold” to be misclassified “above the threshold” and vice versa. To determine how the frequency of hypoglycemic results in a simulation dataset influences the misclassification rates at 32, 36, and 47 mg/dL thresholds for neonatal hypoglycemia, a second simulation dataset was generated by subtracting 20 mg/dL from each value in the first dataset (see supplemental file).

Impact of Positive Bias at the Clinical Decision Thresholds

Figure 2 illustrates the effect of imprecision and positive biases of up to +20 mg/dL on false normoglycemia: misclassification of data from <32 mg/dL (1.8 mmol/L) to ≥32 mg/dL (panel (a)), from <36 mg/dL (2.0 mmol/L) to ≥36 mg/dL (panel (b)) and from <47 mg/dL (2.6 mmol/L) to ≥47 mg/dL (panel (c)). The approximately horizontal isocontour lines in Figure 2 panels indicate that misclassification rates, only slightly affected by method imprecision, are primarily influenced by method bias. The misclassification rates were comparable for the 32 and 36 mg/dL thresholds. Biases up to +10 mg/dL were predicted to cause a 0.2% to 0.8% false-normoglycemia (misclassification from below to above the threshold values) at the 32 mg/dL threshold. In contrast, at the 47 mg/dL (2.6 mmol/L) threshold in Figure 2(c), a bias of +11 mg/dL caused a 5.1% false-normoglycemia rate.

Figure 2.

Contour plots of the fraction of glucose misclassification rates (“positive bias” resulting in undertreatment of hypoglycemia) as a function of analytical bias and imprecision. Panel (a) shows isocontour lines that demonstrate the percentage misclassification from <32 to ≥32 mg/dL. Panel (b) illustrates isocontour lines that demonstrate the percentage misclassification from <36 to ≥36 mg/dL. Panel (c) depicts the isocontour lines that demonstrate the percentage misclassification from <47 to ≥47 mg/dL.

Impact of Negative Bias at the Clinical Decision Thresholds

Figure 3 illustrates the effect of imprecision and negative biases on false hypoglycemia: misclassification of data from ≥32 mg/dL (1.8 mmol/L) to <32 mg/dL (panel (a)), from ≥36 mg/dL (2.0 mmol/L) to <36 mg/dL (panel (b)) and from ≥47 mg/dL (2.6 mmol/L) to <47 mg/dL (panel (c)). Similar to positive bias contour plots in Figure 2, the isocontour lines in Figure 3 negative bias plots are almost horizontal indicating that misclassification was significantly influenced by method bias. The isocontour lines in Figure 3 are slightly convex showing greater influence of method imprecision than in Figure 2. The rates of negative misclassification (false hypoglycemia) were much greater (up to 24.9% in Figure 3(c)) than the predicted rates for positive biases in Figure 2. Biases ranging from −10 to −20 mg/dL combined with method imprecision up to 20% were predicted to produce misclassification rates from 3% to 12% at the 32 mg/dL (1.8 mmol/L) threshold. Similar rates of misclassification were observed at the 36 mg/dL (2.0 mmol/L) threshold with comparable bias levels. The largest misclassification rates were observed with the 47 mg/dL (2.6 mmol/L) threshold. Bias of −20 mg/dL with no imprecision was observed to result in 25% misclassification of initial results from ≥47 to <47 mg/dL. Less bias (~ −16 mg/dL) was required to achieve a similar misclassification rate when combined with an imprecision of 20%.

Figure 3.

Contour plots of the fraction of glucose misclassification rates (“negative bias” resulting in over-treatment of hypoglycemia) as a function of analytical bias and imprecision. Panel (a) shows the isocontour lines that demonstrate the percentage misclassification from ≥32 to <32 mg/dL. Panel (b) illustrates the isocontour lines that demonstrate the percentage misclassification from ≥36 to <36 mg/dL. Panel (c) depicts the isocontour lines that demonstrate the percentage misclassification from ≥47 to <47 mg/dL.

Discussion

Given the increasing trend of POCT for neonatal glucose, the high frequency of glucose measurements performed in NICU, and the significant implications derived from glucose POCT, it is imperative for clinical decision-makers to appreciate the analytical constraints of glucose POCT. The clinical performance of POCT is predicated on its inherent methodological bias(es), precision, and the resultant cumulative analytical error. With an abundance of POCT glucose meters available, the majority of which have been optimized for hyperglycemia detection rather than hypoglycemia, bedside providers and laboratory clinicians must work in synergy to ensure that patient treatment decisions are made with awareness of the risks of false hypoglycemia and false normoglycemia when using clinical POCT data generated in NICUs.

With the simulation models, we demonstrated the relationships between errors in glucose measurement and clinical misclassifications: (i) false hypoglycemia leading to overtreatment in the presence of negative analytic bias (up to 25% of results in Figure 3(c)) and (ii) false normoglycemia leading to undertreatment in the presence of positive analytic bias (up to 5% of results in Figure 2(c)). The simulations show that methodological bias, not precision, has a greater impact on clinical misclassification, as indicated by the predominantly horizontal isocontour lines in Figures 2 and 3.

These simulation outcomes have significant implications at the level of the clinical chemistry laboratory, the bedside clinician, and the patient. At the laboratory level, these data highlight the importance of ensuring adequate recognition of POCT biases, familiarity with output variance between different POCT glucose meters, and the importance of selecting the optimal POCT glucose meter for the relevant clinical outcome and patient population. For bedside clinicians, careful scrutiny of POCT glucose results within the context of the clinical picture must be employed to produce sound clinical decision-making. Finally, at the patient level, misclassification may result in both over- and undertreatment. For those overtreated, interventions such as formula supplementation or intravenous glucose therapy may be unnecessarily ordered, leading to disruption of the critical period of maternal-infant physical bonding and breastfeeding initiation. For those undertreated, unrecognized hypoglycemia may be permitted which would delay recommended intervention, further investigations, and delay the mitigation of harm to the patient. While no clinical test is absolutely definitive, these data behoove clinicians to interpret POCT glucose with caution and with sound clinical acumen.

This study indicates a disproportionate effect of bias within total analytical error. There is a growing body of literature describing factors that introduce bias in POCT glucose measurement. The effect of endogenous factors such as hematocrit, oxygen, pH, bilirubin, and uric acid as well as exogenous factors like acetaminophen, galactose, maltose, and ascorbic acid has been reported to interfere with glucose measurement, depending upon the principle of glucose detection employed.^10-13 A very common explanation for falsely low glucose concentrations for central laboratory testing is ex vivo glycolysis. Red blood cells metabolize glucose at a rate of 5% to 7% per hour as long as the cells are in contact with the plasma/serum. Ex vivo glycolysis can be an important pre-analytical factor influencing glucose measurement depending upon the time it takes to analyze the specimens in the clinical laboratory or with a blood gas analyzer.¹⁴ Based on the patient’s hematocrit level, either falsely increased or falsely decreased levels of glucose have been reported.^10,13,15 Similarly, oxygen tension has been reported to influence glucose measurement.¹⁶ High oxygen tension can falsely decrease measured glucose levels with glucose oxidase meters, whereas low tensions have been reported to have the opposite effect. Highly relevant to a neonatal population, the exposure to galactose through nutrition can result in artificially elevated glucose levels, depending on the glucose measurement technique being utilized.^12,16 Thus, in a neonatal population who may be exposed to numerous endogenous and exogenous factors, careful selection of a POCT glucose meter is paramount and an awareness of situations where bias may be significant is necessary. While all extraneous factors cannot be eliminated, an understanding of the clinical methodology being used in POCT glucose meters and the factors that may affect results is critical.

The last decade has led to a significant rise in the number of POCT glucose meters available for hospital and/or patient self-monitoring. While the devices purportedly have improved precision and accuracy, an understanding of their specific intrinsic limitations is also necessary. An appreciation of the extent of analytic agreement between POCT glucose meters to laboratory glucose measurements and any predominant inherent biases that might affect clinical decision making would be beneficial. The simulations performed in this study were conducted to assess a wide range of imprecision and bias values that meet or exceed reported POCT device performance. Readers are advised to review recent studies that describe the variation in glucose results between POCT and traditional lab methodologies as well as the variation between POCT devices that continues to improve.^17-23

There are several limitations to this simulation study. The focus of the current study was to assess the effect that analytical error introduces to the misclassification of glucose results, not misdiagnosis. As simulation models were used in this study, it is important to acknowledge that their usefulness is limited by the appropriateness of the model tested and the data analyzed. The original NICU dataset was derived from a single model of glucose meter. The misclassification rates observed in the simulations were dependent on the original distribution of local NICU glucose values so the rates of misclassification may not be transferable to other institutions with different neonate population and practices.

In conclusion, the pervasiveness of neonatal blood glucose monitoring by POCT warrants critical analysis of the limitations inherent to POCT. Using simulation models to scrutinize both bias and imprecision, our analysis demonstrated that glucose result misclassification at hypoglycemia action thresholds almost exclusively relates to bias, not imprecision, and this has important clinical implications in neonatal hypoglycemia management. The demonstration of bias as the primary source of error emphasizes the need to better identify, recognize, and account for POCT glucose meter biases and interpret results appropriately within the entire clinical context.