Probabilistic Prediction of Laboratory Test Information Yield

Abstract

Low-yield repetitive laboratory diagnostics burden patients and inflate cost of care. In this study, we assess whether stability in repeated laboratory diagnostic measurements is predictable with uncertainty estimates using electronic health record data available before the diagnostic is ordered. We use probabilistic regression to predict a distribution of plausible values, allowing use-time customization for various definitions of “stability” given dynamic ranges and clinical scenarios. After converting distributions into “stability” scores, the models achieve a sensitivity of 29% for white blood cells, 60% for hemoglobin, 100% for platelets, 54% for potassium, 99% for albumin and 35% for creatinine for predicting stability at 90% precision, suggesting those fractions of repetitive tests could be reduced with low risk of missing important changes. The findings demonstrate the feasibility of using electronic health record data to identify low-yield repetitive tests and offer personalized guidance for better usage of testing while ensuring high quality care.

Introduction

Laboratory diagnostics constitute the highest volume of medical procedures. Previous studies have shown that up to 50% of inpatient laboratory tests might be medically unnecessary and contribute significantly to healthcare waste (1–5). Besides increased healthcare costs, unnecessary laboratory tests lead to medical complications and other indirect harm to patients, such as sleep fragmentation (6). The recent supply shortage of blood collection tubes also demonstrates the urgency of mitigating unnecessary laboratory utilization (7).

Although many laboratory utilization interventions such as creation of guidelines, targeted education, audit feedback and modifications to test ordering pipelines have been attempted, data-driven approaches have the potential to address the problem more objectively. One such approach is to predict the expected information yield for laboratory tests and present it to ordering clinicians at the point of order entry.

Previous studies on predicting information yield of medical tests found that machine learning models achieve good performance at predicting whether test results are within “normal” reference ranges (8). Based on a survey with clinicians however (9), many felt it more clinically useful to predict whether test results are “stable” even if they are not “normal.” We conceptualize laboratory test results as stable if there is no clinically relevant change from their most recent value, which we further detail in later sections. Aikens et al. (10) predicted laboratory diagnostic stability, but the laboratories were limited to single-component tests and fixed definitions of stability for specific cohorts. Another study (9) focused on implementation science and concluded using utility predictive algorithms as a promising paradigm for curbing overutilization, but it only used a simple prediction model.

The overall objective of this study is to determine whether advanced machine learning algorithms can generate probabilistic regression predictions of laboratory diagnostic results with greater precision than existing practices of flagging repetitive lab testing, using only features which can be automatically extracted from electronic health records (EHR) that are feasible for real-time prediction deployment.

The contributions can be summarized as following:

1. This study approximates the diagnostic yield using the stability of measurements, which is more clinically relevant compared to using whether measurements are within reference ranges. A flexible definition of lab test “stability” is also provided which can be applied in multiple clinical settings.

2. Instead of point estimates, the proposed method can give predictions with uncertainty estimates by using a technical framework named probabilistic regression. Moreover, it can provide a single model allowing adjustments for various stability thresholds during deployment without retraining the model.

3. Evaluations on retrospective data show promising performances for predicting stability. The model can achieve a sensitivity of 29% for white blood cells, 60% for hemoglobin, 100% for platelets, 54% for potassium and 35% for creatinine at 90% positive predictive value. These results establish the benchmark of what is feasible to achieve or predict given typical information available in electronic health record data.

Methods

The overall development pipeline is shown in Figure 1, together with a potential deployment workflow. The publicly available DEPLOYR-dev package(11) and NGBoost packages(12) were used throughout the development pipeline. The DEPLOYR-dev package contains common pipeline codes for supervised learning and data extraction against Stanford’s clinical data warehouse (STARR) (13), and the NGBoost package contains implementations of natural gradient boosting built on top of Scikit-learn.

Figure 1.

Overall pipeline for the development phase and one potential deployment workflow. As probabilistic regression models output probability distributions, an additional step is needed to convert distributions into “stability” predictions.

Cohort

Seven years (January 1, 2015, to December 31, 2021) of inpatient electronic health records (EHR) data from Stanford Hospital were used for this study. These data were extracted and deidentified by the STARR (STAnford Research Repository) project (13). Cohorts were created based on orders for complete blood count with differential (CBC) and metabolic panel diagnostic tests were the focus of our study due to their high prevalence, which we summarize in Table 1.

Table 1.

Characteristics of CBC cohort and metabolic panel cohort.

	CBC			Metabolic Panel
	Training Set	Test Set	Test Set w/o Overlapping Patients	Training Set	Test Set	Test Set w/o Overlapping Patients
Test orders	120,000	20,000	14,315	120,000	20,000	14,719
Unique individuals	37,356	7,373	5,727	23,132	4,440	3,554
Age (avg. ± SD) at order time	60.9 ± 18.7	61.9 ± 18.3	61.8 ± 18.5	59.7 ± 18.2	60.5 ± 17.8	60.4 ± 18.1
% Female	48.2%	48.5%	48.5%	46.5%	45.9%	46.4%

All laboratory diagnostics ordered within three days of a previous order for the same diagnostic were extracted, as the focus was on repetitive laboratory tests. Diagnostics with missing measurement values or non-standard reference units were removed. Moreover, tests with extreme last measurement values were excluded, as a follow-up test was likely necessary. The specific extremity thresholds were advised by a practicing physician at Stanford Hospital, which we show in Table 2.

Table 2.

Ranges and stability thresholds for individual laboratory test components. Reference ranges refer to boundaries of test results based on a healthy population. Stability thresholds refer to half of the stability interval widths. Measurements not within acceptable prior result ranges are considered as extreme, and those test orders with extreme prior values were excluded.

Lab Diagnostic	CBC			Metabolic Panel
Components	WBC	HGB	PLT	NA	K	ALB	CR
Units	K/uL	g/dL	K/uL	mmol/L	mmol/L	g/dL	mg/dL
Reference range	4~11	12~16	150~400	135~145	3.5~5.5	3.5~5.2	0.5~1.0
Stability threshold	1.75	1	62.5	2.5	0.5	0.425	0.125
Acceptable prior result range	1~20	7~25	50~1000	120~155	3~6	2~7	0.1~3.0

White blood cells (WBC), hemoglobin (HGB) and platelets (PLT) components from CBC tests and sodium (NA), potassium (K), albumin (ALB) and creatinine (CR) components from metabolic panel tests were selected for analysis due to their clinical relevance. For each laboratory test, a random sample of 20, 000 test orders per year was drawn. To mimic the deployment settings, the tests ordered from 2015 to 2020 were used as the training set and those ordered in 2021 were used as the test set. The rationale is we only have access to retrospective data for model training, and the tool will be deployed to make predictions on prospective data.

For some patients in the training set, there might be lab test ordered for them in the test set due to later visits. We refer to these patients as overlapping patients because they have lab test orders both in the training and test set. To gauge the generalizability of models, a subset of the test set with overlapping patients removed was also used for evaluation. In this case, the models were evaluated on two scenarios: with and without overlapping patients between the training and test set.

Outcome Measures and Evaluation Metrics

The objective was to predict the results (stable vs unstable) of each laboratory test component using information available before the order was placed. Stability was defined as a function of two consecutive laboratory diagnostic measurements of the same result component. Given the last measurement result m_l and stability threshold , the stable interval was defined as (m_l − α, m_l + α). The current results were defined to be stable if they lied within the stable interval.

The design of the probabilistic regression allows for flexible run-time adjustment of what this stability threshold is for different clinical scenarios. For simplicity in this study, we illustrate the results at 25% of reference range widths used as stability thresholds for the experiments, as they account for various scales across test components. Reference ranges refer to upper and lower limits of a lab diagnostic based on a group of healthy people, and they are shown in Table 2. The stability labels were generated using the stability thresholds shown in Table 2. For example, a hemoglobin test has a normal reference range of 12 to 16g/dL, so the reference range width is 16 − 12 = 4g/dL, and the stability threshold was calculated to be 4 × 25% = 1g/dL. For a patient whose last hemoglobin value was 9g/dL, it would be considered stable if the next value was within 8~10g/dL.

To provide clinically relevant performance characteristics, models were evaluated on their ability to predict whether laboratory test results would be stable compared with respect to their last measurement values. Precision-recall curves and calibration plots were used as the evaluation metrics, with “stable” defined as the positive class.

Feature extraction

Hundreds of candidate features were engineered through nine feature extractors. The main features included demographics (age, race, sex, ethnicity), diagnosis codes, medication orders, laboratory test orders, laboratory test results and flowsheet records. Numerical features from demographics and flowsheet records were binned into five groups using percentiles as per (14), while laboratory test results were kept at their raw values. Missing categorical features were treated as absence. The structure of the binning numerical values into quantile presence vs. absence converts numerical features into categorical, so no additional explicit missing value imputation was necessary. In total, 644 raw features were used for CBC tests, and 632 raw features were used for metabolic panel tests.

Model Development

There are two steps for model development. The first step is to train machine learning models for probabilistic regression. Current measurement results were used as the labels.

Probabilistic regression outputs the conditional probability distribution P(y|x) instead of point predictions which traditional regression algorithms output. The conditional distribution was assumed to be Gaussian, parametrized by the mean and standard deviation. Given scalar values as labels, a scoring rule needs to compare the estimated probability distribution and the observed labels. In this study, continuous ranked probability score (CRPS)³ was used as the loss function since it is a robust alternative to negative logarithmic score. Given a cumulative density function F and observation y (outcome), the CRPS is defined as $C\left(F,y\right)={\displaystyle {\int }_{-\infty }^{y}F{\left(z\right)}^{2}dz+{\displaystyle {\int }_y^{-\infty }{\left(F\left(z\right)\right)}^{2}dz}}$. Intuitively, minimizing CRPS would lead to a sharp distribution concentrated around the observation.

Natural gradient boosting (NGBoost) (12) was used for probabilistic regression. NGBoost is based on gradient boosting, just like XGBoost, except that it produces a full probability distribution over the outcome space rather than a point estimate. For most hyper-parameters, default values were used for training models as they worked well empirically. Decision tree regressors with a maximum depth of three were used as the base learner. The number of boosting iterations was 500, and the learning rate was 0.01. Ten percent of the most recent orders in the training set were held-out and used as a validation set.

A probabilistic regression model was trained for each of seven laboratory test components across our two panel diagnostics. Those models were evaluated individually.

The second step is to convert the predicted probability distribution into a stability score. A stability score is defined as the probability of the lab test results being within the “stable” interval of last measurements. Figure 2 shows an illustration with the area of the shaded region being the stability score. Isotonic calibration was also conducted for stability scores using the validation set. It fits a stepwise non-decreasing linear function which minimize ${{\displaystyle {\sum }_i\left(y_i-{\widehat{y}}_i\right)}}^2$.

Figure 2.

Translating probability distributions into stability score. The orange curve represents a probability density function across the possible range of values that the next value could take on. The blue lines represent the range around the last value that would be considered clinically “stable” compared to the last value. The green area under the curve represents the probability that the model predicts the next value will be within the “stable” range. For example, given the last value of hemoglobin was 20g/dL, the stability score is the probability of the next result being within 19 to 21g/dL, as illustrated in the diagram.

After the conversion, each lab test case has a probability predicted for how likely it is for the next result to be stable. We then evaluate the accuracy of this binary classification for each lab component in terms of standard metrics including precision-recall curves, recall at 90% precision and calibration plots, with being stable as the positive class.

Unlike traditional classification algorithms which require retraining upon redefinition of the outcome, the proposed method allows end users to adjust the stability threshold based on patient-specific circumstances. The stability thresholds can also vary across different test components based on end users’ judgment.

Results

Prevalence of Repetitive Tests

Figure 3 shows the distribution of time intervals between consecutive CBC and metabolic panel tests. It highlights the high volume of repeated laboratory tests within short periods of time at Stanford Hospital during the period.

Figure 3.

Distribution of intervals between consecutive test orders within three days normalized by number of patients. (a). CBC orders (b). metabolic panel orders.

Figure 4 shows the proportion of stable test results against different stability thresholds. To allow comparison among individual components, stability thresholds are shown as a percentage of their respective reference ranges in the x-axis. The vertical lines highlight the stability threshold used for the experiments in this study. For example, for platelet tests repeated within three days of each other, the green curve crosses the blue dotted line threshold at 92.2%. It indicates that 92.2% of platelet test repeats are “stable” compared to their last values, where “stable” is defined as being within +/- 25% of the reference range width for platelet values.

Figure 4.

Proportion of stable results versus different stability thresholds. (a). CBC tests (b). metabolic panel tests.

Model Performance

Figure 5 and Figure 6 show scatter plots of predicted expectations from probabilistic regression models versus actual numerical values, and there is clear correlation shown. However, a point prediction is not good enough for an end user to know what to do with it. In this case, using the predicted distributions, stability scores can be calculated based on previous measurements and stability thresholds.

Figure 5.

Scatter plot of real measurement results versus predicted expectation values for CBC test components.

Figure 6.

Scatter plot of real measurement results versus predicted expectation values for metabolic panel test components.

After converting the distributions into calibrated stability scores, the performances can be evaluated as a binary classification problem. Figure 7 shows the precision-recall curves and calibration curves for predicting the stability of CBC test components, while Figure 8 shows those for metabolic panel test components. Both performances evaluated on test set and test set with overlapping patients removed are shown. Areas under the precision-recall curves are included in the legend. Here, the positive class refers to results being “stable” compared with prior values.

Figure 7.

Precision recall (PR) curve and calibration plot for CBC test components, evaluated on both test set and test set with overlapping patients removed.

Figure 8.

Precision recall (PR) curve and calibration plot for metabolic panel test components, evaluated on both test set and test set with overlapping patients removed.

Table 3 shows the proportion of low-yield tests identifiable by using the models, at 90% precision. For example, this means that ~60% of the repetitive Hgb testing in the hospital are predictably “stable” compared to their prior values, and those predictions are correct 90% of the time.

Table 3.

Recalls at 90% precision for different test components.

Recall at 90% PPV	CBC			Metabolic Panel
	WBC	HGB	PLT	NA	K	ALB	CR
Test Set	29.3%	59.5%	100%	<1%	53.7%	99.4%	35.0%
Test Set w/o Overlapping Patients	22.3%	57.0%	100%	<1%	54.3%	99.3%	36.9%

Discussion

Overall, this study quantifies the prevalence of repetitive laboratory testing in the inpatient setting and how often they are “stable” based on a flexible definition. Moreso, we demonstrate that it is possible to use existing electronic health record data to generate probabilistic predictions into the likely range of future lab diagnostic values, before they are even ordered. This opens up a pathway to implement personalized clinical decision support to guide clinicians on when diagnostic tests are more or less likely to be stable vs. yield new information to inform more effective choice of testing intervals.

The models were evaluated on two scenarios: with and without overlapping patients between the training and test sets. The performances under both scenarios are generally similar, demonstrating the generalizability of the models. It is of note that the subset with overlapping patients removed is biased in favor of new patients, as those patients with previous encounters were removed. In a hypothetical deployment scenario, model inference would be triggered on new patients and patients seen during model training.

To assess interpretability, feature importance was calculated. As expected, the main predictor by far for a future value is the last value, but many other components are contributory. For example, other contributors include vital sign abnormalities such as temperature, heart rate and SpO2, which makes sense as they are indicative of dynamic patient conditions that are worth monitoring with repeated testing.

Results show the models are slightly mis-calibrated, especially at the low predicted score regions. One possible explanation is the misspecification of modeling assumptions. For example, the noise was assumed to be Gaussian, which might not model the real world very well. It is of note that the models are well calibrated for cases of high predicted scores, which reflect the potential deployment settings. Those cases are most relevant as feedback to clinicians is likely to be triggered only if we are confident that the lab test results are going to be stable.

The findings show promising performances for predicting stability for most of the lab test components. This implies a considerable amount of the laboratory testing in one hospital alone could be safely and effectively reduced. A common concern is the potential harm of false positives and the potential risk of missing an important diagnosis or early warning signs which might override all cost saving benefits. One potential solution is to space out the ordering intervals for patients predicted to have stable test results, as it is not a one-shot decision making. For example, instead of testing daily (which is very common as shown in Figure 3) or multiple times per day, it could be reduced to testing every other day.

The capability for use-time customization is desirable as the end users such as clinicians can use their own expertise to choose different levels of stability thresholds. Consequently, the decisions can be more personalized as the patient-specific circumstances and risk tolerances can be considered. It is also noteworthy that this framework with uncertainty estimates is general and can be applied to other lab tests and other predictive tasks.

We were not able to predict the stability of sodium at 90% positive predictive value. One possible explanation is that sodium levels are intrinsically more volatile thus harder to predict. If the clinicians accept a lower positive predictive value such as 80%, the model can still capture a significant portion of those low-yield sodium tests. More robust feature engineering can be conducted in the future to improve performances.

The inability to predict certain test components might raise the concern that there is no actionable plan as the whole panel must be ordered. It is straightforward to consider the case of the entire panel being stable, as defined by all components being stable. It occurred around 20% to 30% of the time in the cohort we studied, but this is a coarse binary label as compared to the more flexible probabilistic regression, thus we emphasize the component results that the end-user clinician can integrate into a holistic decision framework. Furthermore, a clinician might not be particularly interested in all lab test components when ordering a panel test. For example, a clinician is likely to need to closely monitor a patient’s creatinine (CR) levels for acute kidney injury, while common fluctuations in sodium (NA) values on a day-to-day basis in the hospital rarely change acute management. Further, we emphasize tracking trends in WBC and HGB that a clinician is likely to monitor for patients at risk of infection or bleeding, and specifically ignore that a CBC generates many other values like RDW, MCV, MCHC that rarely change acute management and are thus not the high yield components to monitor.

Limitations in this study include that the data for evaluation are limited to retrospective data, so prospective evaluation in multisite settings are pending to have a more comprehensive understanding of real-world performances. This speaks to the importance of why we used only readily available structured EHR data at the time of test ordering, to ensure that real-time prospective deployment and evaluation is feasible. Another issue is that the hospital is not a uniform entity but treats many different sub-groups of patients from different departments (e.g., intensive care unit vs. surgical vs. medical wards) with different risk and clinical profiles that may warrant different monitoring and diagnostic approaches. This ultimately speaks to the value of our approach to offer more personalized guidance based on individual patient circumstances, as opposed to one-size-fits-all blanket lab test ordering policies that institutions are otherwise constrained. After thorough evaluation and validation, a potential deployment workflow needs to be figured out, especially on the format of feedback to clinicians and integration within the EHR system.

Conclusion

We find that low-yield, repeated laboratory diagnostics that unnecessarily burden patients and increase cost of care is common at an academic hospital. This study demonstrates the feasibility of using machine learning approaches to identify low-yield laboratory diagnostics at order time using readily available electronic health record data. Deployment of such models has the potential to discourage low-value repeated tests which incur cost and indirect harm.