Using machine learning to develop smart reflex testing protocols

Abstract

Objective

Reflex testing protocols allow clinical laboratories to perform second line diagnostic tests on existing specimens based on the results of initially ordered tests. Reflex testing can support optimal clinical laboratory test ordering and diagnosis. In current clinical practice, reflex testing typically relies on simple “if-then” rules; however, this limits the opportunities for reflex testing since most test ordering decisions involve more complexity than traditional rule-based approaches would allow. Here, using the analyte ferritin as an example, we propose an alternative machine learning-based approach to “smart” reflex testing.

Methods

Using deidentified patient data, we developed a machine learning model to predict whether a patient getting CBC testing will also have ferritin testing ordered. We evaluate applications of this model to reflex testing by assessing its performance in comparison to possible rule-based approaches.

Results

Our underlying machine learning models performed moderately well in predicting ferritin test ordering (AUC=0.731 in reference to actual ordering) and demonstrated promising potential to underlie key clinical applications. In contrast, none of the many traditionally framed, rule-based, hypothetical reflex protocols we evaluated offered sufficient agreement with actual ordering to be clinically feasible. Using chart review, we further demonstrated that the strategic deployment of our model could avoid important ferritin test ordering errors.

Conclusions

Machine learning may provide a foundation for new types of reflex testing with enhanced benefits for clinical diagnosis.

Introduction

Appropriate clinical laboratory test selection is critical to high-quality clinical care.¹ Since many, if not most, clinical decisions rely in significant part on clinical laboratory data, a physician’s failure to order necessary tests can lead to delayed, overlooked, or incorrect diagnoses; unnecessary downstream testing; and improper or foregone treatments.¹ For example, a failure to order iron studies in patients with findings suggestive of iron deficiency may lead to an overlooked diagnosis of iron deficiency and delayed treatment. This may in turn subject the patient to unnecessary symptoms and potentially even significant morbidity and mortality.² Ordering unnecessary tests is also problematic.³ Test over-utilization not only wastes physician, nursing, and laboratory resources, potentially subjecting patients and the healthcare system to financial harm, but it can also lead to false positive results, unnecessary patient anxiety, unnecessary downstream workups, and in some cases even inappropriate treatments. In addition, physicians may overlook important test results among unneeded ones in patients with excessive testing data.^4,5 To improve test utilization and patient care, clinical laboratories and health systems can pursue strategies including the development of institutional guidelines and best practices, the delivery of focused clinician education, the deployment of clinical decision support, and the offering of “reflex testing” protocols.^1,4,6

Reflex testing enables the laboratory to automatically add on and perform second line tests on patient samples, as clinically indicated based on the results of initial tests.⁴ For example, an institution might offer a thyroid testing reflex protocol where laboratories first test thyroid stimulating hormone (TSH) and then add on additional second line testing needed for diagnosis (eg, free thyroxine [free T4] and/or triiodothyronine [T3]) as warranted based on the TSH results. Laboratories offering institutional reflex testing will generally offer clinicians a choice between a standalone test or the reflex protocol (eg, clinician could choose TSH alone or TSH with reflex to T3 and free T4). Reflex testing solves several test utilization challenges. First, without reflex testing, clinicians might be inclined to order second line tests upfront, just in case they will be needed. This is likely to lead to test overutilization. Conversely, if clinicians do not order second line testing upfront, they will need to use manual processes to add on the needed second line test. In practice, testing cascades, where second line test selection requires knowing the results of first line tests, in the absence of reflex protocols, will at best unnecessarily consume laboratory and clinician resources and at worst lead to oversights in which important second line tests do not get ordered or get delayed.

At present, reflex testing protocols typically involve one or several “if-then” rules to determine whether to add second line tests to patient specimens based on first line test results (eg, if TSH is high then add Free T4). However, reliance on simple if-then rules limits the scope of reflexing protocols to test ordering decisions that are reducible to simple rules. In practice, many test ordering decisions are nuanced and patient-specific⁷ and would be difficult, if not impossible, to support using traditional reflex testing frameworks. This may explain, at least in part, why laboratories typically only offer a narrow menu of reflex testing options. Moreover, rule-based reflex protocols do not scale well since every testing scenario will generally require a custom, often institutionally developed set of rules. Improved reflex testing protocols, to the extent that they could appropriately encompass nuanced test ordering and be applied to widely varied situations, could provide a valuable tool in supporting optimal test selection and ordering.

A potential solution to some limitations of rule-based reflex protocols is to use machine learning to determine when to reflexively add on various tests, a concept we explore in this manuscript and call the “smart” reflex protocol. For example, machine learning based models could predict which second line tests a typical clinician would order and then automatically order these to be performed on existing specimens.

In this article, we explore this concept of the “smart” reflex protocol, using ferritin, a diagnostically important blood-based biomarker of iron stores,⁸ as an exemplar. Low ferritin is virtually diagnostic of iron deficiency. Iron deficiency is most often suspected based on CBC test results; iron deficiency is a common cause of anemia, as identified by abnormally low hemoglobin and hematocrit parameters. Other CBC parameters including a low or low-normal MCV (or a decrease in MCV from a patient’s baseline) and a high RDW may further suggest iron deficiency. However, no CBC finding in isolation can definitively diagnose or exclude iron deficiency, and no reflex testing protocols currently exist to add on ferritin studies based on the CBC results alone (neither in our lab nor in any other clinical lab, to the best of our knowledge).

Here, we develop a proof-of-concept machine learning based smart reflex testing algorithm, designed to decide whether to reflexively add a ferritin test to a patient’s CBC, using CBC results and other prior patient data. Our machine learning model is trained to predict whether a clinician would have manually ordered a second line ferritin test given the CBC results, using retrospective electronic medical record (EMR) data as the ground truth. That is, we assume that if a clinician manually ordered a ferritin on a patient after receiving the CBC results, then they would also have wanted to use a reflex protocol, were it deployed, to order ferritin on that patient. We validate our model’s performance both against the retrospective ground truth and in comparison to simple if-then rules of the type that might be considered were a lab exploring traditional types of reflex protocols for ferritin.

Methods

Our methods are summarized in Figure 1. As described in greater detail below, we extracted sets of clinical laboratory test results from patients who, at minimum, had CBC testing. Using this set of laboratory test results along with other basic patient data, we developed a machine learning model to predict whether a patient would have a ferritin test ordered concurrent with and/or within 30 days of each CBC.

Figure 1.

Methods overview. Shown is a flowchart providing an overview of the dataflow and methods we used to develop and test our prediction model.

Site, dataset, patient inclusion, and ethics

Data for this study were derived from clinical laboratory testing performed at the Massachusetts General Hospital (MGH) and affiliated health centers. MGH is a large, tertiary care hospital based in Boston, MA and is a member of the Mass General Brigham (MGB) health system. All data used received institutional review board approval.

Data were extracted from the MGB clinical laboratory information system (LIS) via an established laboratory data mart. Patients were included in this study if they had at least one outpatient CBC test during a 1-year period (study year) performed in the clinical laboratories at MGH. In addition to CBC test results, we extracted additional laboratory test results as shown in Table 1.

Table 1.

Analytes included as predictors or target variations.

Analyte group	Analytes included	Used as a targeta	Concurrent results used as a predictorb	Aggregate historic results used as predictorsc
Ferritin	Ferritin	✓		✓
CBC	Hemoglobin, hematocrit, platelet count, MCV, MCH MCHC, RDW, RBC, WBC		✓	✓
Routine chemistry	Alanine transaminase; Albumin; Alkaline phosphatase; Anion gap; Aspartate transaminase; Bicarbonate; Blood urea nitrogen; Calcium; Chloride; Creatinine; Globulin; Glucose; Magnesium; Phosphorus; Potassium; Sodium; Total bilirubin; Total protein			✓
Coagulation	aPTT; D-dimer; INR; PT			✓
Hematology (nonroutine CBC)	Absolute basophil count; Absolute eosinophil count; Absolute lymphocyte count; Absolute monocyte count; Absolute neutrophil count; Bands; Metas; Myelos; Percent basophils; Percent eosinophils; Percent lymphocytes; Percent monocytes; Percent neutrophils; Percent nucleated RBCs; Reactive lymphs; Retic; Schistocytes			✓
Iron studies (besides ferritin)	Iron; Total iron-binding capacity;			✓
Lipids	Cholesterol; HDL; LDL; Triglycerides			✓
Other	Amylase; B12; CRP; CEA; Creatine kinase; Direct bilirubin; ESR; Folic acid; Free T4; LDH; Lipase; NT-ProBNP; Osmolality; PTH; Lactic Acid; PSA; Testosterone; Troponin-T4; TSH; Uric acid; Urine total protein; Vitamin D			✓

^aTarget variable was dichotomous describing whether a ferritin was performed concurrent with or within 30 days of the CBC.

^bThe test results itself was used (or in rare cases where not reported on the current CBC, the most recent prior result within 30 days).

^cThe patients’ the mean value of the analyte; the standard deviation of the values of the analyte on the patient; the number of times the analyte was tested on the patient (count); the minimum and maximum values of the analyte; the sum of all results for the analyte (mean times count). All calculated across all of the patient’s results from analyte collected from 2 years (in some cases as long as 3 years) prior up until 1 month prior to the CBC.

^d4th generation troponin T, only; excluded more recent 5th generation results.

Data temporal aggregation and processing

The unit of analysis for our study was a “CBC-event” defined as an outpatient CBC during the study year. Each CBC was treated as a separate case for the purposes of our analysis. For each CBC-event, we defined and captured a dichotomous target variable (ferritin ordered) denoting whether a ferritin was performed at the time of and/or within 30 days after the CBC. Additionally, we captured predictor variables including age, gender, and laboratory testing data as summarized in Table 1. As shown, we included current CBC parameters as predictors, but no non-CBC lab results from the 30 days prior to the CBC. We chose not to include concurrent (and up to 30 days prior) results from non-CBC tests in part to avoid potential biases (for example that iron studies tend to be ordered together) and to facilitate the development of an algorithm that would perform well regardless of what additional labs (if any) beyond the CBC were ordered. We additionally included predictor data across a range of analytes (summarized into defined features as described in Table 1), for test results occurring between 2 years prior to the CBC up until 30 days before the CBC. These data were aggregated temporally by taking the mean, count, minimum, maximum, and sum (equivalent to mean times count) of values observed during the historical period. All data were centered and scaled to have zero mean and unit variance. Missing values were imputed to the nearest value for the recent CBC feature-set, and for the historical lab values any labs that were never measured were imputed to the population mean and the fact that no data were reported for that feature was captured in the count statistic. As a result of our technical approach to data extraction and aggregation, for cases in which a patient had multiple CBCs during the study year, prior results were considered up to 2 years prior to the first CBC during the study year; thus for a patient with a CBC both early in the study year and late in the study year, the prediction in reference to the later CBC would include predictor data as early as 2 years prior to the earlier CBC and thus up to 3 years before the index CBC event.

Of note, in very rare cases, individual CBC parameters may be ordered by the clinician without a full CBC being performed; for the purpose of this analysis, we assume the patient had a CBC if results were reported for any of the CBC parameters. The sample collection date and time were used as the timestamp for all lab results.

Data partitioning

Data were randomly split into training, hyperparameter tuning, and testing partitions in an 80:10:10 ratio. Individual patients, even if enrolled multiple times due to multiple CBCs, were not split across partitions. Using a Monte-Carlo cross validation framework, we split the data into 10 random such partitions, and repeated the same preprocessing, training, hyperparameter tuning, and validation procedure on each split, and results represent cross validation performance reported using the test sets for the optimal model configurations according to hyperparameter tuning in each split separately. After assembly of our initial data table, we performed all processing steps separately for the training and testing partitions including result scaling and mean imputation of unordered predictor test, to ensure that there was no leakage of information from the training to the testing partition. The initial assembly involving data extraction and dataset assembly that was performed prior to the training/testing splits relied only on individual patient level data and did not perform any computations that utilized data from multiple patients and so there was no risk of information leakage.

Model training and validation

We built logistic regression and random forest-based models using Scikit-learn.⁹ Hyperparameter tuning was performed via a programmatically automated grid search over a range of regularization parameters for both the Logistic Regression and Random Forest model types, and final results are based on whichever model type and hyperparameters maximized performance for the hyperparameter tuning partition for each partition. The hyperparameters evaluated are shown in the Supplementary Material. In all cases, random forest models outperformed logistic regression baselines, so final results use that model type, with mildly different regularization parameters per partition.

Chart review

In a subset of cases where the model predictions disagreed with clinician decisions in the EMR, we additionally performed chart review. In particular, we identified the 200 cases where ferritin was ordered that had the lowest predicted probability of ferritin ordering and the 200 cases where ferritin was not ordered that had the highest predicted probability of ferritin ordering (probability cutoffs for each group are included below in the “Results” sections). For the purposes of chart review, we considered cases that were included in the test partition for at least 1 of the first 3 (of the total 10) runs of the model and used the median predicted probability of ferritin ordering across any of the 3 runs for which the case fell into the test partition. From each set of 200 cases, we randomly selected 20 for chart review. All cases selected for manual chart review were reviewed by 1 pathologist (J.M.B.); half of the cases were also independently reviewed by a second pathologist (A.S.D.) to capture interobserver agreement. Pathologist review involved reviewing the electronic health record, including narrative notes and other findings to decide whether the patient should have been tested for ferritin (even if they were not actually tested in reality). The pathologist scored the cases on an ordinal scale as follows: Ferritin Clinically Indicated; Ferritin Possibly Indicated; Ferritin Probably Not Indicated; Ferritin Not Indicated. Considerations in the pathologist review included additional explanations for ordering or not ordering ferritin offered in the clinical notes and evidence of possible iron deficiency (or other reason to order ferritin) based on holistic review of relevant clinical notes and laboratory test results.

Comparison to rule-based reflex testing protocols

We compared model performance to a series of hypothetical rule-based triggers analogous to the type of rules that laboratories typically use in reflex testing protocols. Hypothetical rule sets used are shown in Table 2. Although to the best of our knowledge, no lab actually has a ferritin reflex testing protocol (perhaps due to the inadequacy of rule-based triggers in this context), we present this analysis to show why an algorithm would present a practical strategy even in cases such as this where rules fail.

Table 2.

Hypothetical reflex testing rules used for comparison to the model.

Rule	Definition
Low HCT	Patient’s HCT is below the reference limit
Low MCV	Patient’s MCV is below the reference limit
Prior Ferritin	Patient had ferritin test in the 2 years prior to the CBC
Prior Ferritin and Low HCT	The patient had ferritin test performed in the 2 years prior to the CBC AND the patient’s current HCT is below the reference limit
Low MCV and High RDW	The patient’s MCV is below the reference limit AND the RDW is above the reference limit
Low HCT or MCV	The patient’s HCT is below the reference OR the patient’s MCV is below the reference or both
Prior Ferritin and Low MCV	The patient had ferritin test performed in the 2 years prior to the CBC AND the patient’s current MCV is below the reference limit
Low HCT and MCV, High RDW	The patient’s HCT and MCV are both below their respective reference limits and the RDW is above the reference limit

These are representative of the types of rules a laboratory might use to develop a reflex protocol for ferritin. In practice, we are not aware of any labs with such a ferritin reflex protocol, perhaps due to the inadequacy of these rules (and hence the motivation for this article). Abbreviations: MCV, mean cell volume; HCT, hematocrit; RWD, red cell distribution width. Gender specific, adult reference ranges were used (regardless of actual patient age); HCT: male 41%, female 36%; MCV 80 fL; RDW 14.5%.

In particular, we assessed the concordance between patients who would (and would not) have had ferritin ordered under each hypothetical set of reflex rules and the actual ferritin ordering in practice. In assessing predictive model performance in this subanalysis, only test partition predicted probabilities were considered and for patient-CBCs included in the test partition on multiple training-test splits, the mean predicted probability across runs was used. The hypothetical rule based reflex protocols are shown in Table 2.

Analytics and statistics

Modeling was performed in Python using Scikit-learn.⁹ Outcome metrics were assessed using auROC, auPrC, and Brier loss, all calculated using Scikit-learn and aggregated over the Monte-Carlo cross validation partition test sets. Analyses comparing the model to hypothetical reflex rules were performed in R¹⁰ and plotted using the ggplot2 package¹¹ with confidence intervals for positive and negative percent agreement (PPA and NPA) calculated using the R binom package and Wilson method.

Results

The final dataset included 288 427 unique “CBC-events” across 137 451 unique patients. Thirty-five thousand one hundred thirty-two of the CBCs (12.2%) were associated with a ferritin within 30 days.

Model performance

Model performance is shown in Figure 2. As shown, the models provided moderate discriminative power in predicting whether ferritin would be ordered within 30 days as shown in the ROC curves (Figure 1A) and precision-recall curves (Figure 1B). The overall AUC was 0.731 (standard deviation = 0.004) and AUPRC was 0.349 (standard deviation = 0.011). The models were generally well-calibrated as shown in Figure 2B (Brier loss = 0.094, standard deviation = 0.003). We performed a confirmatory analysis where we treated rare ferritin tests timestamped shortly before the CBC as concurrent with the CBC as described in Supplementary Appendix B; as shown in this Supplementary Material, this confirmatory analysis was highly consistent with our primary findings shown in Figure 2. We likewise assessed performance with patients subgrouped by sex and age quartile (<40, 40–58, 58–70, and >70 years); there were few subgroup differences with subgroup AUC results ranging from 0.71 to 0.75.

Figure 2.

Model performance. Shown is a calibration curve (A), a receiver operating characteristic curve (B), and precision recall curve (C) for the test partition aggregated across the 10 runs. Ribbons represent mean plus/minus standard deviation across all 10 runs.

Figure 3 shows the features that were most important in model predictions. In general, these closely overlap with the clinical features that might suggest iron deficiency or that a physician might consider in deciding whether to order ferritin. Of note, in addition to clinical features, the model found whether the patient had a ferritin test within the prior 2 years (excluding the prior month) to be highly informative. This is in large part likely capturing the fact that patients with a history of iron deficiency or prior signs and symptoms requiring ferritin testing are more likely to undergo ferritin testing. It may also in part capture clinician style with respect to whether the patient is seeing a physician who tends to order ferritin.

Figure 3.

Feature importance. Aggregated feature importance (higher is more important) across all models (mean plus/minus standard deviation), as judged by the average decrease in the gini entropy when this feature is used across all decision trees in the random forest. “(prior)” indicates the historical value of the feature (older than 30 days ago) is being referenced, rather than the current results for that test.

Comparison of machine learning to rule based reflex testing

While to the best of our knowledge, no laboratory supports a ferritin reflex testing protocol (presumably due to the inadequacy of traditional rules for doing so) here we compared hypothetical rules involving hematocrit, MCV and RDW of the type that a lab might consider for reflex testing in other context to the performance of our machine learning model (Figure 4). In this analysis, we evaluate the PPA and NPA. PPA and NPA parallel sensitivity and specificity, but these terms recognize the lack of a true gold standard; as defined here, PPA indicates the agreement between actual ordering and the algorithm or rule set when ferritin was actually ordered and NPA indicates the agreement when ferritin was not actually ordered. As shown, the rule-based reflex testing protocols exhibit inadequate performance to be used clinically. In contrast, our smart reflex protocol demonstrates potential to serve key clinical applications (as elucidated in the discussion). For example, as illustrated in Figure 4, the most sensitive rule set tested (Low MCV or Low HCT) only achieved a 51% PPA at an NPA of only 63%. In contrast, our smart reflex protocol can achieve an NPA of about 80% at that rule’s PPA, or a PPA of 63% at that rule’s NPA. While model performance would likely require further optimization (see “Discussion”) for general clinical application, the current model may offer support for key applications as detailed in the discussion.

Figure 4.

Comparison to hypothetical rule-based reflex testing protocols. Shown is an ROC-like plot describing the concordance between actual ferritin test ordering and both: (i) the predictive model (blue curve; looks across various probability thresholds in the test data) and (ii) various hypothetical rule-based testing triggers (labeled white points). Performance of randomly adding on ferritin (red line) is also shown as a negative control. One minus negative percent agreement (1−NPA, x-axis) represents the proportion of patients who did not have ferritin in actual practice who would have using the model (at a given cutoff) using the specified hypothetical reflex testing rules. Positive percent agreement (PPA, y-axis) represents the proportion of patients who had ferritin in actual practice who would have using the model (at a given cutoff) or specified rules. Line width (blue and red curves) represent 95% confidence intervals. For hypothetical rules, white points represent point-estimates and black boxes surrounding white points represent 95% confidence intervals. By way of example, the most sensitive rule shown (Low HCT or MCV) has a PPA and NPA of only 51% and 63%. Our smart reflex protocol could offer a PPA and NPA of 51% and 80% or of 63% and 69% (among other combinations shown in the blue curve). (HCT, hematocrit; MCV, mean cell volume; RDW, red cell distribution width).

Association between predicted probability of ferritin and ferritin result

As a secondary goal of this manuscript (alongside our primary goal of developing a proof of concept for smart reflex testing), we evaluated the association between model predicted likelihood of ferritin ordering and ferritin results as described in the Supplementary Material A. We unsurprisingly find evidence (Figure S1) that ferritin test results are not missing at random.

Clinical appropriateness of predictions

While we trained our models to predict whether ferritin would be ordered, an ideal reflex protocol would capture whether ferritin should or should not be tested. To determine to what extent our protocol captures this, we performed chart review on selected cases (as described in the methods) where model predictions and actual test ordering were particularly discordant. We found that in many cases the model predictions were more “correct” (ie, appeared to be better in line with ideal clinical practice) than the actual clinical ordering (Table 3). Indeed, in multiple cases reviewed where the model predicted ferritin would be ordered but it was not, the failure to order ferritin appeared to be a simple oversight (eg, note indicates intention to order ferritin and ultimate order includes other iron studies but not ferritin). Likewise, in many cases where ferritin was ordered in contradiction to model predictions, the ferritin appeared to be unnecessary. Interpathologist concordance in assessing the need for ferritin is shown in Table 4. In all but one case, the 2 pathologists arrived at the same conclusion or a conclusion within one level on the ordinal scale.

Table 3.

Chart review findings.

Pathologist assessment	Cases where the model predicted a relatively high probability of ferritin testing but ferritin was not testeda	Cases where the model predicted a relatively low probability of ferritin testing but ferritin was testedb
Ferritin clinically indicated	10	1
Ferritin possibly indicated	2	4
Ferritin probably not indicated	3	4
Ferritin not indicated	5	11

^a20 randomly selected cases from the 200 with the greatest predicted probability of ferritin testing where ferritin was not tested.

^b20 randomly selected cases from the 200 with the lowest predicted probability of ferritin ordering where ferritin was tested.

Shown are the findings of a pathologist in reviewing cases. Twenty of the 200 test cases where the model most confidently predicted ferritin would be tested as well and 20 of the 200 test cases where the model most confidently predicted ferritin would not be tested. Cells represent the number of cases.

Table 4.

Pathologist chart review concordance.

		Pathologist 2
		Ferritin clinically indicated	Ferritin possibly indicated	Ferritin probably not indicated	Ferritin not indicated
Pathologist 1	Ferritin clinically indicated	4	0	0	0
	Ferritin possibly indicated	2	0	1	0
	Ferritin probably not indicated	1	1	0	1
	Ferritin not indicated	0	0	0	10

Shown is the number of cases based on the assessments of the 2 pathologists. Numbers shown indicate the number of cases. Pathologist 1 (J.M.B.) is the primary pathologist whose findings are shown in Table 3. Green cells represent complete agreement while yellow cells represent agreement within one level on our ordinal scale. Only a subset of the cases shown in Table 3 were reviewed by both pathologists and shown here.

Discussion

Here, we provide a proof-of-concept regarding a new machine learning based approach to reflex testing: the “smart” reflex protocol. By offering improved performance over the types of rule-based approaches currently used in clinical practice, machine learning models may have multiple applications to reflex testing. As shown in Figure 5, the general approach would be for a clinician to order a CBC with possible reflex to ferritin (and other iron studies). The algorithm would automatically add on ferritin as appropriate based on model computations performed after the CBC results become available. The performance of the current algorithm is likely inadequate for it to be used to fully replace manual ferritin test ordering (although we anticipate being able to improve performance through subsequent work as below). Nonetheless, even with current performance our algorithm could support important clinical scenarios pending additional prospective validation (Figure 5). The first of these (Variation 1), would be the clinician who would order CBC plus iron studies upfront on a patient, anticipating the iron studies to be needed and not wanting to overlook them. In this case, the clinician could order a reflex protocol using a highly sensitive algorithm cutoff that when in doubt will perform ferritin. This algorithm could “cancel” the ferritin order when it can confidently predict it to be unneeded. A second application (Variation 2) would be in patients where the clinician is simply ordering a CBC without a suspicion of iron deficiency. The algorithm could run at a highly specific cutoff helping to reduce the risk of overlooked iron deficiency.

Figure 5.

Clinical applications to smart reflex testing. Potential applications to smart reflex testing are illustrated. The general case would require further model development and variations 1 and 2 would require further validation prior to implementation in clinical practice.

Another advantage of our approach is potential generalizability. Unlike rule-based reflex testing that often requires each alert be set as a one-off, adaptations of the prediction models described here could potentially be trained across hundreds or thousands of test orders using the same approach developed herein. Of course, for many tests, it will be critical to pull nonlaboratory clinical factors into the model for use as predictors.

Implementing a machine learning-based reflex testing protocol would likely require creative deployment strategies that go beyond the use of the standard LIS or laboratory middleware functionality that laboratories currently use for rule-based reflex testing. One approach would involve translating a trained underlying logistical regression or random forest algorithm into a set of stepwise arithmetic calculations and if/then rules. Since commonly used middleware and LIS systems have calculation modules that can apply arithmetic calculations and if/then rules to drive workflow including reflex testing, in theory, our algorithm could be implemented in this way. However, given the number of variables, and rules that would be required, whether there would be practical barriers to implementing an algorithm in this way in some or all systems remains uncertain. Another approach would involve algorithm implementation functionality native to some EHR systems and/or “third-party” add on algorithm engines. Nonetheless, initial implementation of this type of model will likely represent a substantial challenge, but one that we feel fairly confident is addressable through future work.

Prior to model implementation, substantial additional evaluation and validation would be needed. This would need to include an assessment for potential racial or other biases. Although we evaluated our model for performance based on patient age and sex, the dataset we extracted did not have race; we will extract race for future iterations of model development so that we can assess performance across racial groups to identify any potential biases. However, simply testing model performance by race will not alone be sufficient since the underlying clinician ordering patterns used to train and test the model could reflect racially inequitable care. Additional qualitative assessments would also be needed to fully ensure that the model is not biased. Indeed, since our model did not have explicit racial information, we would hypothesize that our model may produce less biased test ordering decisions than clinicians (although it is possible that racial biases would be indirectly reflected through predictors applied within the model).

A laboratory would also need to consider some administrative, logistical, and more subtle clinical considerations prior to implementation. For example, it would likely be most appropriate to offer such a protocol with a clinician override option allowing the clinician to either order a ferritin regardless of the algorithm or just a standalone CBC without the possibility of a reflex ferritin. This would parallel typical current processes for institutional (rule-based) reflex protocols as needed for good clinical care and compliance. Likewise, institutional reflex protocols would typically require approval by hospital medical policy committees. Finally, questions around billing and other regulatory compliance, ethics and liability for potential harm often come into play when considering clinical implementation of machine learning algorithms.¹² Future work towards implementation of these types of protocols will require exploration of these questions.

One additional practical consideration is that after implementation of a smart reflex algorithm, the laboratory would cease to develop new data on which to update the model (eg, there would no longer be new clinician ordering data to use for updated training or validation). This could be a problem, if, for example, clinical practice were to shift. Laboratories may need to employ a combination of strategies including ongoing evaluation of the algorithm using cases where the clinician opts out of the reflex protocol (although these may be fundamentally different from those where the algorithm is applied), use of data from other laboratories not employing the protocol and/or manual review to ensure continued validity of the model.

Although we chose ferritin as an initial proof of concept, we anticipate that we would likely achieve similar performance with other iron studies such as total iron binding capacity (TIBC), and indirect marker of transferrin and iron. In our dataset, 84% of CBCs associated with ferritin within 30 days were also associated with TIBC and 91% associated with TIBC were associated with ferritin, implying that a model trained on one analyte could be substantially applicable to the other. In practice, we would likely train a more general “iron studies” reflex protocol for implementation that determined whether a typical set of iron studies including ferritin, iron, and TIBC should be ordered together.

While the primary aim of this article was to explore smart reflex testing algorithms, we envision a secondary application of the algorithm we developed: reducing imputation bias when imputing missing laboratory test results in the development of machine learning models. A common framework for the development of machine learning based prediction models involves first imputing missing predictor data (eg, missing laboratory test results) and then applying actual and imputed data to a downstream prediction model. However, most imputation methods assume missing data are missing at random (MAR) and may introduce biases if the MAR assumption is violated (as would typically be the case with missing lab results).^13–17 Indeed, our findings that the rate of abnormal ferritin results correlates with the likelihood of ferritin ordering (Figure S1) suggests that the distribution of missing ferritin values differs from the distribution of measured values and in turn might suggest that typical imputation methods are likely to produce biased imputation results. We anticipate that use of models such as this for predicting whether a test will be ordered may help to correct for imputation bias. We aim to explore this in future work.

An important consideration is that clinicians use information, including clinical factors to decide whether to order ferritin, that go beyond the information available to our model. Thus, we envision being able to improve this model through future work by incorporating additional nonlaboratory predictors.

Limitations

This study is subject to several limitations. First, despite the interpathologist concordance, use of chart review for clinical validation remains subjective. However, chart review is only used as a final feasibility assessment and many of our model evaluation metrics (eg, those described in Figures 2-4) are not subjective. More importantly, for the approach developed in this manuscript to generate clinically optimal test ordering, the training data and hence current test ordering practice would need to represent clinically optimal test ordering. This consideration impacts not only our proposed smart reflex testing framework, but other intelligent decision support systems developed in the past.¹⁸ In practice, actual test ordering is imperfect. Nonetheless, even if imperfect, the model may be sufficient to offer improvement on current practice, a hypothesis that may in part be supported by the chart review. Future solutions may also use hand-developed rules to filter out clearly nonoptimal test orders (or missed orders) from the training data to improve clinically targeted performance. More rigorous validation studies would be needed prior to clinical implementation. Another limitation is that this study is based on a single center. Finally, we did not incorporate nonlaboratory clinical predictors into our prediction models. We aim to explore doing so in future iterations and expect that this may improve performance.

Conclusion

This article offers a framework to enhance clinical laboratory diagnostics through the development and deployment of machine learning-based “smart” reflex testing. We anticipate that models such as the one developed here have the potential to expand reflex testing across a wide range of patients and tests.

Author contributions

Project framework and design: All authors. Model development: M.M. Data structuring and analysis and model evaluation: M.M. (primary) and J.M.B. (secondary). Chart review: J.M.B. and A.S.D. Manuscript initial drafting: J.M.B. and M.M. Manuscript revision: All authors.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was support by National Library Medicine (grant number R01 LM013337).

Conflict of interest

J.M.B. is an employee of Roche Diagnostics in addition to his academic role.

Data availability

Raw patient data cannot be shared due to privacy constraints and institutional policies.