Development of criteria to optimize manual smear review of automated complete blood counts using a machine learning model

Abstract

Background

Manual blood smear review (MSR) to complement automated CBC results is a labor‐intensive process. Efforts have been made to use criteria based on automated hematology analyzer data to identify samples warranting MSR. These efforts have coincided with the emergence of modern data science and machine learning.

Objective

In this study, we aim to determine if machine learning can reduce manual smear review (MSR) rates while meeting or exceeding the performance of traditional MSR criteria.

Method

9938 automated CBCs with paired MSRs were performed on samples from rhesus and cynomolgus macaques. The definition of a positive (abnormal) smear was determined. Two expert‐derived MSR criteria were created: criteria adapted from published, standardized human laboratory criteria (Adapted International Consensus Guidelines[aICG]) and internally generated criteria (Center Consensus Guidelines [CCG]). An ensemble machine learning model was trained on an independent subset of the data to optimize the balanced accuracy of classification, a combined measure of sensitivity and specificity. The resulting machine learning model and the two expert‐derived MSR criteria were applied to a test dataset, and their performance compared.

Results

aICG criteria demonstrated high sensitivity (80.8%) and MSR rate (74.2%) while CCG criteria demonstrated lower sensitivity (57.1%) and MSR rate (36.1%). The machine learning model integrated with CCG criteria had a superior combination of both sensitivity (76.8%) and MSR rate (45.1%) achieving a false negative rate of 1.6%.

Conclusion

Machine learning in combination with expert‐derived criteria can optimize the selection of samples for MSR thus decreasing MSR rates and labor efforts required for CBC performance.

1. INTRODUCTION

The use of automated hematology analyzers for CBCs has become ubiquitous in both human and veterinary clinical pathology laboratories. However, despite the increased use of automated analyzers, manual smear review (MSR) remains a necessary part of CBC analysis due to the inability of automated analyzers to consistently detect clinically relevant abnormal morphologies. ¹ Morphologic assessment of leukocytes, erythrocytes, and platelets is important for both clinical patients and research subjects. Although some abnormal morphologies may be detected or suggested by the results of the automated CBC using various numerical results, indices, and flags, some important morphologies are not detectable by these means and require MSR. ¹ This selection can, therefore, be either universal (all samples receive MSR) or selective (samples predicted to be the most likely to have abnormal morphologies receive MSR). The latter approach has the potential to significantly reduce labor‐intensive MSR rates. Therefore, identification of criteria that use automated analyzer results to successfully select abnormal samples is critical.

Increases in the quality and speed of automated results combined with lower costs compared to manual methods have led to optimization efforts in human medicine to decrease the number of MSRs. ² , ³ , ⁴ In studies involving human patient samples, MSR rates vary widely ² , ⁵ , ⁶ , ⁷ , ⁸ with the mean rate from the College of American Pathologists survey in 2004 at 28.7%. ⁶ The practice of optimizing MSR rates is less common in veterinary laboratories, although automated analyzer cytograms and flag information have been used to determine when MSRs are required for cats, dogs, and horses with reported MSR rates of 20%–32%. ⁹ , ¹⁰ A coordinated effort to create standardized MSR criteria for human laboratories based on a selected set of rules (eg, thrombocytopenia below an established threshold) occurred in 2005 with the development of the International Consensus Group for Hematology (ICG) criteria for further action after automated CBC analysis. ³ When applied to 13298 samples analyzed at 15 different laboratories, the criteria demonstrated 79.3% sensitivity in the identification of positive (abnormal) smears with recommendations that false negative rates for MSR criteria validation be <5% (the highest level acceptable to maintain patient safety). ³ Several independent human laboratories have attempted to validate the ICG criteria and found it necessary to modify the rules to better fit their specific laboratories and patient populations in order to reach the desired performance. ⁴ , ⁷ , ⁸ , ¹¹ In veterinary clinical pathology, Flatland and Vap ¹ have published workflow guidelines for actions after automated CBC analysis in domestic species that required MSRs for all samples, and the ASVCP Guidelines: Principles of Quality Assurance and Standards for Veterinary Clinical Pathology ¹² has recommended that rules must be established to determine when MSR is required. In this study, we incorporated MSR criteria and optimization methodologies employed in human laboratories rather than those practices more commonly found in veterinary laboratories. While, adapting established guidelines is a known method for optimizing MSR criteria rates in human laboratories, ² , ⁴ machine learning is a less commonly explored option utilized in this study.

Machine learning provides a means to objectively analyze large quantities of data with the goal of identifying useful patterns. The subset of machine learning focused on determining the classification of a parameter given a known training dataset containing labeled samples is termed supervised learning. ¹³ The goal of supervised learning is to generate a model using known paired data (eg, paired automated hematology results and MSR results) to predict the classification of new data. The implementation of machine learning in healthcare has been reviewed by Liu et al. ¹⁴ In veterinary medicine, applications of machine learning have included the diagnosis of feline infectious peritonitis and canine skin tumors, chronic kidney disease risk prediction in cats, evaluation of respiratory sounds during exercise tests in canines, and retrospective epidemiological surveillance. ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ In human medicine, machine learning approaches, such as convolutional networks, have been applied during the MSR process to identify and/or classify hematologic conditions, including abnormal leukocytes and leukemias, ²⁰ , ²¹ , ²² neutrophil cytoplasmic abnormalities, ²³ and malarial parasites. ²⁴ However, the application of machine learning to the identification of automated hematology results requiring MSR has not been as thoroughly studied. In our work, we sought to optimize this critical decision point in the CBC workflow through the application of supervised learning to the identification of patient samples requiring MSR. By creating and evaluating the performance of a machine learning model and two expert‐derived blood MSR criteria for use in a research colony of nonhuman primates, we demonstrate that machine learning can be used to minimize the number of MSRs needed, thus reducing labor efforts while maintaining appropriate quality goals for patient safety (<5% false negatives). ³

2. MATERIALS AND METHODS

2.1. Animal use statement

The hematology data in this study were generated from animals housed at the Wisconsin National Primate Research Center. Animal care and use were in accordance with the Guide for the Care and Use of Laboratory Animals ²⁵ and protocols approved by the University of Wisconsin‐Madison Institutional Animal Care and Use Committees. Animals were indoor‐housed on a 12‐h light/dark cycle with controlled temperature and humidity and provided fresh water ad libitum, commercially formulated monkey chow, fresh produce, and approved enrichment items. Animals were housed as groups or pairs where possible.

2.2. Study population and sample collection

Data were collected from a non‐continuous subset of routine CBC testing performed in the Center's clinical pathology laboratory from 2016 through 2024. Blood samples collected from rhesus macaques (Macaca mulatta) and cynomolgus macaques (Macaca fascicularis) were included in this study and were reflective of the samples routinely received by the clinical pathology laboratory, including wellness checks, pre‐assignment and baseline evaluations, diagnostic workups for ill animals, and pre‐determined research study time points. EDTA whole blood samples were collected via femoral, saphenous, or brachial venipuncture using manual restraint methods or during anesthesia. Samples were primarily collected using the vacutainer method into K2EDTA/K3EDTA blood tubes (BD, Franklin, NJ; Greiner, Kremsmunster, Austria).

2.3. Hematologic analysis

2.3.1. Automated hematology

CBCs were performed as part of routine laboratory workflow and in accordance with standard operating procedures. Automated analysis was performed using a Sysmex XS‐1000i automated hematology analyzer with IPU(E) software version 23.04–00 (Sysmex Corporation, Kobe, Japan). Samples were thoroughly mixed prior to evaluation, and clotted samples were not evaluated. Testing was typically performed within 2–4 h of collection or on samples stored at 4°C until analysis (< 24 h after collection).

2.3.2. Manual smear review (MSR)

All samples received MSR regardless of automated hematology results. Blood smears were prepared at time of analysis and stained with Wright‐Giemsa stain using an automated slide stainer (Wescor Aerospray Hematology Slide Stainer; Wescor Inc., Logan, UT). MSRs were performed by a trained laboratory technician or anatomic veterinary pathologist with hematology training. All reviewers followed standardized MSR procedures and morphology grading scales, which for all samples included a 100–300 manual WBC differential, manual platelet estimate (except in the presence of marked platelet aggregates), and a review for abnormal WBC, RBC, and platelet morphologies as well as hemotropic parasites. Abnormal morphologies seen on MSR were scored as slight, moderate, or marked. Scoring for toxic change included Döhle bodies, toxic granules, cytoplasmic basophilia, and cytoplasmic vacuolation.

The definition of a positive (abnormal) smear (Table 1) consisted of morphologic features of cellular elements important for disease characterization, diagnosis, and prognostication that are not consistently or specifically detected by automated procedures. These morphologic features were selected with input from Center veterinarians, a clinical pathology laboratory Research Program Manager, an ACVP‐boarded veterinary anatomic pathologist, and a long‐time collaborating ACVP‐boarded veterinary clinical pathologist. These features were selected as their presence may signal the need for specific interventions such as immature neutrophils or neutrophils with toxic change, which may be associated with inflammatory disease; atypical leukocytes, which may be associated with inflammatory or neoplastic processes; nucleated erythrocytes, requiring correction of the automated WBC count to exclude these cells; abnormal erythrocyte morphologic features, which aid in diagnosis of hemolytic disorders and iron deficiency; and platelet aggregates and macroplatelets that in combination with thrombocytopenia impact interpretation and clinical interventions for thrombocytopenic patients.

TABLE 1.

Definition of a positive (abnormal) blood smear.

Parameter	Value
Bands	>1%
Metamyelocytes/myelocytes/promyelocytes	≥1%
Atypical cells/blasts	≥2%
Nucleated RBC count	≥5/100 WBCs
Toxic change	Moderate to Marked
Abnormal RBC morphology*	Moderate to Marked
Platelet aggregates	Moderate to Marked (if automated platelet count <100 × 103/μL)
Macroplatelets	Moderate to Marked (if automated platelet count <100 × 103/μL)
Hemotropic parasites	Present

^*Morphologies evaluated included: polychromasia, spherocytosis, schistocytosis, microcytosis, RBC inclusions (eg, heinz bodies, howell jolly bodies, basophilic stippling, hemoglobin crystals, and siderotic granules) and poikilocytosis (eg, codocytes, acanthocytes, dacrocytes, eccentrocytes, keratocytes, hypochromic cells/leptocytes, ovalocytes, elliptocytes, and stomatocytes).

2.4. Expert‐derived MSR criteria

Two sets of expert‐derived MSR criteria (Adapted International Consensus Group for Hematology [aICG] criteria [Table 2] and Center Consensus Guidelines [CCG] criteria [Table 3]) were created to identify samples likely to meet the definition of a positive smear based on their automated hematology results and flag them for subsequent MSR. These criteria could then be compared to the current laboratory method (review all smears) as well as a machine learning model (Figure 1). Rules incorporated into each set of criteria included numeric value thresholds, incomplete results based on automated analyzer results, or automated hematology analyzer flags. Any sample meeting at least one of these rules triggered MSR.

TABLE 2.

Adapted International Consensus Group for Hematology (aICG) criteria for manual blood smear review.

Parameter	Value
WBC Differential	Absent/incomplete
WBC count	<4 × 103/μL or >30 × 103/μL
PLT count	<100 × 103/μL or >1000 × 103/μL
HGB	<7 g/dL
MCV	<63 fL or >84.2 fL
RDW	>22% or no value given
Neutrophil count	<1.0 × 103/μL or >20.0 × 103/μL
Lymphocyte count	>5.0 × 103/μL
Monocyte count	>1.5 × 103/μL
Eosinophil count	>2.0 × 103/μL
Basophil count	>0.5 × 103/μL
Suspect WBC Flags
Blasts?	Present
Left shift?	Present
Atypical lympho?	Present
Abn lympho/blasts?	Present
Suspect RBC flags
NRBC?	Present
RBC agglutination?	Present
Turbidity/HGB interf?	Present
Iron deficiency?	Present
HGB defect?	Present
Fragments?	Present
Suspect PLT flags
PLT Clumps?/PLT clumps (S)?	Present
Other Flags
WBC unreliable flag (WBC Abn Scattergram)	Present
Dimorphic RBC Flag (dimorphic population)	Present
PLT Flags except platelet clumps (PLT Abnormal)	Present

Abbreviations: HGB, hemoglobin; MCV, mean corpuscular volume; PLT, platelet; RDW, red cell distribution width.

TABLE 3.

Center Consensus Group (CCG) criteria for manual blood smear review.

Parameter	Value
WBC differential	Absent/incomplete
WBC count	<3.5 × 103/μL or >21 × 103/μL
Hemoglobin	<10 g/dL
Automated platelet count	<100 × 103/μL
Monocytes	>15%
Eosinophils	>20%
Basophils	>20%

FIGURE 1.

A proposed workflow for CBC performance, including identification of samples requiring manual smear review (MSR) based on MSR criteria of automated hematology results and flags using either expert‐derived criteria (adapted International Consensus Group for Hematology [aICG] or Center Consensus Group [CCG]) or a model combining CCG criteria with Machine Learning (CCG + ML).

2.4.1. Adapted International Consensus Group for hematology (aICG) criteria

aICG criteria were derived from the published consensus rules established for human laboratories in Barnes et al, ³ with modifications to account for differences in species and data availability. These adapted criteria consisted of 25 rules compared to the 41 rules in the original ICG study. Adaptations included changes to source rules requiring knowledge of historic patient results or demographics, which were incompatible with the laboratory's established workflow. This included the removal of rules using delta checks (exceeding a predetermined threshold on consecutive samples from the same patient), adjustment of rules applicable only to the first sample from a patient, or the first‐time occurrence of a parameter falling above or below a set threshold in an individual patient to applying the rule to all occurrences, and in cases where age‐specific rules existed, as with lymphocytosis and monocytosis, the lowest value was chosen. Adjustments were also made to the mean corpuscular volume (MCV) thresholds to be relevant to the macaque species housed at the Center; the relative margin above and below the reference interval for human MCV ²⁶ values used by the source rules were applied to the laboratory's established macaque MCV reference interval to create the aICG rule thresholds. The presence of nucleated RBCs was not included in the adapted criteria because the hematology analyzer used did not report this. For flag‐based rules, the automated flags reported by the Sysmex XS‐1000i that most closely matched the flags applied in the source material were used for the adapted criteria.

2.4.2. Center Consensus Group (CCG) criteria

CCG criteria were historic, internally derived rules adjusted by the Center's clinical pathology laboratory Research Program Manager, ACVP‐boarded veterinary anatomic pathologist and long‐time collaborating ACVP‐boarded veterinary clinical pathologist in consultation with Center clinical veterinarians to be both clinically relevant and mandatory for inclusion in any MSR criteria. These eight rules were based on numerical results of an automated CBC targeting samples with the potential to have clinically relevant blood cell morphology changes or samples with incomplete or presumed to be inaccurate information that required confirmation (in the case of absent, incomplete, or unlikely automated WBC differentials). Samples with WBC counts <3.5 × 10³/μL or >21 × 10³/μL fell outside the Center's reference intervals and were more likely to have immature granulocytes in circulation in response to an inflammatory process in the patient. Anemic patients (hemoglobin <10 g/dL) required MSR to determine the presence of increased numbers of polychromatophilic erythrocytes as evidence of a potential regenerative response and subsequent consideration for a manual reticulocyte count for an objective classification as regenerative or nonregenerative anemia (as the Sysmex XS‐1000i does not perform automated reticulocyte counts). CBCs with thrombocytopenia at a clinically significant level (<100 × 10³/μL) required MSR to exclude the presence of platelet clumps. Through years of experience with the Sysmex XS‐1000i hematology analyzer, Center laboratory staff recognized that samples with a numerical result of >15% monocytes often included large or atypical lymphocytes requiring microscopic identification. Samples with eosinophil counts >20% and basophil counts >20% were suggestive of inaccurate WBC differentials requiring verification.

2.5. Data analysis

2.5.1. Data integration

Data for this study were obtained from both the Sysmex XS‐1000i hematology analyzer and the Center's Labkey Electronic Health Records (EHR) system. Automated CBC results including numeric results, WBC differential scattergrams, Q flags, IP (Interpretive Program) flags, and Judgment Calls were obtained from the analyzer. This flag data included numeric values triggering Suspect flags at set points based on the probability of their presence (Q flags), abnormal flags based on user‐defined count criteria or abnormal scattergram appearance as well as Suspect flags based on Q flags (IP flags), and Positive or Negative categorizations of samples requiring further review based on potential abnormalities in the count, morphology, or differential (Judgment Calls). MSR results (morphology and score) and patient demographics were obtained from the EHR system. Extracted data from the hematology analyzer and the EHR system were merged using Python and Pandas (version 1.4.3), and matched (automated analyzer and EHR) records from rhesus and cynomolgus macaque samples collected during the study periods were retained. Samples with absent or incomplete differentials were identified through image analysis of the Sysmex differential scattergrams using Pillow (Python Imaging Library‐Fork, v.9.4.0) to identify scattergrams containing gray pixels (RGB[192, 192, 192] or RGB [128, 128, 128]).

2.5.2. Partition of training and test datasets

The merged dataset generated as described above was then partitioned randomly to a training dataset (75% of samples) or a test dataset (25% of samples) (Figure 2) using train_test_split from the sklearn.model_selection library. This ratio was chosen to ensure that the performance metrics of the machine learning models reflected performance on new data.

FIGURE 2.

The study population dataset (n = 9938) was divided into a machine learning (ML) training dataset (75%; n = 7453) and a test dataset (25%; n = 2485). The training dataset was further subdivided into a dataset for cross‐validation and a holdout dataset for validation of the model prior to application on the test dataset. All manual smear review criteria for identification of positive (abnormal) blood smears (expert‐derived and machine learning model‐based) were evaluated using the test dataset.

2.5.3. Performance measures

Several measures were used to assess criteria performance in predicting MSR positivity given automated hematology results. After application of the criteria to a dataset (eg, aICG criteria applied to test dataset), a confusion matrix consisting of True Positive (TP, Criteria+, MSR+), True Negative (TN, Criteria‐, MSR‐), False Positive (FP, Criteria+, MSR‐), and False Negative (FN, Criteria‐, MSR+) was calculated using sklearn.confusion_matrix. Subsequently, these values were used to calculate additional performance measures, including sensitivity, specificity, MSR rate, and balanced accuracy.

Sensitivity and specificity were calculated for each MSR criteria evaluated using the standard definitions below in Python. As a measure of efficiency, MSR rate was calculated as the number of samples triggering MSR for each set of criteria divided by the total number of samples. Balanced Accuracy (calculated as Youden's J statistic, also known as Youden's index), was calculated with scikit‐learn metrics.balanced_accuracy_score(adjusted = True). Youden's J statistic ranges from −1 to 1 with a value of 0, resulting from classifiers with no discriminative power and classifiers with perfect identification of all positive and negative samples, resulting in a value of 1.

$$\mathit{Sen}\mathrm{s}\mathit{\text{itivity}}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$$

$$\mathit{\text{Specificity}}=\frac{\mathit{TN}}{\mathit{TN}+\mathit{FP}}$$

$$\mathit{MSR}\mathit{\text{Rate}}=\frac{\mathit{TP}+\mathit{FP}}{\mathit{TP}+\mathit{FP}+\mathit{TN}+\mathit{FN}}$$

$$\mathit{\text{Balanced Accuracy}}\left(\mathit{\text{Youde}}{n}^{\prime }sJ\mathit{\text{Statistic}}\right)=\mathit{\text{Sensitivity}}+\mathit{\text{Specificity}}-1$$

2.6. Machine learning

Machine learning models were built in Python with auto‐sklearn. ²⁷ This library simplifies model creation by performing many of the data preprocessing steps that would typically be required to generate a machine learning model in Python. In addition, it implements ensemble methods, wherein multiple classifiers (eg, Random Forest, Passive Aggressive, or other machine learning classifiers) are trained on a dataset ¹³ and then combined to create a better performing model. ²⁸ This was advantageous for this study for two reasons. First, the automated hematology data used as input for the models consisted of differing data types including binary (IP messages and Judgment calls [Diff, Morph, Count]) as well as integer (Q‐Flags, absolute measurements, and percentage measurements) values. This presents a challenge because the performance or application of some models is limited by the data type. Using an ensemble method allowed for different models to be used for different input datatypes. Second, by simplifying the data preprocessing and selection of models, auto‐sklearn is more approachable to the veterinary clinical pathology laboratory than the more complex programming that would be required using Python and scikit‐learn.

The training dataset was split to create a separate holdout dataset to assess performance of the model during training (the model was not trained on this independent holdout dataset). Auto‐sklearn was then used to train models on the automated hematology analyzer parameters contained in Table S1 and the MSR results (morphology and score) from the remaining training dataset to create the ensemble model which used an adjusted balanced accuracy metric (Youden's index, described above) and was cross validated on multiple splits of the training dataset. Cross‐validation (k = 4) was used during training to improve the reliability of performance metrics used for training, ¹³ and the model was retrained on all folds at the conclusion of training. Performance of the model (metric: adjusted balanced accuracy) was assessed both before and after refitting using the holdout dataset. Additional detail of the relative weights of the leading members of the final ensemble model and the balanced accuracy scores achieved by the model are provided in Tables S2 and S3.

After training, the final machine learning model was applied in combination with the CCG criteria to the test dataset (the independent data partition created before training). If either the CCG criteria or the machine learning model criteria were met, the sample was flagged as warranting MSR. Functionally, this ensured that the machine learning model could add samples to the set warranting MSR (to improve identification of false negatives from CCG criteria) but would not remove samples that the CCG criteria flagged for review.

2.7. Statistical analyses

Means of animal ages and corresponding bootstrapped 95% confidence intervals were determined with NumPy (v1.23.2) and Scipy. stats (v.1.9.0). Statistical tests were performed in Python using Statsmodels (v0.13.2) for McNemar's Test in comparisons of the sensitivities and specificities of each MSR criteria and Scipy. stats (v1.9.0) for the Chi‐Square tests to compare MSR rates.

3. RESULTS

The study dataset comprised a total of 9938 non‐consecutive samples with automated hematology and paired MSR results from a colony of rhesus and cynomolgus macaques. 84.7% (n = 8417) of samples were from rhesus macaques with 15.3% (n = 1521) from cynomolgus macaques. 51.7% (n = 5134) of samples were male, 45.7% (n = 4543) were female, 2.0% (n = 201) were oophorectomized female, and 0.6% (n = 60) were orchiectomized male. Mean age of all macaques in the dataset was 8.7 years (95% CI 8.6–8.8, range: 0–36.8). Rhesus macaque mean age was 8.4 years (95% CI 8.3–8.6, range: 0–36.8), and cynomolgus macaque mean age was 10.3 years (95% CI 10.1–10.5, range: 0.8–23.2). Animals with a reported age of 0 represented animals sampled between the day of birth to ~18 days of age. The full dataset was randomly partitioned into a training (75%) and test (25%) dataset. A descriptive summary of the demographics of the entire study population and these partitioned datasets can be found in Table 4. The median frequency of CBC results from a single individual was two for both datasets. 7.1% (n = 177) of samples in the test dataset met at least one aspect of the positive smear definition. 26.3% (n = 664) of samples in the test dataset had absent/incomplete WBC differentials, with only 6.3% (n = 42) of these meeting at least one aspect of the definition of a positive smear.

TABLE 4.

Demographic characteristics of the study population, including the full dataset, machine learning model training dataset and manual smear review test dataset populations (N = 9938).

Characteristic	Full dataset	Training dataset	Test dataset
Total—N (%)	9938 (100)	7453 (75)	2485 (25)
Species—N (%)
Rhesus macaque	8417 (84.7)	6315 (84.7)	2102 (84.6)
Cynomolgus macaque	1521 (15.3)	1138 (15.3)	383 (15.4)
Sex—N (%)
Male	5134 (51.7)	3853 (51.7)	1281 (51.5)
Female	4543 (45.7)	3392 (45.6)	1151 (46.3)
Oophorectomized female	201 (2.0)	162 (2.2)	39 (1.6)
Orchiectomized male	60 (0.6)	46 (0.6)	14 (0.6)
Age—mean in years (95% CI)	8.7 (8.6–8.8)	8.7 (8.6–8.9)	8.7 (8.4–8.9)
Rhesus macaque	8.4 (8.3–8.6)	8.4 (8.3–8.6)	8.4 (8.2–8.7)
Cynomolgus macaque	10.3 (10.1–10.5)	10.4 (10.2–10.7)	9.9 (9.5–10.3)
Age range—years	0*‐36.8	0*‐36.8	0*‐36.5

Abbreviations: CI, confidence interval; N, number; %, percentage.

^*The study included patient samples collected as early as the day of birth.

The performance of the current laboratory MSR criteria (review all smears), the two expert‐derived criteria (aICG and CCG), and the machine learning model combined with CCG criteria (CCG + ML) is summarized in Table 5. The aICG criteria showed high sensitivity (80.8%) but low specificity (26.3%) at detection of positive smears with 74.2% of smears triggered for MSR. Compared to the aICG criteria, the CCG criteria were less sensitive (57.1%, P = 3.3 × 10⁻⁹, McNemar's test) but exhibited a superior specificity (65.5%, P = 7.1 × 10⁻¹⁹¹, McNemar's test) and much lower MSR rate (36.1%, P = 2.5 × 10⁻¹⁶⁰, Chi‐Square test). Both the aICG and CCG criteria met the target of <5% false negatives (1.4% and 3.1%, respectively) (Table 6). However, with the aICG criteria, there was a large percentage of false positives (68.5%) consistent with the high MSR rate.

TABLE 5.

Performance comparison of four manual blood smear review criteria (N = 2485).

Criteria	Sensitivity (%)	Specificity (%)	MSR rate (%)
Review all smears	100	0	100
CCG criteria	57.1	65.5	36.1
aICG criteria	80.8	26.3	74.2
CCG criteria + Machine Learning model	76.8	57.3	45.1

Abbreviations: aICG, adapted International Consensus Group for Hematology; CCG, Center Consensus Group; MSR, manual smear review; N, number; %, percentage.

TABLE 6.

Truth tables of four manual blood smear review criteria (N = 2485).

Review all smears	N (%)
True positive	177 (7.1)
False positive	2308 (92.9)
True negative	0 (0)
False negative	0 (0)

aICG Criteria	N (%)
True positive	143 (5.8)
False positive	1701 (68.5)
True negative	607 (24.4)
False negative	34 (1.4)

CCG Criteria	N (%)
True positive	101 (4.1)
False positive	796 (32.0)
True negative	1512 (60.8)
False negative	76 (3.1)

CCG Criteria + Machine Learning Model	N (%)
True positive	136 (5.5)
False positive	985 (39.6)
True negative	1323 (53.2)
False negative	41 (1.6)

Abbreviations: aICG, adapted International Consensus Group for Hematology; CCG, Center Consensus Group; N, number; %, percentage.

The application of the ensemble machine learning model to samples not selected for MSR by the CCG criteria identified 46.1% of the remaining positive smears by reviewing only 14.1% of the remaining samples. Combining the machine learning model with the CCG criteria significantly increased the sensitivity from 57.1% with the CCG criteria alone to 76.8% when combined (P = 9.1 × 10⁻⁹, McNemar's test), while bringing the MSR rate significantly lower (CCG + ML: 45.1%, aICG: 74.2%, P = 1.2 × 10⁻¹⁰, Chi‐Square) than the similarly sensitive (CCG + ML: 76.8%, aICG: 80.8%, P = .28, McNemar's test) but less specific (CCG + ML: 57.3%, aICG: 26.3%, P = 8.2 × 10⁻¹³³, McNemar's test) aICG criteria. The false negative rate was also improved, decreasing nearly in half from 3.1% to 1.6% (P = .0015, Chi‐Square test). Overall, the combined CCG + Machine Learning Model significantly improved the MSR rate from the current practice of reviewing all smears (P <3 × 10⁻¹⁶⁰, Chi‐Square test).

A complementary approach to compare the performance of competing MSR criteria was to examine the relationship of sensitivity to MSR rate (Figure 3). Using the CCG criteria as a minimum baseline of which smears must be reviewed, all three alternatives evaluated (aICG, Review All Smears, and CCG + ML) increased sensitivity. However, only CCG + ML did so without requiring an outsized increase in MSR rate. Therefore, the combination of CCG criteria with a Machine Learning Model represented an optimal MSR criteria with a sensitivity of 76.8% and a MSR rate of 45.1%.

FIGURE 3.

Sensitivity versus manual smear review (MSR) Rate for four different blood MSR criteria on the test dataset. While the Center Consensus Group (CCG) criteria established a minimum baseline of which smears must be reviewed, some positive smears remained unidentified for MSR. Sensitivity could be increased by randomly reading a fraction of the remaining slides not selected for MSR by the CCG criteria (dashed line). Only the CCG criteria with Machine Learning (CCG + ML) model increased sensitivity without an outsized increase in MSR rate.

4. DISCUSSION

MSR is a critical component of CBC analysis when monitoring the health of veterinary patients. ¹ Yet, this must be balanced with the increased workload and labor costs associated with MSR in the context of shortages of laboratory professionals. ²⁹ Based on favorable results in human laboratories in the generation of MSR criteria, we sought to determine if a similar approach would work for our laboratory and reduce smear review rates from the current practice of reviewing all blood smears as part of CBC performance. Of the two expert‐derived, traditional MSR criteria evaluated, neither criteria alone demonstrated sufficient performance for laboratory needs. Here, we demonstrated the creation and application of combined expert‐derived and machine learning criteria, which succeeded in improving the performance of expert‐derived criteria alone through the reduction of the MSR rate while maintaining appropriate sensitivity. In addition to the improvement to the MSR rate, the addition of the machine learning component to the CCG criteria identified almost half of the false negatives of the CCG criteria alone, improving the rate to 1.6%, a level well below even the more conservative recommended threshold of 3% established by Cui et al ² for patient safety, all while only increasing the overall MSR rate by 25%.

Using the aICG criteria, we were able to achieve a sensitivity comparable with that obtained in the original ICG study. ³ However, the specificity (and therefore MSR rate) was suboptimal. In general, the low specificity of all criteria evaluated was impacted by the high frequency of samples with absent/incomplete WBC differentials that were negative on MSR. This large number of false positives would incur a significant increase in workload through otherwise unnecessary MSRs. An automated hematology analyzer with customizable and/or species‐specific gating could significantly improve WBC differential generation and further minimize MSR rates by reducing MSRs in cases where the only rule met is an incomplete/absent WBC differential. Another limitation that may have impacted the performance of the aICG criteria in this study was the inability to utilize delta check and first occurrence rules due to the design of the laboratory's EHR system. It is likely that this limitation led to additional unnecessary MSRs (false positives) that could have been eliminated if these rules had been applied, as has been described previously. ⁵ This example demonstrates the need to evaluate a laboratory's infrastructure when designing MSR criteria and to identify which evaluations are compatible with the laboratory's instrumentation and electronic data management systems. Further, while the ICG criteria perform well for human patients, they may not be directly or readily translatable to non‐human patients, even for those as closely related to humans as macaques. Conversely, the CCG criteria were created with our colony and clinician requirements in mind, and while the specificity was suitable, the sensitivity was too low for laboratory acceptance. However, as these rules encompassed critical results requiring MSR in all cases, it was an excellent pairing for the machine learning model to ensure that samples meeting these rules still received the needed evaluation regardless of the presence of positive smear‐defining features that drove the training of the machine learning model. Consideration of criteria based solely on machine learning is worth exploring, but efficacy should be carefully evaluated to ensure patient safety.

In the creation and verification of MSR criteria, thoughtful and detailed effort must be placed on how a positive smear is defined, as well as each rule included in the MSR criteria to ensure that they are both specific and relevant to the patient population. Even small changes to these parameters have the potential to significantly impact the performance of the criteria, and if using published or standardized guidelines, adjustments, and optimizations are often needed to fit the rules to the specific population. ⁴ For example, in our study, relative counts (percentages) for monocytes, eosinophils, and basophils were utilized in the CCG criteria. Some laboratories may instead use other measurements, such as absolute counts, which are recommended for application when interpreting results. ³⁰ Other laboratories may choose to further divide their MSR criteria to fit various demographics (eg, species, age, sex, or breed) to better capture features specific to certain populations. Using WBC scattergrams and RBC and platelet histograms when generating MSR criteria is also a worthwhile consideration, as instrument flagging paradigms are not always accurate and can vary between analyzers. ¹¹ , ³¹ Additionally, establishing MSR criteria requires both a sufficient number of positive smears and close coordination with clinical veterinarians to determine relevant rules. The sample number requirement may impact performance if the overall sample numbers or number of positive smears is too small. Below such limits, it may be difficult or impossible for criteria to successfully meet desired performance metrics. In addition, not all parameters defining a positive smear may be present in a specific data set used for the creation and evaluation of MSR criteria, for example, the absence of hemotropic parasites identified in the samples in our study. In these cases, performance in the identification of these features cannot be assessed by the specific models created.

A unique aspect of using machine learning in the development of MSR criteria is that the machine learning model development process can be customized and optimized for the desired metrics of the laboratory. For example, the model can be designed to balance multiple metrics, focus on maximizing sensitivity at the expense of specificity, or take other metrics into account in its design. For this study, we chose to put a greater emphasis on increasing the balanced accuracy of the model, which led to a higher MSR rate than might have been possible with a more efficient model. Ultimately, the laboratory must identify if the MSR criteria will meet their needs for both patient safety and laboratory efficiency. ⁸

Large sample numbers are a limitation in designing machine learning‐based MSR criteria. For the ensemble method used in this specific study, thousands of patient results were required for model training. To ensure the model would be generalizable to other datasets, a large training dataset, an independent holdout dataset, and robust testing of the models within the training dataset were needed. With the large number of unfiltered patient results used in this study, some patients had multiple samples included in the study dataset. In principle, this may result in overfitting of the model to the features of the overrepresented patients in the dataset. While removing repeated individuals might have, therefore, led to improved performance of the model, the median frequency of repeated individuals was low, and the use of the full dataset allows the model to more accurately reflect the clinical reality wherein ill animals often have frequent and recurrent testing. In cases where the available datasets are too small for the approach used in this study, alternative machine learning models may be needed. An additional consideration is that ensemble models such as the one we developed for this study, while straightforward for a computer program to implement, are typically low in interpretability (human readability of underlying logic). This raises two potential concerns: it may not be permissible under some regulatory structures to use ensemble models due to their opaque nature, and such models may be subject to systematic error over time because the distributions of underlying triggering parameters change over time. It may, therefore, be advantageous to randomly select and perform MSR on a small percentage of samples otherwise marked as negative by MSR criteria as part of laboratory quality assurance practices. The performance of the model on these samples could then be monitored over time and changes made (up to and including the generation of a new machine learning model) if warranted. Alternatively, creation and evaluation of human‐interpretable machine learning models for MSR could be a beneficial future undertaking and has the potential to expand understanding regarding which parameters from the automated hematology analyzer are most valuable in identifying patient samples requiring MSR and inform laboratory decision making regarding CBC analysis including adjustments to user defined settings and selection of features needed in an instrument.

We created and evaluated two sets of traditional MSR criteria as well as innovative machine learning‐augmented MSR criteria. In doing so, we successfully demonstrated that machine learning can be used to improve the MSR rate while offering comparable sensitivity to traditional methods. Machine learning should be further explored as a way for laboratories to optimize their MSR criteria to decrease labor efforts required for routine performance of CBCs. If machine learning methods are not feasible for an individual laboratory, an exploration of traditional MSR criteria with an evaluation of performance metrics is a worthwhile pursuit.

CONFLICT OF INTEREST STATEMENT

The authors have indicated that they have no affiliations or financial involvement with any organization or entity with a financial interest in, or in financial competition with, the subject matter or materials discussed in this article.

Supporting information

Data S1.

Hayes JM, Hayes MR, Friedrichs KR, Simmons HA. Development of criteria to optimize manual smear review of automated complete blood counts using a machine learning model. Vet Clin Pathol. 2025;54(Suppl. 2):S95‐S104. doi: 10.1111/vcp.13400