Development and validation of a peripheral blood smear assessment tool using cloud-based full-field digital microscopy

Abstract

Peripheral blood smear review is a fundamental skill that aids in the timely diagnosis of morbid conditions in hematology. There is a lack of existing, evidence-based assessment tools to evaluate learners and guide curricular development both in peripheral blood smear review and analogous visuospatial skills across medical specialties. We collected validity evidence for a novel assessment tool for use in hematology education. We recruited attending hematologists, hematology fellows, and internal medicine residents to participate in the development of this assessment tool. The attending cohort participated in iterative revision of a digital microscopy–based assessment tool encompassing classical and malignant hematology. The validated tool then compared performance across attending, clinical fellow, and resident cohorts. Attending accuracy was poor with peripheral blood smear images alone and improved significantly with the addition of clinical context. Performance increased across training levels with attending and fellow performance significantly outpacing resident performance. Characteristics of morphologic descriptions varied significantly across cohorts. Digital microscopy image quality was considered to be equivalent to light microscopy with minor limitations. This tool offers hematology and pathology educators a validated method to assess learners and evaluate microscopy-based curricular developments. Further investment is required to broaden applications within hematology and pathology education.

Introduction

Mastery of visuospatial tasks is required in many clinical disciplines such as skin examination (dermatology), urine sediment analysis (nephrology), echocardiography (cardiology), radiographic interpretation (radiology), and peripheral blood smear (PBS) review (hematology). Peripheral blood smear review requires accurate identification of cellular morphology, which in turn, is dependent on proper identification of cellular characteristics including size, shape, contour, nuclear and cytoplasmic hue, and granularity among others.^1,2 Expertise in PBS evaluation is essential in the rapid identification of, and early treatment initiation in, some of the most morbid conditions within hematology, including thrombotic microangiopathies (TMAs)³ and acute leukemias (AL).⁴ As such, the Accreditation Council for Graduate Medical Education requires that graduates of hematology fellowship demonstrate competence in PBS review.⁵ Classically, this skill acquisition has occurred through an apprentice model at the compound light microscope within the context of routine patient care. However, this approach may be limited by location-specific factors, including access to microscopes or expert hematopathologists, time to review slides, or an adequate breadth of available blood smears.

Digital microscopy (DM) offers several potential advantages to learners and may foster the democratization of hematopathology education, offering access to cases and expert educators that may otherwise be unavailable to the learner. This can occur through a standard browser, thus alleviating the need for on-site hardware or content experts.⁶ Importantly, in trained hands, concordance between DM and light microscopy (LM) appears preserved, although prior studies have largely focused on surgical pathology applications.⁷ Until recently, inno vation in digital PBS review and education has progressed at a modest pace, given technical challenges with imaging at 100× magnification. The U.S. Food and Drug Administration has now cleared several devices for use in PBS review.^8,9 Despite the availability of commercial PBS products,¹⁰ open-source image banks,¹¹ and PBS review courses,¹² there remains a lack of validated assessment tools to evaluate learners and curricula.

Currently, hematology fellows do not complete any competency assessment or formal evaluation at the program level, or through the American Board of Internal Medicine board certification examination. The lack of validated assessment tools is a major obstacle in the accurate evaluation of fellow skill acquisition and impedes curricular development. We aimed to create and examine validity evidence for a DM-PBS assessment tool capable of discriminating between learners at various training levels.

Materials and methods

The purpose of this study was to examine validity evidence for a DM-PBS assessment tool designed to evaluate microscopy-dependent diagnostic skill acquisition using Kane's validity model. All four elements of Kane's model were assessed in this study. Scoring, which describes how the observation results in a reproducible and quantifiable result, was reflected in the use of an objective rubric. Generalization, a measure of reliability and how the sampled items reflect the whole performance, was assessed using 10 distinct cases that reflect the range of skills determined essential to PBS interpretation.² Extrapolation, the correlation between testing and real-world performance, was assessed by using real-world blood smears generated by routine practice requests rather than simulated or prototypical examples and by providing entire slides for review rather than hand-selected images. The final stage of inference, implications, is the interpretation of, and subsequent action about, the results. This was assessed by assessing the correlation of scores between level of training and years of experience.

The initial study design was proposed by the authors and granted an educational exemption by the Beth Israel Deaconess Medical Center (BIDMC) Institutional Review Board (protocol 2022P000487). It was conducted between September 1, 2021, and October 31, 2023. The list of morphologic findings and diagnostic cases were derived from consensus recommendations for fellow competency in PBS review, which were developed by leaders in hematology education in an iterative and consensus-driven fashion using nominal group technique and recently published.² Convenient and representative cases were collected through clinical encounters on the inpatient consultation service and in outpatient hematology clinics at BIDMC (Boston, MA) between December 2020 and May of 2022 (Table 1). PBS slides were generated as part of standard clinical practice using Wright-Giemsa staining. High-resolution (optical objective equivalent 100×) full-field images were generated using an X100 with Full-Field PBS application instrument (Scopio Labs, Tel Aviv, Israel) and uploaded to a secure cloud-based scan gallery.⁹ Hyperlinks to the gallery were accessible via standard internet browsers (Supplemental Table 1).

Table 1.

Peripheral blood smear images—ground truth diagnoses. Cases used for assessment tool versions 1 and 2. Ground truth diagnosis and pertinent morphological characteristics are for each.

Ground truth diagnosis	Pertinent morphology
Acute myeloid leukemia	Blasts (typical appearance, high WBC count)
Reactive leukocytosis	Bands, leukocytosis, absence of more immature myeloid precursors
Chronic myelogenous leukemia (CML-accelerated phase)	myelocytes, metamyelocytes, bands, leukocytosis, eosinophilia, basophilia
Thrombotic microangiopathy	schistocytes, thrombocytopenia
Warm auto-immune hemolytic anemia	spherocytes, minimal other poikilocytes
Acute lymphoblastic leukemia	Blasts (atypical appearance, low WBC count)
Sickle cell anemia	sickle cells
Iron deficiency anemia	microcytes, hypochromic RBCs, elliptocytes, “pencils,” “cigars”
Myelophthisic anemia (myeloma)	dacrocytes, nucleated RBCs
Immune thrombocytopenia	thrombocytopenia, otherwise normal morphology

RBC: red blood cell; WBC: white blood cell.

Participants were recruited to three cohorts—attending hematologists, clinical fellows in hematology, and internal medicine residents. Attending candidates were required to have both extensive clinical experience in the use of PBS and responsibility in the education of hematology fellows. Publicly available lists of hematology faculty at the BIDMC and the American Society of Hematology medical educator institute were used to identify prospective participants. Clinical fellows in the combined hematology and oncology training program and internal medicine residents with an interest in hematology at BIDMC were invited to participate in the fellow and internal medicine cohorts.

All participants participated in a software demonstration using the web-based PBS viewing platform and completed a demographic survey at the time of enrollment (Table 2). A PBS assessment instrument was constructed from a list of morphologies and diagnoses required for fellow competency upon graduation.² Participants were instructed to list a single most likely diagnosis, other diagnoses under consideration, and morphology seen. An established clinical diagnosis made through routine care was established as the ground truth condition for each case. A “correct” answer was scored if the single most likely diagnosis correctly identified the ground truth condition.

Table 2.

Demographics and self-assessed competency.

	Attending	Fellow	Resident
	n = 9	n = 12	n = 9
Age, years (range)	50.9 (37–73)	31.1 (29–34)	28.2 (27–30)
Female gender, n (%)	1 (11)	3 (25)	5 (56)
Post-graduate year, mean (range)	–	5.1 (4–6)	2.0 (1–3)
Years post fellowship, mean (range)	17.6 (2–40)	–	–
Number of PBS reviewed annually, mean (range)	155.6 (50–400)	30.3 (8–75)	6.4 (1–15)
Received formal training on PBS review, n (%)	–	9 (75)	2 (22)
Responsible for PBS review with fellows, n, (%)	9 (100)	–	–
Self-competency questions, I feel confident in my ability to …, mean (SD)
identify common pathologic morphology on peripheral blood smear	4.4 (0.49)	3.75 (0.43)	2.4 (0.96)
Use PBS morphology to assist in clinical evaluation of benign vs malignant conditions	4.2 (0.42)	3.50 (0.50)	1.9 (0.57)
suspect the diagnosis of thrombotic microangiopathy based on my morphologic evaluation on PBS	4.8 (0.41)	4.17 (0.55)	2.6 (0.94)
suspect the diagnosis of acute leukemia based on my morphologic evaluation of potential blasts on PBS	4.4 (0.49)	4.08 (0.49)	2.1 (0.31)

The results of a baseline demographic survey and self-assessed competency in peripheral blood smear review. Participants answered on a 5-point Likert scale (1, strongly disagree; 2, disagree; 3, neither agree nor disagree; 4, agree; 5, strongly agree).

PBS: peripheral blood smear.

The initial phase of the study was designed to evaluate the ability of the tool to assess attending competency and evaluate interrater reliability. A 75% accuracy threshold was specified a priori based on prior DM data with limited sample sets.¹³

Assessment tool version 1 included only a digital PBS image (Supplemental Table 1). No clinical context was included. The review of images was conducted asynchronously and independently by attending hematologists without a time limit. Version 1 was rejected, and a subsequent revision was required due to attending accuracy failing to meet the prespecified threshold of 75%. Qualitative feedback was solicited from participants between revisions.

A revised version 2 (Supplemental Figs. 1–10) of the tool included a brief clinical vignette with pertinent information (focused history, vital signs at presentation, and commonly available laboratories). These data were extracted from the patient chart in question and linked to the PBS slide from the date of slide preparation. PBS images used in version 1 were randomly organized and reused in version 2 of the tool. Attending participants were blinded to this. A minimum washout period of 60 days was implemented to decrease the likelihood of biasing. Clinical fellows and internal medicine residents were invited to participate after the 75% accuracy threshold was reached in the attending cohort using version 2.

Statistical analysis was completed using SAS Statistical Analysis System version 9.4 (SAS Institute, Cary, NC). Attending performance on versions 1 and 2 was compared using Student's paired t-test. A three-way analysis of variance (ANOVA) with pairwise evaluation was used to assess performance on version 2 across the three cohorts of participants and to explore variability in morphologic description across cohorts.

Results

Ten slides were identified for use in this study. A total of 40 prospective participants were invited. Thirty individuals (75%) participated in the study, including nine attending hematologists, 12 clinical hematology fellows, and nine internal medicine residents (Table 2). The reasons for declining participation included lack of time, current inpatient service responsibilities, and parental leave. Four prospective participants (one attending and three medicine residents) did not respond to the invitation or follow-up messaging. The response rate among the 30 active participants was 100%.

Attending hematologists participated in the iterative revision of the assessment tool. The prespecified score threshold of 75% was not met using version 1 (mean score 50%, range 30%–70%). The addition of a clinical vignette and routinely available clinical data significantly improved participant scores by an average of 28% (P < 0.0001, mean score 78.9%, range 60%–90%; Fig. 1). All participants scored higher on version 2 compared to version 1. Similar improvements were seen in the number of participants identifying the correct diagnosis within a broader differential diagnosis. Thirty-four incorrect answers on version 1 were correctly answered on version 2 (Fig. 2). There were only eight instances where attendings switched from a correct primary diagnosis on version 1 to an incorrect diagnosis on version 2 acute leukemia/acute lymphoblastic leukemia (AL(ALL)), 4; acute leukemia/acute myeloid leukemia (AL(AML)), 2; chronic myeloid leukemia (CML), 1; reactive leukocytosis (RL), 1). Seven of these occurred on slides displaying malignant pathology, and in all these cases, the reported differential diagnosis included other malignant conditions that would necessitate more advanced testing likely to identify the accurate diagnosis in question. There were no significant differences in the overall and relative frequencies of morphologic reporting by attendings between versions.

Fig. 1.

Relative assessment performance by cohort. Attending hematologists showed significantly improved performance from version 1 (A1) to version 2 (A2) (∗P < 0.0001). Overall performance differed significantly across residents (R), fellows (F), and attendings (A2) on version 2 (^†ANOVA, P < 0.0001).

Fig. 2.

Attending answer switches between version 1 and version 2 by peripheral blood smear slide. Thirty-four incorrect-to-correct answer switches and eight correct-to-incorrect answer switches were observed. AIHA: auto-immune hemolytic anemia; AL (ALL): acute leukemia/acute lymphoblastic leukemia; AL (AML): acute leukemia/acute myeloid leukemia; CML: chronic myeloid leukemia; IDA: iron deficiency anemia; ITP: immune thrombocytopenia; MPTH: myelophthisic anemia; RL: reactive leukocytosis; SCD: sickle cell disease; and TMA: thrombotic microangiopathy.

Overall performance increased significantly (ANOVA P < 0.0001) across the cohorts by training level (Fig. 1). Attendings scored highest (78.9%), followed by clinical fellows (63.3%) and residents (45.6%). Pairwise associations were adjusted for multiple hypothesis testing using the Bonferroni method and revealed significant score differences between attending and resident cohorts of 33.3% (confidence interval [CI] 14.8–51.8, P = 0.003) and 17.8% (CI 0.5–35.1, P = 0.04) between fellow and resident cohorts. Pairwise comparison between attending and fellow cohorts did not reach statistical significance (15.6%, CI −1.7 to 32.9, P = 0.089).

Clinical context impacted diagnostic accuracy meaningfully. Among attendings, 34 incorrect answers on version 1 were correctly answered on version 2, while only eight responses changed from a correct to incorrect primary diagnosis. There were no significant differences in the overall and relative frequencies of morphologic reporting by attendings between versions (Supplemental Fig. 11). Exploratory analysis was conducted to evaluate the performance on individual slides and to assess the reporting of PBS morphology. These comparisons were not adjusted for multiple hypothesis testing. The attending cohort scored highest on all but two slides (auto-immune hemolytic anemia and reactive leukocytosis). Significant differences between cohorts were noted for three slides: AL (ALL), CML, and iron deficiency anemia (Fig. 3).

Fig. 3.

Cohort performance by slide on assessment tool version 2. Performance differed significantly across residents (R), fellows (F), and attending hematologists (A2) in the evaluation of acute leukemia/acute lymphoblastic leukemia (∗P = 0.02), chronic myeloid leukemia (^†P = 0.007), and iron deficiency anemia (‡P = 0.01). A indicates attending hematologists, version 2; F, fellows; and R, residents. AIHA: auto-immune hemolytic anemia; AL (ALL): acute leukemia/acute lymphoblastic leukemia; AL (AML): acute leukemia/acute myeloid leukemia; CML: chronic myeloid leukemia; IDA: iron deficiency anemia; ITP: immune thrombocytopenia; MPTH: myelophthisic anemia; RL: reactive leukocytosis; SCD: sickle cell disease; and TMA: thrombotic microangiopathy.

The depth and expansiveness of morphologic description are described in Supplemental Table 2. A wide variation in the number of morphologic descriptors was seen across participants and slides. Red blood cell morphology and white blood cell morphology were most commonly described. Within version 2, significant variation was found between cohorts in the number of morphologic terms reported between attendings and fellows overall and for the most common red blood cell, white blood cell, RBC, and platelet morphology. This variation was driven by interaction between attending-resident cohorts and fellow-resident cohorts. There was a non-significant numerical increase in morphologic reporting between fellow and attending cohorts.

Discussion

Our study provides validity evidence for the application of this PBS assessment tool in clinical hematology training. To our knowledge, this represents the first attempted validation of a tool of this kind. These data describe several differences that appear correlated with level of training. As expected, they suggest a “dose-dependent” relationship where the ability to accurately identify prospective diagnosis by PBS review is strongly associated with training level.

These data also help contextualize the pattern of PBS skill acquisition from medical resident, through fellowship, and into practice as an attending hematologist. The difference in number and type of morphologic features identified by residents compared to fellow or attending cohorts may indicate a shift in the methodology of PBS review that only occurs once a trainee enters fellowship. Trainees appear to broaden their descriptions of morphologic findings through fellowship training to a level that approaches that of attending hematologists. This observation, along with the large number of first-year fellows, may indicate that PBS skill acquisition occurs rapidly early on in fellowship training.

Importantly, there was no significant association between performance and the proportion of practice between classical hematologists (77.5%) and those practicing classical and malignant hematology (80.0%) despite varied practice across attendings (P = 0.7156). Attending hematologists attained high, yet not perfect, scores, suggesting a reasonable level of assessment difficulty.

Attending performance on the AL items was lower than expected. The single-best-answer assessment may underestimate attending recognition of AL. All attendings selected an alternative malignant diagnosis as their top answer on version 2. All still included AL in the differential, suggesting they would likely have pursued the correct diagnostic workup in practice. Several factors may explain the incorrect responses: the performative nature of testing may have prompted participants to favor rarer diagnoses, some may have avoided choosing “acute leukemia” twice in a 10-item assessment, and the intentionally challenging ALL case—with relatively unremarkable laboratory findings—likely made AL less apparent.

We suggest that this assessment tool is a broad assessment of skill in general PBS interpretation that is directly applicable to hematopathology and other areas of pathology education, reflecting the role of PBS interpretation as a foundational skill within clinical pathology. This work has multiple potential implications for Graduate Medical Education (GME) education. Foremost, this is an open-source tool with validity evidence that offers hematology educators an opportunity to both evaluate their trainees, curricula, and future innovations.

There is general agreement that hematology trainees should be educated in the use of DM and limitations thereof.² These data broaden the applications of DM and suggest it as a potential tool to improve delivery of PBS education, perhaps with the most outsized implications for international medical education, and for smaller or less resourced training programs.

The essential value of clinical context in PBS review could have implications for PBS review workflows in clinical practice. An unexpectedly large variation in attending performance on version 1 and version 2 demonstrates the importance of clinical context to interpretation. This has significant implications for the use of both academic and commercial hematopathology review services. The clinical practices of graduates diverge greatly following fellowship, with a majority of graduates from combined training programs focusing on oncology.¹⁴ Hematologists with fewer resources or experience may seek local or send out hematopathology review. Clinical data for a patient in question may or may not be available to the third-party reviewer. This practice of seeking a secondary review may not necessarily improve the accuracy of diagnosis if the clinical context is not provided. Importantly, in our research, the number of morphologic descriptors did not vary between versions 1 and 2, indicating that attendings were equally descriptive in their PBS interpretation but assigned variable weight to the identified morphologic features based on new data provided in version 2. This again supports the clinical context as an important variable in PBS interpretation. To improve accuracy, community providers should both provide clinical information at the time of review and understand there may be a higher than expected error rate by an expert reviewer.

More broadly, the development of this assessment tool and subsequent insights may offer a generalizable roadmap for similar, rigorous investigations of visuospatial skill assessment within subspecialties. The acquisition of the skill of PBS review is not inherently unique to hematology. The most analogous application would be pathology, where the interpretation of imagery, including subtle analysis of hue, shapes, contours, saturation, and spatial relationships between objects, is needed, although similar visuospatial skills are required across many specialties. These methods may be useful in assessment development for echocardiography, radiology, dermatology, fundoscopic examination, and urine sediment analysis, among others.

We must acknowledge several limitations within this work. Our sample size was small, with most participants from a small number of institutions from one geographic region. The assessment tool is modest in size – only using 10 slides. We attempted to capture a broad spectrum of classical and malignant hematology while understanding that educators have generally placed outsized importance on the identification of AL and TMA.² The tool does not fully account for the breadth of the field and likely underemphasizes illnesses with unique geographic distributions, such as malaria and T cell leukemias and lymphomas.

Broad use of this tool is limited to those who have access to these hardware and software resources. Complimentary scanner access, technical support, and cloud-based access were provided by Scopio Labs (Tel Aviv, Israel). Currently, all commercially available DM products require individual user licenses that carry expenses that outstrip standard educational budgets in many training institutions.

There are potential limitations of DM. Eight of the nine attending hematologists felt image quality was equal to or better than standard compound LM. However, the inability to focus the depth of field was a limitation reported by multiple experts. This has specific importance for the differential diagnosis of intracellular inclusions compared to smear preparation artifact.

In summary, we developed an assessment tool for PBS interpretation and collected validity evidence for its use in assessment of fellows in hematology/oncology. It could be further extrapolated to use the tool to assess the adequacy of curricula in GME hematology education. Further work is required to collect validity data in other educational settings and to establish a pass/fail threshold for use in formal competency assessment.

Author contributions

M.L.C. designed and conducted the research, wrote, and revised the article. K.Y.Y. conducted data review, wrote, and revised the article. J.A.F, designed the protocol and revised the article.

Funding

This study received no external funding.

Declaration of competing interest

All authors certify that they have no financial or non-financial conflicts of interest relevant to the subject matter of this article. Scopio Labs provided access to their platform for research purposes only and had no role in study design, data analysis, interpretation, or article preparation. No author has received personal financial compensation related to this work.

Supplementary data

The following are the Supplementary data to this article.