Use of remote radiology support for training and quality assurance in the “focused assessment with sonography for HIV-associated tuberculosis:” A pilot program in Malawi

DESCRIPTION OF THE PROBLEM:

The diagnosis of tuberculosis (TB) in HIV-infected patients can be challenging in resource-limited settings. Extrapulmonary TB is particularly under-diagnosed, with data suggesting that half of patients with disseminated TB are not diagnosed until autopsy.¹

Ultrasound represents a promising diagnostic modality because of its low cost, widespread availability, and minimal risks to the patient. Ultrasound can be used to identify pericardial effusions, pleural effusions, ascites, abdominal lymph nodes, and hepatic or splenic lesions, which all have high specificity for extrapulmonary TB in the developing world.^2–7 Heller et al. developed a “focused assessment with sonography for HIV-associated TB” (FASH) protocol to evaluate for these findings.⁸ They were able to demonstrate the feasibility of a short two-day course to train clinicians in the FASH exam, though long-term follow-up and sonographer development over time were not recorded.⁹

Teleultrasonography, where a geographically removed expert electronically receives and interprets acquired ultrasound images, has been shown to be clinically feasible and useful for operator training in a variety of settings.¹⁰ We performed a prospective observational study of teleultrasonography and hypothesized that remote expert radiology support would improve sonographer technique and interpretation in the FASH exam through the use of real-time quality feedback on image acquisition and interpretation.

WHAT WE DID:

Malawian physicians, clinical officers, radiographers, and medical assistants were enrolled in a four-day FASH training course at three sites in the Central region of Malawi: one public-private medical center (Partners in Hope), one district hospital (Kasungu), and one mission hospital (Madisi). The training sessions consisted of groups of four to five participants, and three groups were trained each week. The course was taught by two physicians from the University of California, Los Angeles (UCLA) who had received formal training in the FASH protocol.

Following the training, the participants performed FASH exams on local “control” subjects who were HIV-positive but had no signs or symptoms of TB as well as “case” subjects who were HIV-positive and had signs or symptoms of TB. Enrollment of control subjects began in February 2016 and concluded in March 2016 at each of the three sites. Enrollment of case subjects began in March 2016 and concluded in April 2017. Enrollment of case subjects was limited to the Partners in Hope clinical site.

For each FASH exam, the participant acquired 9 images corresponding to each position in the protocol.⁸ All images were then sent to a remote United States board-certified radiologist with expertise in ultrasonography. Each image was de-identified and coded with a unique identification number and sent via an encrypted dropbox.

The expert radiologist first evaluated the images on the technical variables of gain, depth, focus, labeling, and usable images (to assess for technical competency). Each of the images per scan was given 1 point for each correct technical variable. Every scan could thus earn up to 9 points per technical variable (as there were 9 images per scan). For example, if 7 of the 9 images in a scan had the correct depth, this would count as 7 out of 9 points for “correct depth” for that scan. “Usable images” was defined by the ability of the expert reader to determine the interpretation variable on the image provided. For example, if the image of the liver was obtained with the incorrect depth but was still interpretable, this image was given 1 point for the technical variable “usable image” but 0 points for the technical variable “correct depth.” Trainees were also given written feedback on the quality of their images within 48 hours of each exam.

Both the participant and the expert radiologist completed the interpretation portion of the exam (to assess for interpretation competency). Interpretation variables including pericardial effusion, right pleural effusion, left pleural effusion, ascites, peri-portal lymphadenopathy, para-aortic lymphadenopathy, liver lesions, splenic lesions, and other abnormalities were evaluated for each exam. These were given a score of “0” if the finding was not seen, “1” if the finding was seen, or “not applicable” if the finding could not be evaluated from the images provided. Agreement and disagreement between the participant and the expert radiologist, or “standard of reference” (SOR), was evaluated. Feedback on the interpretation part of the exam was also provided to the trainees within 48 hours of each exam.

Informed consent for this study was signed by all clinician participants as well as case and control subjects. The study protocol was approved by the Malawi National Health Sciences Research Committee and the Institutional Review Board (IRB) at UCLA.

OUTCOMES:

Eleven participants were trained and performed scans on a total of 183 patients (Table 1). Seven were clinical officers, 2 were physicians, 1 was a radiographer, and 1 was a medical assistant. None of the trainees reported previous experience using ultrasound to diagnose TB. Seventy-five of the scans were on controls (25 at each of the 3 sites) and 108 were on cases.

Table 1:

Clinic Location and Number of Scans Performed Per Participant

ID	Title	Location of Scan			Total
		Kas*	Madisi†	PIH‡
ID1	Radiographer	0	0	1	1
ID2	Clinical Officer	7	0	1	8
ID3	Physician	10	0	1	11
ID4	Clinical Officer	5	0	1	6
ID5	Medical Assistant	3	0	0	0
ID6	Physician	0	6	0	6
ID7	Clinical Officer	0	19	0	19
ID8	Clinical Officer	0	0	2	2
ID9	Clinical Officer	0	0	50	50
ID10	Clinical Officer	0	0	38	38
ID11	Clinical Officer	0	0	39	39
Total		25	25	133	183

^*Kas = Kasungu, district hospital

^†Madisi = Madisi, mission hospital

^‡PIH = Partners in Hope, public-private medical center

Technical Competency:

Graphs showing the progression of each correct technical variable over time, including case and control scans for all participants at all sites, are shown in Figure 1. There was an overall improvement in correct depth, correct focus, correct labeling and number of usable images over time throughout the study. While depth and focus represented the most challenging variables initially, there was a similar slope of improvement between these two graphs over time. There was a slight decrease in correct gain over time throughout the study. Analysis of the data revealed that one individual clinician was responsible for this decrease over time, as he had a few outlier images where he set the gain too high. After real-time feedback via teleultrasonography, he corrected his gain on subsequent images.

Figure 1:

Graphs of Depth, Focus, Label, Gain, and Usable Images over time.

We performed a chi-squared analysis to detect a percentage difference in depth, focus, gain, labeling, and usable images over time. As the PIH site was the only site to enroll patients over the entire study period, this analysis was limited to the PIH site; in addition, given that the control patients were scanned first for all operators, we performed this analysis by comparing the control patients (i.e., the first 25 exams) to all of the following case patients. Of these first 25 scans (“group 1”), 64% demonstrated the correct depth, 64% demonstrated the correct focus, 89% demonstrated the correct gain, and 91% demonstrated correct labeling of the image (Table 2). In contrast, for scans performed later on (“group 2,” the 108 case scans), 87% demonstrated the correct depth, 87% demonstrated the correct focus, 88% demonstrated the correct gain, and 96% demonstrated correct labeling of the image. For the variables of depth, focus and labeling, this thus yielded a statistically significant increase in the percentage of technically competent scans between group 1 and group 2 (p<0.001). There was also a statistically significant difference in “usable images” when the group 1 scans were compared to the group 2 scans (74% vs 96% usable images, respectively, p<0.001).

Table 2:

Technical Competency of Initial Scans versus Subsequent Scans for Participants

	All PIH Scans (% correct)			ID9 (% correct)			ID10 (% correct)			ID11 (% correct)
	Group 1*	Group 2*	p‡	Group 1†	Group 2†	p‡	Group 1†	Group 2†	p‡	Group 1†	Group 2†	p‡
Depth	64	87	<0.001	62	95	<0.001	92	95	0.284	49	74	<0.001
Focus	64	87	<0.001	62	95	<0.001	92	95	0.284	49	76	<.0001
Gain	89	88	0.762	99	84	<0.001	100	77	<.001	91	97	0.012
Labeling	91	96	<0.001	89	98	<0.001	98	96	0.440	89 74	96	0.010
Usable Images	74	96	<0.001	78	96	<0.001	96	98	0.215		95	<0.001

^*Group 1 = the initial 25 scans (on control patients); Group 2 = the subsequent 108 scans (on case patients)

^†Group 1 = the initial 10 scans performed by each individual; Group 2 = the subsequent scans performed by each individual (40 for ID9, 28 for ID10, 29 for ID11)

^‡p-value determined with STATA using a chi-squared test

Three clinicians performed more than 20 exams and were thus included in the individual analysis over time (Table 2). For this, we performed a chi-squared analysis by comparing each individual’s first 10 chronologically-performed scans (after completion of the hands-on training course) with all subsequent examinations. ID9 had 10 years of clinical experience, ID10 had 9 years of clinical experience, and ID11 had 8 years of clinical experience; none of these individuals had prior experience or training in ultrasound. ID9 scanned a total of 50 subjects and had a statistically significant increase in the percent of usable images between the first 10 and the subsequent 40 exams, improving from 78% to 96% (p<0.001). ID11 scanned a total of 39 subjects and also had a statistically significant increase in the percent of usable images between the first 10 and subsequent 29 exams of 74% to 95% (p<0.001). ID10 scanned a total of 38 subjects and had no significant change in percent of usable images between the first 10 exams (96%) and subsequent 28 exams (98%), as this clinician performed well at baseline (p=0.125). Importantly, all three were able to obtain over 95% usable images after only 10 patient scans, suggesting not only a steep learning curve for ultrasonography, but also the teachability of the skill.

Interpretation Competency:

The participant’s interpretation of the ultrasound findings was compared to the SOR’s interpretation in 181 of the 183 exams, as 2 had missing data. Given that each exam included 9 separate images, this allowed for 1629 comparisons between the clinician and the SOR. The clinicians identified 96 (6%) of these images as “abnormal” while the SOR coded 85 (5%) as “abnormal,” revealing an overall agreement of 98% between the clinicians and the SOR.

Analysis of the prevalence and level of participant-SOR agreement between each of the 9 interpretation variables was performed on the 108 case subjects at PIH (Table 3). The learners’ sensitivity, specificity, false negative rate, false positive rate, positive predictive value, and negative predictive value are also listed in Table 3. Overall, pericardial effusions were the most prevalent abnormality, with the SOR identifying them in 30 (28%) of the case subjects. There was only one case of a false-positive effusion (i.e., where the participant erroneously identified a pericardial effusion as present while the SOR evaluated it as absent), resulting in an agreement rate of 99%. Peri-portal and para-aortic lymphadenopathy were the next most prevalent abnormalities, present in 11 (10%) and 9 (8%) cases, respectively. Both had a participant-SOR agreement rate of 98%, with the participants missing 2 cases of peri-portal lymphadenopathy and 1 case of para-aortic lymphadenopathy. Pleural effusions, which were present in 7 (6%) of cases, were over-diagnosed by trainees in 5 cases and missed by them in 1 case. Ascites, liver lesions, and splenic lesions, though relatively uncommon (being present in 4%, 3%, and 0% of cases, respectively), all yielded participant-SOR agreement levels of 99%. Finally, the SOR detected 9 “other abnormalities,” whereas the trainees documented 14, yielding an agreement rate of 94%. Examples of these abnormalities seen by both the clinicians and the expert radiologist included splenomegaly, hepatomegaly, hepatic congestion, a liver hemangioma, old granulomatous disease in the spleen and/or liver, an ovarian cyst, a dermoid pelvic mass, a fibroid uterus, cystitis, mild hydronephrosis, hydroureter, and echogenic kidneys consistent with medical renal disease.

Table 3:

Prevalence of Abnormalities and Interpretation Competency of Participants for Case Scans

	Prevalence*	Sensitivity†	Specificity†	PPV†	NPV†	Participant-SOR % Agreement
Pericardial Effusion	30/108 (27.8%)	100%	98.7%	96.8%	100%	99.1%
Periportal LAD	11/108 (10.2%)	81.8%	100%	100%	98.0%	98.1%
Para-aortic LAD	9/108 (8.3%)	88.9%	99.0%	88.9%	99.0%	98.1%
L pleural effusion	5/108 (4.6%)	100%	99.0%	83.3%	100%	99.1%
R pleural effusion	2/108 (1.9%)	50%	96.2%	20%	99.0%	95.4%
Ascites	4/108 (3.7%)	100%	99%	80%	100%	99.1%
Liver Lesions	3/108 (2.8%)	100%	99%	75%	100%	99.1%
Splenic Lesions	0/108 (0%)	NA	99.1%	NA	100%	99.1%
Other Abnormalities	9/108 (8.3%)	88.9%	93.9%	57.1%	98.9%	93.5%
Any Abnormality	73/972 (7.5%)	93.2%	98.2%	81.0%	99.4%	97.8%

^*Prevalence was based on SOR interpretations

^†= of participant scan interpretations (as compared to the SOR)

LAD = lymphadenopathy, L = left, R = right, SOR = standard of reference

Conclusion

Our prospective observational study suggests that both physician and non-physician clinicians can learn the technical skills of ultrasound image acquisition after a short four-day training course followed by focused teleultrasonography feedback. Importantly, our protocol also led to proficiency in ultrasound interpretation, with the clinicians identifying 92% of abnormalities seen by the expert reader. As is evidenced by the rapid separation between the first 10 scans and the later scans, teleultrasonography appeared most useful during the initial learning period. Its value in later scans likely lies in quality assurance, by ensuring that learners sustain technical skill and also interpret rarer findings correctly. This study thus not only has the potential to improve the diagnosis of extrapulmonary tuberculosis in the global health setting, but also to inspire the creation of future protocols using teleultrasonography to harness radiology expertise for training in resource limited settings.