A pilot training program for point-of-care ultrasound in Kenya

Abstract

Introduction

Ultrasound is an effective and affordable clinical diagnostic tool highly attractive for use in low and middle income countries (LMICs), but access to training programs in these countries is limited. The objective of our study was to develop and pilot a program for healthcare providers in Kenya in the use of point-of-care ultrasound.

Methods

Trainees were recruited in district hospitals for participation in three all-day workshops held every 3–5 months from September 2013 through November 2014. Prior to the initial workshop, trainees were asked to study a training manual, and a knowledge test was administered. Ultrasound-credentialed emergency physicians provided brief didactic lessons then hands-on training for eFAST and obstetric training. This was followed by an observed assessment of scanning image quality (IM) and diagnostic interpretation (IN).

Results

Eighty-one trainees enrolled in four initial training sessions and 30 attended at least one refresher session. Among those trainees who attended refresher sessions, there was an increase in the proportion passing both the knowledge and practical tests at the follow-up, as compared to the initial session. Overall, mean practical skill scores also trended toward an increase over time, with a significantly higher mean score in November 2014 (2.64 + 0.38, p = 0.02) as compared to March 2014 (2.26 + 0.54), p < 0.05. Pre-workshop preparation evolved over time with the goal of maximizing trainee readiness for the hands-on course. A strong correlation was observed between knowledge and practical skill scores illustrating the importance of pre-workshop training.

Conclusions

Our pilot workshop showed promise in promoting knowledge and practical skills among participants, as well as increasing use in patient care. Results also suggest that refresher training may provide additional benefits for some participants. These findings provide a strong rationale for expanding the training program and for measuring its clinical impact.

African relevance

• •Most district hospitals in Kenya have no access to ultrasound training.
• •This training program is appropriate for care providers at different training levels.
• •The program increases knowledge and practical ultrasound skill proficiency.
• •Its format is readily expandable to other low and middle income countries.

Introduction

Ultrasound has increasingly become an effective clinical diagnostic tool in low and middle income countries (LMICs).¹ Compared to other imaging modalities, ultrasound equipment is portable, durable, and its use is safe and relatively affordable. The availability of portable bedside ultrasound equipment in particular, has served to boost the diagnostic capacity of rural healthcare facilities in resource-limited areas,1, 2 and studies have shown that patient management was changed based on ultrasound use.3, 4, 5 Despite its utility, however, an estimated two-thirds of people in LMICs have no access to radiologic imaging.⁶ Availability of ultrasound machines and the proctored skill training needed are the two limiting factors for expanded use of point-of-care ultrasound in LMICs.

Although the World Health Organization (WHO) has established guidelines in ultrasound training for local practitioners, there are no standardized approaches to program length, content, trainer qualifications or mechanism of training.7, 8, 9, 10, 11 Only a handful of ultrasound training programs in Africa including a train-the-trainer program have been described.12, 13, 14, 15, 16, 17 Results from these evaluations show that many basic applications can be competently performed and interpreted by non-radiologists.18, 19 The program we designed and piloted is unique in that it focuses on a few basic but high yield applications and uses a multimedia-based training manual²⁰ to maximize pre-training preparation. Both of these factors are designed for more efficient use of limited instructor time.

Methods

The study was a national pilot of a workshop to train healthcare providers in Kenya in the use of point-of-care ultrasound. Participants were from district hospitals that represented seven of the eight Kenyan provinces. The Health Research Ethics Committee of The Aga Khan University where the training workshops were held approved the study.

The Christian Health Association of Kenya (CHAK) helped to distribute an application form to various healthcare facilities across the country, both to assess interest in the training program and to identify facilities where the use of ultrasound would significantly impact care. Information collected included (a) numbers of inpatients and outpatients, (b) number of providers and their level of training, (c) the level and frequency of obstetric, surgical, and infectious diseases care, (d) whether they had a reliable electrical power source, and (e) an inventory of their radiology equipment.

A total of 38 hospitals were chosen to participate based on the following criteria: (a) return of the application form (measure of interest), (b) identification of at least two providers from the facility with training as doctors, clinical officers (equivalent to U.S. physician’s assistants), nurses, and/or radiology technicians who were willing to sign a contract of intent to participate, (c) facility provides obstetric care and at minimum, minor surgical care for the district (measure of potential impact), and (d) a reliable electrical source (measure of feasibility). The contract of intent indicated that the participant was willing and able to attend the 1-day initial workshop and two refresher workshops.

Four to six weeks prior to the workshop, trainees were provided with training materials²⁰ and asked to study their contents. The content of the pre-workshop materials changed based on participant feedback and trainer perceptions of how well participants were prepared for hands-on training. Specifically, prior to the first session in September 2013, we provided each attendee with a locally developed Point of Care Ultrasound Handbook with instructions to read it before the session began. Because individual facilities may have limited Internet access, we provided all materials off-line.

We found that the participants were not well prepared for this first session. The trainees reported that the manual was difficult to cover in its entirety. Based on this feedback, starting with the December 2013 workshop, we provided subsequent trainees only the manual chapters that covered the ultrasound applications specific to their clinical needs. These topics were basic physics and machine use; the abdominal, pleural and cardiac assessment for free fluid; the thoracic exam for pneumothorax; and an obstetric exam for intrauterine pregnancy, cul-de-sac fluid, fetal heart activity and position.²⁰ We also included more images, illustrations and descriptive narratives. In March 2014, we added multi-media videos to better illustrate the concepts and techniques.

A written exam consisting of questions adapted from the ACEP Emergency Ultrasound Exam was administered online to the trainees before the start of each workshop. As with content, trainee pre-workshop assessment changed over time. For workshops held from September 2013 to June 2014, the trainees were not allowed access to the training materials during the test. We empirically set a passing score of 50% to assess the educational training’s ability to prepare participants for skill training. [For board certification, residents are required to achieve a 70% score.] We found a strong correlation between knowledge and practical skill scores during this period, with trainees scoring over 50% being more likely to achieve practical skill proficiency. Because of the observed correlation and because the goal was trainee preparation not participant exclusion, we changed the educational training format in the final workshop of the pilot study (November 2014). Briefly, the online survey was administered as before, but participants were able to access the material to fill knowledge gaps and re-test until a score of 90% was achieved.

Each workshop was a full day session and sessions were held every 3–5 months from September 2013 through November 2014. Ultrasound-credentialed emergency physicians (credentialed according to American College of Emergency Physician’s (ACEP) 2008 Emergency Ultrasound Guidelines) provided the training. Workshop components included short bedside review of content, instructor demonstrations, trainee scanning of live models, and clinical application discussions, all in groups of 4–5 trainees. Testing of practical skills was performed using the objective structured clinical examination (OSCE) format as established by the Point-of-care Ultrasound in Resource-Limited Environments (PURE) Initiative. Testing involved two skills, assessing image interpretation (IN) and image quality (IM). Each skill was composed of 9 tasks. Scores for each skill were the average of the task scores, with 4.0 and 3.0 being maximum scores for IN and IM, respectively. An average score of 2.0 for both IN and IM was considered passing, defined as the minimum proficiency for proper image acquisition and for interpretation of findings to answer “most” clinically relevant questions.

All data analyses were performed using PRISM™ (Graphpad Software Inc., version 6.0e). For comparisons of continuous variables (i.e. knowledge test scores), a t-test (comparison of 2 conditions) or ANOVA (comparison of 3 or more conditions) was performed. Comparisons of categorical variables (i.e. proportions passed/failed) were performed using the chi-square test or Fisher Exact test (small cell sizes). Linear regression was used to determine whether there was a correlation between written and practical scores. All p values were two-tailed and significance was defined as p < 0.05.

Results

A total of 81 trainees, the majority without previous ultrasound experience, were recruited at 38 hospitals, and 30 of the trainees have completed at least one refresher session (Table 1). Approximately equal proportions of clinical officers, doctors, and nurses participated, with a somewhat smaller proportion of radiographers and one laboratory technician.

Table 1.

Summary of participant training session completion.

Session	Sept 2013	Dec 2013	Mar 2014	Jun 2014	Nov 2014	Total
Initial training	22		36	6	17	81
Refresher #1		12		15	3	30
Refresher #2				3	7	10
Refresher #3					1	1

Table 2 summarizes written score results for trainees participating in their initial workshop. Pairwise comparisons of mean scores using a t-test showed no significant differences between sessions, p > 0.05 in each case. Chi-square analysis comparing the proportion of trainees who passed from September 2013 to June 2014 also showed no significant difference, p = 0.51. As described in Methods, all November participants repeated the exam until achieving a score of 90% or higher. Written scores (mean ± standard deviation) by clinical position were 64 ± 12.3% (Clinical Officer), 68 ± 12.3% (Doctor), 61 ± 11.6% (Nurse), and 55 ± 20% (Radiographer).

Table 2.

Summary of written scores for trainees participating in their first workshop.^a

	Training Session
	Sept 2013	Mar 2014	Jun 2014	Nov 2014
Number of Trainees	22	36	6	17
Knowledge Methodology
Pre-training Materials Num. Exam Questions	Manual 68 questions	Manual + Videos 30 questions
Written Examb	Scores
Median	60%	65%	57%
95% CI of median	46–70%	60–73%	53–80%	Trainees repeated exam until 90% or higher
Mean	58%	64%	64%
SD	16%	14%	12%
95% CI of mean	51–65%	59–68%	49–79%
Written Exam (P/F)c	N (column %)d
Passed	15 (68%)	29 (81%)	5 (83%)	NA
Failed	7 (32%)	7 (19%)	1 (17%)

^aNo new trainees participated in December 2013.

^bPairwise comparisons using both a t-test (parametric) and Mann–Whitney test (non-parametric) showed no differences in written scores by session, p > 0.05 in each case.

^cPass = 50% correct or greater.

^dChi-square comparison of proportions showed no significant difference, p = 0.51.

The average of the practical skill scores in November (2.6 ± 0.41) was significantly higher than for March (2.26 ± 0.54), p = 0.012 (Table 3). A comparison of the proportions of trainees who passed the practical exam in the earliest workshop (Sep 2013, 59%) and the most recent workshop (Nov 2014, 88%) trended toward an increase, but did not reach significance, p = 0.099.

Table 3.

Summary of practical scores for trainees participating in their first workshop.^a

	Training Session
	Sept 2013	Mar 2014	Jun 2014	Nov 2014
Number of Trainees	22	36	5	17
Practical Skills (IN)	Scores
Mean	Raw scores not recorded	2.2	2.5	2.6
SD		0.59	0.43	0.41
95% CI of mean		1.98–2.39	2.04–2.93	2.38–2.80
Minimum		0.6	2.0	1.7
Median		2.3	2.5	2.7
95% CI of median		2.0–2.6	2.0–2.9	2.4–2.9
Practical Skills (IM)	Scores
Mean	Raw scores not recorded	2.34	2.28	2.69
SD		0.58	0.54	0.38
95% CI of mean		2.14–2.54	1.71–2.85	2.49–2.88
Minimum		0.5	1.6	2.0
Median		2.3	2.2	2.7
95% CI of median		2.1–2.6	1.6–2.9	2.5–2.9
Practical Skills (Ave)	Scores
Mean	Raw scores not recorded	2.26	2.38	2.64b
SD		0.54	0.42	0.38
95% CI of mean		2.08–2.45	1.94–2.82	2.45–2.83
Minimum		0.6	1.9	1.9
Median		2.3	2.33	2.65
95% CI of median		2.05–2.45	1.9-2.9	2.5–2.9
Practical Skills (P/F)c	N (column %)
Passed	13 (59%)	23 (66%)	4 (80%)	15 (88%)d
Failed	9 (41%)	12 (34%)	1 (20%)	2 (12%)

^aNo new trainees participated in December 2013.

^bPairwise comparisons using a t-test (parametric) and the Mann–Whitney test (non-parametric) showed a significant difference between March and November; t-test p = 0.012; Mann–Whitney, p = 0.008.

^cPass = Average of IN and IM of 2.0 or greater.

^dChi-square, September vs. November, p = 0.099.

By clinical position, average practical skill scores were 2.3 ± 0.35 (Clinical Officer), 2.7 ± 0.23 (Doctor), 2.3 ± 0.65 (Nurse), and 2.9 ± 0.71% (Radiographer). Linear regression showed a significant correlation between written and practical exam scores for trainees at their initial session. The Pearson r was 0.44 and the p value was 0.0048.

For knowledge, 90% of trainees who completed more than one training sessions (27 of 30) maintained, defined as within 10% of initial training scores, or improved, defined as >10% higher than initial training scores, their scores. Similarly, of the 18 trainees who attended multiple sessions and who had practical skill scores available, 17 trainees (94%) maintained or improved their scores.

Of the 30 trainees who completed one or more refresher sessions, the proportion that passed the written exam increased from 63% (n = 19) to 73% (n = 22) for initial and follow-up training, respectively (Table 4). For the practical skill exam, an increase in the proportion passing was also observed, from 77% (n = 23) for the initial training to 87% (n = 26) for follow-up. All participants who completed more than one refresher session (n = 11) passed both the final written and final practical exams.

Table 4.

Summary of written and practical exam results (Pass vs. Fail) for participants who attended the initial workshop and at least one refresher training session. N = 30.

		Written Exam (Initial/Follow-up)
		Pass/Pass	Pass/Fail	Fail/Pass	Fail/Fail
Practical Exam (Initial/Follow-up)	Pass/Pass	12	3	6	1
	Pass/Fail		1
	Fail/Pass	3		1
	Fail/Fail				3

Discussion

The goal of pre-workshop training was to sufficiently prepare participants for optimal use of the limited time they had with instructors for hands-on training. The training manual, which participants were expected to study in advance, provides the in-depth knowledge needed to perform each application. We found that limiting the required reading to manual chapters most relevant to the learned skills was less overwhelming to the trainees. In addition, we added more images and videos of both normal and common disease states over time to demonstrate proper image gathering and interpretation; and feedback from participants suggested they found this helpful. Watching the videos is thought to elicit an internal representation of the performed action, which allows rehearsal and encourages learning, even before any personal experience.²¹

Two-thirds or more of first-time trainees passed the written exam from September 2013 through June 2014, and scores trended toward an increase over time. However, instructors were still concerned that preparation was not optimal. Thus a new format was introduced in November 2014 where participants had the opportunity to re-study materials until a 90% score was achieved. Initial workshop practical skill scores in November were significantly higher than for earlier workshops, with all participants passing; but additional studies will be needed to see if the new format is consistently better for preparing trainees than earlier formats.

When trainees attended the course, they worked in small interactive groups. The instructor and trainees scanned real human models, including early and late pregnancy models. The framework of the group scanning was peer-assisted learning, through supervised scanning, discussion of clinical significance, and finally the opportunity for trainees to demonstrate competency. If trainees were able to satisfactorily complete knowledge and practical skill testing at the conclusion of the first session, the facility they represented was awarded a portable ultrasound machine with a large curvilinear transducer to practice the applications they had learned.

Practical scores for trainees at their initial workshop increased significantly when comparing March 2014 and November 2014. Notably, the strong correlation observed between passing the written and practical exams validates the importance of pre-workshop study and knowledge acquisition. Although numbers remain low, no significant differences in practical skills have been observed for different clinical positions to date. This indicates that the training can be appropriate and effective for providers at all levels. Anecdotally, instructors have noted that the best predictor of skill proficiency is a strongly expressed interest in and dedication to mastering the techniques.

Refresher sessions were designed to assess ongoing knowledge and skill retention or improvement. Goals of refresher training were to fill any knowledge gaps, strengthen practical skills and encourage additional scanning at their clinical facilities to gain practice. Knowledge retention and even improvement was observed among the majority of refresher participants (December 2013–June 2014). Retention or improvement of practical skills was also observed among refresher workshop attendees. All trainees who completed two or more follow-up sessions passed both their final knowledge and practical skill exams.

Moving forward, only practical skill retention or improvement will be measured, as all participants will have a 90% score for pre-workshop knowledge. Moreover, to reduce bias in assessing refresher workshop effectiveness, analysis will need to control for the level of practice between workshops. In addition, skills could be tested prior to the refresher training (baseline) and after to determine whether the workshop independently contributed to increased proficiency.

Surveys were provided to participants after each refresher workshop to report on ultrasound its use since the last training session. Response rates ranged from 74% to 90%. Questions addressed the applications for which the participants had used ultrasound machines and any impacts they had on the participant’s clinical practice. For individual groups, two-thirds to three-quarters of trainees reported they had used point-of-care ultrasound over 20 times in the past three-month period. Approximately 60–70% of the trainees reported that ultrasound had influenced their clinical care of patients more than half the time. Obstetrical exams, predominately 2nd and 3rd trimester exams, were reported to be the most commonly performed and to have had the greatest impact on their patient management. More specifically, they reported using ultrasound to determine fetal lie and gestational age. The next most frequent uses were 1st trimester obstetric exams and eFAST (Extended Focused Assessment with Sonography for Trauma).

The inspiration for workshop design, particularly providing pre-workshop materials and live training was based on the trainees’ limited access to the Internet at their facilities. Most of the current web-based teaching materials for point-of-care ultrasound are free-access and of high quality. Academic centres are currently using these resources as a foundation for their curricula in post-graduate point-of-care ultrasound training, but most LMICs do not have Internet service capable of accessing this training. Another problem of limited or no Internet access is the inability to send ultrasound scans electronically for professional review to ensure the accuracy of interpretation.

Providers in LMICs often find it difficult to leave their facilities for training, due to limited staffing. Some facilities have not been receptive to excusing their providers for the 3 workshops. Though the trainees have to arrange their own travel to Nairobi, nearly all report that it is a routine destination with established means of transport.

The relatively small number of participants in our pilot studies may reduce the generalizability of our results. However, results are highly encouraging and provide a justification for expanding the training program as resources allow.

Conclusions

We have created and piloted a training program that is designed to recognize the clinical needs and logistic challenges of LMIC providers. The pilot 1-day workshop on point-of-care ultrasound resulted in increased knowledge and clinically relevant practical skill acquisition by participants. Initially, a passing knowledge score was 50%. For those workshops, knowledge scores correlated strongly with practical skill scores. This observation, in addition to trainer perceptions, resulted in a final pre-workshop educational format that allowed participants to re-test until a 90% score was achieved. All participants achieved a passing practical skill score in this final workshop. Additional studies will be needed to further test this pre-workshop format. Refresher training was associated with retention or improvement of practical skill scores. In addition, preliminary survey results suggest that training increased the use of ultrasound at facilities that were part of the program, that a large percentage of participants felt the training had influenced their clinical care of patients, and that ultrasound applications in obstetrics were the most commonly used. Thus, obstetric applications may provide an opportunity to study the clinical impact of point-of-care ultrasound training and use in Kenya.

Conflict of interest

The authors declare no conflict of interest.

Author contributions

GB contributed to the design of the pilot program and to its assessment, including helping to create the training program, pre-training materials and knowledge and skill tests. He also participated in training, in data interpretation, and in the final preparation of this manuscript.

BW similarly contributed to program design, materials and assessment and was the primary creator of the pre-training study materials. He was also a trainer in the pilot program and was primarily responsible for compiling test results. He contributed to the final preparation of this manuscript.

GD performed data analysis and helped in its interpretation. She also prepared the preliminary draft of this manuscript and helped in its final preparation and submission.

Dissemination of results

Progress reports and publications resulting from these studies will be shared with supporters of the study, including CHAK, the DAK Foundation and participating healthcare facilities. Broader distribution of study results will include postings on the EM Kenya Foundation blog (http://www.emergencymedicinekenya.org/emkf-point-of-care-ultrasound-initiative/).