Measuring movement towards improved emergency obstetric care in rural Kenya with implementation of the PRONTO simulation and team training program

Abstract

As the proportion of facility‐based births increases, so does the need to ensure that mothers and their newborns receive quality care. Developing facility‐oriented obstetric and neonatal training programs grounded in principles of teamwork utilizing simulation‐based training for emergency response is an important strategy for improving the quality care. This study uses 3 dimensions of the Kirkpatrick Model to measure the impact of PRONTO International (PRONTO) simulation‐based training as part of the Linda Afya ya Mama na Mtoto (LAMMP, Protect the Health of mother and child) in Kenya. Changes in knowledge of obstetric and neonatal emergency response, self‐efficacy, and teamwork were analyzed using longitudinal, fixed‐effects, linear regression models. Participants from 26 facilities participated in the training between 2013 and 2014. The results demonstrate improvements in knowledge, self‐efficacy, and teamwork self‐assessment. When comparing pre‐Module I scores with post‐training scores, improvements range from 9 to 24 percentage points (p values < .0001 to .026). Compared to baseline, post‐Module I and post‐Module II (3 months later) scores in these domains were similar. The intervention not only improved participant teamwork skills, obstetric and neonatal knowledge, and self‐efficacy but also fostered sustained changes at 3 months. The proportion of facilities achieving self‐defined strategic goals was high: 95.8% of the 192 strategic goals. Participants rated the PRONTO intervention as extremely useful, with an overall score of 1.4 out of 5 (1, extremely useful; 5, not at all useful). Evaluation of how these improvements affect maternal and perinatal clinical outcomes is forthcoming.

1. INTRODUCTION

Global efforts to reduce maternal mortality in the 21st century have resulted in significant decreases in the maternal mortality ratio; however, rates of change are inadequate for achieving global targets (Kassebaum et al., 2014). Access to and utilization of skilled birth attendants have been a critical approach to date for achieving the improvements in maternal and neonatal outcomes worldwide (Alvarez, Gil, Hernández, & Gil, 2009; Kassebaum et al., 2014; World Health Organization [WHO], United Nations Children's Fund, United Nations Population Fund, Bank, & The United Nations Population Division, 2014). The 2014 Demographic and Health surveys (DHS) found a maternal mortality ratio in Kenya of 362 maternal deaths per 100,000 live births (Kenya National Bureau of Statistics [KNBS] et al., 2015). Kenya has seen an increase in facility‐based births, a proxy for skilled birth attendance. Between the 2009 and 2014 DHS, the proportion of facility births in Kenya increased from 43.2% to 61.2% (KNBS and ICF Macro, 2010; Mwangi et al., 2015). Despite this rise in facility‐based deliveries, intracounty variations exist. In Kakamega County, 47% of births occurred in facilities according to the 2014 DHS. This is a 25.3 percentage point increase from 2009 estimates for Western Province—the larger province that formerly contained what is now Kakamega County (KNBS and ICF Macro, 2010; Mwangi et al., 2015).

With successes in increasing the proportion of facility‐based deliveries globally, international stakeholders are now mobilizing efforts to improve the quality of obstetric care within facilities to improve maternal and perinatal health outcomes. In‐service training is a common strategy for impacting the quality of facility‐based maternity services (Austin et al., 2014; Hofmeyr et al., 2009). In‐service training can teach evidence‐based clinical practices and teamwork skills to health care workers. The body of evidence, from high‐resource settings, demonstrates the value of incorporating simulation into in‐service training for clinical decision making, teamwork, and use of evidence‐based practices (Crofts, Winter, & Sowter, 2011; Crofts et al., 2006; Fransen et al., 2012; Merien, van de Ven, Mol, Houterman, & Oei, 2010; Reynolds, Ayres‐de‐Campos, Pereira‐Cavaleiro, & Ferreira‐Bastos, 2010; Siassakos, Crofts, Winter, Weiner, & Draycott, 2009). Simulation training in low‐resource settings has not been well studied. Existing evidence suggests that simulation can be an effective training modality in low‐resource settings when specific methods are adapted to the context. Modifications include low‐cost, low‐tech simulators, modifying training supplies to the local context and creatively using diverse venues for training (Andreatta, Gans‐Larty, Debpuur, Ofosu, & Perosky, 2011; Bergh, Baloyi, & Pattinson, 2015; Hofmeyr et al., 2009; Walker et al., 2014; Walker et al., 2012, 2015).

The Linda Afya ya Mama na Mtoto (LAMMP, Protect the health of mother and child) was designed to improve maternal and perinatal outcomes in Kakamega County, Kenya. Two integrated goals throughout the overall project were (a) to improve obstetric referral linkages between the community and facilities and (b) to strengthen clinic‐based obstetric care in linked facilities. Community–facility linkages were strengthened by developing or strengthening Kenya's community units, training community health volunteers (CHVs) on maternal and newborn nutrition and promoting facility‐based deliveries, and training existing traditional birth attendants to transition to a new role as skilled birth advocates. All facilities received the Ministry of Health's (MoH) knowledge and skills‐based Harmonized Maternal and Neonatal Care training program, based on the “Making it Happen” training program during the baseline period, whereas intervention facilities were supplemented with PRONTO International (PRONTO) low‐tech, highly realistic simulation‐based emergency obstetric and neonatal and team training program (Ameh & van den Broek, 2015; Walker et al., 2012).

This study assesses the impact of PRONTO two‐module simulation and team training program on process indicators including participant knowledge, self‐efficacy of obstetric and neonatal emergency management, teamwork skills, and achievement of strategic training goals as set by participating facilities. First, we hypothesized that the PRONTO intervention would sustainably improve participant knowledge, obstetric self‐efficacy, and teamwork skills. Second, we hypothesized these improvements would create an enabling environment where strategic goal setting and health system improvements could be achieved.

Key messages.

• The PRONTO International (PRONTO) training resulted in sustained changes in provider knowledge, self‐efficacy, and teamwork self‐assessments at 3 months post‐training.

• This study supports the use of highly realistic simulation as methodology for in‐service training in low‐ and middle‐income countries.

• The strategic goal achievement results suggest participants were able to achieve early results based on the Kirkpatrick Model.

2. METHODS

LAMPP is a component of the cross‐country Community‐Based Maternal and Newborn Health Project. Project was a cross‐sector collaboration of practitioners, obstetric and neonatal training content experts, academics, the Government of Kenya, and international not for profit. The Kenyan MoH and the Micronutrient Initiative (MI) provided leadership, oversight, and guidance for the program's development and funding. The MoH also rolled out the standard Harmonized Maternal and Neonatal Care training program that was completed prior to the PRONTO training in all sites, intervention, and comparison. African Medical Research Foundation (AMREF) improved linkages between facilities and communities in intervention sites, by strengthening or developing community units, training of CHVs, and transitioning traditional birth attendants to skilled‐birth advocates. University of Washington (UW) managed the rollout of the PRONTO training and undertook facility‐based impact evaluation. University of Nairobi (UoN) provided external evaluation of LAMMP.

This program utilizes a nonequivalent group, preimplementation–postimplementation design for impact evaluation. Clinics were purposefully selected to receive the PRONTO intervention on the basis of three criteria:

• First, participating clinics were within one of three subcounties in Kakamega County at the time of facility assignment—Kakamega Central, Mumias, and Matungu subcounties. Intervention facilities were originally assigned prior to devolution to a federal government structure in March 2013. Two facilities that were in Kakamega Central prior to 2013 were administratively reassigned to Navakholo subcounty in the federal government structure. The two Navakholo subcounty facilities were managed by a management team and reproductive health coordinator different than those of the Kakamega Central facilities. Despite the redistricting, the two Navakholo subcounty facilities were retained in the Kakamega Central training group, referred to as Kakamega Central–Navakholo and analyzed on the basis of the intent‐to‐treat principle.

• Second, facilities were level 2 or 3 facilities (as defined by the Kenya Essential Package for Health facility pyramid).

• Third, they had conducted 10 or more deliveries per year in 2011 (National Coordinating Agency for Population and Development [Kenya] et al., 2011). Comparison clinics were in two adjacent subcounties, Butere and Khwisero, but met all other inclusion criteria.

In the Kenya Essential Package for Health, level 2 and 3 facilities are primary care facilities that are considered the population's first point of clinical care. These facilities provide basic obstetric care in addition to primary health care treatment and preventative services (National Coordinating Agency for Population and Development [Kenya] et al., 2011). A total of 44 facilities were included in the impact evaluation of the program. Twenty‐six received the PRONTO intervention, and 18 facilities served as comparison sites.

Although all facilities in both intervention (26) and comparison facilities (18) received the MoH Harmonized training package, only the intervention facilities received the PRONTO training. The Harmonized training package is a comprehensive 5‐day skills‐ and drills‐based training and covers all aspects of antenatal, intrapartum, and postnatal care for mother and baby (Ameh & van den Broek, 2015). PRONTO's training covered a subset of the clinical content presented in the MoH Harmonized training package; however, the clinical content was supplemented with team training challenges and highly realistic simulations to support the translation of improved provider knowledge into behavior. This paper reports findings from the training conducted in the 26 intervention facilities only.

As part of the impact evaluation of LAMMP, facilities were surveyed at baseline for staffing level, equipment and supply availability, and rates of maternal and neonatal complications during the previous 2 years (2011 and 2012). With WHO's near‐miss approach, the research team developed a morbidity and mortality tool to measure clinic‐based outcomes for LAMMP (WHO, 2011). The impact evaluation for PRONTO's training is based on the four levels of the Kirkpatrick Training Evaluation Model (Kirkpatrick & Kirkpatrick, 2006). The Kirkpatrick Training Evaluation Model evaluates training programs on four levels: (a) participant reactions, (b) improvements in knowledge, (c) changes in behavior, and (d) results. Programs can evaluate these levels to determine where along this continuum a training either succeeded or failed in achieving the desired results (Kirkpatrick & Kirkpatrick, 2006; Smidt, Balandin, Sigafoos, & Reed, 2009). This study focuses on three of the four levels of the Kirkpatrick Model: measuring participant reaction to the training (level 1), improvements in participant knowledge and self‐efficacy (level 2), and strategic goals achievement (level 4). Impact on clinical outcomes was assessed separately.

The PRONTO curriculum was reviewed by Kenyan MoH officials, and Kenyan Obstetricians and Pediatricians to ensure that it met MoH guidelines. Additionally, all simulations and case studies were adapted to ensure cultural relevance and applicability with input from Kenyan obstetricians, pediatricians, and nurses. Throughout LAMMP, PRONTO trained and mentored local nurses, obstetricians, and pediatricians as trainers. At each training, the majority of training sessions were conducted by Kenyan clinicians under the mentorship of PRONTO master trainers.

The PRONTO training is conducted over 3 days with Module I (2 days) occurring 3 months before Module II (1 day). All training sessions occurred in situ—at intervention facilities. Birth simulations were conducted in situ where deliveries occur. Separate spaces were used for other components of the training—including video‐guided simulation debriefs, teaching sessions, skill validation, or team building and communication activities. A total of six PRONTO training sets were delivered in three subcounties of Kakamega. Matungu and Mumias subcounties received two training sessions each, and two additional training sets were delivered for the Kakamega–Navakholo group.

For disruptions in facility's service delivery to be minimized, only half of the facility staff attended an PRONTO training at one time. CHVs from the targeted facilities were identified by MoH officials and invited to participate in Module I and II training sessions to strengthen the relationship between facility staff and the community. Because not all CHVs were able to participate in the pre‐testing/post‐testing based on literacy and this cohort does not formally provide obstetric care at facilities, data on the CHV cohort are excluded from this analysis. However, all CHVs underwent the same oral consent process as other training participants.

Module I (2 days) covers obstetric hemorrhage, neonatal resuscitation, and teamwork and communication concepts through skills stations, case presentations, teamwork activities, and six simulations. Throughout the training, the impact of maternal nutritional status, particularly maternal anemia, was emphasized as a potential contributing factor. Additionally, basic newborn nutritional interventions such as delayed cord clamping and early initiation of breastfeeding were included in all simulations and case studies. After exposure to these concepts and skills, participants from each facility collaborate on a facility‐specific strategic plan. The strategic planning session is designed for participants to translate the knowledge and skills gained from PRONTO's simulation training into practice at their facilities. Each facility was provided 24,000Ksh (approximately USD $275) to implement their strategic plan including purchase of supplies and equipment; half of the incentive was given at the end of Module I, and the remaining half at the end of Module II. Module II (1 day) occurred 2 to 3 months after Module I and focused on pre‐eclampsia, shoulder dystocia, teamwork and communication, and a review of strategic goal achievement and included three simulations on the clinical topics included in the training. A detailed explanation of the PRONTO training program has been previously published (Walker et al., 2012).

Immediately before and after the Module I and II training sessions, participants completed subject‐specific knowledge tests, self‐efficacy reports on performing delivery‐related skills, and a teamwork self‐assessment (Evaluation materials in the supplemental materials). The self‐efficacy survey used 10 questions about the participants' confidence in performing clinical skills on a scale of 0 (cannot do at all) to 100 (highly certain can do) and was completed at each time point (Cohen, Cragin, Wong, & Walker, 2012). Clinical knowledge was assessed at baseline prior to training on new information/skills and at the conclusion of each module. Module I content was tested pre‐Module I, post‐Module I, and post‐Module II. The 24‐question assessment included content on neonatal resuscitation (n = 9 questions), obstetric hemorrhage (n = 9), and teamwork and communication techniques (n = 6). Module II content was tested pre‐Module II and post‐Module II; it contained knowledge questions on pre‐eclampsia (n = 10) and shoulder dystocia (n = 7).

Participants' perceptions about their facility's teamwork skills were measured using the validated 16‐point Clinical Teamwork Scale (CTS; Guise et al., 2008). It was administered post‐Module I to measure teamwork skills in the last 3 months. The CTS was administered post‐Module I so that participant would have exposure to the teamwork concepts the instrument assessed, and pre‐Module II to gauge teamwork in the time between training not including team performance during the Module II training itself.

The strategic goals set by participating facilities at Module I and evaluated for completion at Module II were finalized by participants in the week following the training. Written goal statements were collected in collaboration with the local subcounty reproductive health officers. Goals were categorized by PRONTO trainers as Supply Procurement, System Change, Teamwork Goals, and Training Goals. Goal achievement was captured through participant self‐report during the Module II training, and later confirmed by PRONTO trainers during a 1.5‐ to 2‐hr site visit at the facility approximately 6 months after Module II. For goals that required supply procurement, supplies were visually confirmed, and noncommodity goals were confirmed using facility records. The numerator for calculating the proportion of strategic goals achieved was based on data collected during the site visits and not simply on self‐report.

Data from the training evaluation, and knowledge and self‐efficacy assessments were entered and managed using the REDCap (Research Electronic Data Capture) electronic data capture tools hosted at the Institute for Translational Health Science (Harris et al., 2009). Descriptive statistics for both facility and participant characteristics were calculated using facility inventory, baseline morbidity and mortality assessments, and training evaluations (Tables 1 and 2). Pearson's chi‐square or Fisher's exact test was used to assess differences in binary variables across subcounties (Tables 1 and 2). For nonparametric facility and training characteristics, Kruskal–Wallis H tests were used to compare variable distribution across multiple subcounties (Tables 1 and 2). For differences in the proportion of training participant characteristics, Pearson's chi‐square test was used. For nonparametric facility and training characteristics, the Kruskal–Wallis H tests were used to compare variable distribution across multiple subcounties.

Table 1.

Baseline characteristics of facilities that received PRONTO training as part of the LAMMP

	Kakamega Central–Navakholo		Mumias		Matungu		Total		p value
	n		n		n		n
Facilities	10	38.5%	9	34.6%	7	26.9%	26	100%	—
Level 3 facilities	4	40%	4	44.4%	3	42.8%	11	42.3%	1
Median time (min) to closest referral hospital (IQR)	24.5	(15)	30	(25)	20	(15)	27.5	(15)	.42a
Supply availability
Bag valve mask (500 or 250–300 mL)	2	20.0%	6	66.7%	5	71.4%	13	50.0%	.065b
Magnesium sulfate	6	60.0%	7	77.8%	5	71.4%	18	69.2%	.865
Syntocinon (oxytocin)	9	90.0%	9	100.0%	5	71.4%	23	88.5%	.245

Personnel	N	Median (IQR)	N	Median (IQR)	N	Median (IQR)	N	Median (IQR)
Total personnel	47	3.5 (5)	60	6 (2)	51	4 (5)	158	4 (4)	.53a
Medical officer	0	0 (0)	0	0 (0)	1	0 (0)	1	0 (0)	.26a
Clinical officer	7	0.5 (0.5)	9	1 (1)	8	1 (2)	24	1 (1)	.79a
Nurse	40	3 (4)	51	5 (2)	42	4 (5)	133	4 (3)	.46a

Baseline complications	n	Rate/ratio	n	Rate/ratio	n	Rate/ratio	n	Rate/ratio
Any maternal or perinatal complication (complication per 1,000 vaginal births)	223/646	345.2	183/1383	132.3	197/1095	179.9	604/3124	193	<.0001
Baseline clinic‐based maternal mortality ratio	0/646	0	2/1353	147.8	2/1061	188.5	4/3038	131.7	.829b
Baseline clinic‐based perinatal mortality ratio	20/646	31.0	22/1383	15.9	39/1095	35.6	81/3124	25.9	.005

Note. IQR = interquartile range.

^aKruskal–Wallis test.

^bFisher's exact test.

Table 2.

Characteristics of Participants who received PRONTO training as part of the Project

	Kakamega Central–Navakholo		Mumias		Matungu		Total		p value
	n		n		n		N
Participants trained by PRONTO, N and median per facility (IQR)	55	3 (6)	51	6 (3)	76	8.5 (13)	182	5 (7)	.0001a
Participants from level 3 facilities (vs. level 2)	37	69.8%	31	63.3%	59	77.6%	127	71.4%	.213
Age (years), median (IQR)b	33.0	(30–46)	40.0	(33–47)	37.5	(30–44)	38	(30–45)	.2521a
Female participants (vs. male)	47	85.5%	41	83.7%	62	81.6%	150	83.3%	.839
Nurses (vs. medical or clinical officer)	45	81.8%	45	91.8%	67	88.2%	157	87.2%	.296
Self‐reported of receiving MoH EMONC Harmonized training program	47	88.7%	41	85.4%	55	75.3%	143	82.2%	.122

Note. EMONC = emergency obstetric and neonatal care; IQR = interquartile range; MoH = Ministry of Health.

^aKruskal–Wallis test.

^bFive participants with missing age information.

For Kirkpatrick's first level to be measured, participants evaluated the training itself after both modules using a 5‐point Likert scale. Lower scores indicated stronger agreement with each statement where 1 was strongly agree and 5 strongly disagree. Participants were asked nine questions including “I understood clearly the purpose and objectives of the training” to “The scenarios were similar to real situations” and “I was supported during the learning process.” Participants were also asked qualitative questions including “Which aspects of the training did you enjoy most?” and “Please list two things you would like to see added or changed to about the training.” For these training evaluations, the median evaluation score and interquartile range were calculated. The Kruskal–Wallis H tests were used to detect differences in participant's reactions to Module I and II training sessions.

To assess Kirkpatrick's second level of training evaluation, learning, we used a variety of methods. For knowledge questions, the percentage of correctly answered items was calculated. For self‐efficacy assessment and CTS, the mean was calculated for each instrument. Changes in scores were analyzed using longitudinal fixed‐effects linear regression models. For assessments conducted in Modules I and II, obstetric hemorrhage, neonatal resuscitation, teamwork and communication concepts, and self‐efficacy were collected at three time points, and post‐Module I and post‐Module II scores were compared with pre‐Module I. Tests conducted at two time points, including pre‐eclampsia, shoulder dystocia, and CTS, were compared at pre‐time points and post‐time points. The fixed effects were modeled to estimate change in knowledge, self‐efficacy, and teamwork for each participant.

Descriptive statistics were calculated for goal achievement including mean goals achieved, which represent Kirkpatrick's fourth level of training evaluation. Differences in the proportion of strategic goals achieved by goal category and across subcounties were tested using the Kruskal–Wallis H test.

Participants in the PRONTO training were informed that their participation in the pre‐testing/post‐testing was voluntary and would not impact their ability to participate in the training. Participants then provided verbal informed consent for the results of their pre‐tests/post‐tests for research purposes. This study was approved by the UoN Ethical Review Board (P257/05/2012) and the UW Institutional Review Board (43069). The overall Project was registered at the Pan African Clinical Trials Registry (PACTR201212000457326). All analyses were conducted with STATA 12 (STATACorp LP, 2015).

3. RESULTS

3.1. Facility characteristics

Participants were clinicians identified from the twenty‐six level 2 and 3 facilities in Kakamega Central, Navakholo, Mumias, and Matungu subcounties. The 26 participating facilities were staffed by 158 clinicians, including one medical officer, 24 clinical officers, and 133 nurses (Table 1). There were slightly more level 2 facilities included in LAMMP; 42.8% of included facilities were level 3 (Table 1). The facility inventory demonstrated slight variability in the time to the nearest referral facility, from a median travel time of 20 min in Matungu to 30 min in Mumias; these differences were not statistically significant (p value = .42; Table 1). There were no significant differences in the proportion of facilities that stocked basic supplies across the subcounties, including pediatric bag valve masks, magnesium sulfate, and oxytocin (Fisher's exact test, p values = .065–.86; Table 1).

The overall proportion of any maternal or perinatal complications at baseline varied by subcounty with a complication ratio of 132.3 complications per 1,000 vaginal births in Mumias to 345.2 in Kakamega Central–Navakholo (Pearson's χ², p value < .0001). Six months of baseline morbidity and mortality data included four maternal deaths from 3,038 vaginal births, suggesting a facility‐based maternal mortality ratio of 131.7 maternal deaths per 100,000 live births across all intervention facilities. This ratio varied by subcounty, with no maternal deaths reported in Kakamega Central–Navakholo, but a maternal mortality ratio of 188.5 in Matungu subcounty (Fisher's exact test, p value = .83). The perinatal mortality rate varied from 15.9 perinatal deaths per 1,000 vaginal deliveries in Mumias to 35.6 in Matungu subcounty (Fisher's exact test, p value = .005; Table 1).

3.2. Training characteristics

The PRONTO training included 182 personnel during six training, two training in each of the three subcounties—Kakamega Central–Navakholo, Mumias, and Matungu. This number is greater than in the personnel inventory because of additional staff hires (6) between the time of the inventory in March/April 2013 and the PRONTO training, and the inclusion of local MoH representatives (17). Each training had a median of five personnel attending from each facility, and a mean of nine facilities was represented at each training. The median number of participants per facility that received the training ranged varied by subcounty, ranging from three to five participants per facility (Kruskal–Wallis test, p value = .0001). The majority of participants were from level 3 facilities (71.4%) and were nurses (87.2%). The median participant age was similar across subcounty training groups and ranged from 33 to 40.0 years (Table 2).

3.2.1. Kirkpatrick training evaluation framework level 1—Reaction

The participant evaluations of the PRONTO training were positive and consistent across module and topics. The median evaluation scores ranged from 1.0 to 1.2 on the 5‐point Likert scale with a high score of 1 (Table 3). There was no difference in evaluation scores between Module I and Module II training sessions (Kruskal–Wallis test, p values = .35–.94; Table 3). In open‐ended qualitative questions, participants reported enjoying the simulation and teamwork components of the training. When asked what aspects of the Module I training they had or would implement in their facilities, the majority reported teamwork and communication skills, neonatal resuscitation, and active management of the third stage of labor (Table 3). The primary changes that participants desired were extended duration–frequency of the training, change in the training space to a larger venue, and expanded clinical content (Table 3).

Table 3.

Participant evaluation results by PRONTO training module

	Module I (N = 165)		Module II (N = 148)		p value
	Median	IQR	Median	IQR
Overall mean score	1.2	0.4	1.1	0.4	.49
I understood clearly the purpose and objectives of the training	1	1.0	1	0.0	.35
I was supported during the learning process	1	1.0	1	1.0	.85
The scenarios were similar to real situations	1	1.0	1	1.0	.68
The feedback was constructive	1	1.0	1	1.0	.69
I would like to participate in simulations in the future	1	1.0	1	1.0	.53
The trainers were well prepared and had a lot of knowledge about the subject	1	0.0	1	1.0	.49
The trainers were experienced in simulation training	1	0.0	1	0.5	.43
I will use the tools I learned in this training in my work	1	1.0	1	0.0	.47
I will use the teamwork concepts in my practice	1	0.0	1	0.0	.94
Which aspects of the training did you most enjoy?	“The fact that the trainers tried to arrange the simulation and look almost real just like the clinical settings”		“The simulations gave me a chance to beprepared in case of an emergency”
	“Leadership skills, communication, post partum hemorrhage management”		“Simulations and communication techniques”
	“Neonatal resuscitation”		‐ “Shoulder dystocia management” ‐ “Neonatal resuscitation ” ‐ “Management of chorioamnionitis” ‐ “Communication”
Please list two things you would like to see added or changed about the training	‐ “Time frame too short” ‐ “Add more simulations”		‐ “Training be done on monthly basis” ‐ “One topic per day”
	1. “Addition of extra day to ensure more understanding” 2. “Addition of other areas of clinical care”		“More time allocated for simulation more space and place for writing (tables)”
Is there anything you learned in Module I PRONTO training that you have (or will) put into practice?	‐ “Leadership/communication skills” ‐ “EMONC updates”		“Yes newborn resuscitation and communication skills”
	‐ “Proper management of third stage of labour” ‐ “Correct decision making on neonatal resuscitation depending on the condition”		‐ “Neonatal resuscitation” ‐ “Estimation of blood loss” ‐ “Management of haemorrhage”

Note. EMONC = emergency obstetric and neonatal care; IQR = interquartile range.

3.2.2. Kirkpatrick training evaluation framework level 2—Learning

The postintervention self‐efficacy, teamwork, and knowledge assessments showed significant improvements. These changes were modeled using longitudinal fixed‐effects linear regression. Participant scores on knowledge of obstetric hemorrhage and neonatal resuscitation improved significantly. Neonatal resuscitation scores rose by 20.3 points (CI [17.9–22.6]) on a 100‐point scale from 65.4% of questions answered correctly pre‐Module I to 85.9%. Obstetric hemorrhage knowledge scores rose 24.1 (CI [21.3–27.0]) points from 61.7% to 86.3%. By the end of Module II, these scores remained significantly higher than those of pre‐Module I, showing an increase of 22.1 points (CI [19.1–25.0]), to 84.3% for obstetric hemorrhage, and 17.6 points (CI [15.1–20.1]) points, to 83.1% for neonatal resuscitation (p values < .0001). Although the knowledge of shoulder dystocia and pre‐eclampsia improved significantly (p values < .0001), the gains were lower, with only 9.9 (CI [6.2–13.5]) and 12.6 (CI [9.3–16.0]) point increases, respectively (Table 4). Although mean participant self‐efficacy scores were high at pre‐Module I measurement (82.4, CI [80.1–84.7]), there were step‐wise increases after each module. The mean overall self‐efficacy score rose significantly to 93.1 (CI [90.8–95.4]) post‐Module I and 94.2 (CI [92.6–95.7]) post‐Module II on the basis of the regression model (p value < .0001; Table 4).

Table 4.

Participant training evaluation results and pre‐test/post‐test knowledge and self‐efficacy scores

	Module I			Module II			p value
	Premodule score (n = 174)	Postmodule score (n = 167)	Change in test score [95% CI]	Premodule score (n = 134)	Postmodule score (n = 147)	Change in test score [95% CI]
Knowledge questions
Obstetric hemorrhage	61.7%	86.3%	24.1% [21.3–27.0]	—	84.3%	22.1% [19.1–25.0]	<.0001
Neonatal resuscitation	65.4%	85.9%	20.3% [17.9–22.6]	—	83.1%	17.6% [15.1–20.1]	<.0001
Teamwork and communication	53.0%	76.4%	23.2% [19.7–26.7]	—	78.7%	26.0% [22.3–29.6]	<.0001
Shoulder dystociaa	—	—	—	43.9%	54.0%	9.9% [6.2–13.5]	<.0001
Pre‐eclampsiaa	—	—	—	46.5%	59.4%	12.6% [9.3–16.0]	<.0001
Self‐efficacy questions
Obstetric hemorrhage and neonatal resuscitation	82.4	93.1	11.2 [9.2–13.3]	91.0	94.2	12.4 [10.3–14.6]b	<.0001
Overall CTS score (1–10 Likert scale)c	—	7	—	8.1	—	1.14 [0.84–14.3]	<.0001
Specific CTS responses (1–10 Likert scale)c
How would you rate teamwork?	——	6.76	—	8.21	—	1.47 [0.95–1.99]	<.0001
Overall communication rating	—	6.63	—	8.04	—	1.39 [0.91–1.88]	<.0001
Orient new members (SBAR)	—	6.23	—	7.66	—	1.44 [0.84–2.04]	<.0001
Transparent thinking	—	6.56	—	8.34	—	1.75 [1.17–2.32]	<.0001
Directed communication	—	6.94	—	8.4	—	1.52 [1.0–2.04]	<.0001
Closed‐loop communication	—	6.55	—	7.8	—	1.42 [0.944–1.89]	<.0001
Overall situational awareness	—	6.94	—	7.99	—	1.04 [0.57–1.5]	<.0001
Resource allocation	—	6.39	—	7.4	—	1.05 [0.47–1.64]	.001
Target fixation	—	5.35	—	6.47	—	0.98 [0.29–1.67]	.005
Overall decision making	—	6.98	—	8.01	—	1.09 [0.63–1.56]	<.0001
Prioritization	—	7.04	—	7.68	—	0.69 [0.14–1.24]	.014
Overall role responsibility rating	—	7.05	—	7.85	—	0.90 [0.39–1.41]	.001
Role clarity	—	7.1	—	7.89	—	0.86 [0.33–1.38]	.002
Perform as a leader/helper	—	7.23	—	7.75	—	0.64 [0.074–1.2]	.027
Patient friendly	—	7.74	—	8.43	—	0.84 [0.35–1.33]	.001
Predelivery team huddle/delivery review	—	7.01	—	7.625	—	0.83 [0.28–1.37]	.003

Note. CTS = Clinical Teamwork Scale; SBAR = Situation, Background, Assessment, Request/Recommendation.

^aContent included and assessed in Module II only.

^bChange in score compares post‐Module II score with pre‐Module I score.

^cAssessment given post‐Module I and pre‐Module II only.

Participants also reported postintervention improvements in both teamwork–communication knowledge and skills. Participant's teamwork knowledge scores improved 23.2 (CI [19.7–26.7]) percentage points between pre‐Module I (53.0%) and post‐Module I (76.4%) and continued to increase to a 26.0 (CI [22.3–29.6]) percentage point improvement between pre‐Module I and post‐Module II (78.7%; p value < .0001). Participants reported overall improvements in clinical team functioning at their facilities by 11% (CI [8.4% to 14.3%]) between Module I and Module II (p value < .0001; Table 4). Self‐reported teamwork skill utilization also increased in all 15 CTS‐specific domains of team function, on a 10‐point Likert scale with 10 being perfect performance. In the 3 months between Modules I and II, participants reported significant improvements in teamwork function with domain‐specific improvements ranging from 1.75 (CI [1.17–2.32]) to 0.64 (CI [0.074–1.2], p values < .0001 to .027; Table 4).

3.2.3. Kirkpatrick training evaluation framework level 4—Results

The strategic planning results demonstrate how improvements in knowledge and self‐efficacy can translate into impact at the facility level. Through the strategic planning sessions, participants developed 192 facility‐specific goals and had achieved 95.8% of these goals by 6 months post‐training. The proportions of category‐specific goal achievement were statistically similar across subcounties, ranging from 92.5% in Matungu to 97.4% in Kakamega Central and Navakholo (p values = .19–.57), except for training‐specific goal achievement where the proportion of training‐related goals achieved varied by subcounty from 77.8% of training goals achieved in Matungu and 100% achieved in Mumias (p value = .032; Table 5). The majority of the 26 facilities set strategic goals related to supply procurement; 129 of 192 goals related to filling perceived gaps in obstetric supplies. Facilities chose to purchase nonconsumables including infection control supplies, suction bulbs, and delivery packs as part of their strategic goals. PRONTO participants also initiated and delivered 32 participant‐led training at their clinics. The majority of these training sessions were conducted through onsite continuing medical education activities, and seven facilities successfully conducted additional training sessions with CHVs (Table 5).

Table 5.

Facility strategic goal achievement—Goals established end of Module I, assessed Module II 3 months later

Category	KakamegaCentral–Navakholo		Mumias		Matungu		Total		p value
	N	% achieved	N	% achieved	N	% achieved	N	% achieved
Overall goal achievement	78	97.4	61	96.7	53	92.5	192	95.8	.34
Training	18	94.4	5	100.0	9	77.8	32	90.6	.032
Neonatal resuscitation training	1	100.0	4	100.0	3	100.0	8	100.0
CHVs	3	100.0	1	100.0	3	66.7	7	85.7
Post‐partum hemorrhage	1	100.0	0	0.0	2	50.0	3	66.7
Referral/emergency preparedness	7	85.7	0	0.0	0	0.0	7	85.7
Infection control	2	100.0	0	0.0	1	100.0	3	100.0
Other	4	100.0	0	0.0	0	0.0	4	100.0
System change	7	100.0	5	80.0	5	100.0	17	94.1	.43
Develop neonatal resuscitation area	1	100.0	0	0.0	2	100.0	3	100.0
Infection control	3	100.0	1	100.0	0	0.0	4	100.0
Other	3	100.0	4	75.0	3	100.0	10	90.0
Teamwork	5	80.0	6	100.0	3	100.0	14	92.9	.571
Supply procurement	48	100.0	45	97.8	36	94.4	129	97.7	.19
Neonatal Ambu bag	6	100.0	4	100.0	1	100.0	11	100.0
BP machine	3	100.0	2	100.0	2	100.0	7	100.0
Clock	0	100.0	2	100.0	3	100.0	5	100.0
Heater	4	100.0	2	100.0	1	100.0	7	100.0
Infection control supplies	4	100.0	3	66.7	6	83.3	13	84.6
Delivery pack	5	100.0	7	100.0	3	100.0	15	100.0
Lighting	2	100.0	6	100.0	0	0.0	8	100.0
MVA kit	6	100.0	2	100.0	4	100.0	12	100.0
Bulb suction	7	100.0	6	100.0	4	100.0	17	100.0
Neonatal resuscitation table	0	100.0	2	100.0	5	100.0	7	100.0
Green towels	5	100.0	4	100.0	1	100.0	10	100.0
Consumable supplies	0	0.0	1	100.0	3	100.0	4	100.0
Other	6	100.0	4	100.0	3	66.6	13	92.3

Note. BP = blood pressure; CHV = community health volunteer; MVA = manual vacuum aspiration.

4. DISCUSSION

Using the Kirkpatrick Training Evaluation Model, PRONTO measured impact at levels 1, 2, and 4 of the evaluation framework—participant reaction, learning, and results (Kirkpatrick & Kirkpatrick, 2006). These results demonstrate that participants valued PRONTO training; showed statistically and practically significant improvements in knowledge, self‐efficacy, and team function; and were able to successfully complete self‐determined strategic goals. Although these process‐level indicators are not a guarantee of improved patient outcomes, they indicate that the PRONTO intervention successfully created conditions for improving the quality of clinical care. Only with the appropriate clinical knowledge, confidence, and availability of critical supplies is it possible for providers to successfully manage emergency situations.

The low pretest knowledge scores, ranging from 43.9% for knowledge of pre‐eclampsia to 65.4% for knowledge of neonatal resuscitation, suggest that basic knowledge of obstetric emergency identification and management may be a critical barrier to improving clinical quality at level 2 and 3 health facilities. These low emergency obstetric knowledge scores were present despite 81.8% of participants reporting participation in the MoH Harmonized training package within the previous 1 to 9 months preceding the PRONTO intervention (Table 4). Further, 13.1% of participants reported participating in another maternal and/or newborn care training program prior to Module I PRONTO training, including Helping Babies Breathe, active management of the third stage of labor training, and manual vacuum aspiration training. Although the timing between a training in the Harmonized training package and PRONTO training differed by subcounty, there were no statistically significant differences in baseline knowledge scores by subcounty. This suggests that emergency obstetric and neonatal knowledge attenuation may occur rapidly after previous training—including training that primarily focuses on knowledge transfer or discrete skill development.

Even with the significant improvements in knowledge at the post‐training assessments, there is need for reinforcement of provider knowledge particularly in regard to the management of shoulder dystocia and pre‐eclampsia/eclampsia because baseline knowledge of these emergencies was lower than that of other emergencies. Simultaneously, however, the relatively high self‐efficacy assessments of 82.4 on a 100‐point scale prior to Module I training suggest that participants may not be aware of gaps in their clinical knowledge or skill. This discrepancy between provider knowledge and self‐efficacy is important for understanding the translation of knowledge into behavior and warrants future study.

Previous studies have suggested that the use of simulation techniques reduces the attenuation of knowledge that often occurs following a knowledge‐based training program (Fransen et al., 2012; Khanduja, Bould, Naik, Hladkowicz, & Boet, 2015; O'Leary, Nash, & Lewis, 2015). In this context, scores for obstetric hemorrhage and neonatal resuscitation knowledge following PRONTO's intervention remained higher than baseline at 3 months postintervention (pre‐Module I vs. post‐Module II). Paralleling other studies comparing traditional training techniques with highly realistic simulation, these data suggest simulations may have allowed participants to immediately practice skills and knowledge gained through case studies and skills stations in the simulations—and thus more effectively retain knowledge gained (Fransen et al., 2012; Khanduja et al., 2015; O'Leary et al., 2015). Although we do not know the impact of the Harmonized training package on provider knowledge, the sustained improvements in knowledge of obstetric hemorrhage and neonatal resuscitation between Modules I and II suggest an added benefit of the PRONTO training.

Despite these individual level improvements, a participant's greater knowledge of and confidence in managing emergencies can only translate into improved care in the presence of, and access to, basic medications, consumables, and equipment. The facility‐specific strategic planning session allows teams to identify concrete opportunities for improvements to implement following the training. A facility's staff created locally specific and relevant plans to improve obstetric and perinatal care at their own facilities. These gaps were identified by facility staff who were responsible for implementing the improvements. The strategic goals developed included nonfinancial training‐based goals including formal continuing medical education, less formal training of colleagues on teamwork techniques, and training CHVs. Some system‐improvement goals included creating spaces for neonatal resuscitation, improving infection control, developing a list of ad hoc drivers in case of a referral, and making baby blankets from existing cloth (Table 5).

Using financial resources to incentivize goal attainment was controversial. Facilitators were carefully trained to focus strategic planning discussions on gaps and goals that local facility teams had control over—not system‐wide problems that were beyond their control. Planning sessions focused on locally owned achievable practical goals. The available financial incentives were used by participating facilities to purchase critical, basic durable supplies and equipment. PRONTO's focus on feasible, locally owned goals that do not require external inputs beyond a small financial incentive was crucial to facilities achieving 95.8% of their goals. This is similar to the model presented by Daniel and Carl Taylor for lasting, locally owned change (Taylor & Taylor, 2002). Strategic planning sessions focused on local ownership may be a powerful mechanism for participants to apply what they have learned through implementation of feasible, locally driven changes. The implementation of these goals also serves as a method for encouraging participants to reflect on and apply the lessons learned when they return to clinical practice, further reinforcing knowledge gains.

The items purchased through the incentive reveal practical challenges when translating new knowledge or skills into routine clinical practice. The majority of facilities used financial incentives to purchase nonconsumable equipment, such as bulb suction, Ambu bags, manual vacuum aspiration kits, and delivery kits. These purchases represent critical supplies necessary to manage normal births and emergency complications. Without routine access to these essential items, it is difficult for training that aim to increase participant knowledge to improve adverse maternal and perinatal health outcomes. Further, although PRONTO trainers did not instruct participants about how to use incentives, the majority of facilities purchased items that will have sustained impact on care quality. Many training programs provide participants with a package of predefined supplies and equipment following the training. Although this method can efficiently improve access to essential emergency supplies, it does not address the complexities of local procurement systems that may result in different needs in each facility. Only four facilities purchased consumable supplies—20‐cm³ syringes, oxytocin, misoprostol, magnesium sulfate, sodium bicarbonate, and maternity files. Although these consumables are relevant for emergency management, the majority of facilities purchased durable supplies that will have a long‐term impact on obstetric health outcomes, given their durable nature.

During emergencies, cohesive team function is essential for utilizing individual knowledge and mobilizing key equipment and supplies. Teamwork and communication training has been identified as a critical component to improving clinical outcomes in high‐resource settings (Bergh et al., 2015; Cornthwaite, Alvarez, & Siassakos, 2015; Crofts et al., 2011; Siassakos et al., 2009). In low‐resource settings, teamwork and communication may be even more important as it can optimize care through improved efficiency and a reduction in errors where resources are limited. The specific contributions of teamwork and teamwork training on reduced maternal and perinatal mortality and morbidity warrant additional analysis.

The 11 percentage point improvement in the post‐training teamwork scores demonstrated in this study is consistent with that of other studies utilizing the CTS tool (Fransen et al., 2012). The disaggregated CTS results mirror the emphasis and time spent in the PRONTO training. With TeamSTEPPS®, the PRONTO training teaches and emphasizes the use of specific communication techniques such as SBAR (Situation, Background, Assessment, Request/Recommendation), Transparent Thinking (Thinking Out Loud), and closed‐loop communication (Check‐back; Powell, 2006). The PRONTO training does not emphasize issues surrounding clinical prioritization or role responsibility. Consequently, it is not surprising to find greater improvements in CTS questions on techniques receiving greater emphasis during the Organization 1 training (Table 4). To our knowledge, the Organization 1 training model is one of the first training programs to combine simulation‐based training together with formal team training in low‐resource settings (Cohen, Cragin, Rizk, Hanberg, & Walker, 2011).

This study has several limitations. Due to small representation of some professions, we were not able to quantify if profession‐specific differences in knowledge, self‐efficacy, or team skill existed (i.e., nurses, clinical officers, or medical officers). Although improved knowledge, self‐efficacy, and reported team practices were maintained at 3 months, it is not yet possible to quantify the magnitude of impact that these improved processes will have on health outcomes or to extrapolate these significant benefits beyond a 3‐month period. By assessing participant's knowledge post‐Module II on Module I‐specific content, we demonstrate that improvements in these topics were sustained at 3 months following Module I. Because we do not have access to the Harmonized training package evaluation results nor were staff in comparison facilities tested, it is not possible to causally attribute the sustained process improvements to PRONTO alone. However, given the similar immediate post‐Module I and post‐Module II self‐efficacy, teamwork, obstetric hemorrhage, and neonatal resuscitation results, it is likely that PRONTO had an impact on the longer term knowledge improvements.

The study represents a large, longitudinal sample and used a validated clinical teamwork tool to capture data (CTS). Further, the knowledge and self‐efficacy results from Kenya parallel results from the PRONTO training in other contexts (Walker et al., 2014; Walker et al., 2012, 2015). These data represent the first analysis of the PRONTO training outcomes outside of Latin America. Contextualizing the PRONTO training program for the Kenyan context required a thorough review of both clinical and cultural contents. These results suggest that a low‐cost, highly realistic simulation‐based training customized to local culture and setting was as well received in Kenya as in Latin America. An analysis of impact on clinical outcomes is forthcoming.

CONFLICT OF INTEREST

JCD, SK, KC, JC, MK, and OG have no conflicts of interest to report. SRC and DW are on the Board of Directors for PRONTO International, a nonprofit organization.

CONTRIBUTIONS

JCD contributed to study design, data collection, program implementation, data analysis, and interpretation of results. SK contributed to data collection, program implementation, and interpretation of results. KC contributed to study design, program implementation, and interpretation of results. SRC contributed to study design, program implementation, and interpretation of results. JC contributed to data collection, program implementation, data analysis, and interpretation of results. MK contributed to study design, data collection, program implementation, and interpretation of results. OG contributed to study design, data collection, program implementation, and interpretation of results. DW contributed to study design, data collection, program implementation, and interpretation of results.

PAN AFRICAN CLINICAL TRIALS REGISTRY NUMBER

PACTR201212000457326

Supporting information

Data S1. PrePost MII post

Dettinger JC, Kamau S, Calkins K, et al. Measuring movement towards improved emergency obstetric care in rural Kenya with implementation of the PRONTO simulation and team training program. Matern Child Nutr. 2018;14(S1):e12465 10.1111/mcn.12465