Identifying and prioritising technical procedures for simulation-based curriculum in paediatrics: a Delphi-based general needs assessment

Abstract

Objectives

To identify and prioritise technical procedures that should be integrated in a curriculum of simulation-based procedural training in paediatrics using the Delphi method.

Study design

National general needs assessment using a Delphi process was completed among 93 key opinion leaders in paediatrics in Denmark. Delphi round 1 identified technical procedures. Round 2 explored frequency of procedures, number of paediatricians performing the procedures, risks and/or discomfort for patients and feasibility for simulation-based training. Round 3 included final elimination and reprioritisation.

Results

Response rates in the Delphi rounds were 73%, 71% and 72%. We identified 37 procedures in Delphi round 1, preprioritised in round 2, resulting in a final list of 19 procedures in round 3. Strong correlation between the prioritisation from the second and third Delphi rounds was identified, Spearman’s r of 0.94 (p<0.0001). Top five on the final list were acute neonatal airway management, acute non-neonatal airway management, non-neonatal peripheral intravenous and intraosseous access, neonatal vascular access and advanced heart lung resuscitation.

Conclusion

We identified and prioritised 19 technical procedures in paediatrics that are suitable for simulation and may be used as a guide for the development of simulation-based curriculum in paediatrics.

What is known about the subject?

• Paediatricians are expected to possess the skills of managing acutely ill children in a broad specialty in high-stakes time sensitive acute situations.

• Rarity of acute events limits exposure and it is impractical and unethical to rely entirely on apprenticeship and clinical interactions to develop the skills required.

• Simulation-based training is often driven by opinion of local educators, available simulators or coincidence. Development of a curriculum should follow a structured and systematic process.

What this study adds?

• A needs assessment using a Delphi process to identify and prioritise technical procedures that are suitable for simulation in paediatrics.

• The procedures represent core competencies in paediatric specialist training that should be practised in a simulation-based environment before applied on patients.

• Educators may use this list to develop, plan and implement simulation-based training curricula for paediatric trainees.

Introduction

Paediatricians are expected to possess the skills of managing acutely ill children. However, the volume of acute events in paediatric emergency departments often limits exposure to many procedures. In addition, paediatrics is a broad specialty considering the broad range of patient age, different subspecialties and rare clinical conditions. Consequently, it is both impractical and unethical to rely entirely on apprenticeship and clinical interactions to develop and master the skills required to manage the treatment of acutely ill children.

Current paediatric training in Denmark includes work-based, supervised and apprenticeship learning blended with elements of simulation-based training. Simulation-based training is often driven by opinion of local educators, available simulators or coincidence,¹ which has several limitations such as limited training time due to work-hour restrictions, patient safety considerations and training programmes that are not necessarily aligned with the current needs.^1–3 Simulation-based training of procedures in paediatrics has been shown to improve the acquisition of skills,^4–8 promote patient safety and reduce errors.^{2 9} Nonetheless simulation-based training is a complex and resource-demanding educational intervention.¹⁰

The development of a simulation-based procedural training curriculum should follow a structured and systematic process, for example, as described by Kern¹¹ in a six‐step approach: (1) Problem identification and general needs assessment, (2) Targeted needs assessment, (3) Goals and objectives, (4) Educational strategies, (5) Implementation and (6) Evaluation and feedback. Unfortunately, the first step is often disregarded or is decided based on availability of simulators or local interests.¹² There is a need to bridge the gap between this unstructured way of developing a simulation-based procedural training curriculum and a needs-driven approach. The results may aid the development of simulation-based training programmes as part of residency curricula in paediatrics.

The objective of this study was to perform a national general needs assessment in paediatrics using a Delphi process to identify and prioritise technical procedures that should be integrated in a simulation-based training curriculum in paediatrics.

Materials and methods

Study design

The study consisted of a three-round Delphi process to establish consensus and prioritise technical procedures for simulation-based training (figure 1). The Delphi technique is widely accepted as a valid tool to obtain expert opinion, obtain consensus and support decision making.^13–15 During each Delphi round, the preceding results are presented to the participants in an anonymous manner to re-evaluate and explore. The Delphi technique allows involvement of participants from across different locations. This means that they do not have to meet in person, that anonymity is maintained, which is key to the Delphi process and avoids strong characters dominating the process. In this modified Delphi, we had planned on the number of rounds a priori but were open to another round if consensus was not achieved. In the most cases, consensus is reached after three rounds,¹³ and therefore, we decided on three rounds. The results from previous rounds were fed to the next round for review and exploration. This method has been used to develop curricula for simulation-based training in various specialties.^16–22 The study was conducted from December 2017 to September 2018.

Figure 1.

The Delphi process.

Participants

The participants were key opinion leaders in paediatric education and were identified based on their roles in the specialist training and education of paediatric trainees across Denmark. The study was nationwide and all paediatricians who are involved in residency training and education were invited. These include postgraduate clinical associate professors, heads of clinical education and training of all paediatric departments, heads of paediatric departments, conveners of mandatory courses for paediatric specialisation, professors, members of educational committees, board members of The Danish Society for Paediatrics and board members of The Danish Young Paediatricians Association. Participants were invited individually and introduced to the study via emails. The second and third round were administered to all the participants regardless if they responded or not in first and second round. Participation was voluntary.

Patient and public involvement

No patients participated and there was no public involvement in the study.

Data collection

The Delphi process was conducted through online survey questionnaires using survey software (Survey Monkey, San Mateo, California, USA) and were distributed by email. Reminder emails were sent up to three times. A multidisciplinary research steering group was formed to facilitate data collection and analysis. It consisted of five members, all investigators and coauthors of the study: two paediatric consultants and postgraduate clinical associate professors (TB and NTH), head of research at a simulation centre (CP), an assistant professor (RDJ) and a paediatric resident, who is also a member of the educational committee in paediatrics and a board member of The Danish Society for Paediatrics (ST). Anonymity of responses was maintained. The primary investigator (ST) was responsible for data organisation, ensuring that the answers were blinded from the rest of the steering group. All suggestions from the participants were considered and were only eliminated when consensus was not reached

Delphi round 1: brainstorming phase

The Delphi process started with a brainstorming phase, where the participants were asked to suggest all technical procedures which a paediatrician who had just completed specialist training in paediatrics should be able to perform. All suggested procedures were reviewed by the research steering group and non-technical skills were removed (eg, team training and communication skills). The remaining technical procedures were consolidated into categories where similar procedures were grouped. This consolidated list of technical procedures was used in Delphi round 2.

Delphi round 2: survey

We used a previously developed needs assessment formula (NAF) from the Copenhagen Academy for Medical Education and Simulation (CAMES) to prioritise the included procedures from the first Delphi round (1, 20). This formula explores the need for simulation-based training of a given technical procedure by investigating four different factors.

1. Frequency; the number of procedures performed annually.

2. Operators; number of physicians that should be able to perform the procedure.

3. Impact; discomfort/risk if the procedure is performed by an inexperienced physician.

4. Feasibility; suitability for learning the procedure in simulation-based environment.

Each technical procedure/procedure group from Delphi round 1 was evaluated according to these four factors. Frequency, operators and impact were stated as multiple‐choice items on a five-point rating scale ranging from 1 to 5. The fourth element of the formula (feasibility for simulation-based training) was challenging for the participants to estimate and was therefore evaluated by the steering group. Feasibility was calculated as the mean score of three equally weighted factors: suitability for simulation-based training, equipment availability and associated costs. We used the means and not medians for ranking the procedures because using the medians would create many ties in the ranking. The steering group explored feasibility of each technical procedure from Delphi 1 using a five-point Likert scale ranging from 1 to 5.

The CAMES NAF score for the individual technical procedure was the average of the mean scores (1–5) of the four factors (frequency, physicians, impact and feasibility) giving each factor equal weighting of 25%. Alternative weighting was considered but decided against because arguing for and quantitating alternative unequal weights was difficult, that is, is it more important to practice a high-frequency low risk procedure or a very rare high-risk procedure. The resulting total CAMES NAF score from 1 to 5 points determined the procedures’ ranking on the preliminary prioritised list used in the third Delphi round.

Delphi round 3: elimination and prioritisation

The preliminary prioritised list of technical procedures from Delphi round 2 was sent to the participants for re-evaluation. The participants were asked to eliminate technical procedures that they found unsuitable for learning in a simulation-based environment, simple technical procedures that should have been learnt in pregraduate medical education or very advanced technical procedures that are learnt in the later part of specialist training. The participants also had the opportunity to reprioritise the remaining procedures.

Data analysis and statistics

In Delphi round 1, we eliminated non-technical skills and grouped similar procedures into categories. We grouped them together when the procedures could be combined and trained in one training programme. In Delphi round 2, the mean score for each factor was calculated and the CAMES NAF score was used to produce a preliminary ranked list of technical procedures used in Delphi round 3. In Delphi round 3, we calculated the number of occurrences for each response using frequency analysis and applied a degree of agreement of ≥70% to establish consensus. The procedures that had <70% support from the participants were eliminated from the final list. Spearman’s r was calculated to explore the alignment between prioritisation according to the CAMES NAF (Delphi round 2) and the final decisions of the participants (Delphi round 3). All analyses were performed using STATA/IC V.15.1 (StataCorp).

Results

Results of Delphi round 1

A total of 93 participants were identified and invited to the study. Round 1 of the Delphi process had a response rate of 73% (68/93) and 68 clinical procedures were suggested in the brainstorming process. These were reduced to 37 procedures/procedure groups.

Results of Delphi round 2

The response rate in Delphi round 2 was 71% (66/93). A preprioritised list of technical procedures was generated using the CAMES NAF score. In this Delphi round, we identified acute airway management of neonates as highest priority and biopsy of skin, tendon and muscles as least priority (table 1).

Table 1.

Technical procedures identified in Delphi round 1

	Procedure group	No of procedures performed (frequency)	No of doctors	Impact	Feasibility	Total CAMES NAF score
1	Airway management, acute, neonatal	4.36	4.63	4.52	4.33	4.46
2	Lumbar puncture	3.81	4.79	4.51	4	4.28
3	Airway management, acute, non-neonatal	3.4	4.82	4.19	4.33	4.19
4	Peripheral intravenous and intraosseous access, non-neonatal	4.79	4.91	4.34	2.67	4.18
5	Intubation	2.57	4.07	4.78	4.33	3.94
6	Peripheral intravenous access, neonatal	4.27	4.14	4.54	2.67	3.91
7	Advanced heart lung resuscitation	2.16	4.88	4.4	4	3.86
8	Nasogastric tube insertion	3.66	4.31	4.09	3.33	3.85
9	Neurological examination of the newborn	4.73	4.87	3.79	2	3.84
10	Intraosseous access, neonatal	2.97	4.71	4.82	2.67	3.79
11	Injections; intradermal, subcutaneous, intramuscular	3.39	4.19	4.87	2.67	3.78
12	Barlow and Ortolani test; congenital hip dislocation	4.31	4.85	3.84	1.67	3.70
13	Pneumothorax management	2.16	3.61	4.6	4.33	3.68
14	Suprapubic bladder aspiration	3.49	4.49	4.13	2.33	3.61
15	Ear examination including otoscopy and tympanometry	4.57	4.85	3.91	2	3.56
16	Sample collection, microbiology	3.61	3.96	3.97	2.67	3.55
17	Collecting vital signs	4.54	4.74	3.46	1.33	3.52
18	Neurologic examination of the child, not newborn	4.61	4.69	3.76	1	3.51
19	Rectal examination	4.1	4.75	3.08	2	3.48
20	Foreign bodies of the airway; airway management	1.76	4.52	4.1	3.33	3.43
21	Bladder catheterisation	2.72	4.1	4.18	2.33	3.33
22	Amplitude integrated electroencephalography	2.79	2.91	3.79	3.33	3.21
23	Asthma related technical procedures	4.24	3.69	3.79	1	3.18
24	Peripheral inserted central catheter, neonatal	2.42	2.45	4.67	2.67	3.05
25	Bedside Fast Assessment Diagnostic Echocardiography	3.16	2.22	4.13	2.33	2.96
26	Bedside bladder ultrasound	3.81	3.48	3.33	1	2.91
27	Ventilator settings	2.43	2.12	4.6	2.33	2.87
28	Diabetes related technical procedures	3.55	2.51	4	1.33	2.85
29	Bone marrow biopsy and aspiration	2.1	1.72	4.34	2.67	2.71
30	Bedside neonatal cranial ultrasound	3.01	2.27	3.84	1.67	2.70
31	Ophthalmoscopy	1.82	2.01	4	2.67	2.63
32	Arthrocentesis	2.21	1.49	4.39	2.33	2.61
33	Handling percutaneous endoscopic gastrostomy feeding tubes	2.66	2.63	3.84	1	2.53
34	Bedside lung ultrasound	1.75	2.3	3.64	2.33	2.51
35	Transabdominal ultrasound measurement of rectal diameter	3.01	2.31	3.45	1	2.44
36	Bedside abdominal ultrasound	1.39	1.63	3.75	2.33	2.28
37	Biopsy of skin, tendon and muscles	1.75	2.75	3.21	1.33	2.26

Ranked according to CAMES Needs Assessment Formula in Delphi round 2.

CAMES, Copenhagen Academy for Medical Education and Simulation; NAF, needs assessment formula.

Results of Delphi round 3

The response rate in Delphi round 3 was 72% (67/93). Eighteen technical procedures were eliminated. The final list included 19 technical procedures (table 2).

Table 2.

Final prioritised list of technical procedures in paediatrics that should be integrated in simulation-based curricula

	Procedure group	Description
1	Airway management, acute, neonatal	Securing airway including optimal positioning, face mask ventilation, use of T-piece resuscitator (eg, Neopuff), oropharyngeal airway, nasopharyngeal airway, suction and optimal positioning. Use of mask‐CPAP and DUOPAP.
2	Airway management, acute, non-neonatal	Securing airway including optimal positioning, face mask ventilation, use of oropharyngeal airway and suction.
3	Peripheral intravenous and intraosseous access, non-neonatal	Peripheral iv-access including use of different kinds of vein finders. Use of scalp veins for access in infants. Intraosseous access including site selection in different ages.
4	Vascular access, neonatal	Peripheral intravenous access, umbilical vein catheterisation, umbilical artery catheterisation and peripherally inserted central venous catheters (lines)
5	Advanced heart lung resuscitation	Chest compressions and face mask ventilation for adolescents, children and neonates. Use of defibrillator—manual and automated external defibrillator
6	Lumbar puncture	For adolescents, children and neonates
7	Intraosseous access, neonatal	Intraosseous access in neonates including correct intraosseous needle positioning and the avoidance of dislodgement.
8	Pneumothorax management	Needle decompression of pneumothorax.
9	Foreign bodies of the airway; airway management	Airway management, removing visible foreign body, use of back blows and abdominal thrusts.
10	Neurological examination of the child less than 1 year of age	Neurological examination including ‘the 180° examination’.
11	Suprapubic bladder aspiration	Suprapubic aspiration to take a urine sample, with or without ultrasound guidance.
12	Injections; intradermal, subcutaneous, intramuscular	Different kinds of injection including the use of an epinephrine auto injector.
13	Neurological examination of the child above 1 year of age	Examination of sensory function, motor function, reflexes, cranial nerves, cognition and development. Hyperventilation test for Absence seizures.
14	Bladder catheterisation	Urinary catheterisation, both gender and all paediatric ages.
15	Asthma-related technical procedures	Use of inhalation devices and pulmonary function tests.
16	Amplitude integrated electroencephalography	Initiating amplitude integrated electroencephalography.
17	Bedside bladder ultrasound	Measure the volume of urine.
18	Transabdominal ultrasound measurement of rectal diameter	As a diagnostic tool in childhood constipation.
19	Diabetes-related technical procedures	Handling different insulin pumps and use of insuflon (angled injection port).

A Spearman’s r of 0.94 (p<0.0001) showed a strong correlation between the prioritisation order of the technical procedures from Delphi round 2 and the final list in Delphi round 3 (figure 2).

Figure 2.

Correlation between the preliminary prioritisation after Delphi round 2 and the final prioritisation after Delphi round 3.

Discussion

We performed a general needs assessment to identify and prioritise technical procedures that should be integrated in a curriculum of simulation-based procedural training in paediatrics. A prioritised list of 19 technical procedure was produced using a three‐round Delphi process among key opinion leaders in education in paediatric. All paediatric departments involved in the specialist training programme were represented, and the response rates were consistently high in all three rounds, indicating strong support from the participants and increasing the credibility of our results.

Airway management of both neonatal and non-neonatal topped the final list with highest priority. Airway management comprises several hands-on procedures and some of the airway management procedures are performed infrequently, making it relevant for simulation-based procedure training. Improvement in airway management skills through simulation-based training among paediatric residents has been shown in several studies.^{5 23 24} There is a growing body of evidence finding that a simulation-based airway management curriculum is superior to no intervention and to non-simulation intervention for education outcomes.²⁵

Non-neonatal peripheral intravenous and intraosseous access and neonatal vascular access ranked third and fourth, respectively, in the final prioritised list. Neonatal intraosseous access ranked lower in the list, probably because it is rarely performed due to umbilical vein and artery availability. Nevertheless, it has been shown that intraosseous access can be performed more quickly in neonates than umbilical vein catherisation, especially for paediatricians who do not routinely place umbilical vein catheters.²⁶ A low-cost model with real umbilical cord has been described to provide a more realistic training model than currently available commercial simulators.²⁷

Advanced heart lung resuscitation was the fifth procedure on the final prioritised list. Advanced heart lung resuscitation comprises several hands-on procedures in form of cardiopulmonary resuscitation with rescue breathing and chest compressions as well as airway management and defibrillation.²⁸

The top five procedures in the final prioritised list are all part of the Airway, Breathing, Circulation, Disability, Exposure approach of paediatric advanced life support and resuscitation.²⁹ These procedures represent core competencies in paediatric specialist training. The skills are required in high-stakes time sensitive acute settings, and thereby emphasises the need for simulation-based training of these procedures.

The strong correlation between the results from Delphi rounds 2 and 3 indicates minimal changes in the ranking order of the procedures (figure 2). However, a few interesting changes were observed, for example, the change in rank for lumbar puncture and intubation which were both in top five on the preprioritised list from round 2. Lumbar puncture rated second in Delphi round 2 and changed to a sixth placement in Delphi round 3, most likely because lumbar puncture is a less acute procedure compared with airway management, vascular access and advanced heart lung resuscitation which ended up in top five. Intubation ranked fifth after the second Delphi round, but was eliminated in the final round. This is probably because the participants were asked to eliminate the very advanced technical procedures that are not expected for a newly specialised trainee to perform.

Ultrasonography only appeared as no. 17 and 18 in the final list in contrast to several ultrasound related procedures which were found in Delphi round 2. We assume that they were regarded as specialised procedures that only some paediatricians should master. In a recent study on general needs assessment in radiology, the top five procedures included ultrasonography.²¹ In addition, ultrasonography is listed among the top-10 procedures in cardiology,¹⁶ urology,¹⁹ pulmonology²⁰ and anaesthesia.²² The relatively low ranking of ultrasonography in our study compared with that seen in other specialties may indicate that experience with ultrasonography among the paediatric participants is relatively low at this point. We may expect a stronger need for training of ultrasonography in paediatrics in years to come as in international proposals.³⁰

The combination of CAMES NAF score and a three-round Delphi not only identified but also prioritised the technical procedures for simulation-based training. CAMES NAF score prioritises procedures if they are frequent, performed by many physicians, potentially harmful to patients if performed by a non‐competent physician, and very feasible to train simulation based.¹ The participants were free to remove and reprioritise the procedures from the second round if they did not agree with the prioritisation resulting from the CAMES NAF score. However, we found a strong correlation between the ranking order after the second and third Delphi rounds, indicating that the participants widely agreed with the results based on CAMES NAF score in the second Delphi round. The same strong correlation has been found in previous needs assessment studies in other specialties.^{18 20}

It is a strength of this study that all departments involved in the paediatric specialist training programmes in Denmark were represented, indicating strong support from the departments and increasing the generalisability and credibility of the study results. Furthermore, this study used a well-known method for curriculum development¹² and followed international methodological criteria for reporting of Delphi studies.¹⁴ Additionally, a strength of the Delphi process is that surveys are sent electronically which reduces the risk of participants influencing each other and allows participants across different geographical locations to participate.

We are aware of the limitations in this study. The disease panorama is different from country to country, influencing the needs and some of the procedures on the list may not be transferable to other countries. However, we believe that the generalisability of this study is quite high, particularly because the top five procedures, on our final prioritised list represents core procedures in acute situations and therefore may be transferable. In perspective, it would be interesting to perform the need assessment in other countries and compare results. Another limitation is the three-round survey design, which takes time for the participants to respond to, and therefore, may have caused some potential participants to decline participation. The response rate in this study varied in the three rounds, with the lowest response rate in Delphi round 2. Overall, we had high response rates as seen in similar studies.^16–22 In relation to skills in ultrasound, we have noted above that a change in experience and practice might change ranking of ultrasound skills over time. This may also relate to other skills, both those included and not included in the current report, including views on what is feasible in paediatric skill training may thus change over time. As such, the current results may reflect what is perceived as skills needed today and not skills needed in the future. Educating the future’s clinicians, it is important to bear future needs in mind. Asking the respondents to focus on this aspect may have shifted their responses and priorities somewhat, however, the acute skills rated at the top are unlikely to change markedly over time.

It is important to note that only technical skills for simulation-based procedural training were asked for and identified in this study. Factors that impact training such as communication, leadership and other non-technical skills should be incorporated when developing more team-based simulation-based programmes. Finally, it is important to acknowledge that we only explored which procedures to train in a simulation-based environment. Exploration of other important factors in curriculum development such as volume of training and how to implement training is outside the scope of the current study.

Implementing simulation-based training is associated with economic costs related to doctors spending working hours away from patients and sometimes to training facilities and equipment. The costs involved in the development and implementation of simulation-based training remains incomplete.³¹ However, the gains related to better training in human and economic terms should also be considered. The cost-effectiveness of simulation-based training in paediatrics according to the procedures identified in this study remains to be studied.

In conclusion, this needs assessment identified and prioritised a list of clinically relevant technical procedures suitable for simulation-based training in paediatrics. The procedures represent core competencies in paediatric specialist training that should be practiced in a simulation-based environment before applied on patients. Educators may use this list to develop, plan and implement simulation-based training curricula for paediatric trainees.