The European Respiratory Society led training programme improves self-reported competency and increases the use of thoracic ultrasound

Abstract

Thoracic ultrasound has become a well-implemented diagnostic tool for assessment and monitoring of patients with respiratory symptoms or disease. However, ultrasound examinations are user dependent and sufficient competencies are needed. The European Respiratory Society (ERS) hosts a structured and evidence-based training programme in thoracic ultrasound. This study aimed to explore and discuss the self-reported activity and self-reported competency of the participants during the ERS course.

Online surveys were sent to the training programme participants before the second part of the course (practical part of the course), and before and 3 months after the third part of the course (final certification exam). A total of 77 participants completed the surveys. The self-reported frequency of thoracic ultrasound examinations increased during the course, and in the final survey more than 90% of the participants used thoracic ultrasound on weekly basis. The self-reported competency (on technical execution of the thoracic ultrasound examination and overall competency) also increased.

The ERS thoracic ultrasound training programme forms the basis of broad theoretical knowledge and sufficient practical skills that seem to lead to behavioural changes, whereby a large proportion of the participants implemented ultrasound in their clinical practice.

Tweetable abstract

Structured and evidence-based training programme in thoracic ultrasound https://bit.ly/46rkKpw

Background

More and more clinicians use thoracic ultrasound (TUS) because of its high diagnostic accuracy and ability to answer clinically relevant point-of-care questions at the bedside [1, 2]. However, ultrasound is user dependent and sufficient theoretical and practical competencies are needed to be able to achieve high diagnostic accuracy [3] Previously, recommendations for education and training included an arbitrary number of ultrasound examinations be performed [4]. In recent years training programmes including formative and summative assessments have been presented advocating for mastery learning [5–7].

At the core of the mastery learning concept is the idea that every candidate can learn a competency, although the pace of learning will differ between people due to several factors like previous experience, reception of the teaching style, and motivation [8, 9].

A three-step training programme in TUS has been developed by the European Respiratory Society (ERS) [10]. The training programme consists of three parts, beginning with a theory-based module with material available online for self-study, including TUS images and videos, and ending with a theoretical test. The second part is a practical hands-on course with subsequent TUS applications integrated into students' usual clinical practice. For the third and final part, students must pass an observed objective structured clinical examination (OSCE) in achieve final certification [11].

Despite the above, little is known about the effect of the ERS training programme on participants' use of TUS in their clinical daily life, how comfortable they are with the procedure during the training programme, or whether there is a correlation between the final OSCE results and the frequency with which ultrasound examination performed.

The primary aim of this study was to explore the self-reported activity and self-reported competency assessments during the TUS course (prior to part 2, prior to part 3 and 3 months after completion of the course). We also sought to investigate the correlation between OSCE scores and the self-reported activity and self-reported assessment.

Methods

Study design and setting

The study was carried out as a cross-sectional study based on online surveys sent out to participants of the ERS TUS training programme between September 2022 and January 2023.

The survey was sent to one group of participants before part 2, and another group of participants before part 3 and to the same group 3 months after the part 3 (see figure 1).

FIGURE 1.

Overview of the components of the European Respiratory Society thoracic ultrasound training programme and time of surveys.

Questionnaire and variables

The survey was developed in two phases. An initial pilot study was conducted by the author group, which consisted of TUS and medical educational theory/implementation experts. The questionnaire was then tested on a study group of physicians training in ultrasound, all of whom were radiology residents at Odense University Hospital, Denmark. Subsequently, minor revisions were made to increase readability of the questionnaire.

In addition to demographic information (gender, age, country, and current career stage), the participants of the ERS TUS training programme were asked to answer the following.

• How often they had performed TUS within the last 3 months (one answer possible).

• In which setting(s) they had used TUS (more than one answer possible).

• For which patient categories they had used TUS (more than one answer possible).

• If they had performed ultrasound examinations other than TUS.

• On a five-point Likert scale (1: very poor; 2: poor; 3: fair; 4: good; 5: very good), how they would rate their technical skills (execution of the ultrasound examination) (one answer possible).

• On a five-point Likert scale, how they would rate their overall TUS competence (including technical execution, interpretation and diagnoses) (one answer possible).

The final questionnaire is available in the supplementary material. No questions were asked regarding ultrasound-guided procedures (e.g. pleural drainage or thoracentesis).

Data collection

Data collected before participants' part 3 OSCE was carried out on paper and was handed out before entering the examination. Further distribution of the survey was undertaken online, using SurveyMonkey (https://www.surveymonkey.com). Results were managed in Microsoft Excel (Microsoft Corporation).

Statistical analyses

The study demographics and characteristics were expressed as mean±sd or frequency (%). A t-test was used to compare self-assessment means between the part 2 group and the before part 3 OSCE and 3 months post-part 3 group. A paired t-test was used to compare the repeated self-assessment means between participants before the part 3 OSCE and 3 months post-part 3. All statistics were performed using Stata 18 (StataCorp LLC., College Station, TX, USA).

Results

Participants

A total of 77 participants filled out the questionnaire: one group of 32 participants before part 2 of the TUS course and another group of 45 participants before part 3 of the TUS course, of which 29 participants completed the 3 months post-part 3 TUS course follow-up. Participant demographics are presented in table 1.

TABLE 1.

Participant characteristics

	Total	Part 2	Before part 3	After part 3
Participants, n	77	32	45	29
Age (years)	37.3±7.7	37.7±8.7	37.1±7.1	38.3±7.8
Gender
Female	30 (39.0%)	14 (43.8%)	16 (35.6%)	8 (27.6%)
Male	47 (61.0%)	18 (54.2%)	29 (64.4%)	21 (72.4%)
Career stage
Not yet under specialisation	3 (3.9%)		3 (6.7%)	2 (6.9%)
Under specialisation	51 (66.2%)	23 (71.9%)	28 (62.2%)	19 (65.5%)
Specialised physician	22 (28.6)	9 (28.1%)	13 (28.9%)	7 (24.1%)
Not a medical doctor	1 (1.3%)		1 (2.2%)	1 (3.5%)
Specialty working in
Respiratory medicine	63 (81.8%)
Internal medicine	2 (2.6%)
Emergency medicine	4 (5.2%)
Paediatrics	2 (2.6%)
Cardiology	1 (1.3%)
General practitioner	1 (1.3%)
Thoracic surgery	1 (1.3%)
Anaesthesiology and ICU	2 (2.6%)
Other, not specified	1 (1.3%)
Country
Portugal	14 (18.2%)
Greece	9 (11.7%)
UK	8 (10.4%)
Germany	5 (6.5%)
Ireland	4 (5.2%)
Romania	4 (5.2%)
Spain	4 (5.2%)
Albania	3 (3.9%)
Brazil	3 (3.9%)
Poland	3 (3.9%)
Austria	2 (2.6%)
Belgium	2 (2.6%)
India	2 (2.6%)
Malaysia	2 (2.6%)
Columbia	1 (1.3%)
Croatia	1 (1.3%)
Denmark	1 (1.3%)
Egypt	1 (1.3%)
France	1 (1.3%)
Iceland	1 (1.3%)
Israel	1 (1.3%)
Lebanon	1 (1.3%)
Luxembourg	1 (1.3%)
Palestine	1 (1.3%)
Saudi Arabia	1 (1.3%)
Switzerland	1 (1.3%)

Data are presented as mean±sd or n (%), unless otherwise stated. ICU: intensive care unit.

Most of the participants were doctors training in respiratory medicine, but surveys were completed by a range of grades from pre-specialisation trainees to professors in respiratory medicine, as well as a physical therapist working with patients with respiratory difficulties. The participants represented eight different specialities (table 1).

Most participants answered that they used TUS on inpatients hospitalised with respiratory illnesses (n=65, 84%), with fewer using it in the emergency department (n=33, 43%), endoscopy suite (n=29, 38%), or outpatient clinic (n=27, 35%). Sporadically, it was used in the intensive care unit (n=7, 9%) and in a prehospital setting (n=3, 4%).

The patient groups examined with TUS by the participants emerged during the course. At part 2 most of the participants used TUS solely for assessment of patients suspected of pleural effusion (n=23 out of 32, 72%); however, 3 months after part 3 most participants used TUS for more than one patient category (n=23 out of 29, 79%). Most often for patients with suspicion of pulmonary infection, pleura effusion and/or unclear patients with general dyspnoea.

29 participants (38%) had experience in ultrasound applications other than TUS, covering point-of-care cardiac ultrasound, basic abdominal ultrasound, ultrasound of lower extremities for deep vein thrombosis, and vascular access.

Frequency of TUS examinations

The frequencies for using TUS in the three groups are shown in figure 2. Overall, as participants advanced in the course an increase in use was observed. The number of participants using TUS several times a week began at 44%, increased to 62%, and at 3 months after part 3 rose again to 66%. After the 3-month survey, all participants used TUS and more than 90% performed TUS weekly.

FIGURE 2.

Frequency of how often the participants used thoracic ultrasound. All participants use thoracic ultrasound and more than 90% use thoracic ultrasound weekly 3 months after the course.

Competence and self-assessments

The self-reported competence levels of the participants' technical skills and overall competence were nearly equivalent. As seen in figure 3, there was a slight increase in the self-reported skill assessments and overall competence assessments during the course, from part 2 to part 3 and then subsequently at 3 months after part 3. There was no statistically significant difference between the technical self-reported assessments between part 2 and before the part 3 OSCE (3.25±0.98 versus 3.6±0.63, maximum 5 points, p=0.059).

FIGURE 3.

Mean self-assessment score in each survey. Participants rated their own performance on a scale from 1–5 in each survey sent out. A slight increase in mean self-assessment score was seen during the course. The technical skill mean±sd scores were 3.25±0.98, 3.6±0.63, and 4.3±0.47, respectively (green line). The mean±sd overall competence self-assessment scores were 3.25±0.88, 3.6±0.62, and 4.17±0.54, respectively (grey line).

In the self-reported overall competence assessment, a statistically significant difference was seen from part 2 to part 3 (3.25±0.88 versus 3.6±0.62, p=0.043).

Likewise, a significant increase in self-reported assessment was seen between before the part 3 OSCE and at 3-month follow-up in both the technical skill ratings (3.6±0.62 versus 4.3±0.47, p<0.01), and in the self-reported overall competence assessment (3.6±0.62 versus 4.17±0.54, p<0.01).

OSCE scores and self-assessed rating

There was no correlation between the OSCE scores as a dependent variable and the self-assessed ratings as the independent variable (see the scatterplots in figure 4).

FIGURE 4.

The correlation between the part 3 final objective structured clinical examination (OSCE) score (independent variable) and a) the self-reported technical skill scores and b) self-reported overall competency score. No correlation was seen between the final examination score and the self-reported scores.

Discussion

We aimed to gather responses from participants at parts 2 and 3 of the new ERS TUS training programme, as well as 3 months after completing the certification. The results revealed a self-reported increase in the frequency of performed ultrasound examinations, with the vast majority of participants using TUS on at least a weekly basis. A slight increase in self-reported technical skills and overall TUS competence was also noted.

The increase in frequency of TUS examinations performed would suggest that the participants use their obtained skills and that the training programme was useful. Kirkpatrick's model of educational evaluation is well-recognised and commonly used for assessment of training programmes [12]. The model of continuing professional development consists of four levels and evaluates the trainees' reactions to the training programme (level 1), course learning outcomes (level 2), participant behaviour and/or attitudinal changes (level 3), and impact on the institution as a consequence of learning (level 4). Ideally, learning should lead to behavioural changes and subsequently to institutional changes such as, in the case of TUS, faster diagnostics and improved patient-related outcomes.

The ERS TUS training programme is, as mentioned, a three-step evidence-based training programme with proven solid validity evidence of the included assessments [10, 11]. The training programme emphasises mastery learning [8] – meaning that all participants will be able to pass the training programme but at different learning paces. In this study, we sought to explore course participants' behaviour and explore if they were able to transfer the newly learned competence into their clinical setting. The results suggest a self-reported increase and improvement in both self-reported technical skills and overall self-reported competence. A limitation of the study is the cross-sectional design, a longitudinal study design would have been more optimal, where all participants were followed from signing up to part 1 until after part 3. However, participants do not always follow consecutive parts of the training programme straight away, most participants take part 1 then wait a year or two before continuing to the next part, etc. The longitudinal part of the results, the group investigated before the final certification (part 3) and 3 months after part 3 also revealed a slight increase in participants using TUS weekly. Self-reported data are used as a proxy of competence even though the authors are aware that self-reported data do not always reflect real life, especially regarding skills and competence assessment and the well-known Dunning–Kruger effect reveals the cognitive bias in which people with limited competence over- (or under-) estimate their abilities in a specific competence [13, 14]. Therefore, the results presented are managed with caution. For an exact and objective assessment of the participants' competencies structured and evidence-based tools are needed, but unfortunately it was not possible to include them in this study. In the current study, we also did not explore sensitivity and specificity of the diagnostic performance, institutional changes, or patient-related outcomes. Several studies have provided evidence that the use of basic TUS (for example, in emergency departments) can lead to faster diagnostics and/or improved patient pathways [15, 16]. During the recent coronavirus disease 2019 pandemic, TUS was used for initial assessment of patients, with ultrasound changes able to some extent predict risk of worse patient outcomes [17, 18]. To explore patient-related outcomes of an ultrasound training programme would require a significant controlled trial and a standardised environment that was not practical following the ERS programme, which attracts participants from all over the world.

The ERS TUS training programme was created based on recognised principles, with solid validity evidence indicating that there were sufficient training outcomes corresponding to a successful Kirkpatrick's level 2 [10, 11]. For level 3, behavioural changes, many variables affect the implementation and use of TUS in the daily clinical life – both on an institutional and personal basis. For example, the availability of ultrasound equipment and the opportunity for discussion and supervision by colleagues. Also, institutional political directions and behaviour influence TUS use with, for example, many centres finding limitations regarding how examination findings can be documented and images stored for medical records, either electronic or otherwise. Implementation theory and theory of skill transfer have been a focus of discussions and research in recent years to ensure the integration of newly obtained skills into a clinical setting [19]. Studies in TUS indicate that the procedure is not very difficult technically, but requires sufficient theoretical knowledge and the capability to integrate the ultrasound findings into a clinical context [3]. To the best of our knowledge, no studies have been published exploring implementation or transfer of TUS skills from training programmes to the clinical context. However, in other ultrasound examinations, simulation-based training programmes have been explored and found to positively correlate with higher objective skills assessments and patient-reported outcomes. Simulation-based training programmes in endobronchial ultrasound increased clinical objective assessment scores [20], and the same increase was seen in simulation-based abdominal ultrasound training [21]. Concerning patient-reported outcomes, one study explored the effect of simulation-based training on patient-reported outcomes in vaginal ultrasound. This study found an increase in quality care, a decrease in discomfort, and a reduction in the need for trainee supervision [22].

TUS is, like other point-of-care ultrasound examinations, a valuable tool that can rapidly confirm or exclude specific pathologies, and can assist the physician in clinical decision-making for patients with respiratory symptoms. The ERS TUS training programme can form the basis of a broad theoretical knowledge and sufficient practical skills that seem to lead to behavioural change in which a large proportion of participants implement TUS in their clinical practice.

Supplementary material

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Supplementary material EDU-0160-2023.supplement ^{(68.2KB, pdf)}