A competency-based test of bronchoscopic knowledge using the Essential Bronchoscopist: An initial concept study

Abstract

Background and objective

Competency-based training and assessment are increasingly replacing the traditional structure- and process-based model of medical education. The web-based Essential Bronchoscopist (EB) is an open access, laddered, competency-based curriculum of question-answer sets pertaining to basic bronchoscopic knowledge, accessible in five languages. The purpose of this study was to use consensus to evaluate whether question-answer sets (items) from the EB could provide material from which to devise competency-based tests of bronchoscopic knowledge that could be used in countries with different health-care environments.

Methods

Ten junior and five senior bronchoscopists in Argentina and the USA were asked to identify the material in each of the 186 items of the EB as either ‘not necessary’, ‘necessary’ or ‘absolutely necessary’ for inclusion in a competency-based test of bronchoscopic knowledge. The kappa statistic was used to analyse inter-rater agreement.

Results

More than 80% of the 30 participants rated 150 of 186 items as necessary or absolutely necessary. Seventy items were rated as absolutely necessary by more than 50% of participants and as necessary or absolutely necessary by more than 90%. There was no statistically significant difference between items selected, based on level of training or environment of practice.

Conclusion

Consensus among junior and senior bronchoscopists from two culturally different health-care environments demonstrated that question-answer sets from the EB can be used to provide material for competency-based tests of bronchoscopic knowledge.

INTRODUCTION

In the past decade competency-based instruction has been recognized as improving medical trainee performance and enhancing patient safety, particularly in invasive procedural skills.¹ This has led to a new wave of competency-based curricula being used by medical educators in different fields.^2–5 In the UK, the Royal Colleges that represent 29 medical specialties and subspecialties have published respective competency-based curricula.⁶ In the USA, accountability to the public to ensure patient safety and physician competence has driven the Accreditation Council for Graduate Medical Education, the American Board of Medical Specialties and the Association of American Medical Colleges to initiate competency-based curricula for all physicians.^2,4,7,8 These new competency-based guidelines have been incorporated into the residency training requirements for Residency Review Committees.^7,8 The Association for Medical Education in Europe has likewise developed guidelines for competency- and outcome-based education.⁹ Medical educators in many other nations have similarly embarked on competency-based projects.^10–13

Assessment methods and tools determine the worth of an instructional program¹⁴ and hence competency-based methods have been developed to evaluate the utility and efficiency of competency-based instruction.^15–17 The need for this is under scored by studies demonstrating a poor correlation between faculty evaluations and competency-based assessments.^18,19

Web-based e-curricula are transforming the traditional educational model in differing fields of science, engineering and medicine.^20–22 In medicine, e-curricula are already common in radiology,^23–25 radiation oncology,²⁵ anaesthesiology,²⁶ emergency medicine,²⁷ evidence-based medicine,^28,29 and in surgical and procedural specialties.^30,31 Strategies for better delivery of e-curricula and their impact on patient outcomes have been investigated.^32,33 The next generation of physicians is increasingly accustomed to using web-based search engines, online resources and electronic learning materials rather than traditional texts.

More than half million flexible bronchoscopies are performed annually in the USA alone.³⁴ They are performed by chest physicians, anaesthesiologists, thoracic and trauma surgeons and otorhinolaryngolgists. New bronchoscopy-based procedures such as navigational bronchoscopy, bronchial thermoplasty and endobronchial valve insertion, as well as interventional bronchoscopic procedures such as transbronchial needle aspiration, flexible and rigid bronchoscopic resection, laser bronchoscopy and stent insertion, are also increasingly performed. Despite this, significant heterogeneity exists in the quality and extent of bronchoscopy training.^35,36 This heterogeneity, along with an emphasis on competency-based instruction, has stimulated new learning curricula and educational initiatives for the teaching of bronchoscopic skills.^35–39

The only currently available web-based curricular resource pertaining to bronchoscopy education is the Essential Bronchoscopist (EB), hosted by the University of California (Irvine). The EB is an open-access, laddered, competency-based curriculum of basic bronchoscopic knowledge that can be accessed free of charge in five different languages: English, French, Spanish, Portuguese and Japanese (http:// bronchoscopy.org/ under the link ‘Essential Bronchoscopist’). It has been officially endorsed as a complementary educational tool by several national and international bronchology and pulmonary organizations (from Argentina, Singapore, Belgium, Malaysia, Spain, France and Brazil and by the World Association for Bronchology).

The EB consists of six modules, each with a module-specific competency-based learning objective, totalling 186 multiple-choice question-answer sets, viewable online (Fig. 1a,b) and also download able as portable document files (pdf). The question-answer sets contain information pertaining to the major topics presented in traditional textbooks of bronchoscopy (anatomy and airway abnormalities, patient preparation, indications, contraindications and complications, techniques and solutions to technical problems, lung cancer and infections, BAL, lung biopsy techniques, therapeutic and interventional bronchoscopy, anaesthesia and medications, equipment and its maintenance, as well as history and education). The aim of the EB is to complement the conventional apprenticeship model of training in bronchoscopy.

Figure 1.

(a) Screenshot from the Essential Bronchoscopist, Q/A set regarding correlation between fluoroscopic and bronchoscopic landmarks. (b) Screenshot from the Essential Bronchoscopist, Q/A set regarding laryngeal lesions and artifacts.

At present, there are no comprehensive validated tests of the theoretical or practical knowledge of bronchoscopy. The purpose of this study, therefore, was to systematically evaluate whether the question-answer sets in the EB could be used to devise tests of bronchoscopic knowledge, and whether such a test would be applicable in different health-care environments. The study was designed to explore (i) what information contained in the EB was considered essential, (ii) whether physicians’ expertise impacts on what they perceive as necessary knowledge, and (iii) whether physicians from two very different medical cultures have different perceptions of what is necessary bronchoscopic knowledge (Fig. 2).

Figure 2.

Creating a competency-based test of bronchoscopic knowledge. Competency-based curricula and the tools required to assess their efficacy, along with web-based education, are major driving forces in medical education. Knowledge is the base of Miller's clinical skills assessment pyramid; it is the fundamental prerequisite of competence, which in turn leads to optimum performance and correct actions.

METHODS

The study population consisted of 30 bronchoscopists in the USA and Argentina. These two countries differ in many of the factors affecting the practice of medicine, including but not limited to culture, language, availability of technology, average incomes of patients and physicians, and the financing and administration of health care. In each country, 15 pulmonary clinicians (five senior physicians and ten trainee chest physicians) practising bronchoscopy were randomly selected from among a larger list of available bronchoscopists. No participant had contributed to the creation of the EB. The EB was not part of the training curriculum in either country, and none of the participants had completed all six modules.

Recruitment of senior chest physicians and pulmonary trainees was through a convenience sample methodology. Doctors who agreed to participate were given a packet that contained a consent form, a summary explaining the rationale, methodology and instructions for the study, a CD-ROM containing all six modules of the EB in the appropriate language (English or Spanish), and six questionnaires corresponding to each module of the EB. The instructions asked the participant to read each of the 186 question-answer sets, and decide whether the information included in each item was: (i) ‘not necessary’ for a comprehensive test of bronchoscopic knowledge, (ii) ‘necessary’ for such a test, or (iii) ‘absolutely necessary’ for such a test, because all bronchoscopists regardless of their experience should know this information.

The participants returned the completed questionnaires within 4 weeks, and the data were entered into a database for statistical analysis. Questionnaires were anonymous and were collected separately from the informed consent forms; the only information included was whether the participant was a trainee or senior chest physician, and if a trainee, their level of training.

Statistical methods

Summary ratings for each of the 186 items were established separately, using descriptive statistics such as frequencies and percentages. A ‘best items’ subset of question-answer sets (Group 1) was selected based on whether (i) more than 50% of all raters scored the material as absolutely necessary and (ii) more than 90% scored the material as necessary or absolutely necessary. An ‘intermediate items’ subset (Group 2) was selected based only on criterion (ii) above, that is, being selected by more than 90% of raters as necessary or absolutely necessary. Items that did not meet the criterion for Group 2 were relegated to Group 3 (Table 1). A non-response was considered by default as a rating of not necessary.

Table 1.

Percentage of question/answer items rated by clinicians as not necessary, necessary or absolutely necessary, in the three groupings

	Group 1, best items (n = 70) % (range)	Group 2, intermediate items (n = 38) % (range)	Group 3, remaining items (n = 78) % (range)	Total (186 items) % (range)
Not good/not necessary	2.5 (0–7)	4.3 (0–7)	21.7 (3–77)	10.9 (0–77)
Good/necessary	32.4 (10–47)	53.9 (43–70)	44.4 (17–70)	41.8 (10–70)
Absolutely necessary	65.0 (53–90)	41.7 (27–50)	32.7 (0–63)	46.7 (0–90)
Total responses	100	100	100	100

Agreement between trainee and senior chest physicians and between clinicians in the USA and those in Argentina was investigated using chi-square tests. A significance value less than 5% (unadjusted for the number of comparisons) was used as the criterion for further review of an item for inclusion in the final subset. Items were excluded from the final subset only if there was a significant difference by location or level of expertise when responses were dichotomized (not necessary vs necessary or absolutely necessary).

In addition, the 186 question-answer sets of the EB were grouped into 16 categories by subject (Table 2). The degree of agreement between the participants on which items in each category were considered necessary or absolutely necessary was analysed separately for every category.

Table 2.

Average percentage of question/answer items rated necessary or absolutely necessary within each subject matter category

Subject matter category	Total number of items in category	Average percentage rated necessary or absolutely necessary	Range	Number of Group 1 ‘best Items’ in category	Average percentage rated necessary or absolutely necessary
Anatomy	27	91.36	60–100	9	96.67
Atlas/recognition of airway abnormalities	23	93.48	80–100	10	97.00
BAL	6	96.67	90–100	5	98.00
Lung cancer	5	95.33	80–100	4	99.17
Endobronchial biopsy	3	94.45	90–97	2	96.67
Equipment/ maintenance	17	71.77	37–93	0	—
History of bronchoscopy	2	28.33	23–33	0	—
Indications/ contraindications/ complications	22	94.55	77–100	12	97.22
Infections	6	71.67	57–100	0	—
Therapeutics/ interventional bronchoscopy	13	94.36	90–100	2	95.00
Patient preparation	6	96.11	90–100	4	99.17
Medications/ anaesthesia	6	90.56	77–100	3	97.78
TBLB	6	93.89	80–100	3	100.00
TBNA	9	90.00	83–97	3	96.67
Teaching	6	63.33	43–93	0	—
Technique/ technical solutions	29	90.46	50–100	13	96.67
Total	186	88.53	23–100	70	97.33

TBLB, trans-bronchial lung biopsy; TBNA, trans-bronchial needle aspiration.

RESULTS

The majority of question-answer sets were rated necessary or absolutely necessary by the majority of the 30 bronchoscopists (mean percentage rated necessary or absolutely necessary 88.5%, SEM 1.0%; range 23–100%). A total of 150 items out of 186 were rated necessary or absolutely necessary by more than 80% of participants, while 108 items received those ratings from more than 90% of participants. Only nine items were rated not necessary by 15 or more participants (≥50%), and 52 were rated not necessary by four or more participants (≥13%) (Fig. 3).

Figure 3.

(a) Number of question/answer sets rated necessary or absolutely necessary (broken down by percentage); (b) Number of question/answer sets rated absolutely necessary (broken down by percentage); (c) Number of question/answer sets rated not necessary (broken down by percentage).

Seventy items were rated as absolutely necessary (best items, Group 1) by more than half the bronchoscopists and at least necessary by more than 90% (Table 1), whereas 38 items were only rated as necessary by more than 90% (intermediate items, Group 2).

There was no significant disagreement between clinicians in the two locations for any items included in the best items or intermediate items subsets (Groups 1 and 2). A significant difference was noted, however, for seven of the 78 items in Group 3 (remaining items subset). Junior and senior level bronchoscopists differed significantly (P < 0.05) in their opinions on four items included in Group 1 and one in Group 2. Yet the only difference in opinion between junior and senior level clinicians on these items was in the percentage of study participants that rated the item as necessary compared with the percentage that rated the item absolutely necessary. Because none of these items was rated not necessary by more than one clinician, they were not excluded from the respective subsets.

There was no statistically significant difference in the way clinicians perceived the importance of the 16 different topic categories based on level of experience and location of practice (see summary of question-answer set ratings by subject area in Table 2). The majority of items from 12 of the 16 subject areas were consistently rated as necessary or absolutely necessary, and many of these met selection criteria for the 70 ‘best items’. Technical topics such as BAL, endo-bronchial biopsy, transbronchial lung biopsy, trans-bronchial needle aspiration, lung cancer, indications and patient preparation, anatomy and airway abnormalities and issues pertaining to technique and anaesthesia were high scorers among the categories of necessary knowledge. None of the 31 items from the remaining four subject areas (equipment and its maintenance, history of bronchoscopy, pulmonary infections and bronchoscopy education) met selection criteria based on rating agreement between the bronchoscopists.

DISCUSSION

By definition, competency-based education requires that each competency be teachable, learnable and measurable.⁴¹ This study, based on consensus of junior and senior bronchoscopists practising in two culturally different health-care environments, identified material from which to design written competency-based assessments of bronchoscopic knowledge. More than 80% of the sampled bronchoscopists considered the material included in 150 out of 186 question-answer sets as necessary or absolutely necessary knowledge. The content of the EB was further studied by determining whether a significant degree of agreement existed between clinicians practising bronchoscopy in two culturally diverse settings (USA and Argentina), and at different levels of expertise.

The high degree of inter-rater agreement among bronchoscopists at different levels of experience showed that what clinicians view as necessary knowledge for the safe practice of bronchoscopy did not change with experience.

Similarly, the high degree of inter-rater agreement among bronchoscopists from two different medical settings and cultures suggested that what clinicians view as necessary bronchoscopic knowledge did not differ significantly according to geographic, language, economic and educational divides. As more educators teach bronchoscopy in foreign countries, sensitivity to potential cultural and experiential differences among learners is important. For example, using the EB in Mauritania and Mozambique demonstrated that learners preferred being tutored rather than independent study.⁴²

Although most of the EB was seen as containing necessary knowledge by practising bronchoscopists, a large number of the sampled bronchoscopists in the present study did not regard topics pertaining to equipment and its maintenance, infections and teaching, as material that should be known.

This finding suggests that bronchoscopy educators, until recently, may have under-emphasized instruction in these topics. Education in these topics probably deserves greater attention and a lack of awareness could have an adverse impact on patient safety.

One limitation of this study was the small number of physicians and trainees who took part and the assessment of physicians from only two countries. Practical issues, including the amount of time required by participants to carefully review all 186 question-answer sets, precluded the use of a larger pool of participants from more countries.

There is a critical distinction between bronchoscopic knowledge and bronchoscopic competence. Knowledge, which is the base of Miller's pyramid for the assessment of clinical skills (Fig. 2), is the basis upon which the next three levels of the pyramid, competence, performance and action, rest.⁴⁰ Hence, in the same way that competence is essential for optimal performance and action, knowledge is the fundamental prerequisite for competence.

The results from this study demonstrated that question-answer sets from the EB provide material from which to devise tests of bronchoscopic knowledge. Validation studies of questions derived from this material need to occur to explore internal consistency, reproducibility and discriminant validity. Studies to develop and validate measurement tools to assess technical bronchoscopic skills⁴³ and problem-solving abilities are evolving and are necessary.