Exposure or expert? Setting standards for postgraduate education through a Delphi technique

Abstract

Objectives

The aim of this study was to determine if a syllabus of upper limb anatomy for emergency clinicians composed by an expert panel reflected clinical practice as experienced by those at whom the syllabus would be directed.

Methods

A three round Delphi study was performed using an expert group. We compared this with a measure of the exposure to anatomical concepts in the day to day practice of trainees in emergency medicine.

Results

In total, 404 separate anatomical facts relating to the upper limb were reviewed by both groups. There was poor agreement (κ = 0.348) between the expert group and the trainees.

Conclusion

We have shown disparity between what an expert group believes trainees should know and what trainees are actually exposed to in clinical practice. We believe this demonstrates that curriculum development must strike a balance between important (expert) and common (exposure) information. We have shown how an expert:exposure matrix may be used to inform curriculum development.

As with other specialties, postgraduate training in UK emergency medicine (EM) has now fallen under the auspices of the Postgraduate Medicine Education and Training Board. One of the key implications of this transition has been the requirement to develop curricula and assessments that are fit for purpose. The current assessment structure for trainees in EM comprises a membership examination at the start of specialist training, followed at the point of completion by the fellowship examination. Both examinations aim to establish key clinical competencies appropriate to the candidate's assumed experience. In addition the membership examination contains, within part A, a structured examination with multiple choice questions, testing knowledge of applied basic science.

The derivation of the content of the part A examination has followed historical collegiate trends, based as it is upon the views of an expert UK panel that includes emergency clinicians and examiners in other college examinations.

It is well established that models of curricula where trainees are themselves invited to help determine content can flounder because it is not possible to express opinions regarding subjects not yet known. Furthermore, the strengths of trainee involvement in curriculum planning are mainly related to the delivery of the required teaching to enable effective knowledge acquisitions rather than content.¹

However, there is a real possibility that where content is shaped by an “expert panel”, the assumed importance of a particular item is not in fact reflected in its clinical usefulness in practice. This concern rests upon the fact that expert panels may well comprise people who are removed from regular clinical practice and therefore unable to properly prioritise key knowledge requirements. In such circumstances, there may well be a viable role for trainees in terms of the refinement of curricular content based upon their experience of the need for basic science knowledge on the "shop floor".

Our aim in this study was to test this by determining if a syllabus of upper limb anatomy for emergency clinicians derived by an expert panel reflected clinical practice as experienced by those at whom the syllabus would be directed.

METHODS

We conducted a three round Delphi study using a panel of 10 experts in EM anatomy. We defined experts as being practising emergency physicians at specialist registrar (SpR) or consultant level with: (a) a higher qualification in anatomy and/or (b) ⩽6 months' experience as an anatomy demonstrator.

Round 1 consisted of a comprehensive list (404 items) of anatomical structures and relationships for the upper limb derived from a standard textbook of anatomy by the authors. Participants were asked to score these items on a three part scale: (a) not useful in clinical practice, (b) useful to know, and (c) essential to know. If there was >70% agreement in any one round we did not reiterate the question into the next round.

Having defined the opinions of the experts, we invited a group of 35 EM trainees to grade the same anatomical structures and relationships using the following criteria: (a) almost never used in my clinical practice (for example, once a year or less), (b) occasionally used in my clinical practice (for example, once a month), and (c) regularly used in my clinical practice (for example, weekly). The trainees were invited to give details of their level of training, but the vast majority chose to complete the questionnaire anonymously.

We compared the results of the expert group and the trainees by cross tabulation of the mode (most commonly occurring) for each anatomical structure or relationship and by plotting the mean score for each variable. We used the kappa statistic as a measure of agreement between the two groups.

RESULTS

All 10 SpRs and consultants recruited to the expert group and all 35 participants recruited to the trainee group completed both rounds of the study.

A scatter plot of the mean scores between the trainee and expert groups is shown in fig 1. In the determination of expert opinion, 79 items achieved consensus in the first round and were not reiterated. The mode values are shown in tables 1 and 2, and the mean values have been plotted in fig 2. A complete set of data is available online (http://www.emjonline.com/supplemental).

Figure 1 Mean scores.

Table 1 Results of the expert group in terms of final modal scores.

Final mode		Experts		Trainees
1		129 (31.9%)		258 (63.9%)
2		87 (21.5%)		35 (8.7%)
3		188 (46.5%)		111 (27.5%)

Table 2 Comparison of modal scores between the two groups.

Trainees group		Expert group						Total
		1		2		3
1		124		71		63		258
2		1		9		25		35
3		4		7		100		111
Total		129		87		188		404

κ = 0.348 indicating a low level of agreement between the two groups.

Figure 2 Expert:exposure matrix

DISCUSSION

Statement of principal findings

Our findings demonstrate a significant difference in agreement between the two groups. We have also demonstrated that the trainee group overall score was less than that of the experts. However, there are several points to be borne in mind.

• Trainees may be demonstrating a lack of ever having known certain anatomical information. If they were never taught, they can never know it (an unknown unknown).

• Trainees may subconsciously know important anatomical information but not be able to express it in the terms given. For example, experts scored the location of the long thoracic nerve as a 3 but trainees rated it as 1. Anatomically, it is important that emergency physicians know that this may be damaged if the incision for a chest drain passes the mid‐axillary line (that is, knowing that this line should not be passed, and knowing the anatomical reason why). However, if a trainee knows never to pass the line, but does not know why, then the clinical outcome may be similar but there is no demonstration of understanding (that is, the trainee knows not to do it, but not why). This is arguably a lower level of understanding.

• Trainees may have never known and be unaware of the importance of the information. This may lead to a poor technique, and in the above example, may leave the patient with a winged scapula.

• There is a difference between exposure and need to know. Our findings show that there are a number of conditions where anatomical knowledge was considered to be important, but whose frequency in clinical practice may be low. For example, knowledge of the branch to the phrenic nerve from the brachial plexus may be rarely recalled in daily clinical practice, but is arguably an essential piece of knowledge when dealing with spinal injuries.

• The expert group may have rated certain anatomical information highly in areas where they consider it vital to their practice. In contrast, the trainees may not have the anatomical knowledge but may retain safety through alternative, perhaps more pertinent, knowledge. For example, an anatomist may rate the characteristics of the deltopectoral triangle as essential as it is a landmark for subclavian venous puncture, identifying it through palpation of the underlying muscles. However, the same location may be identified by measuring the distance along the clavicle (approximately two thirds of the distance from the sternoclavicular joint) without any knowledge of what is actually “there”, apart from the knowledge that it is a pertinent landmark.

Strengths and weaknesses of the study

We believe our definition of expertise is valid as it uses a group who will have had a high level of anatomical knowledge in the past and our assumption is that they will have retained this knowledge. Our second group, SpRs, are educationally active clinicians operating at an autonomous level in the speciality, on average for over 30 hours per week of direct clinical contact. We chose SpRs in preference to consultants as they have on average more clinical exposure and also because their training needs are directly applicable to the aims of this study.

Owing to concerns among the trainees about identification, we delivered the questionnaire anonymously and so were unable to determine the specific characteristics of the trainees who completed the study (although all must have completed at least 1 year of EM training). With such a small study population, subgroup analysis in the trainee group would not have been valid even if such information had been available.

Strengths and weaknesses in relation to other studies, particularly any differences in results

Although the Delphi² technique has been used in other settings to define required knowledge,³,⁴ we believe that this is the first time that a panel of experts has been used in this way with a comparison against clinical practice as experienced by clinicians working towards higher qualifications.

Practical implications

This study has highlighted the practical implications of using this methodology to derive a basic science syllabus for EM. We chose to focus on one aspect of anatomy, the upper limb. If it were to be rolled out in its current form to devise a complete document, a larger number of people, with varying degrees of expertise, would have to be recruited. In addition, defining expertise in other subjects such as physiology or pathology may be more difficult than in anatomy.

Another potential difficulty with this approach is that it may not be possible to derive curricular content that does not include some of the low scoring topics. For example, certain branches of the brachial plexus were considered of low value in learning, but it would be difficult to teach the anatomy of the brachial plexus in a piecemeal fashion.

Meaning of the study: possible mechanisms and implications for clinicians or policymakers

A curriculum with content that is fit for purpose should aim to equip trainee emergency physicians with anatomical knowledge that reflects balanced clinical practice (exposure) in the context of expert panels. One method of expressing this balance can be seen in fig 2. The matrix allows a balance to be drawn between topics that are commonly seen (exposure rating) and those that are important to particular clinical problems (expert rating). The focus of curricular delivery and assessment can be mapped against their overall importance as demonstrated by the colour codes red/orange/yellow/green/blue.

We also believe that this model of curricula development illustrates the importance of using learners themselves in curricula design, both to inform learning outcomes and in the future to assess and moderate such curricula.⁵

Unanswered questions and future research

We have only been able to classify the anatomical structures of the upper limb. Further work must address the remaining areas of basic science knowledge and determine an appropriate line on the continuum of the expert:exposure matrix upon which to base content.

Work is now needed to develop curricular content for the remainder of the key basic sciences and to devise a method of translating this into an accessible format for trainees, trainers, and examiners. In addition, although this methodology is capable of defining the factual outcomes that are desirable with regard to anatomical knowledge, it cannot tell us how to teach the subject, as it may in fact it may be necessary to discuss elements of no perceived value in order to teach important facts in the right context. We must guard against the results of studies such as this being used as an atomistic approach to learning (whereby the individual elements are learned but the overall structure of the subject is distorted), but must ensure that a holistic approach is maintained in the way that we educate and assess our trainees.⁶

Further curricula design must use the findings of this type of project with appropriately constructed educational aims and objectives⁷ that provide emergency physicians with clinically relevant anatomical knowledge.

CONCLUSION

We have shown disparity between what an expert group believes trainees should know and what trainees are actually exposed to in clinical practice. We believe this demonstrates that curriculum development must strike a balance between important (expert) information and common (exposure) information. We have shown how a expert:exposure matrix may be used to inform curriculum development.

Abbreviations

EM - emergency medicine

SpR - specialist registrar