Essential Facets of Competence That Enable Trust in Graduates: A Delphi Study Among Physician Educators in the Netherlands

Abstract

Background

There is a need for valid methods to assess the readiness for clinical practice of recently graduated physicians. To develop these methods, it is relevant to know the general features of trainees' performance that facilitate supervisors' trust in their ability to perform critical clinical tasks.

Objective

To discover such essential facets of competence (FOCs), based on the opinion of experienced physician educators.

Methods

We conducted a Delphi study, consisting of 2 rounds, among 18 experienced physician educators in the Netherlands. Mean, standard deviation, level of agreement, and skewness were calculated for the importance of FOCs for making entrustment decisions. The study yielded a list of 25 FOCs.

Results

In the first round, means were between 6.50 and 7.00 on a 7-point Likert scale (SD, 0.42–2.18); in the second round, means ranged from 5.45 to 6.90 (SD, 0.3–2.02). The level of agreement was high for 92% of the FOCs in the first round and 100% of the FOCs in the second round.

Conclusions

Our Delphi study found consensus among experts about FOCs that are important for clinical entrustment decisions.

Editor's Note: The online version of this article contains the following 3 tables: adapted calculation levels of agreement according to the method of De Loe, expert ratings for completeness and clearness, and describing judgments for importance for entrustment decisions of facets of competence in Delphi study ^{(105KB, doc)} .

What was known

Clinical supervisors need information on attributes of clinical trainees to facilitate decisions to entrust trainees with clinical tasks with indirect supervision or “after the fact” supervision.

What is new

A Delphi study of experts in the Netherlands found consensus on facets of competence that are important for clinical entrustment decisions.

Limitations

Small sample size and use of particular competency frameworks may limit the ability to generalize.

Bottom line

Experts in physician education agreed on general attributes of competence for medical trainees that contribute to a robust approach that allows supervisors to make clinical entrustment decisions.

Introduction

Obtaining a correct impression of medical residents' and fellows' readiness for clinical practice is important for medical educators. This requires a valid method for the assessment of clinical competence, a topic that has occupied the minds of medical educators for decades. Despite multiple reviews and authoritative proposals,^1–6 clinical educators, evaluators, and examiners in practice still have difficulty assessing medical trainees.⁷ Instruments to assess separate domains of competence in the workplace are scarce⁸ and instruments that focus on specific tasks usually rely on simulated conditions to ensure standardization. The mini-clinical evaluation exercise⁹ and other direct-observation instruments¹⁰ are valuable feedback instruments but the documentation of their psychometric properties is limited to date.^10,11 One problem may be a lack of validity of those assessment approaches because they do not directly consider the important question of “whether a medical trainee is ready for independent practice” but rather focus on the assessment of particular skills. Traditional checklists established for these purposes may not capture the essence of such entrustment decisions as they insufficiently align with this essential construct.¹² In clinical settings supervisors decide on a daily basis whether to trust a medical trainee with a specific task and to what extent supervision is needed.^13–16 Specific tasks pertain to a given procedure or skill that is to be carried out; general features pertain to task-independent characteristics. These general features can be called facets of competence (FOCs).

The aim of this study was to uncover general features of trainees that facilitate clinicians' trust of trainees with critical clinical tasks. The research question was: What do experienced clinical educators consider as essential facets of competence that determine decisions to entrust a trainee with critical clinical tasks? Our goal was not to answer this question for specific tasks, but for critical clinical tasks in general. The answer to this question is relevant when developing an instrument to evaluate medical graduates' readiness for practice.

Methods

Design

We used a Delphi technique to investigate consensus and the amount of agreement among experts. The Delphi technique is a widely accepted method for identifying desired features of professionals by eliciting expert opinions in successive rounds.^17–19 An advantage of the Delphi process is that face-to-face meetings are not required so that there is no risk of peer pressure, and experts from different regions can easily participate in the study. The Delphi technique in our study comprised the following procedure: experts were interviewed with electronic questionnaires, answers were collected, aggregated, and refined throughout 2 rounds. Although panel members did not know the individual answers of the other participants, after each round general feedback was provided to each panel member by summarizing all judgments of the previous round.

At the start of our study, participants received instructions about the study's general objectives. Next, they were asked to judge preliminary descriptions of general facets of competence of medical trainees and subsequently to judge successive revisions in 2 further rounds.

Participants

Participants comprised experienced physician-educators, acquainted with competence levels of starting residents in the Netherlands. We approached all 24 experienced clinicians in the Netherlands who met the following criteria: (1) holding an academic chair in medical education, (2) working in clinical practice, and (3) supervising medical residents.

Instrument Development

We developed a draft questionnaire with a preliminary list of FOCs drawn from a review of the literature on selected competency frameworks and other relevant publications. Three competency frameworks were analyzed, that is, the Canadian CanMEDS framework,²⁰ the Dutch “Blueprint of Objectives for Medical Schools,”²¹ and the competency framework of the General Medical Council in the United Kingdom.²² A qualitative analysis of the FOCs in the different frameworks resulted in an initial list with the relevant FOCs.

The first draft of the relevant FOCs was completed through a literature review regarding important FOCs for clinicians. The literature yielded 3 articles,^13,23,24 of which two^13,23 used a holistic approach to the evaluation of medical graduates' competences. Ginsburg et al²³ described characteristics of trainees that influence assessments by supervisors. Sterkenburg et al¹³ focused on factors that guide decisions of supervisors to trust residents with critical clinical tasks. The third article by Kearney²⁴ described the results of a Delphi study exploring the features of professionalism.

We compared these 6 sources (the 3 frameworks and the 3 articles) and found many similarities in content (table 1). We also found differences, mainly in the way the descriptions were ordered and labeled, and in their level of detail. From these sources, we constructed a list of 24 FOCs that met the following criteria: content correspondence with the original source, observable and assessable level of detail, and applicability to medical graduates.

table 1.

Final List of Facets of Competence With References

table 1.

Final List of Facets of Competence With References Continued

Each of the resulting FOCs was scored on 2 significant aspects for valid competence descriptions: (1) clarity and completeness of formulation; and (2) importance for the entrustment of critical clinical tasks to starting residents. For each FOC a 7-point Likert scale²⁵ was developed that pertained to the following statements: (1) The description of this FOC is complete and clear. (2) This FOC is important for the entrustment of critical clinical tasks to beginning residents. Only the 2 endpoints of the scale were labelled: 1 (strongly disagree) and 7 (strongly agree). Respondents were asked to substantiate their answers with comments and to propose improvements for the FOC descriptions. At the end of the questionnaire, respondents were asked to add any new FOCs that they felt were missing.

Data Gathering

Our Delphi study had 2 rounds. For the first round, the list of 24 FOCs was electronically provided to the panelists. On the basis of scores regarding completeness and clearness of the descriptions and the experts' suggestions for improvement of the descriptions, all FOC descriptions were slightly modified. Further, for 2 of the FOCs this resulted in each being divided into 2 separate FOCs, and 2 other FOCs were combined into 1. In a second round, a new list consisting of 25 FOCs, together with a summary of the judgments of the experts in the first round, was sent to the panelists. Participants were asked the same questions as in the first round. For 3 FOCs, multiple-choice questions were added to solicit opinions concerning contradictory or unclear aspects that were made during the first round. Round 2 started 2 weeks after the closing of round 1. In both rounds, the respondents had 2 weeks to complete the questionnaire.

Data Analysis

After each round, a key issue was the decision about whether particular FOCs should be accepted, revised, or deleted. This decision was based on expert ratings on the completeness and clearness of formulation and importance for entrustment decisions. We used experts' mean score to calculate their endorsement of the proposed FOCs.

Next we established the degree of consensus among the experts' judgments by using standard deviations and levels of agreement. For the calculation of the levels of agreement we used a method described by De Loe^26–28 for a 4-point scale, adapted to a 7-point scale. De Loe designates consensus as “medium” if 70% of the scores are given in 2 (of 4) contiguous scale levels. Because of our 7-point scale, we adapted this criterion to 3 (of 7) contiguous scale levels, which is a slightly more stringent judgment (provided as online supplemental material). However, consensus can only be assumed if judgments tend to be unidirectional.²⁹ Therefore, the skewness of the distributions of the ratings was computed to check on symmetry and to check whether the experts' judgments tended to be in 1 direction.

Our purpose was to revise the FOCs, from experts' judgments and comments, until the list satisfied most experts. Therefore, we made a list of all written comments from the experts, which we used to modify the descriptions of the FOCs. To test the differences between ratings in the 2 rounds on the completeness and clearness of the formulation, we used 2-tailed dependent t tests.³⁰

In accordance with established national practice in the Netherlands, ethical approval is not required for education studies. We made sure that participants could not be identified from any materials presented to other participants, that participation would cause no plausible harm and would be fully voluntary, and that refusal or withdrawal would not incur any adverse consequences.

Results

Eighteen physician educators from 8 different hospitals and 11 disciplines participated in this study. In the first round, 14 participated in the Delphi study (58%), 13 of whom were men. The average age was 59 years (range, 52–66 years). In the second round, 15 experts responded (63%), which included 13 men. The average age was 59 years (range, 49–66 years). The distribution in sex and age in the sample reflects the distribution in the population of physician educators in the Netherlands. Responders in both rounds represented a wide range of surgical and nonsurgical disciplines (anesthesiology, cardiology, general practice, geriatrics, gynecology, internal medicine, neurology, oncology, pediatrics, rheumatology, and surgery). Eleven experts participated in both rounds, 3 in the first round only and 4 in the second round only.

table 2 shows the expert ratings on “comprehensiveness and clarity” for the descriptions of the FOCs in both rounds of the Delphi study. In the first round means ranged from 4.77 to 6.46 and standard deviations ranged from 0.78 to 2.07. The results of the second round were comparable, with means between 4.82 and 6.27 and standard deviations between 1.03 and 2.23. For the first and the second round, the level of agreement among the experts was medium or high for 83% and 76% of the FOCs, respectively. To test the differences in ratings on comprehensiveness and clarity of the formulations between round 1 and 2, we used dependent t tests. This difference was statistically significant for only 1 FOC (“active listening to patients”), based on a 2-tailed t test (t₁₀  = 1.5, P  =  .03, r  =  .64). The mean rating for this FOC in the second round was higher than in the first.

table 2.

Expert Ratings for “Completeness and Clearness” of Facets of Competence in Delphi Study^a

We also asked the panel members to indicate on a 7-point Likert scale the importance of the FOC for entrustment decisions. In the first round these means ranged from 6.50 to 7.00, and in the second round, from 5.45 to 6.90. The level of agreement was high for 92% of the FOCs in the first round and for 100% of the FOCs in the second round (table 3). In both rounds and for both the judgments of “comprehensiveness and clarity” and the judgments of “importance,” all ratings tended toward the “strongly agree” side of the scale, such that in all cases the skewness was negative. After round 2, minor changes were recommended regarding the description of 11 FOCs to complete the final list of 25 FOCs (table 1).

table 3.

Expert Ratings for “Importance for Entrustment Decisions” of Facets of Competence in Delphi Study^a

Discussion

This study exemplifies which facets of the general competence of trainees seem to inform the decisions by clinical supervisors to trust them with critical tasks. It is important to map these facets, in light of current approaches to assessment in competency-based medical education.^31–34

We made a preliminary list of FOCs, based on competency frameworks and the literature reflecting an international perspective.^13,20–24 After a Delphi study with 2 rounds, we constructed a list with 25 FOCs. The most important finding was that the experts agreed about the formulation of these FOCs. They strongly agreed that these FOCs are important or very important for the entrustment of critical clinical tasks to starting residents.

Earlier studies about entrustment decisions have shown that these decision processes are complex and relate to several factors.^13,14 These factors include the characteristics and achieved level of competence of the trainees. In the current study we clarified which FOCs of the trainees appear relevant for entrustment decisions.

Our study is limited by the small number of participants. Further evaluation of the identified FOCs in other countries and different educational climates will be important to enhance generalizability of these findings and inform understanding of entrustability.^35,36 The validity of our findings could be further supported by evaluating actual entrustment decisions and their relationships with trainees' FOCs. Finally, our population was selective and not representative of clinical supervisors in general. Participants' views were informed by their academic background, which could have led to the listing of certain FOCs, although we do not have specific hypotheses in mind to support this concern.

Conclusions

We found consensus support by experts for a list of 25 important FOCs for the entrustment of critical tasks to medical trainees. Our results identified general features of medical graduates that enable supervisors to entrust them with critical clinical tasks. The findings are useful for the development of a valid method for assessing medical graduates' readiness for clinical practice. The findings may also be useful for frame-of-reference training for clinicians who must regularly make entrustment decisions.⁶