Defining competency-based evaluation objectives in family medicine

Abstract

Objective

To develop evaluation objectives for assessing competence in procedure skills using a key-features approach. This was part of a multiyear project to develop competency-based evaluation objectives for Certification in Family Medicine.

Design

Nominal group technique.

Setting

The College of Family Physicians of Canada in Mississauga, Ont.

Participants

An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience.

Methods

Using a nominal group technique, the expert group developed the general key features for procedure skills. The expert group also linked the key features to already established skill dimensions in the domain of competence, to the 4 principles of family medicine, and to the CanMEDS roles.

Main findings

The general key features were developed after 5 iterations. Ten key features were outlined and were shown to reflect all the essential skill dimensions in the domain of competence for family medicine. The key features were linked to 2 of the 4 principles of family medicine and to 4 of the CanMEDS roles.

Conclusion

The general key features for procedure skills were developed to assess competence in procedure skills in family medicine.

For many years, “see one, do one, teach one” has been the accepted philosophy for teaching and learning procedural skills in medicine. This concept primarily referred to teaching and learning the technical steps necessary for successful procedure performance. Learning the technical steps of a given procedure in family medicine is fairly easily accomplished during training by repetitive practice.^1,2 True competence in procedure skills, however, involves much more than just learning the technical aspects of any procedure.

With the current focus on patient safety and medical error, and the move to competency-based education, the “see one, do one, teach one” principle has been recognized as inadequate for training for most procedures in medicine.^3,4 Furthermore, there has been considerable variability in the procedures taught during training in both Canadian and American family medicine training programs.^1,5

Given this variation in procedures taught and the challenges to fundamental procedures teaching concepts, there was a need for uniform evaluation objectives that would more accurately assess competence in procedure skills that could be used by all teaching programs of family medicine in Canada.

In this paper we have focused on procedure skill teaching using a key-features approach and we describe the development of the general key features for procedure skills. We also point to their utility for evaluating competence in procedure skills.

Background to the process

In 1998, the College of Family Physicians of Canada’s (CFPC) Board of Examiners decided to identify what constituted clinical competence for the purposes of Certification in Family Medicine. The process of developing the definition of competence has been described elsewhere, and began with a survey of practising family physicians to elicit a description of competence in family medicine.⁶ From this initial survey 5 essential skill dimensions, 8 phases of the clinical encounter, and 99 priority topics were identified, constituting the domain of competence. The 5 skill dimensions were a patient-centred approach, communication skills, clinical reasoning skills, selectivity, and professionalism.⁶ Procedure skills were not initially identified among the skill dimensions. The CFPC Board of Examiners, however, recognized that Certification in Family Medicine did imply competence in certain procedures. Therefore, competence in procedure skills was added to the definition of competence in family medicine, as a sixth essential skill dimension.

With respect to competence in procedure skills, a list of 65 core procedures, germane to family physicians practising in any setting, and an enhanced list of 15 procedures was then developed by Wetmore et al.⁷ The core procedures list was included as part of the domain of competence, and was considered as equivalent to the priority topics previously mentioned.

Thus, the domain of competence consisted of 6 skill dimensions, 8 phases of the clinical encounter, 99 priority topics, and a core list of procedure skills. This description was, however, thought to still not be sufficiently detailed for the purposes of assessment. More detailed evaluation objectives for each of the 99 priority topics were developed using a key-feature approach, as described by Lawrence et al.⁸ Similarly, because the core procedures list, by itself, was not sufficient to define competence in performing procedures, the general key features for procedure skills were needed to enhance this definition. Competence in procedure skills could then be evaluated using the core procedures list and the general key features for procedure skills.

Key-feature approach

The key-feature approach is a practical method of defining competence for the purposes of assessment, first described by Bordage and Page.⁹ Page and Bordage described a key feature as a critical point in the resolution of a problem, where examinees are most likely to make errors and which is a difficult aspect of the identification and management of the problem in practice.¹⁰ The overall objective of the key feature approach is 2-fold. The first aim is to identify these essential or critical steps specific to the problem; the second is to determine why they are difficult and what processes are involved in successfully completing them. Page and Bordage identified that key features for a given problem are not typically generic; they vary according to the clinical presentation of the problem relative to other issues, such as age and sex.¹⁰ A general skill might be used in any given key feature; however, an individual key feature is problem specific. Generally, key features are observable actions; they are not simply knowledge. They are generated from practical experience, not theoretical analysis or published references. Key features are pragmatic, suggesting where assessment should be concentrated in order to be both effective and efficient. They are useful tools when planning assessment.

Overall competency frameworks

This paper also describes how the general key features for procedure skills are related to the overall competency frameworks that exist for Canadian physicians: the 4 principles of family medicine, from the CFPC,¹¹ and the CanMEDS roles, developed by the Royal College of Physicians and Surgeons of Canada.¹² Although useful, such frameworks do not describe competence in sufficient detail to usefully direct assessment.¹³ Others suggest that the CanMEDS roles are so intertwined that assessing each of them separately makes little sense.¹⁴ It might be reassuring, however, to know that any new method of assessment of competence in procedure skills can be linked back to these 2 overall competency frameworks.

METHODS

An expert group of 7 family physicians and 1 evaluation consultant used a nominal group technique¹⁵ to derive the general key features for procedure skills, using multiple iterations to achieve consensus. Box 1⁸ describes the nominal group technique used by the expert group. All members of the expert group had experience in assessing competence in family medicine, and represented the Canadian context for region and language. Both sexes were represented. The group collectively provided the full scope of family practice including community, inpatient, intrapartum, and emergency care, and all of the family physicians had experience performing common procedures in family medicine. They met several times a year, and membership remained unchanged during the 4 years in which the work was conducted.

Box 1. Key-features generation process for a single topic.

Step 1: One group member is assigned a topic. This author independently identifies and lists all the potential key features for the assigned topic. In addition, all other group members identify the 1 or 2 key features they think are most critical for this same topic. This preparation occurs individually, and the results are compiled by topic before a meeting (first iteration). Step 2: The assigned member leads a discussion among all group members. This discussion is centred on his or her list of key features, as well as on other members’ additions. The purpose of this step is to ensure that the tasks or competencies presented are truly the critical ones and that none was overlooked. This also requires clarification of the clinical cognitive processes involved for each key feature (eg, gathering data selectively is not the same as interpreting given data). The responsible member then integrates the discussion from step 2 into his or her list of key features (second iteration). Step 3: The revised list of key features is then discussed again, led by the member responsible for the particular topic. At this time, the list of key features is revised to ensure that the starting clinical point, appropriate task, and rationalization are articulated for each key feature, and that the wording is sufficiently precise to guide assessment. This process is repeated until a consensus is achieved (third iteration). Step 4: Group members code the key features independently, focusing on the skill dimensions most essential for successful resolution of the problem and on the phase of the clinical encounter in which the problem occurs. Almost all problems require many of the skills in a number of phases. Coding is limited, however, to a maximum of 2 skill dimensions and 2 phases for each key feature, namely those most critical for the competent resolution of the problem. The individual coding results are tabulated, and the entire group then meets to discuss discrepant codes and reach consensus for each of the key features. Occasional final corrections occur in the wording at this iteration (fourth iteration). In this study the key features were coded concurrently to the 4 principles of family medicine and to the 7 CanMEDS roles.

Adapted from Lawrence et al.⁸

As the final step of key-feature generation, each was coded or linked by the expert group back to the skill dimensions most essential for successful resolution of the problem. In addition, the key features were each coded to the 2 existing overall frameworks of competency—the 4 principles of family medicine and the CanMEDS roles. A consistent iterative method was used, alternating individual work with group discussion until consensus was achieved. A maximum of 2 skills, principles, or roles was permitted per key feature, as part of the linking exercise.

FINDINGS

Ten general key features for procedure skills were generated and are presented in Table 1. Each key feature stands on its own, but some are grouped under a stem statement reflecting a phase in the preparation for or performance of a procedure. Table 1 also documents how the key features are linked to all of the other skill dimensions and to 2 of the 4 principles of family medicine and 4 of the CanMEDS roles.

Table 1.

The general key features for procedure skills

KEY FEATURE	SKILL DIMENSION	4 PRINCIPLES	CanMEDS ROLE
To decide whether you are going to do a procedure consider
• The indications and contraindications to the procedure	Clinical reasoning	Skilled clinician	Medical expert
• Your own skills and readiness to do the procedure (eg, your level of fatigue and any personal distractions)	Clinical reasoning Professionalism	Skilled clinician	Medical expert Professional
• The context of the procedure, including the patient involved, the complexity of the task, the time needed, the need for assistance, and the location	Clinical reasoning Selectivity	Skilled clinician	Medical expert
Before deciding to go ahead with the procedure
• Discuss the procedure with the patient, including a description of the procedure and possible outcomes, both positive and negative, as part of obtaining consent	Communication skills Clinical reasoning	Skilled clinician	Medical expert Communicator
• Prepare for the procedure by ensuring appropriate equipment is ready	Clinical reasoning	Skilled clinician	Medical expert
• Mentally rehearse the following: -The anatomic landmarks necessary for procedure performance -The technical steps necessary in sequential fashion, including any preliminary examination -The potential complications and their management	Clinical reasoning	Skilled clinician	Medical expert
During performance of the procedure
• Keep the patient informed to reduce anxiety	Patient-centred approach	Doctor-patient relationship	Medical expert Communicator
• Ensure patient comfort and safety always	Patient-centred approach	Doctor-patient relationship	Medical expert Professional
When the procedure is not going as expected, reevaluate the situation, stop, or seek assistance as required	Clinical reasoning Professionalism	Skilled clinician	Medical expert Collaborator
Develop a plan with your patient for aftercare and follow-up after completion of a procedure	Patient-centred approach	Doctor-patient relationship	Medical expert Professional

DISCUSSION

Clearly, performing procedures in family medicine is much more than just being able to complete the technical components of a procedure. The general key features for procedure skills described in this paper are crucial to competent performance of procedures in family medicine. It is important to note that these do not detail all there is to know about any one procedure but do reflect an important overall approach to procedure skills competence. For example, anatomy might vary from procedure to procedure, but not knowing the anatomy and failing to review it ahead of time could get one into trouble, no matter what the procedure. Similarly, when the procedure is going wrong, the not-yet competent physician might push on stubbornly to complete the procedure with the resulting complications. The competent physician would stop, reevaluate the situation, and seek assistance in order to correct the situation or halt the procedure. Thus, the general key features for procedure skills point to a different level of competence than just mastery of the technical steps of any procedure, and incorporate the skills, attitudes, and knowledge necessary for overall competence in procedure skills.

Assessment of competence using the general key features for procedure skills is also measuring competence in all 5 of the other essential skill dimensions and reflects the 4 principles of family medicine and the CanMEDS roles. As such, evaluation objectives based on a key-feature approach are framework-independent, and reflect what truly needs to be assessed, in day-to-day practice, to determine competence.

Overall competence in procedure skills could legitimately be inferred based on evaluation using the general key features and observation over a sufficient number of the core procedures. This approach should also enable a physician to learn new skills and procedures throughout his or her career.

The strength of this work is that the key features were derived by an expert group of practising family physicians. As such they constitute a practical tool, grounded in clinical practice. Their validity is supported by the coding that showed that all the other skill dimensions of competence were captured in the key features. We also believe that the general key features for procedure skills are generic to performing procedures in any medical discipline and that they could be adapted by any discipline, after suitable testing or review.

The next steps in this work would include looking at the individual procedures on the core procedures list and developing technical key features for each procedure. These specific key features for each procedure, combined with the general key features for procedure skills, could form the basis for tools to enhance the evaluation of procedure skills competence during training and as part of any examination process. The general key features for procedure skills should also be evaluated for their utility at assessing competence in procedure skills at different levels of training. Appropriate measures should be developed to determine if the key features were being successfully incorporated into practice.

Limitations

The nominal group process used in this research was an excellent process for building consensus but lacked the additional step of external validation that would have added strength to the validity of the general key features. The specific content of these general key features for procedure skills has not been validated. Their validity is suggested by both their method of development and by the results of the linking exercise to the 4 principles of family medicine and to the CanMEDS roles. The general key features for procedure skills might be criticized for lacking the specific technical components for any individual procedure, but this was not the intent. Procedure-specific key features will need to be developed.

Conclusion

The general key features for procedure skills are crucial to the competent performance of any procedure in family medicine and, together with the list of core procedures, could form the basis for defining competence in procedure skills at the start of independent practice in family medicine. Further research will be necessary to validate the content of the general key features for procedure skills and their incorporation into training and practice.

EDITOR’S KEY POINTS

• The general key features for procedure skills were developed to assist in evaluating competence in procedure skills in family medicine.

• The key features reflect all of the skill dimensions in the domain of competence.

• The general key features for procedure skills along with the core procedures list can form the basis for evaluating competence in procedure skills.

POINTS DE REPÈRE DU RÉDACTEUR

• Les caractéristiques clés générales correspondant aux habiletés techniques ont été développées afin de faciliter l’évaluation de la compétence relative à ce type d’habiletés en médecine familiale.

• Ces caractéristiques clés reflètent tous les aspects des habiletés relativement à la compétence.

• Les caractéristiques clés générales correspondant aux habiletés techniques peuvent, avec la liste des techniques principales, servir de base à une évaluation de la compétence en matière d’habiletés techniques.