Defining competency-based evaluation objectives in family medicine

Abstract

Objective

To develop key features for priority topics previously identified by the College of Family Physicians of Canada that, together with skill dimensions and phases of the clinical encounter, broadly describe competence in family medicine.

Design

Modified nominal group methodology, which was used to develop key features for each priority topic through an iterative process.

Setting

The College of Family Physicians of Canada.

Participants

An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian family medicine context with respect to region, sex, language, community type, and experience.

Methods

The group used a modified Delphi process to derive a detailed operational definition of competence, using multiple iterations until consensus was achieved for the items under discussion. The group met 3 to 4 times a year from 2000 to 2007.

Main findings

The group analyzed 99 topics and generated 773 key features. There were 2 to 20 (average 7.8) key features per topic; 63% of the key features focused on the diagnostic phase of the clinical encounter.

Conclusion

This project expands previous descriptions of the process of generating key features for assessment, and removes this process from the context of written examinations. A key-features analysis of topics focuses on higher-order cognitive processes of clinical competence. The project did not define all the skill dimensions of competence to the same degree, but it clearly identified those requiring further definition. This work generates part of a discipline-specific, competency-based definition of family medicine for assessment purposes. It limits the domain for assessment purposes, which is an advantage for the teaching and assessment of learners. A validation study on the content of this work would ensure that it truly reflects competence in family medicine.

In 1998, the College of Family Physicians of Canada’s (CFPC’s) Board of Examiners chose to identify what constituted clinical competence for the purposes of Certification in family medicine. The board’s intent was to describe the domain of competence in family medicine in terms of detailed and operational competency-based evaluation objectives that could direct future assessment for Certification purposes.

Kane has defined 2 major components of professional competence: it is limited to an area of practice, and the competent professional can “handle the encounters or situations that arise in this area of practice.”¹ Kane stresses the importance of defining a professional’s area of practice whenever competence is discussed. In order to assess competence effectively and efficiently, one must define the domain to be assessed¹ and focus on factors predictive of competence.

Through an extensive review of the literature, Epstein and Hundert built on several previous definitions of competence in medicine, defining it as “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.”² While this definition includes many components of competence, competent performance in each component is defined only to the “habitual and judicious” level.

Albanese et al have proposed 5 criteria to define a competency.³ A competency should do the following:

• focus on the performance of the end product or goal state of instruction,

• reflect expectations that apply what is learned in the immediate instructional program,

• be expressible in terms of measurable behaviour,

• use a standard for judging competence that is not dependent on other learners’ performance, and

• inform learners, as well as other stakeholders, about what is expected of them.³

These criteria are also critical components of a discipline-specific definition of competence that directs assessment effectively.

In any profession, many daily activities are routine problems with clear-cut solutions and do not require a high level of competence. Schön has argued that professional competence is the ability to manage ambiguous problems, tolerate uncertainty, and make decisions with limited information.⁴ Regehr agrees that true competence manifests in unfamiliar situations, albeit within a particular domain.⁵ He has also defined competence as one’s “capacity to demonstrate cognitive flexibility and adaptability when faced with novel situations in a given domain, rather than a ritualized set of responses to a predictable set of stimuli.”⁵

In follow-up to Regehr’s comments, Patel and Kaufman⁶ commented that medical diagnostic problems can be characterized as poorly structured: not all the relevant information is available to the problem solver, the potential causes are numerous, and often no definite solution exists. McWhinney summarizes several characteristics of family practice patients, which highlight the relevance of this description of diagnostic problems for family medicine.⁷ He states that patients present with multiple undifferentiated problems, often at an early stage of the disease when the symptoms and signs are not fully developed or obvious. In addition, all types of problems present to family physicians, and so one cannot assume that the problem is within a limited specialty.⁷ Patel and Kaufman comment that solving problems of this nature requires deliberate reasoning, not reflex reaction or simple recognition, which they call “low-road transfer.”⁶ Deliberate reasoning involves the conscious abstraction from one context to another, or the “high-road transfer” of knowledge and skills. The highly competent individual is able to generalize abstracted knowledge across a range of situations.

The domain of practice in family medicine centres on the interaction between patients and physicians. Competence in family medicine therefore must be considered with reference to the clinical encounter. The encounter between a practitioner and a patient has many elements. The content and order of the elements in such an encounter vary; however, the encounter generally consists of 2 phases. In the first (diagnostic) phase, the patient describes the problem and the practitioner asks a series of questions, supplemented by physical examination and investigation, to characterize the problem as one of several diagnostic possibilities (the differential diagnosis). In the second (management) phase, the practitioner proposes an appropriate plan of treatment (eg, drug therapy, surgery), observation, further testing as necessary, and subsequent visits. The relative importance of the different elements of the clinical encounter to the resolution of a clinical problem is problem specific; however, components of the encounter in the diagnostic phase are more likely to involve high-road transfer. Many of the decisions and choices in the management phase are more specific⁸ and involve low-road transfer. While competence obviously requires the demonstrated ability to manage problems, a valid and efficient evaluation process needs to concentrate efforts at the higher level.

Initial work on defining competence in family medicine

In 1998, the CFPC’s Board of Examiners undertook a postal survey of practising family physicians, in which respondents were asked to describe competence in family medicine. Subsequent analysis of the data resulted in a description of competence in terms of 99 priority topics, 5 skill dimensions (a patient-centred approach, communication skills, clinical reasoning skills, selectivity, and professionalism), and 7 clinical encounter phase dimensions (the history, physical examination, investigation, diagnosis, management, referral, and follow-up).⁹ The Board of Examiners also stated that, although it was not identified through the survey, a procedural dimension of competence was important to family medicine. Board members therefore elected to add a sixth dimension of competence, procedure skills, to the 5 other essential skill dimensions.

This work did define the domain of competence as discussed above, but like other attempts to describe competence, it did not provide enough detail. It did not define the skills, abilities, and other qualities needed by practising family physicians with enough specificity to guide assessment. The board therefore chose to define competence in further detail by using a key-features approach.

The key-features approach

The key-features approach is a practical method of defining competence for the purposes of assessment. Key features represent the critical or essential steps in the resolution of a clinical situation or problem. Bordage and Page first published this term.¹⁰ The idea arose from discussions at the first Cambridge Conference, Directions in Clinical Assessment (1984), on the lack of valid and reliable assessments of clinical problem solving using patient-management problems and global rating scales.

The concept of key features was further refined in the Medical Council of Canada’s Q4 Project.^11–14 This work led to the generation of key-features problems for Part I of the Medical Council of Canada Qualifying Examination. Two subsequently published reports of examinations, which were based on key features, showed them to be reliable for the purposes of assessment.^15,16

Page et al have described a key feature as a critical point in the resolution of a problem, where examinees are most likely to make errors and which is a difficult aspect of the identification and management of the problem in practice.¹⁴ The overall objective of the key-features approach is 2-fold. The first aim is to identify these essential or critical steps specific to the problem; the second is to determine why they are difficult and what processes are involved in resolving or failing to resolve the problem. Page et al said that key features for a given problem are not typically generic; they vary according to the clinical presentation of the problem relative to other issues, such as age and sex.¹⁴ A general skill might be used with any given key feature; however, an individual key feature is problem specific.

Generally, key features are observable actions; they are not simply knowledge. They are generated from practical experience, not theoretical analysis or published references. The approach is intentionally selective, covering only what is important, and so the number of key features per topic varies greatly. The quantity of key features is determined by the number of different elements considered essential for dealing with the clinical topic competently; a reflective, iterative process using a group of practising peers best determines these elements.

In addition to inclusion in the Medical Council of Canada Qualifying Examination, Part I, key-features problems have been incorporated into the Royal Australian College of General Practitioners’ fellowship examination for certification¹⁷ and into clinical clerkship examinations at McMaster University in Hamilton, Ont.¹⁵ A specialty certification oral examination in emergency medicine in the province of Quebec was based on key features and proved extremely reliable—more reliable than 2 other similar North American examinations that were not based on key features.¹⁶ Farmer and Hinchy comment that a key-features approach has also been used in other evaluations¹⁸; however, we were unable to identify specific publications related to these situations. Doucet et al used key-features problems to compare the effect of a problem-based learning format with that of a traditional lecture format in the context of continuing professional learning.¹⁹ In 2008, Pinnock and Jones described the development of key features for specific pediatric presentations as part of a process to develop an undergraduate curriculum²⁰; however, they did not articulate the process used in generating their key features. There also do not appear to be instances in which key features for assessment purposes have been developed independent of the method of assessment.

The intent of this phase of the CFPC Board of Examiners’ project was to transform the priority topics into evaluation objectives by identifying the critical elements for competent resolution, using a key-features approach. Each key feature would subsequently be classified according to the most critical skill dimensions and phases of a clinical encounter. The goal was to use the transformed topics as part of a set of competencies that would define the specialty of family medicine for the purposes of assessment. A further intention was that a rigorous process for the development of key features would emerge through this work.

METHODS

Clinical competence is a complex issue that is not well defined by quantitative data. Qualitative approaches, such as the Delphi process and consensus methods, have been recommended for such situations.²¹

An expert group of 7 family physicians and 1 educational consultant used a modified Delphi process to derive a detailed operational definition of competence, using multiple iterations until consensus was achieved for the items under discussion. The process is detailed in Box 1. All discussion of the key features occurred independent of the consideration of assessment tools.

Box 1. Key-features generation process

• Step 1: Each working group member was assigned 1 topic from the list of previously identified priority topics. This author independently identified and listed all the potential key features for his or her assigned topic. In addition, every group member identified the 1 or 2 key features he or she thought were most critical for each of the topics assigned to other group members. This preparation occurred individually, and the results were compiled by topic before a meeting (first iteration).

• Step 2: The assigned author led a discussion among all group members. This discussion centred on his or her list of key features, as well as on other members’ additions. The purpose of this step was to ensure that the tasks or competencies presented were truly the critical ones and that none were overlooked. This also required clarification of the clinical cognitive processes involved for each key feature (eg, gathering data selectively is not the same as interpreting given data). The author then integrated relevant points from the discussion into his or her list of key features (second iteration).

• Step 3: The revised list of key features was discussed in a subgroup of 2 to 3 members, led by the author for the particular topic. At this time, the list of key features was revised to ensure that the starting clinical point, appropriate task, and rationalization were articulated for each key feature, and that the wording was sufficiently precise to guide assessment. Once this was completed, the author presented the list to the entire group for further refinement as necessary (third iteration).

• Step 4: Group members coded the key features for each topic independently; coding was focused on the skill dimensions most essential for successful resolution of the problem and on the phase of the clinical encounter in which it occurred. Almost all problems required many of the skills in several phases. Coding was limited, however, to a maximum of 2 skill dimensions and 2 phases for each key feature, namely those most critical for the competent resolution of the problem. The individual coding results for each topic were tabulated, and the entire group then met to discuss discrepancies and reach consensus for each of the key features. Occasionally final wording was corrected at this iteration (fourth iteration).

All members of the expert group had experience in assessing competence in family medicine. The group had representation from male and female, rural and urban physicians who collectively provided full-scope family practice, including intrapartum, inpatient, and emergency care. There were both Anglophone and Francophone members, and the group included physicians in their early, mid-, and late careers. The group met 3 to 4 times a year from 2000 to 2007. The group experienced little turnover: 4 people were members throughout this period, and 4 others were members for at least 4 years.

To develop key features for each priority topic, the group used a modified nominal group method.

FINDINGS

The expert group analyzed 99 priority topics and defined 773 key features. The number of key features for each topic ranged from 2 to 20, with an average of 7.8 per topic; most topics had between 5 and 9 key features. Boxes 2 and 3 provide a sampling of the key features for 2 topics (in children and hypertension). The entire list of priority topics and key features is available on the CFPC Education webpage.²²

Box 2. Key features for the priority topic in children

1. When evaluating children, generate a differential diagnosis that accounts for common medical problems that can present differently in children (eg, urinary tract infections, pneumonia, appendicitis, depression).

2. Because children, especially adolescents, generally present infrequently for medical care, take advantage of visits to ask about

   • unverbalized problems (eg, school performance),
   • social well-being (eg, relationships, home, friends),
   • modifiable risk factors (eg, exercise, diet), and
   • risk behaviour (eg, use of bike helmets and seat belts).

3. At every opportunity, directly ask questions about risk behaviour (eg, drug use, sexual activity, smoking, driving) to promote harm reduction.

4. In adolescents, ensure confidentiality of visits and, when appropriate, encourage open discussion with their caregivers about specific problems (eg, pregnancy, depression and suicide, bullying, drug abuse).

5. In assessing and providing treatment for children, use age-appropriate language.

6. In assessing and providing treatment for children, obtain and share information with them directly (ie, do not talk to the parents only).

7. When investigation is appropriate, do not limit it because it might be unpleasant for those involved (the child, parents, or health care providers).

Box 3. Key features for the priority topic hypertension

1. Screen for hypertension.

2. Use correct technique and equipment to measure blood pressure.

3. Make the diagnosis of hypertension only after multiple blood pressure readings (ie, at different times and during different visits).

4. In patients with established diagnoses of hypertension, assess and periodically re-evaluate overall cardiovascular risk and end-organ complications:

   • Take an appropriate history.
   • Do the appropriate physical examination.
   • Arrange appropriate laboratory investigations.
5. In suitable patients with hypertension (eg, young patients requiring multiple medications, patients with abdominal bruit, patients with hypokalemia in the absence of diuretics),

   • suspect secondary hypertension and
   • investigate appropriately.

6. Suggest individualized lifestyle modifications to patients with hypertension (eg, weight loss, exercise, limiting alcohol consumption, dietary changes).

7. In patients diagnosed with hypertension, treat the hypertension with appropriate pharmacologic therapy (eg, consider the patient’s age, concomitant disorders, other cardiovascular risk factors).

8. In patients with the signs and symptoms of hypertensive urgency or crisis, make the diagnosis and provide treatment promptly.

9. In all patients diagnosed with hypertension, assess response to treatment, medication compliance, and side effects at follow-up visits.

Early in the process of coding the key features for each topic (fourth iteration), we recognized that the diagnosis phase was not explicit enough to describe the competence identified in many of the key features. We identified and labeled an eighth dimension in the phase of the clinical encounter, hypothesis generation, which involves generating a differential diagnosis from a predominantly undifferentiated starting point.

The frequencies of coding for each skill dimension and phase of a clinical encounter appear in Tables 1 and 2. History, physical, and investigation all refer to the appropriate gathering of data; diagnosis refers to the interpretation of available data. With approximately two-thirds of the key features, only 1 skill dimension and 1 phase were identified as most critical for competence; with the other third, 2 of each were identified.

Table 1.

Frequencies of coded key features, according to the skill dimension assessed: Total number of key features = 773; average number of skill dimensions per key feature = 1.4.

SKILL DIMENSION	N (%)
Clinical reasoning skills	650 (60.0)
Selectivity	173 (16.0)
Communication skills	48 (4.4)
Patient-centred approach	147 (13.6)
Professionalism	54 (5.0)
Psychomotor or procedure skills	11 (1.0)
Total	1083 (100.0)

Table 2.

Frequencies of coded key features, according to the phase of the clinical encounter assessed: Total number of key features = 773; average number of phases per key feature = 1.5.

CLINICAL ENCOUNTER PHASE	N (%)
Hypothesis generation	246 (21.7)
Data gathering
• History	160 (14.1)
• Physical	50 (4.4)
• Investigation	74 (6.5)
Data interpretation
• Diagnosis	186 (16.4)
Clinical management or treatment	337 (29.8)
Referral	19 (1.7)
Follow-up	60 (5.3)
Total	1132 (99.9)*

^*The final percentage is not 100 because of rounding.

DISCUSSION

The identification of these 773 key features sufficiently concentrates, without limiting, the broad domain of family medicine for assessment purposes—a great advantage for the teaching and assessment of learners. It creates a portrait of family medicine that allows specific assessment of the use of skill dimensions and phases of the clinical encounter essential to competence for each problem addressed. The content of the key features was subsequently validated to ensure that they truly reflect competence in family medicine, as Bordage and colleagues did for the Medical Council of Canada¹²; this work will be presented elsewhere. The clarity of key features for guiding assessment comes from the structure and wording, as well as from coding for the skill and phase dimensions of competence. The competencies described here are discipline specific: each discipline of medicine will have its own key features for similar topics. Some key features for a given topic will be shared with other disciplines, while others will be unique to family medicine. Take, for example, the set of competencies required of a family physician if the topic is breast lump, in comparison with the competencies required of a radiologist, a general surgeon, or an oncologist.

This process of creating evaluation objectives through a key-features approach generated assessment objectives that focus on the higher-order cognitive processes in the clinical encounter. The coding indicated that only 37% of the key features focused on the management or treatment phases of the clinical encounter, while 63% focused on diagnosis. About 74% of this latter group of key features dealt with the active process of generating hypotheses and gathering pertinent data to make adequate diagnoses. As previously discussed, components of the clinical encounter in the diagnosis phase are likely to be much better predictors of overall competence than are components of the management phase; many of the decisions and choices in the management phase are more specific⁸ and involve low-road transfer. While competence obviously requires the demonstrated ability to manage many problems in family medicine, concentrating efforts on the higher levels is much more important to the production of valid and efficient evaluation processes.

This work moved us closer to a discipline-specific, competency-based definition of competence in family medicine in Canada. The key-features analysis identified the critical tasks necessary for competent clinical reasoning, selectivity, and use of the patient-centred approach, as well as opportunities to assess these skills. Fewer opportunities were present for assessment of communication skills and professionalism. Furthermore, for these 2 dimensions and for the patient-centred approach, the individual key features did not sufficiently identify the steps that were critical for performing these skills competently; rather, they identified only circumstances in which the skills could be assessed. The detailed evaluation objectives for these 3 dimensions require development by alternative processes and will be reported elsewhere.

The key-features approach also resulted in a minimal number of situations involving psychomotor skills, which were insufficient to define competence with respect to procedure skills. This dimension of competence required a separate process for definition, which began with a list of core procedures judged necessary for Certification.²³ Additional key features for core procedures were also developed and will be reported in the future.

Conclusion

This project has generated part of a discipline-specific, comprehensive, competency-based definition of family medicine for the purposes of assessment and Certification. It, along with work undertaken by the CFPC’s Working Group on Curriculum Review,²⁴ contributes to the move to competency-based curriculum and assessment for postgraduate family medicine training in Canada. It has not defined all skills to the same degree, but it is transparent, clearly identifying gaps that need further definition. It limits the domain for assessment purposes, which is an advantage for teaching learners and assessing their skills. This project has expanded previous descriptions of key-features generation and has removed the process from the context of developing problems for assessment on written examinations. It has also demonstrated that the process of key-features analysis generates opportunities for assessment that focus on components of the clinical encounter more likely to predict competence.

EDITOR’S KEY POINTS

• The development of key features sufficiently concentrates, without limiting, the broad domain of family medicine for assessment purposes—an advantage for teaching learners and assessing their skills.

• Each competency described through key features is discipline specific.

• The expert group’s work on developing key features has moved us closer to a discipline-specific, competency-based definition of competence in family medicine.

POINTS DE REPÈRE DU RÉDACTEUR

• L’élaboration de caractéristiques clés à des fins d’évaluation restreint suffisamment le vaste domaine de la médecine familiale sans toutefois le limiter – ce qui constitue un avantage pour l’enseignement aux étudiants et l’évaluation de leurs habiletés.

• Chacune des compétences établie à partir de caractéristiques clés est propre à une discipline.

• Grâce à l’élaboration de caractéristiques clés par le groupe d’experts, nous nous approchons d’une définition de la compétence en médecine familiale qui soit spécifique à une discipline et fondée sur la compétence.