Defining competency-based evaluation objectives in family medicine

Abstract

Objective

To develop and describe observable evaluation objectives for assessing competence in professionalism, which are grounded in the experience of practising physicians.

Design

Modified nominal group technique.

Setting

The College of Family Physicians of Canada in Mississauga, Ont.

Participants

An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience.

Methods

Using an iterative process, the expert group defined a list of observable behaviours that are indicative of professionalism, or not, in the family medicine setting. Themes relate to professional behaviour in family medicine; specific observable behaviours are those that family physicians believe are indicative of professionalism for each theme.

Main findings

The expert group identified 12 themes and 140 specific observable behaviours to assist in the observation and discussion of professional behaviour in family medicine workplace settings.

Conclusion

Competency-based education literature emphasizes the importance of formative evaluation and feedback. Such feedback is particularly challenging in the domain of professionalism because of its personal nature and the potential for emotional reactions. Effective dialogue between learners and teachers begins with clear expectations and reference to descriptions of relevant, specific behaviour. This research has generated a competency-based resource to assist the assessment of professional behaviour in family medicine educational programs.

Medical educators have devoted a great deal of attention to the teaching and assessment of professionalism. Educational approaches to the domain of professionalism emphasize the need for both teaching the cognitive base of professionalism and providing opportunities for the internalization of its values and behaviours.¹ Educational theories such as situated learning or experiential learning suggest that learning should be embedded in authentic activities that help to transform knowledge from the abstract and theoretical to the usable and useful.² In this paper, we focus on the development of observable behaviours specific to a description of what resident professional behaviour looks like in practice settings.

Others have emphasized the importance of context and setting in the teaching and assessment of professionalism. Respect for context begins with how we define professionalism in a given discipline. In developing a normative definition of professionalism, “the concept of medical professionalism must be grounded both in the nature of a profession and in the nature of physicians’ work.”³ Such a discipline-specific definition of professionalism means that skills are best learned in settings that approximate actual practice environments. This is a foundational principle for most competency-based approaches to medical education.

In the context of family medicine in Canada, the move toward competency-based education began in 1998, when the College of Family Physicians of Canada’s Board of Examiners chose to identify what constituted clinical competence for the purposes of Certification in family medicine. Competence is described in terms of 6 skill dimensions (a patient-centred approach, communication skills, clinical reasoning skills, selectivity, professionalism, and procedure skills), 7 clinical-encounter phase dimensions (history, physical examination, investigation, diagnosis, management, referral, and follow-up), and 99 priority topics.^4,5 However, the level of definition was not operational, as it did not provide sufficient detail to inform assessment and feedback adequately. This level was reached for the priority topics and their interactions with the other elements of competence by using a key-feature analysis,⁶ but this method did not provide adequate definition for 2 of the essential skills: communication skills and professionalism.

Below, we describe a qualitative study in which an operational description of professional behaviour was derived from the experience of practising clinicians. We also explain how this description reflects their contexts and the real-world setting of family medicine.

METHODS

An expert group of 7 family physicians and 1 educational consultant used a modified nominal group technique to derive a detailed operational description of competence in professionalism. All members of the expert group had experience in assessing competence in family medicine and represented the Canadian context with respect to region, sex, language, community type, and experience. The nominal group technique is 1 of 2 recommended by Jones and Hunter to come to decisions about issues, such as the appropriateness of clinical criteria in judging a situation, when there is a lack of quantitative or objective data to guide the decisions.⁷ The other technique is the Delphi approach.⁷

The group first reviewed 576 statements about characteristics that describe competence as far as professional behaviour is concerned for a newly practising family physician. These statements had been previously generated by a postal survey of randomly selected practising family physicians, answering a series of questions about how they defined competence in family medicine.⁴ The statements were first analyzed and reviewed to identify the emergent themes of professionalism. The process then shifted to the generation of specific observable behaviours related to each theme. The participants were directed to provide examples of learner behaviour illustrative of the various professionalism themes. The behaviours could be suggestive of either good or poor performance. In all cases multiple iterations were used until consensus was achieved.

RESULTS

Twelve themes emerged as organizing categories in the skill dimension of professionalism (Table 1). Consensus was achieved as all 576 initial statements were reviewed and accounted for. Although some responses could justifiably fit under more than one theme heading, the final version achieved the goal of coherence and comprehensiveness.

Table 1.

Themes of professionalism with observable behaviours

THEMES	OBSERVABLE BEHAVIOURS
	APPROPRIATE	INAPPROPRIATE
Day-to-day behaviour reassures one that the physician is responsible, reliable, and trustworthy	Comes to clinic when expected Answers pages when on call Notifies attending colleague if he or she is going away and has a maternity patient due or is following an inpatient Notifies others when away for illness or emergencies as soon as possible Sets up systems for follow-up of patients Does not lie	Does not look up questions after specific requests Leaves early or arrives late without advising Inappropriately double-schedules activities Switches schedules to personal advantage Does not do patient rounds appropriately (eg, too infrequent, too cursory) Is unavailable for clinical responsibilities for personal reasons, without consideration of the needs of the patient or team Allows chart completion to back up unreasonably Does not document laboratory results as normal or abnormal; does not document follow-up Does not do letters or summaries Cheats on examinations or quizzes (eg, ALSO, NRP) Goes into SOOs with foreknowledge of cases (ie, cheats on examinations) Does not check allergies or interactions when prescribing Fails to follow up in a timely fashion with patients when investigations are pending (eg, skin biopsy) or in potentially serious clinical situations (eg, depressed adolescent who does not show up for an appointment) Lies about previous experience with a procedure to get to do it Signs in for others when attendance is taken at academic events Plagiarizes on projects
The physician knows his or her limits of clinical competence and seeks help appropriately	Seeks opportunities to address limitations to improve knowledge and skills (electives or continuing education) Does not use the excuse of limited clinical competence to avoid challenging clinical problems	Argues about deficiencies in clinical competence in spite of examples to illustrate concerns Ignores clinical problems to mask clinical limitations Refers cases even when he or she has the skills and resources to perform the tasks (does not take the time to do appropriate medical procedures) Does not initiate the management of complex or difficult problems when a patient presents—defers to an attending physician or a consultant Does not prepare adequately for a procedure
The physician demonstrates a flexible, open-minded approach that is resourceful and deals with uncertainty	In patient encounters, consistently demonstrates a willingness to explore the patient’s ideas of cause and take steps to include or exclude these from the ensuing differential diagnosis Is willing to adapt diagnosis or plan when provided with an alternative view, information, or perspective (willing to change his or her mind) Provides time to deal with the emotion related to an uncertain diagnosis Does not unnecessarily limit patient options (ie, does not display paternalism) Is satisfied with “symptom diagnosis” (eg, says “dyspepsia,” not “peptic ulcer disease”) when information is limited or diagnosis is not confirmable Formulates a patient-centred, stepwise plan to deal with a situation even when he or she does not know the answer	Cuts patients off Refuses to deal with a serious problem during an office visit because of time Refuses to see a patient who arrives slightly late for an appointment Shows anger or rigidity when patients do not follow a prescribed course of action Becomes dismissive of patient ideas when they do not fit his or her own Uses manipulative techniques to influence patient behaviour (“I won’t be able to take care of you if you choose to do …”)
The physician evokes confidence without arrogance, and does so even when needing to obtain further information or assistance	Says, “I don’t know but I know how I am going to find out” Management discussions with patients are clearly helpful to the patient with “value added,” even without a certain diagnosis or final opinion about available treatment Projects appropriate confidence in nonverbal communication: looks patients in the eye when he or she says, “I don’t know”	Uses own experience to devalue the patient’s experience (eg, “I didn’t have to have an epidural”) Tells patients what to do without understanding their circumstances (displays arrogance or paternalism)
The physician demonstrates a caring and compassionate manner	Allows patients time to verbalize their concerns without cutting them off; listens for a while before talking—actively listens before talking Does not belittle the patient’s losses or fears Asks patients about their feelings, worries, and hopes Sits down with patients whenever possible while communicating Addresses issues or behaviours with patients rather than confronting them personally or judgmentally Expands on healthy options or choices with patients Keeps patients’ needs foremost when faced with own personal concerns about medical errors, disasters, or accusations Is willing to acknowledge the patient’s emotions within the encounter Does not blame patients for difficult situations they encounter When dealing with a difficult patient, recognizes his or her own feelings and avoids expressing anger inappropriately Despite time and workload pressure, maintains a pleasant, compassionate approach
The physician demonstrates respect for patients in all ways, maintains appropriate boundaries, and is committed to patient well-being. This includes time management, availability, and a willingness to assess performance	Respects the patient’s time as if it were his or her own: does his or her best to be on time; acknowledges when he or she is not Does not impose personal religious, moral, or political beliefs on patients Does not ask for or accept offers of dates from patients Does not ask patients for favours Does not accept inappropriate gifts Does not make jokes at a patient’s expense Respects a patient’s lifestyle choices as his or hers to make Appreciates the power differential in the physician-patient interaction Maintains personal appearance to facilitate patient comfort and confidence for individual patients, or for specific patient populations Comments and behaviours reinforce and enhance the patient’s abilities and capabilities Does not lend patients money (or borrow money from patients) Recognizes the difference between maintaining confidentiality and seeking appropriate professional advice when needed in difficult situations Actively looks at his or her practice with assessment tools, and implements appropriate changes Thinks and speaks about patients in a positive manner Attempts to understand patient issues that precipitate difficult behaviour or noncompliance, and adapts his or her response accordingly	Always seems rushed or burdened by too many demands Complains about other team members in front of patients Blames others for a personal lack of organization or harried approach (eg, “Who took my stethoscope this time?” “Where’s my pen?” “I’m late because there are no parking spots.” “The secretary didn’t remind me I had to be there.” “My charts weren’t out.”) Is reluctant or refuses to see some patients
The physician demonstrates respect for colleagues and team members	Does not undermine and avoids making negative comments about other providers, especially those who might have seen patients in different settings or contexts When consulted or asked for help, listens to concerns and tries to respond positively and to be available (“How can I help?” vs “I don’t need to see this patient”) When needing to talk to someone unexpectedly, waits and picks the right moment; does not interrupt unduly Thinks and speaks about colleagues in a positive manner; respects their time as if it were his or her own Arrives on time Pays attention when others are speaking Lets others speak or continue; hears them out and stays respectful even if he or she might not agree with topics or points of view	Provides inappropriate feedback in an insensitive manner (eg, nonspecific; wrong place, wrong time) Leaves early, picks the easy tasks, leaves tasks unfinished, etc, such that others have more work Discusses contentious issues in public, or gossips Avoids the discussion of contentious issues that are having or might have important effects on team dynamics and outcomes Argues with other team members Does not make personal adjustments in spite of repeated messages from others about performance in the workplace A male trainee does not accept feedback from a female colleague or faculty Does other things (ie, does not pay attention) while a colleague is speaking (eg, text messages, reads paper, does charts)
Day-to-day behaviour and discussion reassures that the physician is ethical and honest	When an error has been made, acknowledges his or her own contribution, discusses it with the appropriate parties, and tries to clarify why the error was made and apply corrective action for the future Obtains informed consent; asks about privacy, communication, or confidentiality Respects patient autonomy, and assesses whether patient decision making is impaired Provides honest estimates concerning time, services, and billing	Discloses patient information against his or her expressed wishes, especially with respect to adolescents, the elderly, and patients with different cultural issues Discusses patients in “public” places Provides medical treatment inappropriately to colleagues, including writing prescriptions Claims (to colleagues, patients, others) to have done something that has not been done (eg, history, physical examination, laboratory tests, telephone calls, follow-up) Takes credit for work done by others (for monetary reasons, for prestige, for any reason) Has inappropriate prescribing practices (eg, puts in the name of someone with a drug plan instead of the patient, prescribes inappropriately for self-gain, prescribes without sufficient assessment) Makes unjustifiable claims on insurance or other forms
The physician practises evidence-based medicine skillfully. This implies not only critical appraisal and information-management capabilities, but incorporates appropriate learning from colleagues and patients	Does not give undue weight to evidence-based medicine: incorporates the patient’s and family’s expertise about the uniqueness of their situation; incorporates the experience and expertise of colleagues and team members, as well as his or her own When a patient questions care or makes suggestions, is open to respectful discussion; responds positively to patients who bring materials from the Internet When using guidelines or the results of clinical trials (on large populations), customizes and adapts them to ensure applicability to the individual patient in question Does not change a current treatment plan when temporarily dealing with someone else’s patient; if he or she thinks changes are desirable, discusses them first with the regular provider Checks as to whether practice is consistent with recent evidence, and makes changes consistent with this evidence Identifies knowledge gaps in own clinical practice, and develops a strategy to fill them; frames clinical questions that will facilitate the search for “answers” to these gaps	Does not use resources to acquire up-to-date information about specific cases Following a group discussion and decision, does not incorporate agreed-upon changes into clinical practice Relies too much on a limited set of inappropriate information resources (eg, drug company representatives, unselected Internet material, The Medical Post, “expert” opinion) Does not critically question information
The physician displays a commitment to societal and community well-being	Does not dismiss concerns raised by patients on local issues that have an effect on their health (eg, safe walking areas, pollution) Tries to empower the patient who raises concerns about community issues; acts in a confidential manner Responds positively to community requests for participation: will dedicate some time and experience, some resources (eg, put a poster up)	Does not respect the duty to report in situations in which there is a clear danger to others (eg, meningococcal disease, capacity to drive, child abuse) Does not report inappropriate behaviour (eg, substance abuse) of professional colleagues to the appropriate supervisor or authority
The physician displays a commitment to personal health and seeks balance between personal life and professional responsibilities	Takes appropriate time to fulfil personal needs Is willing to discuss observations from colleagues or team members when behaviour suggests difficulty because of stress When a conflict between professional and personal activities is brought to his or her attention, discusses it, makes an appropriate adjustment or not Sometimes puts the patient first, ahead of personal need, and demonstrates satisfaction and appreciation of the value of this action Has a healthy lifestyle: does not smoke, does not drink to excess, drives reasonably	Takes frustration, etc, out on colleagues or staff (eg, is rude and inappropriate) Fails or refuses to recognize or deal with significant illness or a condition that might have an effect on professional activities, especially when concerns are identified by others Stays overtime inappropriately, comes to work sick, is unwilling to take time off Burdens co-workers when taking care of own needs (ie, leaves many things undone without communicating with colleagues) Transfers tasks to colleagues without clear justification, without adequate communication; changes availability for professional tasks “frequently” or “at the last minute” Seeks medical care from friends or colleagues outside of a normal physician-patient relationship; acts as own physician
The physician demonstrates a mindful approach to practice by maintaining composure and equanimity, even in difficult situations, and by engaging in thoughtful dialogue about values and motives	Given a difficult situation, maintains composure and is able to act appropriately (eg, with angry patients, an unexpected clinical turn of events, an overwhelming demand, examinations) Is consistently attentive to a patient or colleague throughout any interaction Tries to understand the behaviour of others without getting mad or being hurt Does not display anger, inappropriate humour, or other emotions when this could undermine constructive work with patients or colleagues When emotions are intense or visible, can nevertheless explain or suggest a constructive plan of action Does not lose his or her cool—even when the other person in the room loses it Can allow for multiple perspectives from various participants in complex situations; entertains or solicits other viewpoints Is willing to engage in dialogue, in order to learn from experience and others, when -a bad or unexpected outcome occurs; -there are conflicting ideas; or -he or she is asked questions (does not perceive these as a threat; makes time to discuss them vs being “too busy to talk about it”) When a mistake appears to have been made, acknowledges it and looks first for personal responsibility rather than directing blame elsewhere

ALSO—Advanced Life Support in Obstetrics, NRP—Neonatal Resuscitation Program, SOO—simulated office oral.

A total of 140 observable behaviours were generated, distributed among the 12 themes of professionalism (Table 1). For this task, the focus group achieved consensus by presenting the observable behaviours as important examples of resident professional behaviour, rather than a comprehensive list.

DISCUSSION

Education in the skill dimension of professionalism is promoted using 3 recommended approaches: role modeling, knowledge acquisition, and experiential learning. All are worthy of increased attention. Professionalism is a construct with multiple dimensions and meanings. This paper focuses on professionalism as a skill dimension, pertaining to the aspects of professional behaviour that can be learned and improved, especially through reflection in clinical settings. Learning and assessing professionalism shares with other skill dimensions an important reliance on effective formative feedback. Clinical settings provide numerous daily examples in which a learner’s professional behaviour can be reflected upon. However, similar to the old clinical adage “You only see what you look for,” both learners and teachers can be reminded of opportunities for observation and discussion by having access to clear descriptions of desirable and undesirable professional behaviours. Reference to published themes and observable behaviours might facilitate discussion of value-laden subjects. An important role of the observer is to guide self-assessment by the learner by presenting the observations as informative feedback. This implies valued behaviours are positively reinforced or opportunities for improvement are described.

Presenting a list of observable behaviours poses a risk of misinterpretation of their intended use. They are not intended as a checklist. The authors undertook their work aligned with the principles of competency-based assessment. The competency-based assessment movement in medical education has itself been criticized for promoting a reductionist approach. Medical educators who promote competency-based approaches acknowledge the risk:

Competence does not equal a list of learning objectives or reductionist tasks; it is a broad objective that necessitates an integration of knowledge, skills and behaviours in practice.⁸

Among the potential perils and challenges of competency based medical education is the threat of reductionism. In an effort to address the challenges of defining and assessing competencies, some have resorted to breaking them down into the smallest observable units of behaviour, creating endless nested lists of abilities that frustrate learners and teachers alike.⁹

Keeping these cautions clearly in mind, and to avoid reductionist pursuits, the authors promote a qualitative approach to working with a resident that is analogous to participatory action research.¹⁰ In the case of in-training evaluation, the teacher and learner embark on an exploration of the learner’s developing and changing competence. True to such methods, values are expressed early in the process, and the reference to observable behaviours serves to assist this purpose. In the domain of professionalism, observations and feedback must include the opportunity to explore the learner’s motives and relevant values. Methods of evaluating professionalism should go beyond observable behaviours to include the reasoning behind them.¹¹ However, even when observed behaviours trigger discussion of underlying reasoning, we should not expect to achieve reliable numerical scoring.¹² A qualitative approach to assessment more aptly pursues trustworthy and accurate assessments of performance. These are the qualitative equivalents for the quantitative goals of reliability and objectivity.

Our own approach is to engage learners with qualitative approaches directed toward constructing a mutual understanding of a learner’s professional behaviour and its determinants. Such methods acknowledge the value-laden nature of the assessment and require participants to make underlying values as transparent as possible. Evaluation objectives contribute to articulating program values and are especially helpful for formative feedback when they are specific and observable.¹³

Giving feedback about professional behaviour raises some specific concerns and has specific requirements. Assessment of professionalism can be more personally threatening, and learners perceive themselves as especially vulnerable when their professionalism is being judged.¹⁴ The feedback process should allow mutual interpretation of events, an acknowledgment of different perspectives, and an exploration of the meaning of observed behaviour. Observation must be coupled with conversation, so that students’ professional behaviours and attitudes can be assessed more fairly.¹⁵

Improving the effectiveness of the experiential learning of professionalism will require not only attention to the skill of providing feedback, but also greater clarity about expectations. Expectations must be clearly expressed as a starting point to guide observation, reflection, and subsequent dialogue. We are reminded that assessment tools will be better if they define professionalism as behaviours expressive of value conflicts, and permit us to investigate the resolution of these conflicts and recognize the contextual nature of professional behaviours.¹⁶

Limitations

The accompanying table presents the language emanating from focus groups (eg, use of acronyms). The table remains largely unedited so as to accurately convey the original work. Readers are invited to adapt or modify these descriptions as appropriate for their own use.

Conclusion

The College of Family Physicians of Canada has developed a resource for the assessment of professionalism that reflects the context of practitioners. Themes and observable behaviours can guide observation and discussion as part of the experiential learning of professionalism. The results of the current research have been presented as guides to facilitate meaningful observation, feedback, and discussion. We encourage teachers and learners to take ownership of these evaluation objectives. Ownership might involve the rephrasing or reorganization of the themes or behaviours. Through this process, value differences can be recognized and adjustments made or limits established.

EDITOR’S KEY POINTS

• Family physicians have identified professionalism as one of the essential skill dimensions for competence in their practice settings.

• In a previous postal survey, family physicians provided a rich and detailed description of how professionalism manifests itself in their practice settings. This description helped to identify themes and formed the basis for the generation of specific observable behaviours related to each theme.

• Giving and receiving feedback about professionalism can be challenging. Access to clear descriptions of professional behaviour in practice settings can assist the observation and dialogue necessary for experiential learning in this domain.

POINTS DE REPÈRE DU RÉDACTEUR

• Les médecins de famille ont indiqué que le professionnalisme est un des aspects essentiels de la compétence dans leur milieu de pratique.

• Dans une enquête postale antérieure, les médecins de famille ont décrit de façon détaillée comment se manifeste le professionnalisme dans leur travail. Cette description a facilité l’identification des thèmes et a servi de base au développement des comportements spécifiques observables propres à chaque thème.

• Il peut s’avérer difficile de donner ou de recevoir des commentaires sur le professionnalisme. L’accès à des descriptions claires relatives à un comportement professionnel au travail peut faciliter l’observation et le dialogue qu’exige une formation expérientielle dans ce domaine.