The Desired Concept Maps and Goal Setting for Assessing Professionalism in Medicine

Abstract

Due to the multi-dimensional characteristics of professionalism, no single assessment modality has shown to reliably assess professionalism. This review aims to describe some of the popular assessment tools that are being used to assess professionalism with a view to formulate a framework of assessment of professionalism in medicine.

In December 2015, the online research databases of MEDLINE, the Educational Resources Information Center (ERIC), Elton Bryson Stephens Company (EBSCO), SCOPUS, OVID and PsychINFO were searched for full-text English language articles published during 2000 to 2015. MeSH terms “professionalism” AND “duty” AND “assessment” OR “professionalism behavioural” AND “professionalism–cognitive” were used. The research articles that assessed professionalism across medical fields along with other areas of competencies were included. A final list of 35 articles were selected for this review.

Several assessment tools are available for assessing professionalism that includes, but not limited to, mini clinical evaluation exercise, standardised direct observation of procedural skills, professionalism mini-evaluation exercise, multi-source feedback and 360 degree evaluation, and case based discussions. Because professionalism is a complex construct, it is less likely that a single assessment strategy will adequately measure it. Since every single assessment tool has its own weaknesses, triangulation involving multiple tools can compensate the shortcomings associated with any single approach. Assessment of professionalism necessitates a combination of modalities at individual, interpersonal, societal, and institutional levels and should be accompanied by feedback and motivational reflection that will, in turn, lead to behaviour and identity formation. The assessment of professionalism in medicine should meet the criteria of validity, reliability, feasibility and acceptability. Educators are urged to enhance the depth and quality of assessment instruments in the existing medical curricula for ensuring validity and reliability of assessment tools for professionalism.

Introduction

Professionalism is a multi-dimensional construct that has been shown to demonstrate variations across educational, regional and cultural contexts [1]. There is a growing evidence that professionalism in medical medicine has a potential role in developing professional excellence in medical students and physicians [2,3]. However, the absence of a comprehensive and universally acceptable definition of professionalism has currently limited its operationalisation [4]. There have been a proliferation of definitions of professionalism [5–7], that perhaps follow the escalating community concerns and public disquiet over clinical incompetence, and unprofessional behaviours by medical fraternity [8]. This urges educators and policy makers to develop effective teaching and assessment strategies of professionalism that can be conveniently embedded into medical curricula. “Professionalism needs to be assessed if it is to be viewed as both positive and relevant” [2]. Assessment of professionalism is a key dimension of the recommended guidelines by the General Medical Council (GMC) for undergraduate curricula and all four domains for appraisal as well as re-validation [9]. Nevertheless, literature has provided simple outcome measures of the teaching and learning experiences with positive impact on attitudes towards professionalism and professional behaviour in medicine [10,11]. Nevertheless, these findings fail to validate the validity and reliability of assessment modalities that are used to assess professionalism in medicine.

This review elaborates on the effectiveness of the currently used popular tools that can be applied for assessing professionalism in medicine. The gaps in the assessment strategies are also highlighted with an attempt to advance the knowledge about how to effectively assess professionalism in medical field.

Research Design

Data sources

In December 2015, two authors independently searched the online research databases of MEDLINE, the Educational Resources Information Center (ERIC), Elton Bryson Stephens Company (EBSCO), SCOPUS, OVID and PsychINFO for full-text English language articles published during 2000 to 2015. MeSH terms used in this search included “professionalism” AND“duty” AND “ethics” AND “assessment” OR “professionalism–behavioural” AND “professionalism–cognitive”. Furthermore, a manual search was conducted on the reference lists of retrieved articles and literature reviews. Only those articles pertaining to medical education providing empirical evidence were selected in this research. Editorials, short communications, opinions, letters to the editor, abstracts and descriptive papers without original data were excluded. The research articles that assessed professionalism across medical fields along with other areas of competencies were included. The reviewers pooled their findings and thus generated a single list of selected articles.

Data Analysis and Results

Initial search retrieved 498 articles. Search on the manual reference list yielded another 49 references. Of these 547 articles, 310 were excluded as they did not meet selection criteria. During further detailed review of their abstracts, 190 publications were excluded. Finally, 35 articles were selected for this literature review [Table/Fig-1].

[Table/Fig-1]:

Schematic presentation of selection of studies the strategies for assessing professionalism in medicine.

General principles of assessment in medicine

In medical education, competencies are task-based performances that a qualified medical professional should be able to perform successfully [12]. Medical competence encompasses a mix of measurable constructs such as knowledge, skills, problem-solving and attitudes [13]. van Mook et al., have argued that assessment of medical competence should measure performance in everyday practice [14]. Four fundamental domains of assessment should be met with; validity, reliability, acceptability and feasibility [15]. The weightage and application of each instrument need to be balanced depending on the context and objective of assessment. In high-stake examinations, reliability may be given priority to the choice of assessment modality [16]. In formative assessments, where the final decision is based on triangulation of a variety of assessments, the educational impact of assessment can be favoured at the expense of reliability [17].

The assessment of surgical competence is even further complex and challenging as this involves a construct of assessment tools to measure surgical knowledge, skills, and attitudes. Guraya et al., gathered the preferred modes of surgical education and training from the participants of a state-of-the-art laparoscopic surgical training center, Advanced International Minimally Invasive Surgery Academy (AIMS) in Milan Italy and concluded that the surgical trainees preferred a blend of training and assessment modes including hands-on training and training by skilled tutors [18]. This blend of preferred learning styles, instead of a single stand-alone entity, has been well documented in several studies [19,20]. A very popular assessment tool, the Objective Structured Clinical Examination (OSCE) has the potential to objectively assess the knowledge, skills and practice in medical field with fair degree of validity and reliability [21]. However, examiner bias has been shown to negatively influence the desired outcome.

The value of measuring professional behaviour in medicine

The main challenge of assessing professionalism is its multidimensional construct that requires a diversity of approaches for valid and reliable assessment [1]. The cornerstone of assessment of professionalism is the measurement of professional behaviour as this domain has been acknowledged to be reflective of the underlying cognitive, attitudinal, and personality characteristics of professionalism. However, from the perspective of socio-cognitive psychology, attitudes have been considered to be poor predictors of behaviour particularly in the presence of strong external constraints and social pressure [22]. “An individual’s behaviour is more likely to be influenced by situational and contextual phenomena arising during learning and practice than by their underlying attitudes”[23]. This phenomenon of ignoring contextual background has the potential to unfairly label the physicians and medical students as ‘unprofessional’[24]. These situationally-dependent professional characteristics must be given due attention while assessing the context driven nature of professional behaviours.

While selecting a mode of assessment, this seems imperative to be aware of the main purposes; to provide feedback to students that will enable them to improve and to measure the achievement of course learning objectives [25]. Failure to assess the core values of professionalism can deliver conflicting messages to all stakeholders including students and practising clinicians that may lose the benefits of teaching and assessing professionalism [26]. The process of assessing professionalism will provide a deep insight into the knowledge and understanding of learners about their professional competency, honesty and confidentiality with patients, improved quality of care, scientific knowledge, professional responsibilities, and trust [27].

The assessment of professionalism in medicine

No single instrument can be employed to assess each competency pertinent to complex multi-dimensional constructs of professionalism. A blend of instruments and triangulations, therefore, should be used [28]. A range of assessment tools as suggested by Goldie is shown in [Table/Fig-2] [1].

[Table/Fig-2]:

The range of modalities for assessing professionalism.

Assessment tool	Types of assessment tool
Observed clinical encounters	-Mini-CEX -Professionalism mini-evaluation exercise -Standardised direct observation assessment tool
Collated views of co-workers	-360 degree evaluation
Records of incidents of professional lapses	-Incident reporting form
Critical incident reports
Simulations	-Ethical dilemmas with patient simulations -OSCE
Paper-based tests	-Defining issues test -Objective structured video examination -Critical incident report
Patient surveys	-Patient assessment questionnaire -Simulated patient rating scales -Humanism scale
Global observer ratings	-Global rating form -Professionalism assessment instrument -Amsterdam attitudes and communication scale
Self-administered rating scales	-Time management inquiry form -Pharmacy professionalism instrument -Cross-cultural adaptability inventory -Cultural competence self-assessment questionnaire -Interpersonal reactivity index

Precisely, five most popular assessment modalities that are currently used to measure professionalism in medicine are elaborated in the following section.

1. Mini- Clinical Examination Exercise (mini-CEX)

An assessor examines a trainee–patient interaction in a hospital and such clinical encounters last for about 15 minutes where the trainee is expected to take a focused history and/or physical examination within a defined time period [29]. Finally, the trainee proposes a working diagnosis and management plan and the performance are ranked using a structured evaluation form. At the end, a constructive feedback is provided. Assessors use a nine-point Likert scale ranging from ‘unsatisfactory’ to ‘superior’, which gives a six domain-specific ratings and one final rating of clinical competence. During one year, the trainees appears in about six clinical assessments with a different assessor for each session [30].

2. Standardised Direct Observation Assessment Tool (SDOPS)

SDOPS is perceived to be an effective and reliable learning and assessment strategy for trainees to enhance their work-place based assessment and performance [31]. SDOPS assessments are customized to be conveniently incorporated into the trainees’ and assessors’ daily practice that speaks volumes of this assessment tool in terms of feasibility and effectiveness [32]. During SDOPS, the assessor examines a trainee’s performance while performing a procedure in real-time environment. This event is followed by a face-to-face feedback and the trainee gets a scored evaluation of his/her performance for clinical and procedural skills [33].

3. Professionalism mini-evaluation exercise (P-MEX)

The development of P-MEX has been based upon the Mini-CEX [34] and the reliability and validity of this assessment instrument has been found to be reasonably fair [35]. The items included in the P-MEX were originally generated at workshops held at McGill University. The mini-MEX assesses the clinical skills of residents, in which the examiner rates residents about their performance on professionalism, clinical reasoning, interviewing, physical examination, counseling, unit organizational skills, and efficiency using a nine-point rating scale [36].

4. Multi source feedback and 360 degree evaluation

The 360-degree tool consists of a set of questions that can explore an individual’s knowledge about professionalism, communication skills, interpersonal style, leadership, and teamwork [37]. The evaluation of performance from a single source, such as a supervisor, subordinate, or patient, can have inherent inaccuracies and can be affected by bias with the “halo and horn” effect [38]. To circumvent this potential element of undue leniency and unfairness in assessment, the concept of the 360⁰ feedback has been coined that gathers feedback from several sources with varying roles in an individual’s work environment and serves to generate a comprehensive perspective on performance [39]. The 360⁰ feedback solicits feedback from multiple sources within a physician’s work environment, including peers, supervisors, and subordinates, carrying a promise of comprehensive global assessment of performance, thus eliminating bias due to race, sex and culture [40]. The resulting dossier of information has been conveniently and effectively used as a means for a framework of professional development and to track employee progress in any given institution [41].

5. Case based discussion (CbD)

During the CbD, the trainee is given the opportunity to select the timing, the records, and even the assessor [42]. A few days before the assessment, the trainee chooses a falls well within the goals and learning objectives of the curriculum that is then discussed using focused questions dedicated to elicit responses for a comprehensive assessment of knowledge, skills, attitudes and behaviours relevant to all domains of professionalism. At the end of discussion, the assessor rates the performance and, finally, provides constructive feedback. On average, trainees are assessed six times during the year. A working plan for a typical CbD encompasses planning the case, main event of discussion and evaluation, and feedback [43].

Traditionally, Miller’s pyramid is regarded as a benchmark in the assessment of clinical skills, performance and competencies [44]. [Table/Fig-3] illustrates a schematic algorithm using the range of assessment strategies that can be conveniently applied for assessing professionalism in medicine.

[Table/Fig-3]:

The range of assessment tools that can be used for the assessment of professionalism in medicine in terms of Miller’s pyramid.

Some models for assessing professional behaviour in medicine

A study by van Lujik et al., has demonstrated three main sets of the required domains for assessing professional behaviours [45];

• - Reliability of the situation (intra-observer), assessors, and assessment modalities.

• - Validity of the situation, judges (qualified), and rating scales (consequential validity).

• - Acceptability by the students and institutions

On the same note, Veloski et al., has argued that “the assessment of professional behaviour should meet the criteria of validity, reliability, feasibility and acceptability” [46].

Professional performance is context specific and student performance will vary from case to case that necessitates broader assessment across a range of contexts [47]. This demands a wide range of assessment tools across the curriculum with rigorous attention to their validity and reliability. Furthermore, its vital to collate these finding and observations over time and carefully triangulate before a holistic judgement can be passed [48]. Such approach signals the use of portfolios collating indicators of performance such as attendance records, multi-source feedback, performance in OSCE, reflective writing on critical incidents that can help capture a positive sketch of a learner’s professional performance across different contexts. This can be prompted by embracing the interactions at individual, interpersonal, societal, and institutional levels.

The School of Medicine and Health at Durham University developed a conscientiousness index that attempted to explore whether students performances completing specific tasks over several domains can be used to assess their professional behaviours [49]. The domains included attendance, timely submission of class work, and participation in evaluation and research. A satisfactory accomplishment of tasks was awarded one conscientiousness point. Each student’s point score was compared to a grade given by a faculty member. A significantly positive correlation has been reported between the conscientiousness index points score and the grade given by the faculty member. “The conscientiousness index has also been found to correlate with a student peers estimation of their professionalism” [50].

The academia at Groningen University introduced peer assessment within small group work for improving student professional behaviours [51]. This peer assessment employed three domains of communication, task performance and personal performance. The students were grouped into control and peer assessment batches once per trimester over a period of two years. Both groups were assessed by tutors and the scores were compared. A significantly enhanced performance was reported in the task performance and personal performance domains of the intervention group. “The study inferred that the peer assessment did have a positive impact on student professional behaviours for the students who were more accustomed to the complex learning environment of undergraduate medicine”.

The University of Michigan developed a self-assessment questionnaire for measuring the professionalism of surgical residents [52]. The instrument contained 15 attributes of professionalism and the respondents were asked to choose one statement from a menu of seven options for each of the attributes. All statements had the extremes of scales such as undesirable examples for every single attribute. Although this instrument was found to be effective and feasible for measuring surgical resident’s professional behaviours, it failed to show reliability for assessment [52].

Recommendations for assessing professionalism in medicine

• - Assessment in multiple settings can help determine the accurate level of an individual’s professionalism and can identify context specific views and knowledge-based skills relevant to learners. Thorough and systematic assessment of professionalism mandates the involvement of several assessors and uses more than one assessment method and assessment in different settings [35]. A body of evidence-based literature has indicated that the engagement of multiple assessors offer several perspectives, thus enhancing the breadth of assessment [53–55]. Per se, this engagement of multiple assessors enhances reliability and effectiveness of assessment. Each assessment strategy has merits and demerits; rating forms are considered relatively easy to use, but are plagued with the ‘halo’ or ‘horns’ effect [56]. The OSCE may overcome this shortcoming but needs huge investment in terms of cost, time, and resources [57]. Consequently, using triangulation that employs more than one assessment tool may help to cover the shortcomings associated with any single approach.

• - “The assessment of medical student professionalism is often delayed until clerkship rotations, however, research indicates that it is both desirable and possible to begin assessing student professionalism during the first year of medical school” [58]. The initial use of this assessment should be formative where the students are invited to present their perspective and this information is used to provide feedback and guide remediation.

• - Persistent patterns of unprofessional behaviour, despite remediation, may be a strong impetus for dismissal [59].

• - The practical strategies to assessing physician professionalism include a 360o evaluation, SDOPS, mini-CEX, OSCE, CbD and a cognitive assessment of professionalism.

• - Faculty development program in medical schools carry great promise in fostering the knowledge and teaching skills of the medical faculty that will, in turn, be reflected in the practice of the future doctors [60].

Conclusion

As professionalism is context based, each institution should develop an explicit definition of professionalism that should resonate with the societal norms and the core elements of professionalism. A judicious combination of assessment methods need to be incorporated in a longitudinal manner using a stepwise approach for assessing professionalism in medicine.

Financial or Other Competing Interests

None.