Multisource feedback: an overview of its use and application as a formative assessment

Abstract

Multisource feedback (MSF), often termed 360-degree feedback, is a formative performance assessment in which data about an individual’s observable workplace behaviors are collected through questionnaires from those interacting with the individual; data are aggregated for anonymity and confidentiality; the aggregated data, along with self-assessment if available, are provided to the individual; and the recipient meets with a trusted individual to review the data and develop an action plan. It is used along the continuum of medical education. This article provides an overview of MSF’s utility, its evidence base and cautions.

Introduction

This manuscript was based on a White Paper submission to the Medical Council of Canada (MCC). The MCC established an Assessment Innovation Task Force to inform the MCC regarding what it should be assessing, how, when, and why. The request recognized that this would require consideration of what the MCC was uniquely positioned to offer to Canadian medical care, how its efforts would ideally fit within the Canadian medical education landscape, and what was missing in its current activity. To that end, the Task Force approached a variety of stakeholders in search of white papers on a variety of assessment topics that could be used to inform the complexity underlying how licensing processes fit (or could fit) into modern healthcare and healthcare education. Multisource Feedback (MSF) was one of the papers solicited. As will be shown, MSF has been used occasionally in Undergraduate Medical Education (UGME), more frequently in Post Graduate Medical Education (PGME) and most commonly for physicians in practice where it is used as part of continuing professional development (CPD) as an approach to facilitate continuing competence or certification.

What is MSF?

Multisource Feedback (MSF) is a formative performance assessment with a four-stage process whereby (1) data about an individual’s observable workplace behaviors are collected through questionnaires from those interacting with the individual; (2) data are aggregated for anonymity and confidentiality; (3) the aggregated data, along with self-assessment if available, are provided to the individual in a report; and (4) the recipient meets with a trusted individual to review the data and develop an action plan.^1,2 The goal of MSF is to provide performance feedback in a manner which enables the recipient to reflect upon it and use for ongoing development.

What is the evidence base for MSF?

MSF provides formative assessment. Assessments of reliability demonstrate it doesn’t achieve a level appropriate to make summative decisions based on the scores and narrative from this assessment tool. As with all assessment tools used in medicine, evidence for validity and reliability are temporal and contextual. This means that as new MSF tools are developed or adapted for different contexts, the psychometrics for each should be examined using Kane’s evidence for validity.^3.4 Nonetheless, MSF can be part of the data used for decision-making⁵ and may trigger other assessments or monitoring.

There are several reviews demonstrating evidence for validity for use of the data (scores/ratings) with practicing physicians;^6,7 however Stevens’ et al.⁷ caution from their review of eight systematic reviews:

This review has demonstrated that the evidence base supporting the statistical and psychometric properties of MSF is sufficient. The internal structural validity of MSF has been repeatedly tested, with feedback instruments often demonstrated to be statistically reliable methods of performance assessment. What is also apparent, although the size of the evidence base is smaller, is that the results of MSF assessments often correlate highly with other WBA [Workplace Based Assessment] methods. Finally, sufficient evidence also exists to demonstrate that MSF is a feasible method of assessing medical performance in terms of cost, time, and response rates. We have also shown however that validity evidence is currently lacking in three areas: (1) how best to ensure that MSF tools measure what they intend to measure (content validity); (2) how best to maximize positive impact on practice (consequential validity); and (3) how to ensure that the process of assessment delivery is rigorous, robust, and free from bias (response process validity).^7p267

Despite Stevens’ et al.⁷ caution, it must be noted that their review was a review of reviews and as such doesn’t capture all facets of validity evidence. As noted by Lockyer et al.,⁸ content validity, consequential validity and response process validity have been examined for two UK and one Canadian instrument drawing on criteria for good assessment.⁹ Recent studies have examined consequential validity and demonstrated the importance of providing quantitative and qualitative data from MSF instruments, along with a facilitated discussion to ensure outcomes.^10-13 While there are other approaches, including providing a peer with a script for discussion¹⁰ , the R2C2 model, which involves a facilitator building a relationship, exploring reactions to the feedback, exploring the content of the feedback and coaching for change,¹⁴ has demonstrated its effectiveness as an approach to facilitated feedback.^11,12

MSF may be combined with other tools.^5,15 As noted by van der Meulen,¹⁵ while MSF tools can provide robust performance information, no instrument is able to capture the whole complex construct of physicians’ professional performance, particularly when decisions are required for promotion, remediation, or suspension/termination.

There are publications examining aspects of validity in PGME⁵ as well as with medical students.^16-20

For these reasons, ongoing evaluation of MSF initiatives are critical to ensure that the MSF tools being used can demonstrate evidence of validity and reliability for the specific context in which they are being used and the data and approaches to feedback discussions and follow-up are optimized to ensure utility for the end-user. Despite the fact that MSF is a formative assessment, attention to the four stages is important. Each stage is important with individual characteristics and requirements. Hence, evidence is required to support each of the four stages of MSF—namely data collection, data aggregation, data provision to recipient and a discussion with a trusted individual to address data and develop an action plan. A lack of rigour in any stage can result in failure of the process and impact on participants acceptance and use of the data.

Why is MSF necessary?

MSF has been used to assess all CanMEDS Roles, either implicitly or explicitly,^21,22 those more commonly and perhaps more appropriately assessed are Communicator, Professional, and Collaborator. There are practical reasons for this. These Roles can be readily observed by other health professionals and patients and are not readily assessed using other approaches. Other tools are available for assessing other Roles. For example, performance in the Medical Expert Role is more readily and accurately assessed using through multiple choice questions, direct observation of procedures, OSCE’s and performance audit.²³ There are specialized instruments for the assessment of Leader.^13,24,25 Health advocate and Scholar Roles are less easily observed and rated using an MSF format. The inclusion of items that can’t be observed has the potential to undermine the overall value for the recipient and can lead to 'guessing' by raters. When those assessed receive feedback on performance which they don’t believe their raters can observe, or data they don’t believe their raters have access to, they question its validity and report itself.

Communicator, Professional, and Collaborator Roles are critical to patient safety.^26-29 It is known that poor communication with patients, colleagues and coworkers and unprofessional behavior negatively influence patient care and outcomes. For example, inadequate handover communication results in errors that impact patient safety. Similarly, unclear or inappropriate communication in any way compromises patient care (e.g., unclear discharge plans, communication with the health care team, and explanation and planning with patients). Likewise, unprofessional behavior that prevents open and respectful communication by all involved in a patient’s care can compromise safe care. Conversely, good communication and collaborative skills handled in a professional way will enhance patient safety.

As noted earlier, MSF’s primary use to date appears to be for practicing physicians. The goal is to guide their ongoing development and creation of learning plans through reflection upon and comparison of feedback data received from various sources.³⁰ In Canada, the Medical Council of Canada has transformed the former Physician Achievement Review Program developed by the College of Physicians and Surgeons of Alberta into the Medical Council of Canada’s MSF program, MCC 360.³¹ The new instruments assess and provide feedback on Professional, Collaborator, and Communicator Roles. It is currently being used by medical regulatory authorities as part of continuing competence programs on a volunteer or mandatory basis in some provinces (British Columbia, Alberta, and Manitoba), The Practice Enhancement Program for Saskatchewan Physicians (PEPSask) as well as by hospitals (The Ottawa Hospital and North York General Hospital).³¹ There are other MSF programs available to physicians, through their health systems or privately, as well. For example, PULSE 360 is used to assess leadership skills across the USA and Canada.^13,32 Internationally, MSF is also being used in PGME (residency) programs.^5,35-39 In Canada, for the most part, the initiatives appear to be localized, being developed within programs for that program and/or that University.^35-38

While MSF has been used less frequently in UGME, two important initiatives have been reported. There is a broadly based instrument being used in Germany that assesses several facets of competence including responsibility, teamwork, empathy, active listening to patients and other competencies that has good evidence of validity.^19,20 In addition, the MCC is currently engaged in a UGME pilot project involving four Canadian universities in a study with up to 25 clinical clerks per school. The rater groups were modified from the MCC 360 for practicing physicians to include self, patient/caregivers, and a mix of supervisor, resident, and peers. The MCC UGME surveys focus on Communicator, Collaborator, and Professional Roles and include three types of questions for raters: frequency (never to always and unable to assess), yes/no options and open-ended questions. Other published applications with medical students include assessments of interprofessional competencies,¹⁶ communication skills¹⁷ and professionalism.¹⁸

What are the opportunities and cautions with MSF?

MSF provides a unique opportunity to provide feedback related to the Communicator, Professional and Collaborator Roles. Due to the interpersonal nature of these roles, assessing and providing feedback on specific behaviours included in each is often challenging. Direct observation is most frequently used, yet the idiosyncratic nature of direct observation and infrequent reporting, can lead to gaps in accurate assessment. Nonetheless, Regulatory Authorities and health systems continue to identify Communication, Professionalism, and Collaboration as areas receiving larger proportions of formal patient complaints than other roles^40-42 suggesting a continuing need to adopt MSF as an approach to continuously monitor and improve physician skills in these areas. Facilitated discussions provide physicians and medical students with an opportunity to gain a further understanding and interpretation of their data reports and use the data to develop an action plan.^11-13,43

However, there are a number of cautions to consider when deciding to include MSF amongst the tools for assessment and feedback.

MSF is expensive in terms of the opportunity costs associated with other methods that might be used as well as the direct and indirect costs associated with implementing a new program. Leaders, raters, and those assessed need to be assured that there will be a potential gain in terms of the information provided to inform decision making.

There are real costs as well. MSF requires a stable infrastructure to sustain. Systems can be developed on an ad hoc or pilot basis. In the longer term, implementation requires leadership to obtain full support from people in the organization to sustaining the program. It also requires participant commitment to an orientation of those being assessed and those doing the assessment to ensure all parties have consistent ideas related to the purpose, rationale, questions, scales, and use of the data. Without a stable infrastructure and commitment, it can be difficult to sustain rigorous MSF initiatives⁴⁴

There are human resources issues to consider as well in terms of the numbers of assessors and the availability of trained individuals for de-briefing sessions. In assessing whether an instrument produces reliable results (i.e., reliability coefficient G > 0.70), the number of items on the questionnaires and numbers of assessors are analyzed. Research across a number of studies and systematic reviews suggests that this can be attained with 5 to 11 physician assessors, 10 to 20 non physician assessors, and 25 to 50 patient assessors.^6,45-47 Too many raters adds workload and runs the risk that the raters haven’t observed the physician to do an accurate assessment of performance. Too few raters may impact on confidentiality and anonymity of results.

Facilitated discussions are critical to guide the physician or learner in the co-development of the learning change plan and its follow-up.^11-13 These studies and other publications^{1,2,30,43,48,49} have demonstrated the importance of discussion and follow-up and are informing best practices for MSF. The de-briefer or facilitator requires training and may be someone appointed for this purpose for medical students and residents so that they are at arms-length from other evaluation and assessment processes or it may be a preceptor or supervisor. For licensed physicians, it may be a division/department head for licensed physicians, or in the case of a medical regulator, it may be another physician who has received training for this role.

Another caution is that MSF can be ‘gamed.’ As Alofs et al.³⁴ noted, if learners and raters perceive the instrument will be used for decision making and not for development, raters selected for MSF may deliberately adapt their scoring or comments compromising the value of the feedback as a formative tool for growth and development. Preventing such bias requires that MSF development and implementation ensure attention is paid to communication throughout the development and implementation of data collection, reporting and de-briefings.

Conclusion

MSF has the potential to offer medical students, residents, and licensed physicians invaluable information about their workplace performance, which is then available for them to use for ongoing development. The goal is continued learning and improvement through application of the feedback. In order to achieve this goal though, particular attention needs to be paid when developing and implementing MSF to ensure (1) high quality instruments that are psychometrically robust, provide evidence for validity and will provide data to guide recipient behavior are created, (2) data are aggregated and anonymized to ensure confidentiality of respondents, (3) the individual receives their data in an easy-to-understand format and (4) the individual is engaged in a facilitated, reflective discussion of their data and guided in using them to develop an action plan, ideally with later follow-up to confirm progress.

Conflicts of Interest

The authors do not have any conflicts of interest.

Funding

None