Abstract
Background:
Modern approaches to clinical evaluation emphasize the value of learner-driven feedback models, where trainees are encouraged to take an active role in the initiation of the evaluation process. In an effort to empower medical students to request evaluations on performance, our medical school developed a web-based application for mobile devices that prompts learners to solicit feedback electronically following a clinical encounter. In 2016, mandatory implementation of this application resulted in a transition from an educator-driven feedback model to a learner-driven feedback model. We aimed to investigate the impact of this innovative system on both the quality and quantity of feedback provided to medical students on their anesthesiology elective.
Methods:
We retrospectively analyzed medical students' feedback data from the sequential academic years prior to and after the implementation of our learner-driven feedback application. Quantitative analysis was performed to compare the frequency of evaluations requested and completed. Free-text commentary was analyzed using conventional qualitative content analysis. Comments were categorized by quality and representative themes based upon Accreditation Council for Graduate Medical Education Core Competency domains.
Results:
A total of 297 evaluations for 72 students were analyzed. Students in the learner-driven model requested feedback more frequently than the previously educator-driven system (13.4 vs 8.9 requests per student, P < .0001). Additionally, a greater proportion of assessments were completed by evaluators when solicited from the learner-driven model (42% vs 34%, P = .0265). The quality of comments solicited from the learner-driven model were of higher quality when addressing students' strengths (71% vs 30%, P < .00001) and of lower quality when addressing areas of improvement (73% vs 59%, P = .0378). Comments from the learner-driven model were more likely to address Patient Care (48% vs 24%, P < .00001) and less likely to address Interpersonal and Clinical Communication (17% vs 28%, P = .0037) compared to the educator-driven model.
Conclusions:
A learner-driven feedback model was successful in improving the quantity of both requested and completed evaluations for students. The quality of feedback was also improved when addressing students' strengths.
Keywords: Feedback, learners, education, evaluation, anesthesia
Introduction
In 1999, the Accreditation Council for Graduate Medical Education (ACGME) endorsed 6 Core Competencies to establish principles for the education and evaluation of physicians.1 Medical knowledge is insufficient alone to determine competency, and medical educators must consider the contributions of professionalism, communication skills, clinical reasoning, technical skills, and systems-based practice to overall clinical performance.1 Contrary to the assessment of knowledge through written examination, the evaluation of these additional attributes involves formative feedback after an observed encounter. A cornerstone of clinical education, feedback is effective and beneficial when it is frequent, timely, and actionable.2,3
Formative feedback in clinical education is an essential tool to improve trainees' performance over time.2,4 However, to maximize its impact, feedback must be of substantive quality and delivered promptly.2,3 These key attributes of effective feedback allow the learner to recall his or her immediate performance, thereby providing the best opportunity for implementing recommended modifications to future practice.2,3,5 Investigating processes to improve the caliber of feedback is a valuable supplement to educational research.
In his historic paper on clinical feedback, Ende declares that those accountable for evaluating a “subordinate” are obligated to provide feedback.6 He further describes that “the hierarchy of the teaching hospital—attending, resident, intern, and student—allows for an orderly flow of information.”6 Traditionally, this tiered approach to feedback has resulted in educator-driven delivery of content from supervisor to trainee,5,7–9 thus minimizing input from the recipient.5
Conversely, more modern approaches to providing clinical feedback emphasize the value of learner-centric models, which encourage the trainee to take an active role in the feedback process.10,11 Reported benefits of student-driven feedback models include enhanced adult and self-regulated learning, as well as promotion of accountability.10,12 In 2017, Tanaka et al published a study of anesthesiology residents at Stanford who expressed a desire to feel empowered and comfortable with initiating feedback from faculty.13 Focus group residents in that study suggested a tool or instrument be created to facilitate requesting feedback from evaluators.13
The Education Design and Informatics Team at Vanderbilt University School of Medicine developed a web-based, integrated learning platform called VSTAR, which is composed of specific applications for course websites (VSTAR Learn), grades (VSTAR Grades), student portfolios (VSTAR Portfolio), and evaluations (VSTAR Compass). In an effort to empower medical students to request feedback on their clinical performance, the VSTAR Compass application prompts learners to solicit feedback electronically through their mobile devices. Immediately following a clinical encounter, students use VSTAR Compass to send an online assessment request to an evaluator.5 Once completed, the form is then viewable by the student, course director, and portfolio coach. The VSTAR learning platform and associated applications were adopted by all clinical courses within the School of Medicine.
The VSTAR Compass application was implemented for our 4-week anesthesiology elective during the 2016–2017 academic year, thus resulting in a transition from our previously educator-driven feedback process to a new, learner-initiated model.5 Importantly, the evaluation form and content remained constant during the years before and after the implementation of the VSTAR Compass application. Therefore, we aimed to investigate the impact of this innovative system on feedback by retrospectively comparing medical students' evaluation data from before and after the institution of VSTAR Compass. We hypothesized that use of the VSTAR Compass application would increase both the quantity and quality of formative feedback compared to our traditional educator-driven assessment method.
Materials and Methods
The Vanderbilt University Institutional Review Board approved this retrospective study for exempt status, and the requirement for written informed consent was waived. This manuscript adheres to the applicable SQUIRE (Standards for Quality Improvement Reporting Excellence) and SRQR (Standards for Reporting Qualitative Research) guidelines.14,15
Feedback evaluation data were obtained for third and fourth year medical students enrolled in a 4-week clinical anesthesiology elective for the 2015–2016 academic year. During this time, feedback was educator-driven, as residents and faculty completed evaluations of students with whom they worked through a REDCap16 survey link that was sent on a weekly basis to appropriate staff paired with the student based upon the rotation schedule. In contrast, for the subsequent 2016–2017 academic year, medical students used VSTAR Compass to initiate and request feedback after an observed clinical encounter. Both feedback modalities consisted of the same evaluation questions and allowed the evaluator to enter free-text comments regarding strengths and areas for improvement.
We retrospectively analyzed feedback data from the sequential academic years, thus comparing VSTAR Compass evaluation data to data from our previous educator-driven assessment process. We employed a triangulation model of mixed-methods research through investigation of both quantitative and qualitative data.17 Quantitative data were analyzed using simple descriptive statistics. Free-text comments were analyzed using conventional qualitative content analysis by two authors (LF, AR). Comments were categorized into representative themes based upon the clinical competencies established by the ACGME.1 Additionally, free-text comments were classified by strength and quality into 1 of 3 groups: high-quality feedback (specific, actionable, addresses knowledge or skill gap), low-quality feedback (non-specific, irrelevant, eg, ‘asks questions’ or ‘keep reading’), or equivocal feedback (eg ‘not applicable’). Comments were reviewed and coded independently using an inductive, grounded approach, and disagreements resolved through consensus.18
Statistical Analysis
Data were entered into GraphPad Software (GraphPad Software, San Diego, California) for simple descriptive statistics. A student t test and Fisher exact test were performed to compare continuous and categorical variables, respectively.
Results
A total of 297 evaluations for 72 students were analyzed. The number of enrolled students and total number of feedback assessments requested versus completed per academic year are listed in Table 1. On average, students in the learner-initiated system (VSTAR Compass) requested feedback more frequently than through the educator-driven system (13.4 vs 8.9 requests per student, P < .0001). Additionally, a greater proportion of assessments were completed by evaluators when solicited from the learner-initiated model (42% vs 34%, P = .0265).
Table 1.
Comparison of Enrollment and Quantity of Evaluations Between the Educator-Driven and Learner-Initiated Feedback Models
| Educator-Driven Feedback Model (Academic Year 2015–2016) | Learner-Initiated Feedback Model (Academic Year 2016–2017) | p-value | |
|---|---|---|---|
| Students enrolled (n) | 43 | 29 | |
| Assessments requested (n) | 384 | 390 | |
| Assessments completed (n) | 132 | 165 | |
| Requested assessments per student (mean, SD) | 8.9 (1.3) | 13.4 (5.9) | <0.0001 |
| Completed assessments per student (mean, SD) | 3.1 (1.5) | 5.7 (2.8) | <0.0001 |
Subgroup analysis was performed to assess for variation among students interested in pursuing anesthesiology residency versus another specialty (Table 2). For both feedback models, the mean number of evaluations requested and completed per student did not significantly differ between learners interested in anesthesiology versus those intending to match into other fields. However, consistent with overall analysis, students in the learner-initiated system both requested and received evaluations more frequently compared to the educator-driven model, irrespective of their interest in anesthesiology. (Table 2)
Table 2.
Comparison of Enrollment and Quantity of Evaluations for Both Feedback Models by Interest in Anesthesiology
| Educator-Driven Feedback Model (Academic Year 2015–2016) | Learner-Initiated Feedback Model (Academic Year 2016–2017) | p-value | |
|---|---|---|---|
| Students Interested in Anesthesiology | |||
| Students enrolled (n) | 15 | 12 | |
| Assessments requested (n) | 140 | 180 | |
| Assessments completed (n) | 53 | 77 | |
| Requested assessments per student (mean, SD) | 9.3 (1.4) | 15 (7) | 0.005 |
| Completed assessments per student (mean, SD) | 3.5 (1.8) | 6.4 (3.1) | 0.005 |
| Students Interested in Other Specialties | |||
| Students enrolled (n) | 28 | 17 | |
| Assessments requested (n) | 244 | 210 | |
| Assessments completed (n) | 79 | 88 | |
| Requested assessments per student (mean, SD) | 8.7 (1.2) | 12.4 (4.8) | 0.0003 |
| Completed assessments per student (mean, SD) | 2.8 (1.3) | 5.2 (2.6) | 0.0002 |
A total of 203 free-text comments were available for 43 students in the educator-driven feedback system, compared to 283 comments for only 29 students in the learner-initiated feedback model. These comments were subjected to qualitative analysis and categorized by ACGME competency domains in Table 3. Comments pertaining to Professionalism were the most common for both educator-driven and learner-initiated feedback models, at 88% and 93% respectively (P = .0791). However, students in the educator-driven system received significantly more comments regarding Interpersonal and Communication Skills than those in the learner-initiated model (28% vs 17%, P = .0037). Conversely, students in the learner-initiated model received twice as many comments regarding Patient Care, which includes evaluation of clinical reasoning, judgment, and technical skills (48% vs 24%, P < .00001).
Table 3.
Qualitative Analysis of Free-Text Comments and Comparison of the Prevalence of Addressing ACGME Core Competencies Through the Feedback Models
| ACGME Core Competency | Educator-Driven Feedback (n = 203 comments) | Learner-Initiated Feedback (n = 283 comments) | p-value |
|---|---|---|---|
|
Medical Knowledge: demonstrates medical knowledge |
36% | 30% | 0.1711 |
|
Professionalism:
timeliness, accountability, demeanor |
88% | 93% | 0.0791 |
|
Interpersonal & Communication Skills:
effective communication with patients, families, and clinical teams |
28% | 17% | 0.0037 |
|
Patient Care: history, clinical reasoning, judgment, technical skills |
24% | 48% | <0.00001 |
|
Problem Based Learning Initiative:
continuous pursuit of learning, seeking and receptivity to feedback |
44% | 36% | 0.0904 |
|
Systems Based Practice:
team building |
16% | 19% | 0.3992 |
Abbreviations: ACGME, Accreditation Council for Graduate Medical Education.
Regarding the quality of feedback, variation was detected between comments that assessed student strengths versus those that assessed areas of improvement (Figures 1 and 2). When addressing a student's strengths, a majority of feedback from the learner-initiated model was considered high-quality feedback, compared to a minority in the educator-driven model (71% vs 30%, P < .00001). Comparatively, when evaluations addressed areas of improvement, the majority of comments in both models were classified as low-quality feedback. Nevertheless, the prevalence of low-quality feedback addressing areas for improvement was significantly higher when solicited from the learner-initiated group (73% vs 59%, P = .0378).
Figure 1:
Comparison between feedback models of the quality of free-text comments that addressed students' strengths
Figure 2:
Comparison between feedback models of the quality of free-text comments that addressed student's areas of improvement.
Discussion
In 2016, Vanderbilt University Medical School transitioned from an educator-driven to a learner-initiated feedback process through use of the VSTAR Compass mobile application. To our knowledge, our study is the first of its kind to examine the impact on both quantity and quality of feedback after such a transition.
We found that our learner-initiated model was successful in improving the quantity of both requested and completed evaluations for students. Our data did not detect a difference in the quantity of requested feedback between students with and without an expressed interest in the field of anesthesiology. This suggests that our learner-initiated model can be a successful platform for any proactive learner with an interest in general self-improvement. Importantly, our data revealed an increase in the proportion of completed evaluations when sought from the learner. As these students were proactively seeking wisdom and advice from those they deemed qualified, it is conceivable that this intentional solicitation of feedback appeals to an evaluator's sense of educational duty and responsibility to provide it.19 However, we acknowledge there are other variables that could also account for the increased response rate, such as the ability to complete the evaluation immediately after an encounter or simply the increased volume of requests.
Feedback, no matter how frequently obtained, is of limited utility if it does not provide specific and actionable direction for improvement. We therefore performed qualitative analysis to determine the strength of free-text comments when addressing both positive attributes and areas of improvement for students. In our previous educator-driven model, only 30% of free-text comments met criteria for high-quality feedback, compared to over 70% from our learner-initiated model when addressing positive merits. We suggest this improvement in quality is likely due to a similar influence on quantity; intentional solicitation of feedback prompts educators to be intentional about providing it.19 Additionally, feedback in our educator-driven system was provided on a weekly basis, whereas it could be immediately requested after a clinical encounter in our learner-initiated model. Evaluators may be able to recall an experience more accurately, thereby providing more effective feedback.2,3
Interestingly, we did not observe an improvement in quality of free-text comments when addressing areas of improvement. The majority of free-text comments were considered of low-quality in both feedback models, confirming that delivering negative feedback can be both difficult and uncomfortable.20,21 However, a greater proportion of feedback met low-quality criteria in the learner-initiated model compared to the educator-driven model (73% vs 59%, P = .0378, Figure 2). We suggest several contributing factors that might account for this difference. It is possible that students who request feedback are perceived as engaged and proactive learners, and these favorable qualities result in a positively skewed impression by evaluators. Additionally, the timing of immediate feedback could be influential, as select education literature establishes that evaluators often delay the delivery of negative feedback.20,21 This is an area for future consideration and implementation of faculty development. Lastly, students could preferentially request feedback from staff with whom they had a positive encounter. While 1:1 daily pairing with faculty or residents may help to mitigate this behavior (the practical option for feedback that day is their paired staff member), team-based environments, eg critical care, provide an opportunity for students to be selective about their evaluators. A proposed solution to this flawed design would be a hybrid feedback model that incorporates both learner-initiated and educator-driven evaluation. We acknowledge the role for nonimmediate, global feedback to provide evaluators an opportunity to comment on a student's progress over time, specifically regarding the incorporation of feedback and suggestions into practice.
Regarding the content of free-text comments, Professionalism was the most commonly addressed Core Competency for both feedback models. Promotion of Professionalism among learners has emerged as a leading trend in medical education since its establishment as an ACGME Core Competency.1,22 In fact, the American Board of Anesthesiology now incorporates formal evaluation of this domain through the Objective Structured Clinical Examination portion of the Applied Exam.23 Despite similarities in content of free-text comments within the 2 feedback models, it is interesting that we detected a significant difference in prevalence of comments pertaining to Patient Care and Interpersonal and Communication Skills. Specifically, comments solicited in the learner-initiated model were twice as likely to address Patient Care, which includes specifics such clinical reasoning and technical skills. We suggest that the timing of immediate feedback allowed the evaluators to recall a clinical experience in greater detail, thus enabling them to provide direct and detailed feedback regarding individual performance. On the contrary, Interpersonal and Communication Skills may be generalized with an overall proficiency rather than related to a precise clinical encounter; perhaps explaining the increased prevalence of these comments in the delayed educator-driven model.
Our mixed-methods approach was designed to enhance the depth and relevance of our findings.24 However, we do acknowledge limitations to our study. Although the sample size is relatively small, it is reflective of the elective's enrollment during the sequential 2 years that the feedback process was revised at our institution. We intentionally chose to limit our study to the years prior to and following the implementation of VSTAR Compass to minimize variation in faculty evaluators. This may have resulted in insufficient power for subgroup variation with secondary analyses. As evaluations could be completed anonymously, we were unable to statistically compare variation between evaluators in both groups. Additionally, our data were derived from the anesthesiology department at a single academic center. The culture of feedback processes can vary widely between departments and institutions; thus, our results may not be applicable to all educational settings. As the data were analyzed retrospectively, inherent confounding bias must be considered. Comments were archived prior to analysis, and therefore member-validation was not performed. The rigor of our qualitative analysis was maintained through strict adherence to methodologic guidelines.18
In summary, we performed both quantitative and qualitative analysis to compare the frequency and quality of feedback obtained from educator-driven and learner-initiated models of evaluation. Transition to a learner-initiated feedback model improved both the quantity of evaluations requested by students and completed by staff. The quality of comments solicited from the learner-driven model were stronger when addressing students' strengths but weaker when addressing areas for improvement.
Footnotes
Attribute Work to: Vanderbilt University Medical Center Department of Anesthesiology
Funding Statement: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Conflicts of Interest: The authors declare no competing interests.
References
- 1.Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next GME Accreditation System — Rationale and Benefits. N Engl J Med. 2012;366(11):1051–6. doi: 10.1056/NEJMsr1200117. doi:10.1056/NEJMsr1200117. [DOI] [PubMed] [Google Scholar]
- 2.Schartel SA. Giving feedback - an integral part of education. Best Pract Res Clin Anaesthesiol. 2012;26(1):77–87. doi: 10.1016/j.bpa.2012.02.003. doi:10.1016/j.bpa.2012.02.003. [DOI] [PubMed] [Google Scholar]
- 3.Bernard AW, Ceccolini G, Feinn R et al. Medical students review of formative OSCE scores, checklists, and videos improves with student-faculty debriefing meetings. Med Educ Online. 2017;22(1) doi: 10.1080/10872981.2017.1324718. 1324718. doi:10.1080/10872981.2017.1324718. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ. 2008 Nov 10;337(2) doi: 10.1136/bmj.a1961. a1961-a1961. doi:10.1136/bmj.a1961. [DOI] [PubMed] [Google Scholar]
- 5.Robertson AC, Fowler LC. Medical Student Perceptions of Learner-Initiated Feedback Using a Mobile Web Application. J Med Educ Curric Dev. 2017;4 doi: 10.1177/2382120517746384. 2382120517746384. doi:10.1177/2382120517746384. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ende J. Feedback in clinical medical education. JAMA. 1983;250(6):777–81. [PubMed] [Google Scholar]
- 7.Archer JC. State of the science in health professional education: effective feedback. Med Educ. 2010;44(1):101–8. doi: 10.1111/j.1365-2923.2009.03546.x. doi:10.1111/j.1365-2923.2009.03546.x. [DOI] [PubMed] [Google Scholar]
- 8.Telio S, Ajjawi R, Regehr G. The “educational alliance” as a framework for reconceptualizing feedback in medical education. Acad Med. 2015;90(5):609–14. doi: 10.1097/ACM.0000000000000560. doi:10.1097/ACM.0000000000000560. [DOI] [PubMed] [Google Scholar]
- 9.Anderson PAM. Giving feedback on clinical skills: are we starving our young? J Grad Med Educ. 2012;4(2):154–8. doi: 10.4300/JGME-D-11-000295.1. doi:10.4300/JGME-D-11-000295.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Rudland J, Wilkinson T, Wearn A et al. A student-centred feedback model for educators. Clin Teach. 2013;10(2):99–102. doi: 10.1111/j.1743-498X.2012.00634.x. doi:10.1111/j.1743-498X.2012.00634.x. [DOI] [PubMed] [Google Scholar]
- 11.Rogers DA, Boehler ML, Schwind CJ et al. Engaging medical students in the feedback process. Am J Surg. 2012;203(1):21–5. doi: 10.1016/j.amjsurg.2011.07.009. doi:10.1016/j.amjsurg.2011.07.009. [DOI] [PubMed] [Google Scholar]
- 12.French JC, Colbert CY, Pien LC, Dannefer EF, Taylor CA. Targeted Feedback in the Milestones Era: Utilization of the Ask-Tell-Ask Feedback Model to Promote Reflection and Self-Assessment. J Surg Educ. 2015;72(6):e274–e279. doi: 10.1016/j.jsurg.2015.05.016. doi:10.1016/j.jsurg.2015.05.016. [DOI] [PubMed] [Google Scholar]
- 13.Tanaka P, Bereknyei Merrell S, Walker K et al. Implementation of a Needs-Based, Online Feedback Tool for Anesthesia Residents With Subsequent Mapping of the Feedback to the ACGME Milestones. Anesth Analg. 2017;124(2):627–35. doi: 10.1213/ANE.0000000000001647. doi:10.1213/ANE.0000000000001647. [DOI] [PubMed] [Google Scholar]
- 14.Ogrinc G, Mooney SE, Estrada C et al. The SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Health Care. 2008;17(Suppl 1):i13–i32. doi: 10.1136/qshc.2008.029058. doi:10.1136/qshc.2008.029058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for Reporting Qualitative Research. Acad Med. 2014;89(9):1245–51. doi: 10.1097/ACM.0000000000000388. doi:10.1097/ACM.0000000000000388. [DOI] [PubMed] [Google Scholar]
- 16.Harris PA, Taylor R, Minor BL et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95 doi: 10.1016/j.jbi.2019.103208. 103208. doi:10.1016/j.jbi.2019.103208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Schifferdecker KE, Reed VA. Using mixed methods research in medical education: basic guidelines for researchers. Med Educ. 2009;43(7):637–44. doi: 10.1111/j.1365-2923.2009.03386.x. doi:10.1111/j.1365-2923.2009.03386.x. [DOI] [PubMed] [Google Scholar]
- 18.Ramani S, Mann K. Introducing medical educators to qualitative study design: Twelve tips from inception to completion. Med Teach. 2016;38(5):456–63. doi: 10.3109/0142159X.2015.1035244. doi:10.3109/0142159X.2015.1035244. [DOI] [PubMed] [Google Scholar]
- 19.Conlee MC, Tesser A. The Effects of Recipient Desire to Hear on News Transmission. Sociometry. 1973;36(4):588. doi:10.2307/2786254. [Google Scholar]
- 20.Scarff CE, Bearman M, Chiavaroli N, Trumble S. Keeping mum in clinical supervision: private thoughts and public judgements. Med Educ. 2018;80(10):S84. doi: 10.1111/medu.13728. doi:10.1111/medu.13728. [DOI] [PubMed] [Google Scholar]
- 21.Tesser A, Rosen S, Tesser M. On the Reluctance to Communicate Undesirable Messages (The MUM Effect): A Field Study. Psychol Rep. 2016;29(2):651–654. doi:10.2466/pr0.1971.29.2.651. [Google Scholar]
- 22.Brotherton S, Kao A, Crigger BJ. Professing the Values of Medicine: The Modernized AMA Code of Medical Ethics. JAMA. 2016;316(10):1041–2. doi: 10.1001/jama.2016.9752. doi:10.1001/jama.2016.9752. [DOI] [PubMed] [Google Scholar]
- 23.Gupta R, Tran MCJ. Anesthesiology Applied Exam Board Review. Vol 1. Oxford University Press; New York, NY: 2018. doi:10.1093/med/9780190852474.001.0001. [Google Scholar]
- 24.Schifferdecker KE, Reed VA. Using mixed methods research in medical education: basic guidelines for researchers. Med Educ. 2009;43(7):637–44. doi: 10.1111/j.1365-2923.2009.03386.x. doi:10.1111/j.1365-2923.2009.03386.x. [DOI] [PubMed] [Google Scholar]