INTRODUCTION
Challenges to feedback in medical education have been attributed to several variables including task performance, feedback reception, observation, and provision.1–6 It is challenging for program directors to obtain reliable and consistent response rates on end of rotation evaluations for residents. Possible reasons include busy faculty with multiple demands on their time, lack of standardized forms, and asynchronous schedules. Prior studies focus on the quality of observation and rating of the performance rather than increasing faculty response rates. Our intervention focuses on feedback documentation and increasing faculty response rates to improve resident skills, assess competency, and meet residency program accreditation.
In our internal medicine (IM) residency program, feedback submissions have transitioned from paper forms to online evaluations on a regional Web portal. Each resident and attending has a unique login for this portal that serves for scheduling, procedural documentation, and evaluations. Our clinical competency committee reviewed response rates between July 2017 and June 2018: ICU and inpatient non-critical care (INCC) subspecialties were identified as rotations with the lowest completed evaluation rates at 45% and 12% respectively.
To improve the quantity and timeliness of our resident rotation evaluations and provide a simplified platform for documentation, our residency program pilot-tested use of QR codes and mobile completion for end-of-rotation resident evaluations.
METHODS
This innovation was conducted at Kaiser Permanente Oakland and included up to 44 IM residents, 23 ICU attendings, and 56 INCC subspecialists. The Research Determination Committee at Kaiser Permanente determined that this project did not meet the regulatory definition of research involving human subjects.
Before July 2018, ICU attendings and subspecialists teaching in our program used a Web-based portal to collect feedback of our residents. Every 2 weeks, the program coordinator reviewed resident schedules and cross matched with attending schedules. Faculty were then sent an email, and asked to follow a link to the Web-based portal and complete the evaluation of residents they supervised in the previous 2 weeks. Reminder emails were sent after 2 weeks if no response was recorded.
On July 1, 2018, unique QR codes were assigned to ICU and INCC subspecialty rotations and disseminated to residents. They were instructed to affix two QR codes to their ID badge and request feedback from their supervising attendings at the end of their rotation. Using their smartphones, attendings opened their camera application and pointed it at the QR code, which instantly took them to the online SurveyMonkey™. The evaluation had drop-down menus for resident and faculty names, evaluation items, and a free text box. Here, attendings could either type or dictate comments using the smartphone’s dictation function. These evaluations were automatically uploaded to the SurveyMonkey website with no resident access. The software collated and aggregated data by resident and rotation.
We measured completion rates 1 year prior to implementation against 9 months post-intervention.
RESULTS/INNOVATION
Within the first 9 months of implementation, evaluation response rates increased from 45 to 76% among ICU attendings and 12 to 65% among subspecialists. Over 90% of evaluations were completed by attendings within 24 h of residents completing their rotations. The quality, timing, and specificity of the free-text feedback improved upon use of QR codes in place of the Web portal, where comments were almost non-existent.
Time-consuming steps were eliminated, as the new process did not require a login or password and the attending and resident names were prepopulated for selection. Additionally, evaluations now took place in real-time rather than retrospectively, and response rates were easily trackable per resident as well as rotation. QR code-initiated evaluations could be completed in approximately 2 min at the end of a rotation and freed up the program coordinator’s time by about 8 h per month.
DISCUSSION
The QR code innovation resulted in a dramatic increase in the quantity and timeliness of feedback delivered to our IM residents. Soliciting and providing feedback in real time has become the standard workflow and increased the frequency of face-to-face feedback. QR code-based evaluations provide simple, affordable, and efficient ways for residents to receive feedback. Given the initial success of this pilot and informal positive feedback from both residents and attendings, this innovation has been expanded and is used for resident rotations in other clinical settings and could easily be implemented at other institutions.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Barnes N, Nowakowski A. The use of QR codes to promote timely feedback. Fam Med. 2018;50(8):635. doi: 10.22454/FamMed.2018.747674. [DOI] [PubMed] [Google Scholar]
- 2.Synder M, Nguyen D, Womack J, et al. Testing quick response codes as an innovation to improve feedback among geographically-separated clerkship sites. Fam Med. 2018;50(3):188–194. doi: 10.22454/FamMed.2018.936023. [DOI] [PubMed] [Google Scholar]
- 3.Faucett E, McCrary H, Barry J, Saleh A, Erman A, Ishman S. High-quality feedback regarding professionalism and communication skills in otolaryngology resident education. Otolaryngol Head Neck Surg. 2018;158(1):36–42. doi: 10.1177/0194599817737758. [DOI] [PubMed] [Google Scholar]
- 4.Mitchell J, Ku C, Diachun C, et al. Enhancing feedback on professionalism and communication skills in anesthesia residency programs. Anesth Analg. 2017;125(2):620–631. doi: 10.1213/ANE.0000000000002143. [DOI] [PubMed] [Google Scholar]
- 5.Sobhani N, Fay E, Schiff M, Stephenson-Famy A, Debiec K. Randomized trial of smartphone-based evaluation for an obstetrics and gynecology clerkship. J Surg Educ. 2018;75(4):1006–1013. doi: 10.1016/j.jsurg.2017.11.009. [DOI] [PubMed] [Google Scholar]
- 6.Reynolds K, Barnhill D, Sias J, Young A, Polite F. Use of the QR reader to provide real-time evaluations of residents’ skills following surgical procedures. J Grad Med Educ. 2014;6(4):783–41. doi: 10.4300/JGME-D-13-00349.1. [DOI] [PMC free article] [PubMed] [Google Scholar]