Six ways to maximize survey response rates: lessons from a medical school accreditation survey in a Canadian setting

Abstract

Background

Surveys are being increasingly used to gather feedback and study data in healthcare professions. However, it may be challenging to achieve high response rates in surveys administered to healthcare professionals. The aim of this paper is to report six strategies that contributed to a high response rate on the Independent Student Analysis at the University of Toronto (U of T), which can be applied to other surveys to achieve strong response rates amongst healthcare professionals.

Methods

In 2019, as part of accreditation for the U of T MD Program, we conducted the Independent Student Analysis, a student-led survey examining a medical student’s experience. We review and critically evaluate the factors that contributed to a robust response rate amongst one of the largest cohorts of medical students in Canada.

Results

Among 1080 students in the MD program, we achieved an unprecedented response rate of 87.2%. Six factors were identified that most contributed to our high response rate, including: faculty support, student representation, eliciting participant feedback, creating protected time for completion, offering incentives, and generating awareness.

Conclusions

Eliciting high survey response rates from medical learners can be challenging. However, with careful consideration of learner feedback and effective employment of the strategies discussed in this paper, medical school faculty may better engage students in survey completion, achieving higher response rates and gathering richer insight, which can be used to more effectively enact meaningful change amongst healthcare professionals.

Introduction

In medical education, online surveys are increasingly used to conveniently gather feedback from students.¹ However, response rates amongst medical learners is low,² with response rates to the medical school Graduate Questionnaire approaching 50%, for example. This undermines results through non-response bias and poor precision, reducing the reliability and generalizability of results.^3,4 This is particularly important for high-stakes evaluations, such as those involved with accreditation. Accordingly, there is a widespread call to identify factors that may bolster response rates amongst medical trainees.² In this paper, we report six evidence-based strategies (Table 1) that we believe contributed to a high response rate on the Independent Student Analysis (ISA) performed at the University of Toronto (U of T) as part of the medical school accreditation process. These recommendations can be implemented at medical schools and beyond across both a Canadian and international context to improve the response rates of high-stakes evaluation activities.

Table 1.

Factors contributing to high survey response rates with illustrative examples as it relates to our setting

Factor	Illustrative example
Enlisting faculty support	Faculty sponsorship on messaging to participants Faculty provisions of physical-space, survey and statisical software, monetary support as incentive for survey completion
Assigning student representatives	Providing medical students with leadership roles in the design and implementation of the survey Dividing the student body into groups (Year 1, Year 2, Year 3, Year 4, MD/PhD) with assigned representatives to gather participant feedback and address concerns
Incorporating participant input into the survey design	Probing the student body for concerns important to them and incorporating these into the survey
Offering protected time for students to complete the survey	Offering dedicated periods of time following mandatory class activities students may use to complete the survey to reduce the barrier to entry of writing the survey
Offering incentives to complete the survey	Chances of winning graded monetary rewards in the forms of gift cards, the value of which increased based on the total number of students who completed the survey Providing a guaranteed monetary reward (gift card) to every survey respondent
Generating student awareness	Routine digital reminders to complete the survey Automatically updating student calendars with protected time to complete the survey Sending pre-notifications Generating awareness with a media campaign consisting of U of T blog posts, U of T news, and a YouTube video

How to get a grip on maximizing survey response rates

Enlisting Faculty Support

Faculty support was essential to overcoming barriers to implementation of our other strategies. Faculty provided crucial access to physical space for meetings, survey software, statistical expertise, and financial support to cover the costs of survey design, dissemination, and incentives. Moreover, survey sponsorship by an academic institution has been found to increase response rates as it accentuates the importance of survey completion.⁵ We used university sponsorship by sending emails to the student body through the official U of T MD Program email.

Assigning Student Representatives

The ISA is mandated by the Committee on Accreditation of Canadian Medical Schools to be student-led and leaves the structure and conduct of the leadership to the discretion of individual medical schools. We therefore identified individuals keenly interested in contributing to the ISA and formed a diverse student team to enlist representation across all years, campus sites, and academies. Enlisting students in the ISA committee leadership was instrumental to incorporating the student perspective and increasing student engagement in survey completion.⁶ Each cohort of students in the MD program had associated leads who were responsible for oversight and management of their year-specific task force (Figure 1). This team structure was advantageous to survey success as we were able to divide the large participant pool of 1080 students into more manageable subgroups and receive insight regarding cohort-specific issues. We were better able to track survey completion rates of each subgroup and provide additional resources and attention to groups with low completion rates.

Figure 1.

ISA Team Structure, with arrows representing lines of communication

Incorporating participant input into survey design

The ISA governing body also suggests medical schools incorporate questions into the ISA elicited from the student body. We capitalized on this opportunity and used the survey as a platform to generate changes specific to our medical school environment and concerns. This generated a sense of ownership and responsibility amongst students, which further incentivized survey completion. Practically, this involved eliciting student feedback about concerns related to medical education at the U of T, as well as providing space in each survey section for students to provide optional free text comments. This consideration must be balanced with survey length as the literature suggests survey response rates decrease with survey length.⁷ Cumulatively, our survey was 125 questions long and had a mean response time of 28.6 minutes, consisting of multiple choice or Likert scale questions with a handful of optional, short answer questions.

Offering protected time for students to complete the survey

We anticipated that medical students would not find time to prioritize survey completion. As such, we sought to naturally incorporate the survey into students’ schedules. We worked with faculty to protect and block off convenient times specific for each year to complete the survey (e.g., preceding or proceeding a mandatory educational activity, such as clinical skills). The duration of protected time was determined through multiple small pilots of survey completion, and we provided students one hour to complete the survey. We found 65.2% of all survey responses originated from these dedicated survey completion periods.

Offering incentives to complete the survey

Incentives to increase response rates appeal to participants’ extrinsic motivation and have been found to be amongst the most significant factors affecting survey response rates.⁸ We utilized a mixed incentive scheme to maximize response rates and minimize cost. Students could choose from a guaranteed $10 gift card, the opportunity to enter a prize draw for one of five gift cards worth up to $300 in value, or a donation on their behalf to a charity. All survey respondents were entered into a draw to win a Toronto Notes 2019 textbook. Students who wrote the survey during protected times would be entered into the draw twice.

We acknowledge that this strategy is not feasible for those working in low-resource settings. Potential alternatives include offering a smaller monetary incentive, only offering a monetary incentive if initial response rates are anticipated to be low and using a lottery system versus fixed payout for survey completion. Prior research shows that a monetary amount between $6.25 and $8.00 USD has the greatest effect on improving response rates, while optimizing for cost efficiency.⁹ Moreover, it is possible that the other factors alone may be enough to improve response rates without needing a monetary incentive. To this end, although all the strategies mentioned work synergistically, they individually all contribute to increasing response rates.

Generating student awareness

Four months prior to survey dissemination, we initiated efforts to increase student knowledge and stress the importance of both accreditation and the ISA with the goal of increasing transparency amongst accreditation processes and generating overall student awareness.¹⁰ These efforts were carried out by the student leadership within the team. Information was inputted into students’ online calendars, in addition to regular emails, online advertising on the electronic resource portal, and university-sponsored blog posts.

We also used a pre-notification strategy, which has been associated with an 8% increase in survey response rates.¹ Providing students a notification a week ahead of survey distribution, with an additional reminder the day before, was an effective strategy.

The medium and content of our messaging was critical. We personalized our emails to the student body, which has been shown to improve response rates in surveys.⁸ This took the form of addressing each cohort specifically, highlighting cohort-specific issues and events, and obtaining university sponsorship on our emails.

Additionally, we sent follow-up emails every two weeks across six weeks total. These emails were personalized and emphasized the importance of maximizing response rate for the ISA while remaining succinct. Follow-up communication has consistently been noted as effective and valuable to employ in maximizing response.⁸

Conclusion

We have outlined six considerations and techniques that we used to achieve a high response rate for the Independent Student Analysis as part of accreditation for the MD Program at the University of Toronto. Collectively, the cumulative effect of various strategies, rather than relying on only a single method, may work to synergistically improve response rates. When institutions and Faculty take steps to effectively engage students, show them their feedback is valued and their concerns will be acted upon, high survey response rates can be achieved even in an era of apathetic aptitudes towards educational surveys. We acknowledge that many of our strategies require external support and are best suited for high-stakes evaluations or those involving institutions. The generalizability of these strategies to small-scale surveys may be limited, but the underlying concepts still hold true. Additionally, it is important that the strategies employed be consistent with the institution’s ethical standards. The development and implementation of the strategies discussed in this paper might offer promising ways forward to bolster survey response rates in undergraduate medical education.

Funding Statement

Funding: None

Conflicts of Interest

None

Authorship

Arshia Pedram Javidan and Yeshith Rai contributed equally to this piece and share first authorship.