Abstract
Objectives
To power the METRIQ (Medical Education Translational Resources: Impact and Quality) Study adequately, we aimed to recruit > 200 medical students, residents, and attendings to complete a 90‐ to 120‐minute survey by leveraging a virtual community of practice (vCoP).
Methods
Participants were recruited using personal (conference campaign and e‐mails) and online (a study website and social media campaign utilizing Twitter, Facebook, blogs, podcasts, an infographic, and a YouTube video) techniques that leveraged relationships within a virtual community or practice. Participants received weekly survey reminders for 4 weeks and at the end of the rating period. Survey completion rates were calculated.
Results
A total of 380 potential participants completed an intake form (139 medical students, 120 residents, 121 attendings), 330 consented to participate, and 309 (81.3% of interested and 93.9% of consenting participants) completed the full survey (121, 88, and 100, respectively). The required sample size was achieved.
Conclusions
The METRIQ Study utilized a multimodal recruitment campaign that targeted a vCoP. It recruited large numbers of participants with high completion rates. Response rates could not be calculated given the uncertainty surrounding the number of individuals invited to participate.
Power calculations for the METRIQ (Medical Education Translational Resources: Impact and Quality) Study1 indicated that we would require > 200 medical students, residents, and attending physicians to complete a survey lasting 90–120 minutes. We anticipated that recruiting so many participants to complete such a long survey would be difficult using traditional approaches. We developed a multimodal recruitment methodology that leveraged our authorship teams’ profiles in a virtual community of practice (vCoP).
Communities of practice are defined as “people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis.”2 Dubé et al.3 subsequently coined the term “virtual communities of practice”: vCoPs share these features with the caveat that their members interact primarily in a virtual environment using novel communication technologies to build relationships, while not excluding possible face‐to‐face contact. An international vCoP of emergency medicine (EM) and critical care clinicians has organized around the Free Open Access Medical education (FOAM) movement.
The FOAM vCoP is engaged via a series of blogs (e.g., http://aliem.com, http://CanadiEM.org), podcasts (e.g., http://emcrit.com, http://thesgem.com), and social media platforms (e.g., Twitter and Facebook) they use for professional development.4, 5 In this paper, we describe the benefits and challenges of targeting a vCoP to participate in a long survey using a multimodal recruitment methodology.
Methods
The data collection technique described was used in the METRIQ Study (http://metriqstudy.org),1 a study designed to evaluate the quality of medical blog posts. The study protocol met the exemption requirements of the University of Saskatchewan's Research Ethics Board.
Participant Recruitment
Recruitment occurred between March 1, 2016, and April 30, 2016. Rating was open between March 1, 2016, and June 1, 2016. All potential participants were directed to the METRIQ Study website (http://metriqstudy.org/) to express interest by completing a study intake form. The use of an intake form allowed block randomization of potential survey participants, ensured the accuracy of contact information so that reminder e‐mails could be sent, and allowed a customized link to an otherwise closed survey to be sent to individual participants. Enrollment information was sent through Fluid Surveys (http://fluidsurveys.com) within 24 hours.
Both personal and social media–based correspondence was used to recruit a convenience sample of medical students, EM residents, and EM attendings. All participants who completed the survey were given the opportunity to be acknowledged as METRIQ Study Collaborators on publications resulting from the data.
The study team communicated with members of the vCoP in several ways. First, prominent members of the vCoP who were known to the study team (http://metriqstudy.org/study-team/) were contacted via e‐mail to request assistance with enrollment. These vCoP members disseminated recruitment materials to contacts they believe would be interested. Second, members of the METRIQ study team were invited by members of the vCoP to present in the iMedEd track at the 2016 Council of Emergency Medicine Residency Directors (CORD) Academic Assembly in Nashville, Tennessee, on the topic of the quality appraisal of FOAM resources. The conference organizers allowed overt study recruitment to occur within iMedEd track during and after the presentation.
The study team also recruited participants on social media via five promotional avenues:
Infographics: An infographic (http://metriqstudy.org/the-recruitment-infographic/) outlining why viewers should sign up for the METRIQ Study was created.
Blog posts and podcasts: Blog posts outlining the study's progress were published on the METRIQ Study website (http://metriqstudy.org/news). A podcast and blog post about the study were hosted on a website commonly visited by members of the vCoP (http://thesgem.com/2016/04/sgemxtra-enter-the-metriq/).
Twitter: A Twitter account for the METRIQ Study (@METRIQstudy) was used to disseminate updates and recruitment information including the infographic (http://metriqstudy.org/the-recruitment-infographic/), blog posts, and podcast. The study team's personal Twitter accounts retweeted these tweets and tagged @METRIQstudy. These tweets frequently included the hashtag of the targeted vCoP (#FOAMed5). After completing the study, some participants independently tweeted about their participation.
Facebook: Multiple members of the METRIQ study team posted recruitment information on personal Facebook accounts.
YouTube: A YouTube video was embedded on the front page of the METRIQ Study website (http://metriqstudy.org/), which featured the primary author (BT) discussing the purpose of the study and encouraging viewers to enroll.
Participants were sent personalized reminder e‐mails directly from the study's primary author (BT) every 6–10 days a maximum of four times. The first and fourth reminders included the recruitment infographic. A final notice was sent to all potential participants within 10 days of the end of the rating period. Participants who completed the full survey were not sent additional e‐mails. The number of e‐mails sent to participants in the study period was tracked.
Statistical Analysis
Descriptive statistics were calculated to describe the participant demographics using Microsoft Excel 2016.
Results
The METRIQ Study was completed by 309 participants. Table 1 outlines the demographics of this group. A total of 1,300 e‐mails were sent to the participants who completed an intake form (a mean of 3.4 e‐mails for each participant who completed the intake form and 4.2 e‐mails per completed survey). A total of 380 potential participants completed an intake form, 330 consented to participate (81.3% of those completing the intake form, including 126 medical students, 95 EM residents, and 109 EM attendings), and 309 completed the survey (93.9% of those who provided consent, including 121 medical students, 88 EM residents, and 100 EM attendings).
Table 1.
Variable | |
---|---|
Age (y) | 31.1 ± 7.3 |
Sex | |
Male | 184 (59.5) |
Female | 123 (39.8) |
Other | 2 (0.6) |
Country | |
Canada | 146 (47.2) |
United States | 111 (35.9) |
Other (25 countries) | 49 (15.9) |
Level of training | |
Medical students | 121 (39.2) |
EM residents | 88 (28.5) |
EM attendings | 100 (32.4) |
Data are reported as mean ± SD or number (%).
Discussion
Two key features of a vCoP were leveraged using the described recruitment methodology: vCoPs are willing to help their members and are interested in specific topics.2, 3 As our authorship team consisted of many vCoP members and participant interest is a key driver in survey completion,6, 7 this vCoP was likely to help our authors and to engage with the survey topic. The described techniques could be used either to study a specific vCoP or to target the vCoP as a convenience sample in a larger survey.
Previously, targeted social media campaigns have successfully recruited survey participants from populations with vested interest in the study topic (e.g., recruiting smokers or patients with HIV to complete surveys on these topics).8, 9 Our methods differ in that we targeted our own nonpatient vCoP9 and did not utilize paid advertisements, a method that has been well described in the literature for recruiting hard‐to‐reach patient populations.8 As members of our study team were integrated into the vCoP, we did not have to pay for access.
Significant research has been devoted to optimizing survey recruitment and completion rates. The Dillman Total Quality Management method, which focuses on personalized and repetitive recruitment,10, 11 is frequently cited, but can result in less than 35% response rates in physician populations.12 Our survey methodology contained multiple recruitment methods and incorporated features to increase participation, including the vested interest of our participants in the topic,6, 7, 8 follow‐up contact,6, 7 a nonsensitive topic,6, 7 personalized invitations,6, 7 and the inclusion of visuals (e.g. infographics) in communications.6 To our knowledge, the intentional targeting of a vCoP and the social media recruitment methods employed (e.g., the use of infographics, blog posts, podcasts, Twitter, and Facebook) in our study have not previously been evaluated. We anticipate that as vCoP mature these interventions be evaluated by isolating each intervention and measuring its effect size.
Limitations
There are multiple limitations to this methodology. First, it requires the existence of a large and relevant vCoP containing members of the study team. Neither of these circumstances may exist for other researchers, limiting its broad applicability. Even if a sufficient vCoP exists, ingratiating investigators into a new vCoP for the purpose of research would be difficult as trust is key to the collaboration that occurs in a vCoP.13 Second, the success of this method is difficult to quantify given the interplay (and confounding effects) of multiple techniques and the inability to quantify the reach of the social media efforts. The number of participants was substantially lower than the number exposed to marketing materials and it is impossible to calculate a traditional response rate. Third, as this technique generates a convenience sample, it is inappropriate for a study that requires a representative sample. This approach is particularly at risk of social desirability bias. Fourth, the number of e‐mails required per participant to increase the response rate could be prohibitive. Finally, while vCoPs do not exclusively exist via information and communication technologies,14 the recruitment of individuals by the investigators through personal interactions may have resulted in participants from outside of the vCoP. However, given that the opportunity to recruit these participants resulted from interactions with the vCoP, these contacts should be leveraged as part of recruitment methods targeting a vCoP.
Conclusion
The METRIQ Study has demonstrated that a virtual community of practice can be leveraged to recruit participation for a survey using a multimodal recruitment strategy incorporating blogs, podcasts, infographics, and social media websites (e.g., Twitter, Facebook). We believe that members of a virtual community of practice will be able to use similar techniques to recruit participants for survey‐based studies on topics of interest to their communities.
AEM Education and Training 2017;1:110–113
Funding for this research was provided by the Canadian Association of Emergency Physicians (Junior Investigator Grant) and the Royal College of Physicians and Surgeons (Robert Maudsley Fellowship for Studies in Medical Education).
The authors have no potential conflicts to disclose.
All authors met the ICMJE minimal standards for authorship.
The manuscript was drafted by a team of authors with a diverse background in medical education research and extensive involvement in a virtual community of practice. It was originally conceived by lead author, Brent Thoma, as a component of the research project for his Masters of Science in Health Professions Education. The co‐authors of the paper made substantive contributions to the refinement of the study design, recruitment of participants, interpretation of data, and/or interpretation of the findings. All authors contributed to the revision of the manuscript and approved the final version to be published. All authors agreed to be named the guarantor of the accuracy and integrity of this work with Brent Thoma serving as the formal guarantor.
References
- 1. Thoma B, Sebok‐Syer S, Krishnan K, et al. Individual gestalt is insufficient for the evaluation of quality in medical education blogs: a METRIQ study. Ann Emerg Med. Available at: http://www.annemergmed.com/article/S0196-0644(16)31662-6/fulltext [DOI] [PubMed] [Google Scholar]
- 2. Wenger E, McDermott R, Snyder WM. Cultivating Communities of Practice. Boston, MA: Harvard Business School Press, 2002. [Google Scholar]
- 3. Dubé L, Bourhis A, Jacob R. The impact of structuring characteristics on the launching of virtual communities of practice. J Organ Chang Manag [Internet] 2005;18:145–66. Available at: http://www.emeraldinsight.com/doi/abs/10.1108/09534810510589570. [Google Scholar]
- 4. Nickson CP, Cadogan MD. Free Open Access Medical education (FOAM) for the emergency physician. Emerg Med Australas 2014;26:76–83. [DOI] [PubMed] [Google Scholar]
- 5. Lulic I, Kovic I. Analysis of emergency physicians’ Twitter accounts. Emerg Med J 2013;30:371–6. [DOI] [PubMed] [Google Scholar]
- 6. Edwards PJ, Roberts I, Clarke MJ, et al. Methods to increase response rates to postal and electronic questionnaires. Cochrane Database Syst Rev 2009;(3):MR0000008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Fan W, Yan Z. Factors affecting response rates of the web survey: a systematic review. Comput Human Behav [Internet] 2010;26:132–9. Available at: http://www.sciencedirect.com/science/article/pii/S0747563209001708. [Google Scholar]
- 8. Lane TS, Armin J, Gordon JS. Online recruitment methods for web‐based and mobile health studies: a review of the literature. J Med Internet Res [Internet] 2015;17:e183 Available at: https://www.jmir.org/2015/7/e183/. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Mendelson C. Recruiting participants for research from online communities. Comput Inform Nurs 2007;25:317–23. [DOI] [PubMed] [Google Scholar]
- 10. Thorpe C, Ryan B, McLean SL, et al. How to obtain excellent response rates when surveying physicians. Fam Pract 2009;26:65–8. [DOI] [PubMed] [Google Scholar]
- 11. Dillman DA, Smyth JD, Christian LM. Mail and Internet Surveys: The Tailored Design Method–2007 Update with New Internet, Visual, and Mixed‐Mode Guide. 4th ed New York: Wiley & Sons, 2008. [Google Scholar]
- 12. Cunningham CT, Quan H, Hemmelgarn B, et al. Exploring physician specialist response rates to web‐based surveys. BMC Med Res Methodol 2015;15:32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Ardichvili A, Page V, Wentling T. Motivation and barriers to participation in virtual knowledge‐sharing communities of practice. J Knowl Manag [Internet] 2003;7:64–77. Available at: http://www.emeraldinsight.com/doi/abs/10.1108/13673270310463626?journalCode=jkm. [Google Scholar]
- 14. Dube L, Bourhis A, Jacob R. Towards a typology of virtual communities of practice. Interdiscip J Inform Knowl Manag [Internet] 2006;1:69–93. Available at: http://www.informingscience.org/Publications/115?Source=/Journals/IJIKM/Articles?Volume=0-0. [Google Scholar]