Table 3.

Lessons Learned and Implications for Future Studies.

Lesson learned	Technical approach and design implications
Participatory research and partnership building	While buying ads on social media platforms has gained momentum as a recruitment source for research, some populations do not find ad-based recruitment acceptable or compelling (as evidenced with study data; Figure 2). Rather, this population (college students) values a trusted source. As such, partnering with existing organizations (trusted by their user base) was a highly efficient and acceptable method to solicit participation. Creating successful collaborations involved engaging organizations in study design decisions and offering data feedback as a value add for collaborators.
Human subject protections and participant safety	Standard practice for implementing participant protections, including use of informed consent and provision of clear notification around safety monitoring can be adapted to an entirely online environment. However, researchers must balance asking survey questions where the responses would warrant a clinical response (eg, acute suicidality) with the fact that participants may not have a direct connection to care—and consider their ethical responsibility to act on information provided as part of a survey. For research that is not able to provide access to a clinician in response to such safety concerns, appropriately scoping the survey is warranted.
Data security and privacy	As with any research, careful attention to data security and patient privacy is vital. Researchers should consider how data collected virtually is stored and transmitted (this is especially true if collecting sensitive data). Using secure servers to host websites and data collection via the REDCap platform, which has built-in functionality to store data securely, were helpful. As email addresses (considered Protected Health Information) were collected for follow-up and remuneration, a process was implemented to store this information separately from survey data (with potential for linkage) to protect against inadvertent disclosure and generate a deidentified dataset.
Use of off-the-shelf tools	Especially in instances of limited budgets or timelines (that would prohibit working with a web developer to build/customize needed software/platforms), existing tools are available and can be adapted to the needs of this study. Use of these existing tools was cost effective and allowed for quick implementation.
Virtual clinical trial	A substantial hurdle to running a trial virtually was the need to randomize participants in real time and in an automated fashion (anticipating that many participants would enter the survey outside of traditional working hours). Although REDCap did not initially have the capacity to perform such automated randomization, the research team worked with their developers to create, implement, and test an algorithm to accomplish these goals. Importantly, hosting the entirety of the trial in a virtual environment likely facilitated participation and enabled heterogeneity in the study sample (in terms of geographic location and other participant characteristics) while carefully designing access routes into the trial and controlling participant experiences during participation helped to protect internal validity.
Recruitment from multiple platforms	As differences are known to exist in the user base of different social media platforms and across the networks of our partner organizations, it was important to diversify recruitment efforts to maximize sample yield and improve external validity. Duplicating efforts across platforms and organizations also likely had a multiplier effect whereby the cost to us was minimal but resulted in participation gains.
Use of online distributed cohorts and issues with generalizability	This and others’ data indicate that adolescents/young adults prefer to be reached/engaged via online channels, the use of which provides important access to diverse and traditionally hard-to-reach groups. Nevertheless, there are limitations in understanding the denominator (underlying population) in many social media/online spaces and so generalizability remains a concern. While this study was strengthened by assessment of sample representativeness and implemented well-defined recruitment paths, efforts are unlikely to yield a perfectly representative sample of larger populations that exist off-line. But just as there are potential biases from the fact that users of our recruitment platforms may differ from nonusers (as well as that responders may also differ from nonresponders), parallel biases exist in other recruitment methods (eg, clinic attendees and nonattendees); so care should always be taken to understand whether factors that may contribute to populations filtering into a sampling frame and then into a sample are likely to be associated with key characteristics under investigation and to select appropriate recruitment strategies accordingly.
Protection against duplicative/false/ invalid records	A major concern of Internet-based research is the potential for false participation, especially when monetary compensation may encourage duplicative entries. Careful application of anticipatory or post hoc protections is recommended to minimize invalid records. These considerations are essential and feasible at the pre- and post-data collection stage; for instance, using browser cookies or allowing one survey entry per IP address (pre) or creating decision rules or implementing test questions (eg, asking specific questions that only the population of interest would have easy knowledge of and eliminating records that provide incorrect or illogical answers) to identify records that are unlikely to be valid (post). Further, instead of offering compensation for each completed record (as done here), remuneration in the form of a lottery or by donation to an outside organization may be less likely to incentivize duplicative/false records.
Academic setting	Rigorous research operating in an outside setting can be done such that it aligns with the core values and practices of human subject protections and IRB governance, an important advance that protects the pathway of academic research undertaken in partnership with private and outside organizations.