Abstract
Background
Recruiting young people for health and intervention studies by traditional methods has become increasingly challenging. The widespread access to the internet may offer new strategies for online recruitment.
Objective
This study aims to assess the feasibility of online recruitment for a randomized controlled trial evaluating the effectiveness of Mindhelper, an online national youth mental health promotion service. The target group was young Danes aged 15-25 in need of mental health promotion.
Methods
Advertisements for recruitment were set up on Facebook and Instagram. Browser history was collected for a subsample of participants. We compared basic characteristics of participants who completed the baseline survey and those who did not, as well as of participants who completed the follow-up survey and those who were lost to follow-up. The significance of these differences was tested with the Pearson chi-square test.
Results
A total of 560 Danes aged 15-25 were recruited within 1 month (ie, had completed the baseline survey). Among these participants, 356 (63.6%) were at risk of developing depression or stress. The average advertisement price per participant completing the baseline questionnaire was 31 DKK (approximately €4 [US $4.2]). The follow-up survey was sent to 545 participants, of whom 318 (58.3%) completed the survey. No statistically significant differences were observed in baseline characteristics of participants who completed the follow-up and those who were lost to follow-up in terms of gender (P=.45), age (P=.35), occupation (P=.17), cohabitation (P=.90), mental well-being (P=.26), mental illness (P=.44; impact of the illness, P=.05), or use of the internet when having a hard time (P=.92).
Conclusions
We conclude that it is feasible to recruit young Danes online for a large-scale randomized controlled trial assessing the effectiveness of Mindhelper.
Trial Registration
ClinicalTrials.gov NCT04650906; https://clinicaltrials.gov/ct2/show/NCT04650906
Keywords: recruitment, web based, online, mental health, young people, well-being
Introduction
Recruiting participants for intervention studies is increasingly difficult, and participation rates in research projects are generally declining, which may hamper data quality, statistical power, and validity of research findings. Recruitment and retainment of young people with mental health problems, for health and intervention studies, are shown to be especially challenging [1-4].
Most young people are active on social networking sites, being online multiple times a day, which may offer an alternative strategy to recruit young participants. In 2020, 93% of 16- to 24-year olds used the internet on a daily basis in Denmark [5], and 96% of all Danes aged 16-39 were social media users [6], of which Facebook had the largest market share [7].
In recent years, an increasing number of studies have applied online recruitment methods, advertising on, for example, social media platforms, through Google search engine, and by website campaigns. Studies have assessed the feasibility of online recruitment among the general population [8,9], and in specific groups, especially among adolescents, and groups that are considered difficult to recruit by traditional means, for example, men who have sex with men [10-18]. Generally, the results show that the cost per participant recruited online is lower compared with offline recruitment methods [13,18-22], and that it is possible to reach populations who are otherwise challenging to enroll [15,19,23]. Studies find that participants recruited online are younger, more highly educated, have poorer self-rated health, and are more likely to be White and female than representative samples [8,18,20,24-26]. However, a systematic review of studies recruiting for health, medical, or psychosocial research using Facebook showed that the majority (86%) of the studies that examined the representativeness concluded that samples recruited through Facebook had similar representation to those recruited through traditional methods [23]. Further, another systematic review examining studies using Facebook for recruiting participants for health research concluded that recruitment though Facebook was more likely than traditional recruitment methods to result in better representation and improved participant selection among adolescents [18].
The widespread access to and use of the internet further opens opportunities for online interventions. Mental health problems are prevalent among young people both in Denmark and abroad [27,28]. Although young people experience the highest rates of mental health problems of any age group, less than half of young people with mental health problems seek professional help [29-32]. Concerns about stigma and confidentiality, low mental health literacy, and difficulties navigating existing mental health services are among the many barriers to help-seeking among young people [33,34]. Online services may overcome some of the obstacles of help-seeking; however, only few studies have examined the effectiveness of unstructured digital mental health solutions [35-40]. Mindhelper [41] is an online, open-access, self-directed youth mental health promotion service that provides information, self-help tools, and guidance to young people in Denmark. Since January 2019, the service has been freely available and disseminated to young people and youth mental health professionals across the country.
The aim of the study was to assess the feasibility of online recruitment for a randomized controlled trial assessing the effectiveness of a website (Mindhelper) targeting young people aged 15-25 years in need of mental health promotion.
Methods
The Intervention: Mindhelper
Mindhelper is a highly scalable, unstructured, multicomponent online mental health promotion service that offers young people information, tools, and support for life problems and mental health difficulties. The site was co-developed (from 2014 to 2017) by the Centre for Telepsychiatry in the Mental Health Services in the Region of Southern Denmark in partnership with young people and 4 Danish municipalities. Mindhelper does not provide psychological or therapeutic treatment. It is designed to provide practical help strategies and tools to support well-being and help-seeking from everyday stressors to more complex mental health issues. The issues range from dealing with family difficulties, depression, and substance use/dependence. Tools and information are derived mainly from the cognitive field, including mindfulness exercises and general strategies to good mental well-being. Mindhelper also offers a supportive outreach service in the form of responding to letters sent by young people. The letters exchanged are published in an anonymized form, so that other young people with similar concerns or worries may benefit from the supportive advice. The website also serves as a national directory to local youth mental health services for further support and help.
The service has been freely available and disseminated across the country since January 2019. In 2020 Mindhelper had more than 1.2 million visits.
Evaluating the Effectiveness of Mindhelper in a Large-Scale Study
In a large-scale study, we aim to evaluate the effectiveness of Mindhelper, contributing to the much needed evidence base for interventions promoting mental health that target young people [42,43]. This feasibility study was undertaken to investigate whether it is possible to recruit young people with mental health problems via social media platforms, to randomize them to use or not to use Mindhelper, and to keep the intervention and control group separate, although the site is open and freely available online. Further, we aimed to explore whether it is possible to retain the participants in the study over time and to assess the validity of self-reported use of Mindhelper.
Browser History
To assess whether self-reported questions on online behavior is a valid measure for actual behavior, a subsample of participants, who had given informed consent to access their browser history, was contacted and invited to the National Institute of Public Health (NIPH). Here they met a project employee, who coded the participants’ browser history (in all available browsers) related to the use of Mindhelper for the intervention period from the devices they bought (usually their laptop and mobile phone). All coding was done manually, and thus nothing was downloaded from the participants’ devices. The participants were present during the coding and had full insight into the process.
Because of COVID-19 restrictions, participants were not allowed to enter the NIPH by the end of the recruitment period, and therefore they were guided through the coding process online, meaning that the coding was partly self-reported. Participants who offered access to their browser history were given a gift card for cinema (200 DKK ≈ €27 [US $28.4]) in appreciation for their participation in the study. The financial compensation for participants was given irrespective of their survey responses and browser history. Hence, there was no reason to believe that the gifts impacted the study in any other way than to promote participation in the study.
The Advertisement Setup
Facebook and Instagram were chosen over other platforms, as the advertisement tools on these platforms enable a more detailed targeting of advertisements than other social media platforms, and because Facebook/Instagram had the lowest cost per eligible contact to participants in a study with a similar aged target group [15]. Advertisements targeted Danish speakers aged 15-25. To allow an easier access to the NIPH, all advertisements further targeted young people living within a 20 km radius from the center of Copenhagen (where the NIPH is located). Age was based on the information listed in the user’s Facebook/Instagram profile, while location was based on the internet protocol address or the address listed on the user’s profile [44].
The advertisements contained a short title (eg, “Help us improve mental well-being among young people”), an image (a photo or drawing of a young male or female looking sad or troublesome), and a main text (eg, “Are you 15-25 years? Help us improve mental well-being among young people. It only requires 2 times 15 minutes of your time”). Examples of the advertisements are displayed in Multimedia Appendix 1. All advertisements were Dynamic Creative Ads, where Facebook’s/Instagram’s algorithms automatically combine title, images, and main text to run based on advertisement performance and the cost per click. In this process advertisements are placed within advertisement sets. For this study, 4 sets of advertisements were constructed with a total of 6 advertisements.
Advertisers can choose to be charged per click on the advertisement, or each time the advertisement is displayed a certain number of times [44]. We chose the cost-per-click option, as we were interested in people clicking through to our website at the lowest possible cost. The cost per click depends on the current competition between advertisers within the target group, and thus may fluctuate over the recruitment period.
The first advertisement was released on Facebook and Instagram on November 2, 2020. Participants were led from the advertisements to a webpage for the study, where information about the study and data collection was provided. From this webpage, participants could click their way further to the baseline questionnaire, if they had given informed consent to participate in the study. We used the online survey tool SurveyXact [45], which allowed secure collection and data protection.
Randomization and Surveys
An automatic randomization was set up allocating everyone opening the baseline questionnaire to either the intervention group or the control group.
Questions on participants’ demographics and use of the internet when they were having a hard time were included. Further, multiple scales were included to assess a broad spectrum of mental health. Well-being was assessed by the Well-Being Index [46,47], psychological distress and daily functioning were assessed by the Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS) [48], while intentions to seek help were assessed by the General Help-Seeking Questionnaire—vignette version (GHSQ-V) [49]. As no Danish version of GHSQ-V exists, the scale was translated by the authors: one in the research group translated the questionnaire from English to Danish, while another translated the Danish version back to English (SHH and DLS). In the few instances where there were inconsistencies between the 2 translations, the best translation was discussed until agreed upon by the researchers.
The survey was pilot tested among 5 young informants within the target group through cognitive interviews, to assess how the posted questions were interpreted and to identify potential challenges in responding to the survey [50]. Special attention was paid to (1) GHSQ-V, as no validated Danish version exists; and (2) new questions formulated by the research group to, for example, measure use of Mindhelper. Based on the informants’ comments, wording was changed to ease the reading, categories corrected to better grasp expected answers, and questions left out due to misinterpretations. However, as no major changes were needed to the questions or structure, it was unnecessary to repeat the pilot testing of the revised questionnaire.
Participants were informed of the follow-up survey, asked to provide their contact details (email or phone number), and informed that they would take part in a lottery of gift cards for the cinema (200 DKK ≈ €27 [US $28.4]) when responding to both surveys. All participants who had completed the baseline survey were invited to answer the follow-up survey 1 week after their completion of the baseline survey. They received the invitation via SMS text message and/or email, depending on the information they had provided in the baseline survey. If they did not respond to the survey within 3 days after the invitation, they received reminders via SMS text message or email. The follow-up survey included questions on the usage of Mindhelper in addition to measures of mental well-being.
Participants were all shown the same questions until the end of the baseline survey. The intervention group was provided with information on Mindhelper and an active link to the website, whereas the control group was informed about the follow-up survey by the end of the baseline survey and received information about Mindhelper only when completing the follow-up survey. On December 2, 2020, the baseline survey was closed; thus, all recruitment was completed within 1 month.
Statistical Methods
We compared basic characteristics of participants who completed the baseline survey and those who did not and used the Pearson chi-square test to assess if differences were statistically significant. Similarly, we compared characteristics of participants who completed the follow-up survey and those who were lost to follow-up and tested the significance of these differences using the Pearson chi-square test.
Ethical Considerations
The design of this study was guided by the CONSORT (Consolidated Standards of Reporting Trials) statement for pilot and feasibility trials [51], and conducted in accordance with the Danish Council for Independent Research’s ethical guidelines. All participants received comprehensive information about the purpose of the project and terms of participation and provided informed consent to participate before responding to the baseline questionnaire. The study was registered at ClinicalTrials.gov (NCT04650906), legally approved by the Region of Southern Denmark (journal number: 20/55262), and ethically approved by The Research Ethics Committee of the University of Southern Denmark (case number: 20/68029).
Consent to Participate
It was clearly stated that participation in the study was voluntary, and participants gave consent that their data could be used for research purposes.
Results
The flow of participants in the study population is presented in Figure 1. A total of 1284 participants opened the baseline questionnaire. There were 48 participants who could not be categorized as within the target group, either because they had missing information on age or residency (which were the only mandatory questions within the questionnaire), or because they were younger than 15 or older than 25 years, or not living in Denmark. There were 671 participants who did not complete the questionnaire. Further, 5 participants completed the baseline questionnaire, but did not provide valid contact information, and therefore could not receive the follow-up questionnaire. Thus, a total of 560 participants within the target group completed the baseline questionnaire. All participants completing the baseline questionnaire should receive the follow-up questionnaire; however, 15 participants were mistakenly never invited due to technical errors. A total of 227 participants who received the follow-up questionnaire did not respond, while 318 respondents completed the follow-up questionnaire, and thus a retention probability of 58.3% (318/545) was achieved.
Figure 1.
Study flowchart presenting participant recruitment.
In total, we were able to track 1009 unique clicks to the baseline questionnaire from the landing page. However, the baseline questionnaire had been started 1284 times, thus several clicks to the questionnaire were not tracked. This may occur if participants shared the link to the questionnaire or because of technical issues related to JavaScript. It was predominantly females who interacted with all sets of ads. The average advertisement price per participant completing the baseline questionnaire was 31 DKK (approximately €4 [US $4.2]).
Baseline characteristics of young people who did and did not complete the baseline and follow-up surveys are presented in Table 1. Most of the participants were females, undergoing postsecondary or higher educations, and living together with both parents. More than half of the participants were at risk of developing depression or stress according to the Well-Being Index. Every second participant had at some point been told by a general practitioner or psychologist that they had a mental illness, of whom nearly half (29/52, 56%) were daily affected by the illness during the past 12 months. More than half of the participants used the internet often or sometimes if they were having a hard time.
Table 1.
Baseline characteristics of young people who did and did not complete the baseline and follow-up surveys, respectively.
| Characteristics | Baseline not completed (n=671) | Baseline completed (n=560) | P value | Follow-up not completed (n=227) | Follow-up completed (n=318) | P value | |
| Gender, n (%) |
|
|
.51 |
|
|
.45 | |
|
|
Male | 87 (13.0) | 61 (10.9) |
|
24 (10.6) | 37 (11.6) |
|
|
|
Female | 573 (85.5) | 488 (87.3) |
|
201 (88.9) | 273 (85.8) |
|
|
|
Other | 10 (1.5) | 10 (1.8) |
|
1 (0.4) | 8 (2.5) |
|
|
|
Missinga | 1 | 1 |
|
1 | 0 |
|
| Age (years), n (%) |
|
|
<.01 |
|
|
.35 | |
|
|
15-17 | 283 (42.2) | 135 (24.1) |
|
60 (26.4) | 69 (21.7) |
|
|
|
18-20 | 139 (20.7) | 146 (26.1) |
|
62 (27.4) | 79 (24.8) |
|
|
|
21-23 | 143 (21.3) | 178 (31.8) |
|
63 (27.9) | 112 (35.2) |
|
|
|
24-25 | 106 (15.8) | 101 (18.0) |
|
42 (18.6) | 58 (18.2) |
|
|
|
Mean age (SD) | 19.2 (0.13) | 20.3 (0.13) |
|
20.1 (0.20) | 20.5 (0.17) |
|
| Occupation, n (%) |
|
|
<.01 |
|
|
.17 | |
|
|
Employed | 99 (14.9) | 85 (15.2) |
|
43 (18.9) | 42 (13.2) |
|
|
|
Unemployed | 24 (3.6) | 28 (5.0) |
|
11 (4.8) | 16 (5.0) |
|
|
|
Student (primary school) | 129 (19.5) | 41 (7.3) |
|
18 (7.9) | 20 (6.3) |
|
|
|
Student (postsecondary education) | 214 (32.3) | 163 (29.1) |
|
71 (31.3) | 86 (27.0) |
|
|
|
Student (higher education) | 167 (25.2) | 215 (38.4) |
|
74 (32.6) | 136 (42.8) |
|
|
|
Other | 30 (4.5) | 28 (5.0) |
|
10 (4.4) | 18 (5.7) |
|
|
|
Missinga | 8 | 0 |
|
0 | 0 |
|
| Cohabitation, n (%) |
|
|
<.01 |
|
|
.90 | |
|
|
Lives with parents (who live together) | 228 (35.3) | 148 (26.9) |
|
61 (27.6) | 84 (26.8) |
|
|
|
Lives with parents in shifts | 52 (8.0) | 20 (3.6) |
|
9 (4.1) | 9 (2.9) |
|
|
|
Lives with 1 parent | 101 (15.6) | 76 (13.8) |
|
32 (14.5) | 39 (12.4) |
|
|
|
Lives with a partner or friend | 116 (18.0) | 123 (22.4) |
|
47 (21.3) | 73 (23.2) |
|
|
|
Lives in a dormitory | 64 (9.9) | 71 (12.9) |
|
26 (11.8) | 44 (14.0) |
|
|
|
Lives alone | 63 (98) | 88 (16.0) |
|
38 (17.2) | 49 (15.6) |
|
|
|
Other | 22 (3.4) | 24 (4.4) |
|
8 (3.6) | 16 (5.1) |
|
|
|
Missinga | 25 | 10 |
|
6 | 4 |
|
| Well-being (WHOb Well-Being Index), n (%) |
|
|
.39 |
|
|
.26 | |
|
|
Not in the risk zone (>50 points) | 167 (33.2) | 198 (35.7) |
|
86 (38.4) | 106 (33.7) |
|
|
|
Risk zone (≤50 points) | 336 (66.8) | 356 (64.3) |
|
138 (61.6) | 209 (66.3) |
|
|
|
Missinga | 168 | 6 |
|
3 | 3 |
|
| Have or have had a mental illness diagnosed by a general practitioner or psychologist, n (%) |
|
|
.17 |
|
|
.44 | |
|
|
Yes | 52 (7.7) | 275 (49.6) |
|
104 (46.4) | 166 (52.7) |
|
|
|
No | 48 (7.2) | 269 (48.6) |
|
116 (51.8) | 143 (45.4) |
|
|
|
Do not want to answer | 5 (0.7) | 10 (1.8) |
|
4 (1.8) | 6 (1.9) |
|
|
|
Missinga | 566 | 6 |
|
3 | 3 |
|
| Impacted by the mental illness within the past 12 months, among participants with a mental illness, n (%) |
|
|
.17 |
|
|
.05 | |
|
|
Yes, daily | 29 (55.8) | 135 (49.1) |
|
44 (42.3) | 87 (52.4) |
|
|
|
Yes, weekly | 16 (30.8) | 77 (28.0) |
|
39 (37.5) | 38 (22.9) |
|
|
|
Yes, but less than weekly | 7 (13.5) | 39 (14.2) |
|
11 (10.6) | 28 (16.9) |
|
|
|
No | 0 (0) | 24 (8.7) |
|
10 (9.6) | 13 (7.8) |
|
| Use of the internet if having a hard time, n (%) |
|
|
.31 |
|
|
.92 | |
|
|
Yes, often | 189 (31.2) | 148 (26.4) |
|
61 (26.9) | 83 (26.1) |
|
|
|
Yes, sometimes | 323 (53.4) | 314 (56.1) |
|
125 (55.1) | 183 (57.5) |
|
|
|
No, not at all | 80 (13.2) | 83 (14.8) |
|
36 (15.9) | 44 (13.8) |
|
|
|
Do not have a hard time | 13 (2.1) | 15 (2.7) |
|
5 (2.2) | 8 (2.5) |
|
|
|
Missinga | 66 | 0 |
|
0 | 0 |
|
aMissing entries were not included in the calculations of percentages, and therefore, percentages are not listed for missing.
bWHO: World Health Organization.
Participants completing the baseline survey were older, more likely to attend a higher education, and to live apart from their parents than those not completing the baseline survey. However, there was no statistically significant differences between participants completing and not completing the baseline questionnaire regarding gender (P=.51), mental well-being (P=.39), mental illness (P=.17; impact of the illness, P=.17), or use of the internet when having a hard time (P=.31).
The 560 participants who completed the baseline survey were equally randomized to the control (n=280) and intervention group (n=280), and no baseline differences were observed between the 2 groups (data not shown).
The follow-up survey was sent to 545 young people, and 318 (58.3%) completed the survey. No statistically significant differences were observed in baseline characteristics of participants completing the follow-up and those who were lost to follow-up regarding gender (P=.45), age (P=.35), occupation (P=.17), cohabitation (P=.90), mental well-being (P=.26), mental illness (P=.44; impact of the illness, P=.05), or use of the internet when having a hard time (P=.92). However, participants who did complete the follow-up survey were slightly older and more likely to be in the risk zone of depression or stress according to the Well-Being Index, compared with those not completing the survey. Furthermore, participants completing the follow-up survey were more likely to have a mental illness and to have been daily impacted by the illness during the past 12 months than those not completing the survey.
During the follow-up period, 21.9% (34/155) of the participants in the intervention group and 3.1% (5/163) of the participants in the control group used Mindhelper (data not shown). The difference in usage between the intervention and control group was statistically significant (P<.01).
In total, 49 participants got their browser history coded. However, 3 participants were subsequently excluded as they could not be identified in the survey data, resulting in 46 usable browser histories linked with survey response (Table 2). Overall, when participants in the survey reported that they had not visited Mindhelper, it was consistent with data from the browser history (an agreement between their survey response and their browser history of 94% [31/33]). When participants reported that they had visited Mindhelper, results were less clear (an agreement between their survey response and their browser history of only 64% [7/11]).
Table 2.
Compliance between data from questionnaire and browser history in a subsample of 46 participants.
| Comparison between self-reported visits at Mindhelper and visits recorded in browser history | Visited Mindhelper (data from follow-up questionnaire), n (%) | |||
| Yes | No | Do not know | ||
| Visited Mindhelper (data from browser history), n (%) |
|
|
|
|
|
|
Yes | 7 (64) | 2 (6) | N/Aa |
| No | 4 (36) | 31 (94) | 2 (100) | |
| Total | 11 (100) | 33 (100) | 2 (100) | |
aN/A: not applicable.
Discussion
Principal Findings
In 1 month, 560 participants within the target group and in need of mental health promotion were recruited. Participants completing the baseline survey were generally older than those not completing the survey, and in line with this, a larger proportion attended a higher education and lived apart from their parents. However, no statistically significant differences regarding mental health and well-being were observed between young people with completed and noncompleted baseline surveys.
A retention probability of 58.3% (318/545) was achieved, which is lower than the average probability of 79% observed in a systematic review of engagement of children and young people in digital mental health interventions (ranging from 16% to 100% in the included studies) [52]. However, the average probability covers a wide variety of target groups and interventions, and thus may not be comparable to ours. The study included in the review most like ours targeted Australians aged 16-25 years and assessed the efficacy of an online self-guided app recommendation service aiming to improve well-being [39]. In that study, a retention probability of 50% (4 weeks after baseline) was achieved. Compared with this, a retention probability of 58.3% (318/545) may be acceptable after 1 week. The 318/545 (58.3%) participants who completed the follow-up survey did not differ significantly from those lost to follow-up in demographic characteristics, mental health, and well-being, thus decreasing the risk of selection bias. However, participants who completed the follow-up survey were more likely to be at risk of developing depression or stress, to have a mental illness, and to be daily impacted by the illness than those not completing the survey. This may indicate that we were able to retain participants in need of mental health promotion, and thus to reach the target group of the intervention. The risk of attrition at follow-up will be accounted for in the sample size calculation in the large-scale randomized controlled trial.
We targeted males and females equally; however, we mainly recruited females (488/560, 87.1%). This tendency is in line with other studies recruiting using Facebook, where the majority of the participants recruited have also been females [23]. The advertisement algorithm favors subparts of the target group who have previously interacted with the advertisement, and thus the advertisements were increasingly shown to females during the recruitment process. This may be corrected for in the large-scale effectiveness study of Mindhelper by re-designing the advertisements, oversampling specific groups, or narrowing down the target group for some advertisements (eg, by only including males). The ability to create multiple advertisements targeting different populations, to closely monitor their real-time performance, and to continuously adjust make online advertising a powerful tool for recruiting according to specific demographic requirements. This will help to ensure a broader representation in the study, but will probably also result in a slightly increased price per recruited participant. With an average advertisement price per participant completing the baseline questionnaire of 31 DKK (approximately €4 [US $4.2]), online recruitment proves economically favorable compared with traditional methods. However, one should be aware of the dynamic nature of advertisement algorithms, and the fact that price per click depends on the current competition between advertisers within the target group. Hence, the achieved advertisement price per participant may fluctuate. Further, the advertisement success of future studies may be affected by changing Facebook algorithms and policies.
According to the follow-up survey, 21.9% (34/155) of the participants in the intervention group and 3.1% (5/163) of the participants in the control group had used Mindhelper during the intervention period. Survey-reported online activity was consistent with browser history when participants reported not to have visited Mindhelper, but results were less clear among participants reporting to have visited Mindhelper, due to the relatively few observations and low overall usage of Mindhelper. However, our results indicated that it was possible to ensure that very few participants in the control group used the website, although the website is open and freely available.
A more active and persistent encouragement of participants in the intervention group is needed to increase the usage of Mindhelper. Currently, the most frequent user flow of Mindhelper is people entering the site from diagnostic and research search phrases in search engines (ie, pull marketing). Recruiting through online advertisement flips the user flow (push marketing). Hence, different measures will be applied in the large-scale effectiveness study of Mindhelper to improve the motivation for interaction with the site. Throughout the intervention period, automated series of emails/SMS text messages introducing participants to learn more about the site, and specific series offering advice on improving mental health (eg, evidence-based advice on self-care) will be applied to remind the intervention group of the site’s possibilities.
Strengths and Limitations
In 1 month, 560 eligible participants within the target group were recruited and thus, we succeeded in recruiting a large study population within a short timeframe. The large study population decreases the impact of nonsystematic errors and improves statistical power. The short recruitment period decreases the risk of other things occurring simultaneously that potentially impact recruited participants and the explored associations.
Some limitations of the study need to be considered. Exposure to the advertisements depended on having a profile on Facebook/Instagram, and the users supplying their correct age in their profile, other users’ interaction with the advertisement (as the advertisement algorithm favors subparts of the target group who have previously interacted with the advertisement), and time spent on Facebook and Instagram. If systematic differences exist between those who were exposed to the advertisements and those who were not, this may give rise to selection bias. Further, there may be considerable differences between those who click on advertisements in Facebook/Instagram and those who do not. Similarly, if young people volunteering to participate in the study vary systematically from those who did not open the survey, self-selection bias may be an issue. As no information was gathered about participants who did not begin the baseline questionnaire, this bias cannot be excluded.
Since recruitment was completed within 1 month, no information was retrieved on longer-term trends in recruitment rates, which may diminish over time. This could be an area for future research.
Young people may access the internet daily from several devices, and they may not have brought all these devices to the NPHI for coding. Additionally, the participants may occasionally browse in incognito mode, and if they did while visiting Mindhelper, this will not show up in the browser history. Therefore, a participant might have actually visited Mindhelper, but it will not show up in his/her coded browser history. This inaccuracy is likely to be highest in the intervention group, as the control group was not provided with information about Mindhelper, and thus the use of Mindhelper may be underestimated in the intervention group.
We were also unable to track all activities from the study website to the questionnaire due to technical issues. In the large-scale effectiveness study of Mindhelper, we will seek to implement the survey directly on the website, which will give us more precise usage data.
Based on the results from this feasibility study, we conclude that it is possible to assess the effectiveness of Mindhelper in a randomized controlled trial and to recruit participants online via social networking sites.
Acknowledgments
This work was supported by TrygFonden (ID:128134).
Abbreviations
- CONSORT
Consolidated Standards Of Reporting Trials
- GHSQ-V
General Help-Seeking Questionnaire—vignette version
- NIPH
National Institute of Public Health
- SWEMWBS
Short Warwick-Edinburgh Mental Wellbeing Scale
Examples of advertisements.
CONSORT EHEALTH Checklist (V 1.6.1).
Footnotes
Conflicts of Interest: None declared.
References
- 1.Liu Y, Pencheon E, Hunter RM, Moncrieff J, Freemantle N. Recruitment and retention strategies in mental health trials - A systematic review. PLoS One. 2018;13(8):e0203127. doi: 10.1371/journal.pone.0203127. https://dx.plos.org/10.1371/journal.pone.0203127 .PONE-D-18-00827 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Iannaccone CK, Fossel A, Tsao H, Cui J, Weinblatt M, Shadick N. Factors associated with attrition in a longitudinal rheumatoid arthritis registry. Arthritis Care Res (Hoboken) 2013 Jul 01;65(7):1183–9. doi: 10.1002/acr.21940. doi: 10.1002/acr.21940. [DOI] [PubMed] [Google Scholar]
- 3.Leonard A, Hutchesson M, Patterson A, Chalmers K, Collins C. Recruitment and retention of young women into nutrition research studies: practical considerations. Trials. 2014 Jan 16;15:23. doi: 10.1186/1745-6215-15-23. https://trialsjournal.biomedcentral.com/articles/10.1186/1745-6215-15-23 .1745-6215-15-23 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Smith KL, Straker LM, McManus A, Fenner AA. Barriers and enablers for participation in healthy lifestyle programs by adolescents who are overweight: a qualitative study of the opinions of adolescents, their parents and community stakeholders. BMC Pediatr. 2014 Feb 19;14(1):53. doi: 10.1186/1471-2431-14-53. https://bmcpediatr.biomedcentral.com/articles/10.1186/1471-2431-14-53 .1471-2431-14-53 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Johnson J. Share of individuals using the internet daily in Denmark 2020, by age group. Statista. [2022-05-05]. https://www.statista.com/statistics/1002046/share-of-individuals-using-the-internet-daily-in-denmark-by-age-group/2021 .
- 6.Tankovska H. Share of social media users in Denmark 2017-2020, by age group. Statista. [2022-05-05]. https://www.statista.com/statistics/560994/share-of-social-media-users-in-denmark-by-age-group/2020 .
- 7.Tankovska H. Most popular social networks in Denmark 2021, by market share. Statista. 2021. May 05, [2022-05-05]. https://www.statista.com/statistics/621343/most-popular-social-networks-in-denmark/2021 .
- 8.Kesse-Guyot E, Assmann K, Andreeva V, Castetbon K, Méjean Caroline, Touvier M, Salanave B, Deschamps V, Péneau Sandrine, Fezeu L, Julia C, Allès Benjamin, Galan P, Hercberg S. Lessons Learned From Methodological Validation Research in E-Epidemiology. JMIR Public Health Surveill. 2016 Oct 18;2(2):e160. doi: 10.2196/publichealth.5880. https://publichealth.jmir.org/2016/2/e160/ v2i2e160 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Kesse-Guyot E, Andreeva V, Castetbon K, Vernay M, Touvier M, Méjean Caroline, Julia C, Galan P, Hercberg S. Participant profiles according to recruitment source in a large Web-based prospective study: experience from the Nutrinet-Santé study. J Med Internet Res. 2013 Sep 13;15(9):e205. doi: 10.2196/jmir.2488. https://www.jmir.org/2013/9/e205/ v15i9e205 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Huybrechts KF, Mikkelsen EM, Christensen T, Riis AH, Hatch EE, Wise LA, Sørensen Henrik Toft, Rothman KJ. A successful implementation of e-epidemiology: the Danish pregnancy planning study 'Snart-Gravid'. Eur J Epidemiol. 2010 May 11;25(5):297–304. doi: 10.1007/s10654-010-9431-y. http://europepmc.org/abstract/MED/20148289 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wise LA, Rothman KJ, Mikkelsen EM, Stanford JB, Wesselink AK, McKinnon C, Gruschow SM, Horgan CE, Wiley AS, Hahn KA, Sørensen Henrik Toft, Hatch EE. Design and Conduct of an Internet-Based Preconception Cohort Study in North America: Pregnancy Study Online. Paediatr Perinat Epidemiol. 2015 Jul;29(4):360–71. doi: 10.1111/ppe.12201. http://europepmc.org/abstract/MED/26111445 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Harris ML, Loxton D, Wigginton B, Lucke JC. Recruiting online: lessons from a longitudinal survey of contraception and pregnancy intentions of young Australian women. Am J Epidemiol. 2015 May 15;181(10):737–46. doi: 10.1093/aje/kwv006.kwv006 [DOI] [PubMed] [Google Scholar]
- 13.McAlindon T, Formica Margaret, Kabbara Karim, LaValley Michael, Lehmer Melissa. Conducting clinical trials over the internet: feasibility study. BMJ. 2003 Aug 30;327(7413):484–7. doi: 10.1136/bmj.327.7413.484. http://europepmc.org/abstract/MED/12946971 .327/7413/484 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Thornton LK, Harris K, Baker AL, Johnson M, Kay-Lambkin FJ. Recruiting for addiction research via Facebook. Drug Alcohol Rev. 2016 Jul;35(4):494–502. doi: 10.1111/dar.12305. [DOI] [PubMed] [Google Scholar]
- 15.Zlotorzynska M, Bauermeister JA, Golinkoff JM, Lin W, Sanchez TH, Hightow-Weidman L. Online recruitment of youth for mHealth studies. Mhealth. 2021;7:27. doi: 10.21037/mhealth-20-64. doi: 10.21037/mhealth-20-64.mh-07-20-64 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kayrouz R, Dear BF, Karin E, Titov N. Facebook as an effective recruitment strategy for mental health research of hard to reach populations. Internet Interv. 2016 May;4:1–10. doi: 10.1016/j.invent.2016.01.001. https://linkinghub.elsevier.com/retrieve/pii/S2214-7829(16)00002-6 .S2214-7829(16)00002-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Amon KL, Campbell AJ, Hawke C, Steinbeck K. Facebook as a recruitment tool for adolescent health research: a systematic review. Acad Pediatr. 2014;14(5):439–447.e4. doi: 10.1016/j.acap.2014.05.049.S1876-2859(14)00186-7 [DOI] [PubMed] [Google Scholar]
- 18.Whitaker C, Stevelink S, Fear N. The Use of Facebook in Recruiting Participants for Health Research Purposes: A Systematic Review. J Med Internet Res. 2017 Aug 28;19(8):e290. doi: 10.2196/jmir.7071. https://www.jmir.org/2017/8/e290/ v19i8e290 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Christensen T, Riis AH, Hatch EE, Wise LA, Nielsen MG, Rothman KJ, Toft Sørensen Henrik, Mikkelsen EM. Costs and Efficiency of Online and Offline Recruitment Methods: A Web-Based Cohort Study. J Med Internet Res. 2017 Mar 01;19(3):e58. doi: 10.2196/jmir.6716. https://www.jmir.org/2017/3/e58/ v19i3e58 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Leach LS, Butterworth P, Poyser C, Batterham PJ, Farrer LM. Online Recruitment: Feasibility, Cost, and Representativeness in a Study of Postpartum Women. J Med Internet Res. 2017 Mar 08;19(3):e61. doi: 10.2196/jmir.5745. https://www.jmir.org/2017/3/e61/ v19i3e61 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Fenner Y, Garland SM, Moore EE, Jayasinghe Y, Fletcher A, Tabrizi SN, Gunasekaran B, Wark JD. Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res. 2012 Feb 01;14(1):e20. doi: 10.2196/jmir.1978. https://www.jmir.org/2012/1/e20/ v14i1e20 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Brøgger-Mikkelsen Mette, Ali Z, Zibert JR, Andersen AD, Thomsen SF. Online Patient Recruitment in Clinical Trials: Systematic Review and Meta-Analysis. J Med Internet Res. 2020 Nov 04;22(11):e22179. doi: 10.2196/22179. https://www.jmir.org/2020/11/e22179/ v22i11e22179 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Thornton Louise, Batterham Philip J, Fassnacht Daniel B, Kay-Lambkin Frances, Calear Alison L, Hunt Sally. Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interv. 2016 May;4:72–81. doi: 10.1016/j.invent.2016.02.001. https://linkinghub.elsevier.com/retrieve/pii/S2214-7829(15)30016-6 .S2214-7829(15)30016-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Etter J F, Perneger T V. A comparison of cigarette smokers recruited through the Internet or by mail. Int J Epidemiol. 2001 Jun;30(3):521–5. doi: 10.1093/ije/30.3.521. [DOI] [PubMed] [Google Scholar]
- 25.Guo X, Vittinghoff E, Olgin JE, Marcus GM, Pletcher MJ. Volunteer Participation in the Health eHeart Study: A Comparison with the US Population. Sci Rep. 2017 May 16;7(1):1956. doi: 10.1038/s41598-017-02232-y. doi: 10.1038/s41598-017-02232-y.10.1038/s41598-017-02232-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Benedict C, Hahn AL, Diefenbach MA, Ford JS. Recruitment via social media: advantages and potential biases. Digit Health. 2019;5:2055207619867223. doi: 10.1177/2055207619867223. https://journals.sagepub.com/doi/10.1177/2055207619867223?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed .10.1177_2055207619867223 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Jeppesen P, Obel C, Lund L, Madsen KB, Nielsen L, Nordentoft M. Mental sundhed og sygdom hos børn og unge i alderen 10-24 år - forekomst, udvikling og forebyggelsesmuligheder. København, Denmark: Vidensråd For Forebyggelse; 2018. Dec 03, [Google Scholar]
- 28.Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, Charlson FJ, Norman RE, Flaxman AD, Johns N, Burstein R, Murray CJL, Vos T. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. The Lancet. 2013 Nov 09;382(9904):1575–1586. doi: 10.1016/S0140-6736(13)61611-6.S0140-6736(13)61611-6 [DOI] [PubMed] [Google Scholar]
- 29.McGorry P, Bates T, Birchwood M. Designing youth mental health services for the 21st century: examples from Australia, Ireland and the UK. Br J Psychiatry Suppl. 2013 Jan;54:s30–5. doi: 10.1192/bjp.bp.112.119214.202/s54/s30 [DOI] [PubMed] [Google Scholar]
- 30.Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psychiatry. 2010 Dec 30;10:113. doi: 10.1186/1471-244X-10-113. https://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-10-113 .1471-244X-10-113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Rickwood DJ, Deane FP, Wilson CJ. When and how do young people seek professional help for mental health problems? Med J Aust. 2007 Oct 01;187(S7):S35–9. doi: 10.5694/j.1326-5377.2007.tb01334.x.ric10279_fm [DOI] [PubMed] [Google Scholar]
- 32.Andrews G, Issakidis C, Carter G. Shortfall in mental health service utilisation. Br J Psychiatry. 2001 Nov 02;179(5):417–25. doi: 10.1192/bjp.179.5.417.S0007125000267792 [DOI] [PubMed] [Google Scholar]
- 33.Clement S, Schauman O, Graham T, Maggioni F, Evans-Lacko S, Bezborodovs N, Morgan C, Rüsch N, Brown JSL, Thornicroft G. What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychol Med. 2015 Jan;45(1):11–27. doi: 10.1017/S0033291714000129.S0033291714000129 [DOI] [PubMed] [Google Scholar]
- 34.Salaheddin K, Mason B. Identifying barriers to mental health help-seeking among young adults in the UK: a cross-sectional survey. Br J Gen Pract. 2016 Sep 29;66(651):e686–e692. doi: 10.3399/bjgp16x687313. http://paperpile.com/b/BDqXMb/LIYL . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Kahl B, Miller H, Cairns K, Giniunas H, Welland L, Nicholas M. A measurable impact: Helping young people to be and stay well. Pyrmont, NSW, Australia: ReachOut; 2020. Jan 01, p. 2020. [Google Scholar]
- 36.Kauer SD, Buhagiar K, Blake V, Cotton S, Sanci L. Facilitating mental health help-seeking by young adults with a dedicated online program: a feasibility study of Link. BMJ Open. 2017 Jul 09;7(7):e015303. doi: 10.1136/bmjopen-2016-015303. https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=28694345 .bmjopen-2016-015303 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Le LK, Sanci L, Chatterton ML, Kauer S, Buhagiar K, Mihalopoulos C. The Cost-Effectiveness of an Internet Intervention to Facilitate Mental Health Help-Seeking by Young Adults: Randomized Controlled Trial. J Med Internet Res. 2019 Jul 22;21(7):e13065. doi: 10.2196/13065. https://www.jmir.org/2019/7/e13065/ v21i7e13065 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Sanci L, Kauer S, Thuraisingam S, Davidson S, Duncan A, Chondros P, Mihalopoulos C, Buhagiar K. Effectiveness of a Mental Health Service Navigation Website (Link) for Young Adults: Randomized Controlled Trial. JMIR Ment Health. 2019 Oct 17;6(10):e13189. doi: 10.2196/13189. https://mental.jmir.org/2019/10/e13189/ v6i10e13189 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Bidargaddi N, Musiat P, Winsall M, Vogl G, Blake V, Quinn S, Orlowski S, Antezana G, Schrader G. Efficacy of a Web-Based Guided Recommendation Service for a Curated List of Readily Available Mental Health and Well-Being Mobile Apps for Young People: Randomized Controlled Trial. J Med Internet Res. 2017 May 12;19(5):e141. doi: 10.2196/jmir.6775. https://www.jmir.org/2017/5/e141/ v19i5e141 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.O'Dea B, Han J, Batterham PJ, Achilles MR, Calear AL, Werner-Seidler A, Parker B, Shand F, Christensen H. A randomised controlled trial of a relationship-focussed mobile phone application for improving adolescents' mental health. J Child Psychol Psychiatry. 2020 Aug;61(8):899–913. doi: 10.1111/jcpp.13294. http://europepmc.org/abstract/MED/32683737 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.The Mindhelper Team Mindhelper. [2022-05-07]. https://mindhelper.dk/
- 42.Moffitt TE, Caspi A. Psychiatry's Opportunity to Prevent the Rising Burden of Age-Related Disease. JAMA Psychiatry. 2019 May 01;76(5):461–462. doi: 10.1001/jamapsychiatry.2019.0037. http://europepmc.org/abstract/MED/30916735 .2729445 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Clarke AM, Kuosmanen T, Barry MM. A systematic review of online youth mental health promotion and prevention interventions. J Youth Adolesc. 2015 Jan;44(1):90–113. doi: 10.1007/s10964-014-0165-0. [DOI] [PubMed] [Google Scholar]
- 44.Facebook Vejledning til Facebook-annoncer. Facebook. [2022-05-06]. https://www.facebook.com/business/ads .
- 45.SurveyXact. [2022-05-08]. https://www.surveyxact.com .
- 46.de Wit M, Pouwer F, Gemke RJ, Delemarre-van de Waal HA, Snoek FJ. Validation of the WHO-5 Well-Being Index in adolescents with type 1 diabetes. Diabetes Care. 2007 Aug 02;30(8):2003–6. doi: 10.2337/dc07-0447.dc07-0447 [DOI] [PubMed] [Google Scholar]
- 47.Bech P, Olsen LR, Kjoller M, Rasmussen NK. Measuring well-being rather than the absence of distress symptoms: a comparison of the SF-36 Mental Health subscale and the WHO-Five Well-Being Scale. Int J Methods Psychiatr Res. 2003 Jun;12(2):85–91. doi: 10.1002/mpr.145. http://europepmc.org/abstract/MED/12830302 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Haver A, Akerjordet K, Caputi P, Furunes T, Magee C. Measuring mental well-being: A validation of the Short Warwick-Edinburgh Mental Well-Being Scale in Norwegian and Swedish. Scand J Public Health. 2015 Nov 03;43(7):721–7. doi: 10.1177/1403494815588862.1403494815588862 [DOI] [PubMed] [Google Scholar]
- 49.Wilson C, Deane F, Ciarrochi J, Rickwood D. Measuring Help-Seeking Intentions: Properties of the General Help Seeking Questionnaire. Canadian Journal of Counselling. 2005;39(1):15. doi: 10.1037/t42876-000. https://ro.uow.edu.au/cgi/viewcontent.cgi?article=2580&context=hbspapers . [DOI] [Google Scholar]
- 50.de Rada VD, de Leeuw ED, Hox JJ, Dillman DA. International Handbook of Survey Methodology. New York, NY: Oxford University Press; 2009. pp. 551–554. [Google Scholar]
- 51.Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, Lancaster GA, PAFS consensus group CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. 2016 Oct 21;2(1):64. doi: 10.1186/s40814-016-0105-8.105 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Liverpool S, Mota CP, Sales CMD, Čuš Anja, Carletto S, Hancheva C, Sousa S, Cerón Sonia Conejo, Moreno-Peral P, Pietrabissa G, Moltrecht B, Ulberg R, Ferreira N, Edbrooke-Childs J. Engaging Children and Young People in Digital Mental Health Interventions: Systematic Review of Modes of Delivery, Facilitators, and Barriers. J Med Internet Res. 2020 Jun 23;22(6):e16317. doi: 10.2196/16317. https://www.jmir.org/2020/6/e16317/ v22i6e16317 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Examples of advertisements.
CONSORT EHEALTH Checklist (V 1.6.1).