Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Mar 1.
Published in final edited form as: Internet Interv. 2014 Mar;1(1):20–25. doi: 10.1016/j.invent.2014.02.003

Participant retention in an automated online monthly depression rescreening program: patterns and predictors

Supria Gill 1, Omar Contreras 2, Ricardo F Muñoz 1, Yan Leykin 2,*
PMCID: PMC4097172  NIHMSID: NIHMS573644  PMID: 25045623

Abstract

Internet-based mental health resources often suffer from low engagement and retention. An increased understanding of engagement and attrition is needed to realize the potential of such resources. In this study, 45,142 individuals were screened for depression by an automated online screener, with 2,539 enrolling in a year-long monthly rescreening study; they received a single monthly reminder email to rescreen their mood. We found that, even with such a minimal cohort maintenance strategy, a third of the participants completed 1 or more follow-ups, and 22% completed 2 or more follow-ups. Furthermore, completion of earlier follow-ups was highly predictive of future completions. We also found a number of participant characteristics (e.g., current depression status, previous depression treatment seeking, education level) predicted follow-up rates, singly or in interactions.

Keywords: online screening, mental health, international, attrition

Internet-based resources for mental health, such as interventions, screening, and other similar services, are proliferating rapidly, providing individuals with opportunities to access mental health resources and services even in places where traditional services are not available or accessible. Compared to traditional face-to-face mental health resources, their online counterparts are inexpensive to develop, highly scalable, widely accessible (Muñoz, 2010), with an ability to offer personalized information that is responsive to individuals’ needs and preferences (Leykin et al., 2011). Importantly, internet-based resources can be effective. Recent meta-analyses have demonstrated the effectiveness of internet interventions for mental disorders, such as depression and anxiety (Andersson & Cuijpers, 2009; Andrews, Cuijpers, Craske, McEvoy, & Titov, 2010; Griffiths, Farrer, & Christensen, 2010; Spek et al., 2007; Van't Hof, Cuijpers, & Stein, 2009); evidence for efficacy also exists for a variety of other conditions such as substance abuse (Chiauzzi et al., 2005; Strecher et al., 2005; Muñoz et al., 2005), obesity (McConnon et al., 2007), diabetes (McKay, King, Eakin, Seeley, & Glasgow, 2001), insomnia (Ritterband et al., 2009), chronic pain (Nevedal, Wang, Oberleitner, Schwartz, & Williams, 2013), and irritable bowel syndrome (Hunt, Moshier, & Milonova, 2009).

Most internet-based resources, especially interventions, rely on consistent engagement from participants, both for evaluation of their efficacy (usually via a clinical trial), as well as for overall effectiveness. As with most interventions, more engagement likely leads to greater effectiveness (An et al, 2008; Palmqvist, Carlbring, & Andersson, 2007; Richardson et al., 2013). However, concern about poor retention has dampened the enthusiasm about internet-based interventions (Christensen, Griffiths, Groves & Korten, 2005; Farvolden et al., 2005; Danaher et al., 2006; Eysenbach, 2005), with some researchers suggesting that high attrition may be inherent in eHealth trials (Eysenbach, 2005); attrition is especially problematic with “unguided” interventions, that is, interventions with little to no human interaction (Christensen et al., 2004; Christensen et al., 2005; Muñoz et al., 2012). Attrition is certainly not unique to the online medium – the modal number of therapy session is low (Connolly Gibbons et al., 2011), and dropout is considerable (Hamilton, Moore, Crane, & Payne, 2011).

For internet-based services to remain relevant and useful, increasing retention and adherence rates is essential. To counteract attrition in efficacy trials, researchers often use either financial incentives or live (e.g., phone-based) follow-up as cohort maintenance strategies (Carlbring et al., 2006; Muñoz et al., 2009). However, both of these approaches introduce significant undesirable confounds. Paying participants to visit the site or to provide data exposes participants to powerful extrinsic motivators which will not be present when the intervention is widely deployed. Indeed, introduction of such motivators can result in undesirable behaviors, such as attempts to “cheat” by registering multiple fake accounts (Prince, Litovsky, & Friedman-Wheeler, 2012). Thus, with financial incentives the ecological validity is reduced and engagement (and efficacy) of an un-incentivized intervention remains unknown. The same argument applies to the provision of phone-based follow-ups: though these may be effective (Fridrici, Lohaus, & Glass, 2009; Leykin, Aguilera, Torres, Perez-Stable, & Muñoz, 2012; Muñoz et al., 2009), they once again introduce variables that will not be present when an intervention is deployed. Instead of relying on these problematic strategies, improving retention should be achieved through attaining a better understanding of it (Eysenbach, 2005; Geraghty et al., 2005) and through offering useful and appealing products.

The present report describes retention in an internet-based mood and depression screener, where participants were asked to return to the site monthly for the next 12 months to rescreen their mood. In an attempt to understand the naturalistic rate of retention, participants were offered no incentives, financial or otherwise, no “live” contact, and no reminders to return beyond the single monthly invitation email. Though many participants did not return for a follow up screening, as expected, a considerable proportion did return at least once. We examine the pattern of retention, as well as explore the predictors of greater retention, with an overall goal of offering data that could be helpful for the designers of internet-based services who wish to improve retention and adherence.

Methods

An internet-based study was conducted worldwide. Consent was obtained over the internet, as approved by the Institutional Review Board of the University of California, San Francisco.

Participants

Inclusion criteria were limited to being at least 18 years of age and the ability to read and understand English. Over the 2 years of recruitment (December 9, 2009 – December 8, 2011), 63,055 persons visited the website, of whom 51,641 were eligible to participate (11,411 were younger than 18, 3 did not give their age); mean age = 31.92 years, SD = 12.17. Of those 51,641, 5.0% (N = 2,567) consented to participate in a rescreening study. However, 25 indicated that their answers may be inaccurate (i.e., they were just experimenting with the site), and an additional three subjects had inconsistent data; data from these participants were there excluded from all analyses, resulting in our final sample of 2,539 individuals. A majority (51.0%) of the participants included in our final sample were aged between 18 and 29, with the average age of participants being 32.64 (SD = 12.13), and 65.65% of participants being women. Participants from 102 countries and territories were present in our dataset. The largest numbers of participants were from United Kingdom, India, and South Africa (29.5%, 15.6%, and 9.0%, respectively).

Measures

Demographics 1 was offered to all visitors to the site, and assessed participants’ age, gender, race, and (since June 23, 2011) country of residence.

Demographics 2 was only administered to participants consenting to participate in a rescreening study; participants were asked about their education, employment, marital status, income, and subjective social status, as assessed by the “social ladder” (Adler et al., 2000), where individuals rate their position in society relative to others, and about the number of other individuals residing in the household. Participants were also asked about their experience with treatment for depression and their knowledge of the availability of mental health resources in their community.

The Major Depressive Episode (MDE) Screener (Muñoz, 1998) is an 18-item measure designed to screen for the presence of current and past MDEs. The MDE Screener, based on the Diagnostic Interview Schedule (Robins et al., 1981), assesses the presence of nine symptoms of depression according to the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV; American Psychiatric Association, 2000) over a period of 2 weeks or more; it also assesses whether Criterion C (significant impairment in functioning) is met within the same time span. Both current MDE (past 2 weeks) and lifetime MDE (any 2-week period excluding the last 2 weeks) are assessed. The MDE Screener has good agreement with the PRIME-MD (Spitzer, Williams, Kroenke, et al., 1994; Muñoz, McQuaid, Gonzalez, et al., 1999) and with clinician-administered diagnostic interviews (Vasquez, Muñoz, Blanco, et al., 2008). MDE Screener items were presented on the same web-page. Numerous studies reported that psychometric properties of questionnaires remain similar whether administered on the internet or on paper (Andersson, Kaldo-Sandström, Ström, & Strömgren, 2003; Holländare, Andersson, & Engström, 2010; Spek, Nyklícek, Cuijpers, & Pop, 2008).

Procedure

The study procedures are described in detail elsewhere (Leykin, Muñoz, Contreras, 2012). Briefly, the worldwide Google AdWords campaign was the most significant recruitment channel (Gross, Liu, Contreras, Muñoz, R. F., & Leykin, 2014); other participants found our website via organic search results, word of mouth, and other avenues. For the Google AdWords recruitment, one of the ads for a “Free Online Depression Screener” was triggered if search terms such as “depression”, “am I depressed” or “sadness” were entered. Clicking on the displayed ad led the individuals to the first page of the Mood Screener website.

The first (landing) page contained information about the nature of this site (i.e., research study), as well as about the acknowledgment that all information entered on the site will be retained. Visitors were then asked to provide their race, gender, age (to determine eligibility) and (since June 23, 2011) country of residence. Eligible participants then completed the “Current” portion of the MDE screener, and (since April 12, 2010) were asked whether their answers were accurate. Participants were then given personalized feedback on their responses, and were offered an opportunity to participate in a monthly follow-up rescreening study, for the next 12 months. Those consenting completed the Demographics 2 questionnaire, as well as the “Lifetime” portion of the MDE Screener, and received additional personalized feedback. Over the next 12 months, participants received a single monthly email inviting them to return to the site and re-screen their mood. Upon returning to the site, participants were asked to indicate whether they have sought help for depression over the past month; they also completed the current portion of the MDE screener and received feedback on their results.

Analytical procedures

To understand the predictors of follow-up response, Poisson regression models (Cameron & Trivedi, 1998) were constructed, predicting the count of follow-ups completed from individuals’ age, gender, employment status, marital status, years of completed education, household size, and depression status, as well as their second-order interactions. Poisson regression is a standard analytical technique for count data, and given that we were attempting to predict the count of follow-ups completed, Poisson modeling seemed appropriate for our purposes. Nonsignificant interactions were iteratively removed until none remained, to examine the main effects. Because of the large sample size, we have elected to only interpret results that reach the significance level of p < 0.001.

Results

Completion of follow-ups

Results showed that 858 (33.8%) participants completed 1 or more follow ups, and 582 (22.9%) participants completed 2 or more follow ups. Only 52 participants (2.0%) completed all 12 follow-ups, and only 225 participants (8.9%) completed over half of follow-ups (7 or more). Interestingly, the rate of drop-off throughout the follow-up year was not especially steep (i.e., 21.6% of participants completed the first follow-up, and 10.1% completed the last; see Figure 1). Furthermore, completing the first several follow-ups greatly increases the likelihood of future follow-up completions. Thus, of the 549 individuals who completed the 1st follow-up, 76.7% completed at least one subsequent follow-up; this number increased to 88.8% among the 320 participants who completed the first 2 follow-ups. Conversely, failure to complete the first follow-ups resulted in very high rate of non-return, with 84.5% of the 1,990 participants failing to complete the 1st follow-up, and 89.8% of the 1,872 participants failing to complete the first 2 follow-ups not returning to any future follow-ups. This pattern suggests that engaging individuals early is likely to greatly increase the likelihood of future engagement.

Figure 1.

Figure 1

Open in a new tab

Proportion of participants completing each follow-up period and total number of follow-ups.

Predictors of follow-up completion

Poisson regressions predicting the count of completed follow-ups from participants’ demographic and clinical characteristics revealed a number of significant main effects and interactions. Depression status significantly predicted the number of complete follow-ups (Wald Chi-Square(1) = 48.23, p < 0.0001), with individuals screening negative for a current depressive episode could be expected to complete 1.29 more follow-ups than those screening positive (Wald Chi-Square(1) = 48.23, p < 0.0001, Exp(B) = 1.29).

Employment status interacted with whether an individual previously sought help for depression to predict the number of complete follow-ups (Wald Chi-Square(1) = 38.27, p < 0.0001). Among those who have previously sought help, employed individuals may be expected to complete 1.21 more follow-ups than the unemployed (Wald Chi-Square(1) = 18.15, p < 0.0001, Exp(B) = 1.21). However, among those who did not previously seek help for depression, employed individuals may be expected to complete fewer (0.80) follow-ups than the unemployed (Wald Chi-Square (1) = 15.06, p < 0.001, Exp(B) = 0.80 (Figure 2).

Figure 2.

Figure 2

Open in a new tab

Average number of follow-ups completed for participants who previously sought help or not by employment status.

Help-seeking for depression likewise interacted with the level of an individual's education (Wald Chi-Square(1) = 16.50, p < 0.0001), such that individuals who have previously sought help may be expected to have a 1.04 times more follow-ups completed for every one year of education attained (Wald Chi-Square(1) = 72.03, p < 0.0001; Exp(B) = 1.04); this is in contrast to individuals with no previous depression help-seeking, who showed no increase in follow-up rate with years of education attained (Wald chi-square(1) = 0.75, ns, Exp(B) = 1.00) (Figure 3).

Figure 3.

Figure 3

Open in a new tab

Average number of follow-ups completed for participants who previously sought help or not by the number of years of education.

Household size also interacted with the level of an individual's education (Wald Chi-Square(1) = 14.43, p < 0.0001), with individuals who live alone being expected to have 1.06 times more follow ups completed for every one year of education attained (Wald Chi-Square(1) = 36.96, p < 0.0001; Exp(B) = 1.06); this is in contrast to those who live with others, for whom the multiplicative effect was still significant, but, at 1.02, about 3 times smaller (Wald chi-square(1) = 21.17, p < 0.0001, Exp(B) = 1.02) (Figure 4).

Figure 4.

Figure 4

Open in a new tab

Average number of follow-ups completed for participants who live alone or live with others by the number of years of education.

Marital status interacted with age (Wald Chi-Square(1) = 42.41, p < 0.001) such that individuals who were married or partnered were expected to have 1.02 more follow-ups completed for every one year of age (Wald Chi-Square(1) = 76.62, p < 0.001; Exp(B) = 1.02); this is in contrast to those who were not married, for whom years of age did not significantly increase follow-up rate (Wald chi-square(1) = 0.0, ns, Exp(B) = 1.00).

Finally, gender also interacted with an individual's age (Wald Chi-Square(1) = 14.62, p < 0.001), with women exhibiting a multiplicative effect of 1.01 on the expected number of follow ups completed for every additional one year of age (Wald Chi-Square(1) = 42.01, p < 0.001; Exp(B) = 1.01); this is in contrast to men, for whom years of age had no effect on the expected rate of follow-ups (Wald chi-square(1) = 0.75, ns, Exp(B) = 1.00).

Discussion

The purpose of this report is to describe the rate of naturalistic follow-up to a study offering individuals a monthly opportunity to be screened for major depression and receive feedback on their results, and to understand the predictors of returning for follow-ups. This study is in response to growing calls to develop the study of attrition (Eysenbach, 2005; Geraghty et al., 2012) from internet-based interventions, and, conversely, the need for the complementary study of retention in these and other technology-based interventions and resources.

Our results show that, while a considerable proportion of participants failed to return for any follow-ups, as could be expected on the internet (Christensen et al., 2005; Leykin, Aguilera, Torres, Perez-Stable, Muñoz, 2011), a sizable proportion of participants – one third – did in fact return for at least one follow-up. Additionally, almost a tenth of all participants completed at least half of all available follow-ups. Though these proportions appear relatively small, it must be noted that the cohort maintenance procedures in this study were minimal – a single follow-up invitation email that was sent monthly. Participants received no phone calls, no contact with a live clinician or a research assistant, no payment for participation, and no reminder emails if they failed to complete a follow-up. Moreover, unlike a bona fide intervention, which commonly contain a variety of materials and tools (Leykin et al., 2012; Leykin, Barrera, & Muñoz, 2010), our follow-ups offered only a depression screening instrument with feedback. Thus, the follow-up rates attained in our investigation are important for three reasons. First, they offer evidence of un-incentivized interest in and demand for an internet-based mental health resource. Whereas in studies where incentives are used to attract participants to return to the site rates of follow-ups might be the function of incentives rather than interest, in this study follow-up rates likely reflect true interest in participation and in utilizing the screening tool. Second, given our minimal cohort maintenance strategies, our results offer a base level of follow-up participation that can be used as a comparison once other strategies (e.g., reminder emails, SMS messaging reminders) are introduced. Finally, our results can be useful for research conducting studies on self-help automated internet interventions as an indication of the minimal rates of follow-up that can be expected from naturalistic users. Well-designed interventions are expected to have engaging content designed in part to build a “relationship” between a participant and the intervention, and which could be revisited upon follow-up; these aspects of interventions likely increase follow-up rates (Kelders, Kok, Ossebaard & Van Germert-Pijnen, 2012; Morrison, Moss-Morris, Michie, & Yardley, 2013). An intervention that fails to attain the follow-up rates found in our minimal contact study may be turning away, rather than attracting, users. Our results also strongly suggest that completion of early follow-ups is highly important for future follow-up retention: participants who completed the first one or two follow-ups were very likely to complete at least one more follow-up. Conversely, those who failed to complete the first one or two follow-ups were very unlikely to complete any future follow-ups. Though there was likely at least some self-selection (i.e., those more likely to complete first follow-ups are different from individuals who elect to ignore all reminders), it is also possible that completion of first follow-ups offered a “foot-in-the-door”, prompting subsequent responses. Thus, for studies with limited budgets, researchers might be encouraged to devote some resources to promote completion of early follow-ups in hopes that this behavior will endure.

We have also explored a number of predictors of follow-up, as well as their interactions. Some of predictors we have identified, such as greater symptoms, education, and age, have likewise been described in other studies (Geraghty et al., 2012). Screening positive for depression reduced the number of follow-ups participants had completed. This was somewhat counterintuitive, as one would expect individuals who are feeling more depressed and down to utilize the depression screening service more. However, it is also possible that individuals with significant symptoms are either too impaired to return to the site, or do not feel that a confirmation of their depression status would be useful.

Help-seeking status emerged as an important predictor of follow-up rate, interacting with two other variables to predict the number of completed follow-up rates. In an interaction with employment, we observed that among individuals who have previously sought help (and who were overall more likely to complete follow-ups), being employed was associated with more completed follow-ups, whereas the reverse was true for those unemployed. It is possible that prior help-seeking has informed individuals about the importance of self-monitoring, resulting in higher overall follow-up rate. Current unemployment may be related to the level of an individual's impairment; making unemployed participants less likely to follow up. However, current unemployment may also indicate a greater need for treatment. Thus, among those who never sought care, unemployed participants may have relied on the rescreenings to a larger extent. Help seeking also interacted with education level to predict follow-up rate. Individuals who never sought care had no relationship between education and completed follow-ups, but those who had previously sought care having a strong association between education level and follow-up rate. Once again, it is possible that previous help-seeking is associated with greater recognition of the importance of self-monitoring, and individuals who are more educated are more able to either have or act on this recognition. Education also interacted with household composition to predict follow-up rate. Though higher levels of education were predictive of more follow-up completion overall (indeed, previous reports suggest that more educated individuals seek depression treatment at higher rates, Mojtabai & Olfson, 2006). This association was especially pronounced among individuals living alone, perhaps because those living with others have more social support (thereby reducing the need for mood checks) or, conversely, less privacy, making visits to our site more challenging.

Age also emerged as an important predictor, interacting with marital status and with gender. Follow-up rate was higher among older women, as well as among older married participants. It has previously been found that women and older individuals are more likely to seek depression treatment (Bayer & Peay, 1997; Bristow & Patten, 2002; Komiya, 2000), and it is possible that these effects extend beyond treatment, which our site did not offer. It is possible that older individuals are more likely to exhibit a stronger social commitment to participation once consented, and be less likely to ignore a commitment. Men, however, may face social pressures to ignore their mood states or consider them less relevant, resulting in age no longer being predictive of follow-up completion among men. Being single in later years may be indicative of longer standing or more impairing mood-related difficulties, which as we found above is predictive of lower follow-up rates.

This study has several limitations. The sample was collected using Google AdWords ad campaigns. Though Google clearly has a dominant position in the English-language search market (comScore, 2013), our sample may not be representative of those who use other search engines or who never respond to or actively suppress Google Ads. Furthermore, participants joined a depression rescreening study; the results may not therefore generalize to individuals suffering from other physical or psychiatric conditions. The sample in this study was worldwide, and it is possible that the feedback about mood states provided by the site was less applicable to individuals from certain countries or cultures, thereby reducing the utility of the rescreenings (and the likelihood of completing follow-ups) for some groups of individuals. Finally, participants’ expectations for the site, which may affect adherence (Boettcher, Renneberg, & Berger, 2013) was not assessed.

As internet-based resources, including interventions, screening services, and information services proliferate, data regarding participants’ engagement and retention is crucial to retain the usefulness and relevance of such resources. Such data can help researchers develop more targeted cohort maintenance strategies, ensuring that limited research budgets are spent in a way that optimize retention in an empirically sound manner through a better understanding of participant engagement. Thus, we hope that this study can facilitate future research on retention, for instance, by informing randomized trials evaluating the benefit of various cohort maintenance procedures, or cost-effectiveness investigations seeking to maximize participant retention while reducing costs. Such work will ultimately produce internet interventions that are more mature and sophisticated, thereby maximizing their potential to provide quality services to more people in need of care.

Highlights.

  • 2,539 participants enrolled in monthly screening study

  • 33.8% completed 1+ follow-up, and 22.9% completed 2+ follow-ups

  • Completion of early follow-ups predicts future completions; reverse is also true

  • Predictors of follow-ups included depression, employment, help-seeking, education

Acknowledgements

This work was supported by Robert Wood Johnson Health and Society Scholars Seed Grant (Leykin, P.I.), by an infrastructure grant from the University of California Committee on Latino Research to the University of California/San Francisco General Hospital Latino Mental Health Research Program (Muñoz, P.I.), by NIMH grant 5K08MH091501 (Leykin, P.I.). This project was also supported by Grant SRG-0-059-11 (Leykin, P.I.) from the American Foundation for Suicide Prevention; the content is solely the responsibility of the authors and does not necessarily represent the official views of the American Foundation for Suicide Prevention. We would like to express our thanks to Google, Inc, for awarding us an AdWords grant (Muñoz, PI), the Brin-Wojcicki Foundation for their generous support.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflicts of interests

None

References

  1. Adler NE, Epel ES, Castellazzo G, Ickovics JR. Relationship of subjective and objective social status with psychological and physiological functioning: preliminary data in healthy White women. Health Psychology. 2000;19(6):586–592. doi: 10.1037//0278-6133.19.6.586. [DOI] [PubMed] [Google Scholar]
  2. American Psychiatric Association . Diagnostic and statistical manual of mental disorders. 4th ed., text rev. Author; Washington, DC: 2000. [Google Scholar]
  3. An LC, Schillo BA, Saul JE, Wendling AH, Klatt CM, Berg CJ, Ahulwalia JS, Kavanaugh AM, Christenson M, Luxenberg MG. Utilization of smoking cessation informational, interactive, and online community resources as predictors of abstinence: cohort study. Journal of Medical Internet Research. 2008;10:e55. doi: 10.2196/jmir.1018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cognitive Behaviour Therapy. 2009;38(4):196–205. doi: 10.1080/16506070903318960. [DOI] [PubMed] [Google Scholar]
  5. Andersson G, Kaldo-Sandström V, Ström L, Strömgren T. Internet administration of the Hospital Anxiety and Depression Scale in a sample of tinnitus patients. Journal of Psychosomatic Research. 2003;55:259–62. doi: 10.1016/s0022-3999(02)00575-5. [DOI] [PubMed] [Google Scholar]
  6. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS One. 2010;5(10):e13196. doi: 10.1371/journal.pone.0013196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Boettcher J, Renneberg B, Berger T. Patient expectations in internet-based self-help for social anxiety. Cognitive Behaviour Therapy. 2013;42(3):203–214. doi: 10.1080/16506073.2012.759615. [DOI] [PubMed] [Google Scholar]
  8. Cameron AC, Trivedi PK. Regression analysis of count data. Cambridge University Press; 1998. [Google Scholar]
  9. Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, Ekselius L, Andersson G. Remote treatment of panic disorder: A randomized trial of Internet-based cognitive behavioral therapy supplemented with telephone calls. American Journal of Psychiatry. 2006;163:2119–2125. doi: 10.1176/ajp.2006.163.12.2119. [DOI] [PubMed] [Google Scholar]
  10. Chiauzzi E, Green TC, Lord S, Thum C, Goldstein M. My student body: a high-risk drinking prevention web site for college students. Journal of American College Health. 2005;53(6):263–274. doi: 10.3200/JACH.53.6.263-274. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. Journal of Medical Internet Research. 2004;226(4):e46. doi: 10.2196/jmir.6.4.e46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: community users of an Internet-based cognitive behavior. Australian and New Zealand Journal of Psychiatry. 2005;40:59–62. doi: 10.1080/j.1440-1614.2006.01743.x. [DOI] [PubMed] [Google Scholar]
  13. 2013 comScore Releases July 2013 U.S. Search Engine Rankings. Retrieved January 20, 2013, from http://www.comscore.com/Insights/Press_Releases/2013/8/comScore_Releases_July _2013_U.S._Search_Engine_Rankings. Archived at http://www.webcitation.org/6JiAav83f.
  14. Connolly Gibbons MB, Rothbard A, Farris KD, Wiltsey Stirman S, Thompson SM, Scott K, Heintz LE, Gallop R, Crits-Christoph P. Changes in psychotherapy utilization among consumers of services for major depressive disorder in the community mental health system. Administration and Policy in Mental Health. 2011;38(6):495–503. doi: 10.1007/s10488-011-0336-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining Participant Exposure Measures in Web-Based Health Behavior Change Programs. Journal of Medical Internet Research. 2006;8(3):e15. doi: 10.2196/jmir.8.3.e15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Eysenbach G. The Law of Attrition. Journal of Medical Internet Research. 2005;7(1):e11. doi: 10.2196/jmir.7.1.e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. Journal of Medical Internet Research. 2005;7(1):e7. doi: 10.2196/jmir.7.1.e7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Fridrici M, Lohaus A, Glass C. Effects of incentives in web-based prevention for adolescents: Results of an exploratory field study. Psychology & Health. 2009;24(6):663–675. doi: 10.1080/08870440802521102. [DOI] [PubMed] [Google Scholar]
  19. Geraghty AW, Torres LD, Leykin Y, Perez-Stable E, Muñoz RF. Understanding attrition from international internet health interventions: a step towards global eHealth. Health Promotion International. 2012;28(3):442. doi: 10.1093/heapro/das029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Griffiths KM, Farrer L, Christensen H. The efficacy of internet interventions for depression and anxiety disorders: a review of randomised controlled trials. The Medical Journal of Australia. 2010;192(11 Suppl):S4–11. doi: 10.5694/j.1326-5377.2010.tb03685.x. [DOI] [PubMed] [Google Scholar]
  21. Gross MS, Liu NH, Contreras O, Muñoz RF, Leykin Y. Using Google AdWords for international multilingual recruitment to health research websites. Journal of Medical Internet Research. 2014;6(1):e18. doi: 10.2196/jmir.2986. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hamilton S, Moore AM, Crane DR, Payne SH. Psychotherapy dropouts: Differences by modality, license, and DSM-IV diagnosis. Journal of Marital and Family Therapy. 2011;37:333–343. doi: 10.1111/j.1752-0606.2010.00204.x. [DOI] [PubMed] [Google Scholar]
  23. Holländare F, Andersson G, Engström I. A comparison of psychometric properties between Internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. Journal of Medical Internet Research. 2010;12(5):e49. doi: 10.2196/jmir.1392. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hunt MG, Moshier S, Milonova M. Brief cognitive-behavioral internet therapy for irritable bowel syndrome. Behavior Research and Therapy. 2009;47(9):797–802. doi: 10.1016/j.brat.2009.05.002. [DOI] [PubMed] [Google Scholar]
  25. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. Journal of Medical Internet Research. 2012;14(6):e152. doi: 10.2196/jmir.2104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Leykin Y, Aguilera A, Torres LD, Perez-Stable EJ, Muñoz RF. Interpreting the outcomes of automated internet-based randomized trials: example of an International Smoking Cessation Study. Journal of Medical Internet Research. 2012;14(1):e5. doi: 10.2196/jmir.1829. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Leykin Y, Barrera AZ, Muñoz R, F. One more tool for smoking cessation: Internet-based interventions. In: Cucciare MA, Weingardt K,R, editors. Using Technology to Support Evidence-Based Behavioral Health Practices: A Clinician's Guide. Routledge; New York, NY.: 2009. [Google Scholar]
  28. Leykin Y, Muñoz RF, Contreras O. Are consumers of Internet health information “cyberchondriacs”? Characteristics of 24,965 users of a depression screening site. Depression and Anxiety. 2012;29:71–77. doi: 10.1002/da.20848. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Leykin Y, Thekdi SM, Shumay DM, Muñoz RF, Riba M, Dunn LB. Internet interventions for improving psychological well-being in psycho-oncology: review and recommendations. Psycho-oncology. 2012;21:1016–1025. doi: 10.1002/pon.1993. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. McConnon A, Kirk SF, Cockroft JE, Harvey EL, Greenwood DC, Thomas JD, Ransley JK, Bojke L. The Internet for weight control in an obese sample: results of a randomised controlled trial. BioMed Central Health Service Research. 2007;7:206. doi: 10.1186/1472-6963-7-206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. McKay HG, King D, Eakin EG, Seeley JR, Glasgow RE. The diabetes network internet-based physical activity intervention: a randomized pilot study. Diabetes Care. 2001;24(8):1328–1334. doi: 10.2337/diacare.24.8.1328. [DOI] [PubMed] [Google Scholar]
  32. Morrison L, Moss-Morris R, Michie S, Yardley L. Optimizing engagement with Internet-based health behaviour change interventions: Comparison of self-assessment with and without tailored feedback using a mixed methods approach. British Journal of Health Psychology. 2013 doi: 10.1111/bjhp.12083. Epub. doi: 10.1111/bjhp.12083. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Muñoz RF. Preventing major depression by promoting emotion regulation: a conceptual framework and some practical tools. International Journal of Mental Health Promotion. 1998 1998 Sep;:23–40. Inaugural issue. [Google Scholar]
  34. Muñoz RF. Using evidence-based Internet interventions to reduce health disparities worldwide. Journal of Medical Internet Research. 2010;12(5):e60. doi: 10.2196/jmir.1463. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Muñoz RF, Aguilera A, Schueller SM, Leykin Y, Pérez-Stable EJ. From online randomized controlled trials to participant preference studies: Morphing the San Francisco stop smoking site into a worldwide smoking cessation resource. Journal of Medical Internet Research. 2012;14(3):e64. doi: 10.2196/jmir.1852. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Muñoz RF, Lenert LL, Delucchi K, Stoddard J, Perez EJ, Penilla C, Perez-Stable EJ. Towards evidence-based Internet interventions: a Spanish/English Web site for international smoking cessation trials. Nicotine & Tobacco Research. 2006;8:77–87. doi: 10.1080/14622200500431940. [DOI] [PubMed] [Google Scholar]
  37. Muñoz RF, Barrera AZ, Delucchi K, Penilla C, Torres LD, Pérez-Stable EJ. International Spanish/English Internet smoking cessation trial yields 20% abstinence rates at 1 year. Nicotine & tobacco research. 2009;11(9):1025. doi: 10.1093/ntr/ntp090. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Nevedal DC, Wang C, Oberleitner L, Schwartz S, Williams AM. Effects of an individually tailored web-based chronic pain management program on pain severity, psychological health, and functioning. Journal of Medical Internet Research. 2013;15:e201. doi: 10.2196/jmir.2296. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Palmqvist B, Carlbring P, Andersson G. Internet-delivered treatments with or without therapist input: does the therapist factor have implications for efficacy and cost? Expert Review of Pharmacoeconomics & Outcomes Research. 2007;7:291–297. doi: 10.1586/14737167.7.3.291. [DOI] [PubMed] [Google Scholar]
  40. Prince KR, Litovsky AR, Friedman-Wheeler DG. Internet-mediated research: Beware of bots. The Behavior Therapist. 2012;35:85–88. [Google Scholar]
  41. Richardson A, Graham AL, Cobb N, Xiao H, Mushro A, Abrams D, Vallone D. Engagement promotes abstinence in a web-based cessation intervention: cohort study. Journal of Medical Internet Research. 2013;15(1):e14. doi: 10.2196/jmir.2277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Ritterband LM, Thorndike FP, Gonder-Frederick LA, Magee JC, Bailey ET, Saylor DK, et al. Efficacy of an Internet-based behavioral intervention for adults with insomnia. Archives of General Psychiatry. 2009;66:692–698. doi: 10.1001/archgenpsychiatry.2009.66. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Robins LN, Helzer JE, Croughan J, Ratcliff KS. National Institute of Mental Health Diagnostic Interview Schedule. Archives of General Psychiatry. 1981;38(4):381–389. doi: 10.1001/archpsyc.1981.01780290015001. [DOI] [PubMed] [Google Scholar]
  44. Strecher VJ, Shiffman S, West R. Randomized controlled trial of a web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy. Addiction. 2005;100(5):682–688. doi: 10.1111/j.1360-0443.2005.01093.x. [DOI] [PubMed] [Google Scholar]
  45. Spek V, Cuijpers P, Nyklcek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behavior therapy for symptoms of depression and anxiety: a meta-analysis. Psychological Medicine. 2007;37:319–328. doi: 10.1017/S0033291706008944. [DOI] [PubMed] [Google Scholar]
  46. Spek V, Nyklícek I, Cuijpers P, Pop V. Internet administration of the Edinburgh Depression Scale. Journal of Affective Disorders. 2007;106:301–5. doi: 10.1016/j.jad.2007.07.003. [DOI] [PubMed] [Google Scholar]
  47. Van't Hof E, Cuijpers P, Stein DJ. Self-help and Internet-guided interventions in depression and anxiety disorders: a systematic review of meta-analyses. CNS Spectrum. 2009;14(2 Suppl 3):34–40. doi: 10.1017/s1092852900027279. [DOI] [PubMed] [Google Scholar]

RESOURCES