Abstract
Background
Computerized, Internet-delivered interventions can be efficacious; however, uptake and maintaining sustained client engagement are still big challenges. We see the development of effective engagement strategies as the next frontier in online health interventions, an area where much creative research has begun. We also argue that for engagement strategies to accomplish their purpose with novel targeted populations, they need to be tailored to such populations (ie, content is designed with the target population in mind). User-centered design frameworks provide a theoretical foundation for increasing user engagement and uptake by including users in development. However, deciding how to implement this approach to enage users in mental health intervention development is challenging.
Objective
The aim of this study was to get user input and feedback on acceptability of messaging content intended to engage suicidal individuals.
Methods
In March 2013, clinic intake staff distributed flyers announcing the study, “Your Feedback Counts” to potential participants (individuals waiting to be seen for a mental health appointment) together with the Patient Health Questionnaire. The flyer explained that a score of two or three (“more than half the days” or “nearly every day” respectively) on the suicide ideation question made them eligible to provide feedback on components of a suicide prevention intervention under development. The patient could access an anonymous online survey by following a link. After providing consent online, participants completed the anonymous survey.
Results
Thirty-four individuals provided data on past demographic information. Participants reported that they would be most drawn to an intervention where they knew that they were cared about, that was personalized, that others like them had found it helpful, and that included examples with real people. Participants preferred email invitations with subject lines expressing concern and availability of extra resources. Participants also provided feedback about a media prototype including a brand design and advertisement video for introducing the intervention.
Conclusions
This paper provides one model (including development of an engagement survey, audience for an engagement survey, methods for presenting results of an engagement survey) for including target users in the development of uptake strategies for online mental health interventions.
Keywords: human centered design, user centered design, health 2.0, suicide
Introduction
Internet-based interventions have the potential to increase the accessibility and efficacy of mental health treatment [1-4]. They are effective in producing change; several meta-analyses have found no significant differences in results when compared to face-to-face interventions [1,5,6]. Most of the current, long-established, research-based mental health treatment options serve a population that is already either seeking treatment and receiving care, or is in crisis and getting services from the emergency department [7]. But the Internet helps us broaden the populations being served and reach individuals who might otherwise be missed in more traditional delivery modalities [8]. The Internet can reach those not seeking traditional care due to logistic difficulties in accessing treatment or stigma around mental illness and has potential for large scale dissemination [9]. However, unsatisfactory uptake and follow-through is a significant barrier to reaching the full potential of online interventions [10-13]. Thus, identifying effective strategies to engage the target population is of critical importance for Internet-based mental health interventions to reach their optimum potential.
The involvement of consumer collaborators in mental health research has the potential to transform care and increase patient engagement [14]. Yet, the peripheral role of the end-user in the development of online mental health interventions has been proposed as a major barrier to utilizing such a service or program [15]. Health 2.0 (and Medicine 2.0, ie, the application of Web 2.0 technologies to health care) and human-centered design (a design process extensively relying on the intended user at all levels of the design process) are useful frameworks for enabling users to have an active role in generating and manipulating Web content and participating in health promotion [16,17]. These frameworks provide a theoretical foundation for increasing user engagement and uptake, but deciding how to implement this approach to increase engagement in mental health interventions is challenging.
Suicide is a devastating consequence of mental illness and major mental health concern in the United States, ranking as the 10th leading cause of death and generating high-cost emergency room visits and hospitalizations [18]. Research has led to the production of mental health interventions efficacious in treating suicidality [19-21], including a Dutch online program designed specifically to decrease frequency and intensity of suicidal ideation [22,23]. However, results from the Dutch study indicate that over half the eligible patients did not return the study consent form [22], thus reinforcing the importance of user engagement.
The aim of this project was to take the first step in attempting to increase user uptake and engagement in an online suicide prevention study by soliciting feedback from patients eligible for the intervention program. Specifically, we surveyed patients with current suicidal ideation about acceptable subject lines, intervention descriptions, project names, and introductory videos in order to gain a better understanding of how to reach the target audience.
Methods
Participant Recruitment
Individuals seen for mental health treatment at Group Health, a large health care organization based in Seattle, Washington, are asked to complete the Patient Health Questionnaire (PHQ-9) before every visit as part of routine care. The PHQ-9 is a 9-item, self-report questionnaire used for screening, monitoring, and measuring the severity of depression over the previous 2 weeks [24]. The PHQ-9 uses a 4-point scale (“not at all”, “several days”, “more than half the days”, “nearly every day”), and the last question asks about presence and duration of suicide ideation. An answer of “more than half the days” or “nearly every day” for suicide ideation has been found to predict subsequent suicide attempts or suicide death [25].
In March 2013, clinic intake staff distributed flyers announcing the study, “Your Feedback Counts”, to potential participants (individuals waiting to be seen for mental health appointments) together with the PHQ-9. The flyer explained that a score of two or three (“more than half the days” or “nearly every day” respectively) on the suicide ideation question made them eligible to provide feedback on components of a suicide prevention intervention under development. The patient could access an anonymous online survey by following a link. After providing consent online, participants completed the anonymous survey.
As a last step, participants could link to a separate survey and provide their contact information to receive a US $10 incentive. No link between survey data and contact information was collected. All study procedures were approved by the Group Health Institutional Review Board.
Survey
The survey included questions about demographics and medical treatment for a suicide attempt or self-injury, as well as acceptability of proposed intervention messaging (see Multimedia Appendices 1 and 2). A patient consultant group of 5 individuals with a history of suicidal ideation and suicide attempts helped to generate the messaging options, as well as a video describing the intervention. There were 26 options for message subject lines introducing the intervention, 20 options for intervention characteristics, 9 intervention brand names, and an option for study participants to make additional suggestions. The survey was programmed using DatStat Ilume and accessible by computer or other portable Internet-connected device (smartphone, iPad, etc) [26]. All questions related to the intervention invitation and description were asked using a 5-point Likert scale (0=strongly disagree; 4=strongly agree). In evaluating the results, we calculated the difference in responders endorsing “agree” or “strongly agree” for a particular option and “disagree” or “strongly disagree”. Descriptive statistical analyses were conducted via SPSS 16 [27].
Results
Participants
Of the 39 participants who visited the online consent page, 38 agreed to participate. Three individuals consented but provided no further information and four individuals provided only demographic information. Participants completed the survey in an average of 20.5 minutes (SD 25.98). Most participants were female, had received medical help for a suicide attempt or self-injury during their lifetime, and were under 64 years old (Table 1).
Table 1.
Demographics and previous treatment for self-harm (n=34).
Characteristics | n (%) | |
Age, years | ||
|
<30 | 16 (47.1) |
|
30-64 | 16 (47.1) |
|
>65 | 2 (5.9) |
Gender | ||
|
Female | 23 (67.6) |
|
Male | 11 (32.4) |
Receiving medical help for suicide attempt/self-injury (lifetime) | ||
|
Yes | 22 (64.7) |
|
No | 12 (35.3) |
Invitation Message Subject Line Preferences
Of the 26 different subject lines proposed, 5 emerged as agreeable to more than 45% of responders and disagreeable to less than 25%—“Checking In”, “Touching Base”, “Between Visits Resources”, “Can We Help?”, and “Invitation to Online Support Program”. Messages advertising support as “free” or “quick” (including words as “instant”, “free”, “simple”) were less appealing to participants (Figure 1).
Figure 1.
“I would like to open a message with Subject Line…”.
Intervention Characteristic Preferences
Twenty intervention characteristics were provided, and an average of 26.5 participants answered each question (Figure 2). The most preferred characteristics were that the program was personalized, that someone who cares personally is involved, that others have found this program helpful, that real people have found the program helpful, and that “you are not alone”. Participants least preferred a “slick” looking and “well made” program.
Figure 2.
Intervention content preferences.
Intervention Name Brand Preferences
The two most preferred options for the intervention name brand were “Now Matters Now” and “You Are Here”. Preference for intervention name varied widely with an agree/disagree gap between –37.5% (“Tiny Matters”) and 30.4% (“Now Matters Now”). Each question was answered on average by 23.5 responders. See Figure 3.
Figure 3.
Intervention name brand preferences.
Video Evaluation
Survey participants were asked to watch a 2-minute video about the intervention and answer a series of 8 follow-up questions (see Figure 4). The purpose of the video evaluation was to identify specific aspects of the video that might need improvement. Participants were asked about technical characteristics (eg, quality of sound) and artistic impression (eg, music, visuals) as well as content itself (eg, liking what the speaker had to say).
Figure 4.
Feedback on advertisement video.
Discussion
Principal Findings
The main goal of this project was to survey patients with current suicidal ideation about acceptable engagement messaging for an online intervention for individuals with suicidal thoughts. Invitation subject lines, intervention descriptions, project names, and an introductory video were reviewed. The most agreeable subject lines for invitation messages were simple inquiries about how the respondent was doing (“Checking in” and “Touching base”). Also agreeable were messages highlighting additional “resources” that “your provider thought would be helpful”. Certain messaging appeared less agreeable to participants, such as messages that proposed something “free” or “instant”. Intervention characteristics that respondents endorsed were personalization, caring, and real people with personal examples. Less important characteristics were appearance and something “new and exciting”. Two brand names relating to the present moment, “Now Matters Now” and “You Are Here”, were rated most agreeable. Overall, most respondents agreed that they could relate to the video introduction, but responses also indicated room for improvement with technical aspects and video comprehension. This project accomplished the goal of giving researchers a good place to start with messaging designed to promote uptake and engagement in an online intervention. Perhaps, more importantly, we now have a better idea about messaging options to avoid.
Limitations
There are limitations to this project. The sample size was small and demographically narrow and thus did not allow for subgroup analyses or comparisons. However, the study sample demographics were similar to those who most commonly attempt suicide in our health care setting [25]. Individuals in the United States most likely to die by suicide, that is, males in their middle and late years, are not represented here. Future research should apply such branding and messaging questions to this specific population, but also a much larger sample in general, given the large deviations in preferences for intervention name/brand. We chose not to advertise more broadly for this survey (on websites or with advertising, which may have resulted in a larger sample) because we were interested specifically in the population our intervention is targeted to—those at increased risk for suicide attempt in the following year in our health care setting [25].
To preserve the anonymity of responders we asked only basic demographic questions. We do not know how many patients were potentially eligible to participate, so we cannot determine the response rate. Participants were informed of inclusion criteria and then were self-selected into the study, which means there may be potential selection bias. We recruited a treatment-seeking population from a mental-health clinic waiting room, and the target population of the future intervention also includes patients receiving only primary care treatment who may have different messaging preferences. We focused on the brevity of the survey, which limited our ability to explore systematic testing or manipulation of messaging content. An alternative explanation for the findings may be that the messages containing “instant/free/simple” were too vague, rather than the terms themselves being unappealing. Last, the survey options were not based on a theoretical foundation or prior hypotheses due to a lack of prior research targeting suicidal individuals for online interventions.
Future Research
Future research should include systematic manipulation of variations around messaging (eg, comparing “Between visits resources” with “Between visits free resources”) in the hope of providing greater understanding of the effectiveness of the various message components. Future research including larger samples and such manipulations could allow us to determine how interventions might be delivered differently for subgroups of the target population (depending on symptoms, preferences, etc).
Conclusions
We consulted with a group of patients with self-reported suicidal thoughts and past attempts to help generate the options we presented to patients. Patients with mental illnesses and suicidal ideation or behavior often report discrimination on many levels (personal, community, institutional, etc) [28]. Receiving feedback from our patient consults was particularly important since we wanted to avoid messaging that would reinforce the stigma of mental health illnesses and prevent or discourage people affected by mental illness from seeking treatment. This paper provides one model (including development of an engagement survey, audience for an engagement survey, methods for presenting results of an engagement survey) for including target users in the development of uptake strategies for online mental health interventions. Finally, large pragmatic clinical trials should be conducted in order to identify whether an online intervention could reduce suicide attempts or deaths on a large scale.
Acknowledgments
We gratefully acknowledge the funding for this work. Funding was awarded to Dr Whiteside in the form of fellowships from the Group Health Foundation and the American Foundation for Suicide Prevention. This work was also supported by an R-34 from the National Institutes of Mental Health MH097836 awarded to Dr Whiteside.
Abbreviations
- PHQ
Patient Health Questionnaire
Multimedia Appendix 1
"Your Feedback Counts"- survey with logos included.
Multimedia Appendix 2
Full survey results.
Footnotes
Conflicts of Interest: None declared.
References
- 1.Barak A, Hen L, Boniel-Nissim M, Shapira N. A Comprehensive Review and a Meta-Analysis of the Effectiveness of Internet-Based Psychotherapeutic Interventions. Journal of Technology in Human Services. 2008 Jul 03;26(2-4):109–160. doi: 10.1080/15228830802094429. [DOI] [Google Scholar]
- 2.Reger MA, Gahm GA. A meta-analysis of the effects of internet- and computer-based cognitive-behavioral treatments for anxiety. J Clin Psychol. 2009 Jan;65(1):53–75. doi: 10.1002/jclp.20536. [DOI] [PubMed] [Google Scholar]
- 3.Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012 Jun;32(4):329–42. doi: 10.1016/j.cpr.2012.02.004. [DOI] [PubMed] [Google Scholar]
- 4.Ybarra ML, Eaton WW. Internet-based mental health interventions. Ment Health Serv Res. 2005 Jun;7(2):75–87. doi: 10.1007/s11020-005-3779-8. [DOI] [PubMed] [Google Scholar]
- 5.Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther. 2009;38(4):196–205. doi: 10.1080/16506070903318960. [DOI] [PubMed] [Google Scholar]
- 6.Cuijpers P, Marks IM, van Straten A, Cavanagh K, Gega L, Andersson G. Computer-aided psychotherapy for anxiety disorders: a meta-analytic review. Cogn Behav Ther. 2009;38(2):66–82. doi: 10.1080/16506070802694776. [DOI] [PubMed] [Google Scholar]
- 7.Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatry. 2002 Jun;159(6):909–16. doi: 10.1176/appi.ajp.159.6.909. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Cartreine JA, Ahern DK, Locke SE. A roadmap to computer-based psychotherapy in the United States. Harv Rev Psychiatry. 2010;18(2):80–95. doi: 10.3109/10673221003707702. [DOI] [PubMed] [Google Scholar]
- 9.Portnoy DB, Scott-Sheldon LA, Johnson BT, Carey MP. Computer-delivered interventions for health promotion and behavioral risk reduction: a meta-analysis of 75 randomized controlled trials, 1988-2007. Prev Med. 2008 Jul;47(1):3–16. doi: 10.1016/j.ypmed.2008.02.014. http://europepmc.org/abstract/MED/18403003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res. 2006;8(3):e20; author reply e21. doi: 10.2196/jmir.8.3.e20. http://www.jmir.org/2006/3/e20/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Eysenbach G. The law of attrition. J Med Internet Res. 2005;7(1):e11. doi: 10.2196/jmir.7.1.e11. http://www.jmir.org/2005/1/e11/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Murray E, White IR, Varagunam M, Godfrey C, Khadjesari Z, McCambridge J. Attrition revisited: adherence and retention in a web-based alcohol trial. J Med Internet Res. 2013;15(8):e162. doi: 10.2196/jmir.2336. http://www.jmir.org/2013/8/e162/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Waller R, Gilbody S. Barriers to the uptake of computerized cognitive behavioural therapy: a systematic review of the quantitative and qualitative evidence. Psychol Med. 2009 May;39(5):705–12. doi: 10.1017/S0033291708004224. [DOI] [PubMed] [Google Scholar]
- 14.Hutchinson A, Lovell A. Participatory action research: moving beyond the mental health 'service user' identity. J Psychiatr Ment Health Nurs. 2013 Sep;20(7):641–9. doi: 10.1111/jpm.12001. [DOI] [PubMed] [Google Scholar]
- 15.van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, Seydel ER. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res. 2011;13(4):e111. doi: 10.2196/jmir.1672. http://www.jmir.org/2011/4/e111/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Eysenbach G. Medicine 2.0: social networking, collaboration, participation, apomediation, and openness. J Med Internet Res. 2008;10(3):e22. doi: 10.2196/jmir.1030. http://www.jmir.org/2008/3/e22/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Salvendy G. Handbook of Human Factors and Ergonomics. United States: Wiley; 2012. [Google Scholar]
- 18.National Institute of Mental Health Suicide in the US. [2014-01-10]. http://www.nimh.nih.gov/health/publications/suicide-in-the-us-statistics-and-prevention/index.shtml.
- 19.Linehan MM, Comtois KA, Murray AM, Brown MZ, Gallop RJ, Heard HL, Korslund KE, Tutek DA, Reynolds SK, Lindenboim N. Two-year randomized controlled trial and follow-up of dialectical behavior therapy vs therapy by experts for suicidal behaviors and borderline personality disorder. Arch Gen Psychiatry. 2006 Jul;63(7):757–66. doi: 10.1001/archpsyc.63.7.757. [DOI] [PubMed] [Google Scholar]
- 20.Williams JM, Swales M. The use of mindfulness-based approaches for suicidal patients. Arch Suicide Res. 2004;8(4):315–29. doi: 10.1080/13811110490476671. [DOI] [PubMed] [Google Scholar]
- 21.Townsend E, Hawton K, Altman DG, Arensman E, Gunnell D, Hazell P, House A, Van Heeringen K. The efficacy of problem-solving treatments after deliberate self-harm: meta-analysis of randomized controlled trials with respect to depression, hopelessness and improvement in problems. Psychol Med. 2001 Aug;31(6):979–88. doi: 10.1017/s0033291701004238. [DOI] [PubMed] [Google Scholar]
- 22.van Spijker BA, Majo MC, Smit F, van Straten A, Kerkhof AJ. Reducing suicidal ideation: cost-effectiveness analysis of a randomized controlled trial of unguided web-based self-help. J Med Internet Res. 2012;14(5):e141. doi: 10.2196/jmir.1966. http://www.jmir.org/2012/5/e141/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.van Spijker BA, van Straten A, Kerkhof AJ. The effectiveness of a web-based self-help intervention to reduce suicidal thoughts: a randomized controlled trial. Trials. 2010;11:25. doi: 10.1186/1745-6215-11-25. http://www.trialsjournal.com/content/11//25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001 Sep;16(9):606–13. doi: 10.1046/j.1525-1497.2001.016009606.x. http://europepmc.org/abstract/MED/11556941. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Simon GE, Rutter CM, Peterson D, Oliver M, Whiteside U, Operskalski B, Ludman EJ. Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv. 2013 Dec 1;64(12):1195–202. doi: 10.1176/appi.ps.201200587. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.DatStat Illume 5.1.1. 1 ed: DatStat; 2012. [2014-01-28]. http://www.datstat.com/ [Google Scholar]
- 27.IBM . SPSS 16.0. IBM; 2007. [2014-01-28]. http://www-01.ibm.com/software/analytics/spss/ [Google Scholar]
- 28.Moskos MA, Olson L, Halbern SR, Gray D. Utah youth suicide study: barriers to mental health treatment for adolescents. Suicide Life Threat Behav. 2007 Apr;37(2):179–86. doi: 10.1521/suli.2007.37.2.179. [DOI] [PubMed] [Google Scholar]