Abstract
Background
Homeless individuals with mental illness are challenging to recruit and retain in longitudinal research studies. The present study uses information from the Vancouver site of a Canadian multi-city longitudinal randomized controlled trial on housing first interventions for homeless individuals. We were able to recruit 500 participants and retain large number of homeless individuals with mental illness; 92% of the participants completed the 6-month follow up interview, 84% the 24-month follow up, while 80% completed all follow-up visits of the study.
Purpose
In this article, we describe the strategies and practices that we considered as critical for successful recruitment and retention or participants in the study.
Methods
We discuss issues pertaining to research staff hiring and training, involvement of peers, relationship building with research participants, and the use of technology and social media, and managing challenging situations in the context of recruitment and retention of marginalized individuals.
Conclusions
Recruitment and retention of homeless participant with mental illness in longitudinal studies is feasible. It requires flexible, unconventional and culturally competent strategies. Longitudinal research projects with vulnerable and hidden populations may benefit from extensive outreach work and collaborative approaches that are based on attitudes of mutual respect, contextual knowledge and trust.
Keywords: Housing first intervention, Homelessness, Mental illness, Recruitment, Retention, Longitudinal study
1. Introduction
Homelessness has become one of the most pressing and complex social, health and political issues in many countries. Studies have consistently demonstrated particularly high prevalence rates of substance use and mental disorders among homeless populations in many Western countries including Canada [1], [2], the US [3], [4], [5], Australia [6], and Europe [7], [8], [9].
Research on how to address this important issue has increased over the past decades but is still fraught with a number of challenges. For example, recruitment can be difficult because potential participants may be withdrawn, socially alienated, distrustful of research projects, or absorbed with the demands of their current challenges. Researchers in this field need to develop effective outreach strategies in order to obtain adequately sized and representative samples [10]. Another issue common to longitudinal studies is participant attrition. Attrition can introduce bias if participants lost to follow-up differ from those retained in the study, particularly if the characteristics distinguishing completers versus non-completers are associated with the outcome measures [11], [12]. Factors such as residential instability, involvement in the justice system, cognitive impairment, and high rates of substance use makes retention of homeless individuals in longitudinal studies particularly challenging.
In recent years, researchers have identified a number of effective strategies to recruit and retain marginalized and hard-to-reach populations in longitudinal studies [13], [14], [15], [16]. Homeless individuals with mental illnesses overlap with several other at-risk groups, e.g., populations with high rates of injection drug use, HIV, and members of ethnic minority groups (e.g., [17], [18]). In 1996, Hough and colleagues [10] summarized successful strategies for the recruitment and retention of homeless mentally ill participants. However, the use of recently developed technologies such as e-mail, text messaging, and social media has received little attention.
The current paper, which was written by both researchers and field interviewers, describes the strategies and practices that we considered as critical for successfully recruiting and retaining homeless participants with mental illness for the Vancouver site of the At Home/Chez Soi Study, a Canadian housing intervention study. Furthermore, we will discuss a number of barriers and challenges faced in recruiting and retaining such participants. The paper may offer direction, recommendations, and inspiration to other researchers who are planning to conduct longitudinal studies with this population.
2. Summary of the At Home/Chez Soi study methodology
The At Home/Chez Soi Study is a randomized-controlled trial (RCT) examining the effects of different housing interventions for homeless adults with mental illness in five Canadian cities. Study design and sample size were determined by the At Home/Chez Soi National Research Team, which monitored activities at the different study sites. Details related to the RCT protocol and study procedures are described in detail elsewhere for both the multi-centre elements [19] and the Vancouver design [20].
Eligibility criteria in Vancouver included legal adult status (19 years and older), current mental disorder and being either absolutely homeless or precariously housed. Current mental disorder was determined using the Mini-International Neuropsychiatric Interview (MINI) [21] for the assessment of the following diagnoses based on criteria of the Diagnostic and Statistical Manual of Mental Disorders: major depressive episode, manic or hypomanic episode, post-traumatic stress disorder, mood disorder with psychotic features, and psychotic disorder. Where possible, mental disorder status was corroborated by physician diagnosis.
“Absolute homelessness” was defined as living on the streets or in an emergency shelter for at least the past seven nights with little likelihood of obtaining secure accommodation in the upcoming month. “Precariously housed” was defined as living in a rooming house, hotel, or other transitional housing; in addition, these individuals must have experienced at least two episodes of absolute homelessness or a single episode lasting at least four weeks during the past year.
In Vancouver, participants were recruited through referral from over 40 agencies available to homeless adults; the majority was recruited from shelters, drop-in centers, outreach teams, hospitals, community mental health teams, and criminal justice programs. We specifically targeted organizations that serve women, youth, aboriginal peoples, and gay/lesbian/transgender individuals in order to obtain a sample as diverse and representative as possible. All participants met face-to-face with a trained interviewer who confirmed study eligibility, explained procedures, and obtained informed consent. An honorarium of $5 was provided for the screening process. Institutional ethics board approval was obtained through Simon Fraser University and the University of British Columbia.
Eligible individuals completed a comprehensive baseline assessment including a variety of psychiatric, psychosocial, and health related measures and an overview of all measures used is published in detail elsewhere (see 19 and 20). Based on the results of the baseline assessment, participants' level of need was categorized as “high” or “moderate” by the following computer algorithm: [participants'] needs were categorized as “high” if they 1) met criteria for a current Psychotic Disorder or Manic/Hypomanic Episode on the MINI, and 2) received a total score of 62 or lower on the Multnomah Community Ability Scale (MCAS), a scale designed to measure daily functioning of individuals with severe mental illness [22], [23], and 3) reported at least one of the following: (a) two or more hospitalizations for mental illness during the last five years, (b) met criteria for current substance abuse or dependence on the MINI, or (c) reported arrest or incarceration in the past six months. All other study participants were categorized as having “moderate” needs. This categorization was grounded on clinical and functional considerations regarding individuals living with psychiatric illness: The presence or psychotic disorders and/or manic/hypomanic episodes, the presence of a concurrent substance use disorder, and repeated hospitalization due to mental illness were considered as indicators for high severity and chronicity of the participants' mental condition. Furthermore, a low score on the MCAS, the Multnomah Community Ability Scale, which measures daily functioning in the community context, as well as recent incarceration were considered as indicators of impaired or low psychosocial functioning.
It was assumed that a high severity and chronicity of the current mental illness and low psychosocial functioning would require (‘need’) more intense and/or more frequent psychosocial support, services and care.
Following the level of need categorization, participants were randomly assigned to receive either a Housing First intervention (i.e., an accommodation plus psychosocial support) or to usual care (i.e., participants received no active intervention through the study but were permitted to access any available form of care and housing by themselves). Participant flow through eligibility screening, need level assessment, and allocation to study arm is visualized in Fig. 1.
Fig. 1.
Participant flow through eligibility screening need level assessment, and allocation to study arm.
All participants were followed for 24 months through four comprehensive and four brief follow-up interviews conducted at three-month intervals. Recruitment and randomization were conducted between October 2009 and June 2011. Because the final follow up assessment is still in progress, we used only data until the 21-month follow up assessment for the current study. The last 24-month follow up interview was conducted on March 25, 2013.
3. Recruitment and retention statistics
Of all five participating sites, Vancouver was a successful study site with timely recruitment (of N = 500 participants) and high attrition [27].
Table 1 describes the retention rates for each follow up at time point 6, 12, 18, and 24 months, where the full interview batteries were administered. The completion rate for each visit ranged between 83.7% and 92.4%. 79.6% attended all four visits, while only 4.6% of the participants did not attend any follow-up visit. Table 2 compares participants who had no follow up visit with participants who had at least one follow up visit on socio-demographic and clinical variables. Table 3A, Table 3B, Table 3C, Table 3D are showing comparisons of socio-demographic characteristics by 6, 12, 18, and 24-month follow up completion status respectively.
Table 1.
Follow-up completion rate for study participants over 24 months by need level.
| High Need (n = 297) N (%) |
Moderate Need (n = 200) N (%) |
Total (n = 497) N (%) |
|
|---|---|---|---|
| Follow up visit completed | |||
| 6 months | 281 (94.6) | 178 (89.0) | 459 (92.4) |
| 12 months | 274 (92.3) | 177 (88.5) | 451 (90.7) |
| 18 months | 257 (86.5) | 173 (86.5) | 430 (86.5) |
| 24 months | 250 (84.2) | 166 (83.0) | 416 (83.7) |
| Number of completed FU visits | |||
| All four visits | 240 (80.7) | 155 (77.5) | 395 (79.6) |
| Only three visits | 18 (6.1) | 15 (7.5) | 33 (6.6) |
| Only two visits | 19 (6.4) | 12 (6.0) | 31 (6.2) |
| Only one visit | 10 (3.4) | 5 (2.5) | 15 (3.0) |
| No follow up visit | 10 (3.4) | 13 (6.5) | 23 (4.6) |
| Reason for no (0) follow up visit | |||
| Death | 5 (1.7) | 4 (2.0) | 9 (1.8)a |
| Withdrew | 0 (0.0) | 2 (1.0) | 2 (0.4) |
| No contact | 5 (1.7) | 7 (3.5) | 12 (2.4) |
All 9 participants died within 7 months of enrolment. In addition, 20 participants died during the follow up period, but completed at least one follow up visit before death.
Table 2.
Comparisons of Socio-demographic and mental disorder related characteristics by follow up completion status.
| Variable | High Need (n = 297, 10 vs. 287) |
Moderate Need (n = 200, 13 vs. 187) |
||||
|---|---|---|---|---|---|---|
| No FU visit n (%)/mean (sd) |
At least one FU visit n (%)/mean (sd) |
P valuea | No FU visit n (%)/mean (sd) | At least one FU visit n (%)/mean (sd) |
P value | |
| Sociodemographics | ||||||
| Age (years): mean (sd) | 35.4 (14.8) | 39.8 (11.0) | 0.224 | 46.8 (9.5) | 42.3 (10.5) | 0.141 |
| Male sex | 9 (90) | 209 (74) | 0.462 | 10 (77) | 131 (70) | 0.760 |
| Ethnicity | ||||||
| Aboriginals | 2 (20) | 42 (15) | 0.817 | 3 (23) | 30 (16) | 0.682 |
| Caucasian | 6 (60) | 164 (57) | 6 (46) | 104 (56) | ||
| Mixed/Other | 2 (20) | 81 (28) | 4 (31) | 53 (23) | ||
| Incomplete high school | 7 (66) | 172 (61) | 0.745 | 5 (38) | 96 (51) | 0.405 |
| Single/Never married | 8 (89) | 206 (72) | 0.453 | 7 (54) | 122 (65) | 0.550 |
| Have children (under18) | 0 (0) | 69 (25) | 0.120 | 5 (42) | 48 (26) | 0.310 |
| Currently employed | 0 (0) | 10 (3) | 1.000 | 1 (8) | 7 (4) | 0.279 |
| Homelessness | ||||||
| Absolutely homeless | 10 (100) | 222 (77) | 0.125 | 8 (62) | 148 (79) | 0.165 |
| Lifetime duration of homelessness (months) | ||||||
| mean (sd) | 62.2 (60.1) | 62.0 (67.3) | 0.993 | 35.4 (68.6) | 61.5 (69.1) | 0.273 |
| DSM-IV diagnosis | ||||||
| Major Depressive Episode | 3 (30) | 92 (32) | 1.000 | 6 (46) | 98 (52) | 0.777 |
| Manic or Hypomanic Episode | 2 (20) | 66 (23) | 1.000 | 1 (8) | 28 (15) | 0.697 |
| Post-Traumatic Stress Disorder | 1 (10) | 62 (22) | 0.694 | 4 (31) | 62 (33) | 1.000 |
| Panic Disorder | 2 (20) | 57 (20) | 1.000 | 1 (8) | 44 (23) | 0.305 |
| Psychotic Disorder | 8 (80) | 203 (71) | 0.729 | 5 (38) | 47 (25) | 0.329 |
| Alcohol dependence | 2 (20) | 70 (24) | 1.000 | 3 (23) | 46 (25) | 1.000 |
| Substance dependence | 4 (40) | 179 (62) | 0.191 | 6 (46) | 99 (53) | 0.776 |
| Suicidality (moderate or high) | 2 (20) | 91 (32) | 0.730 | 3 (23) | 72 (38) | 0.378 |
| Two or more mental disorders | 5 (50) | 143 (50) | 1.000 | 5 (38) | 87 (46) | 0.775 |
p values (for both high and moderate need comparisons) were based on independent sample t-test with equal or unequal variance (for continuous variables) and Pearson chi-square test or Fisher's exact test (for categorical variables) as appropriate.
Table 3A.
Comparisons of Socio-demographic characteristics by 6-month follow up completion status.
| High need (n = 16 vs. n = 281) |
Moderate need (n = 22 vs. n = 178) |
||||||
|---|---|---|---|---|---|---|---|
| 6-month visit-no n (%)/mean (sd) |
6-month visit-yes n (%)/mean (sd) |
P valuea | 6-month visit-no n (%)/mean (sd) |
6-month visit-yes n (%)/mean (sd) |
P value | ||
| Age (years) | mean (sd) | 36.4 (13.5) | 39.9 (11.0) | 0.233 | 46.8 (9.0) | 42.1 (10.6) | 0.044 |
| Gender | Female | 1 (6) | 75 (27) | 0.079 | 4 (18) | 54 (31) | 0.230 |
| Male | 15 (94) | 203 (73) | 18 (82) | 123 (69) | |||
| Ethnicity | Aboriginals | 4 (25) | 40 (14) | 0.507 | 4 (18) | 29 (16) | 0.862 |
| Caucasian | 8 (50) | 162 (58) | 11 (50) | 99 (56) | |||
| Mixed/Other | 4 (25) | 79 (28) | 7 (32) | 50 (28) | |||
| Education | High School or Higher | 4 (25) | 111 (40) | 0.234 | 12 (55) | 87 (49) | 0.616 |
| Less than high school | 12 (75) | 167 (60) | 10 (45) | 91 (51) | |||
| Marital status | Single (never married) | 13 (87) | 201 (72) | 0.369 | 14 (64) | 115 (65) | 0.928 |
| Other | 2 (13) | 77 (28) | 8 (36) | 63 (35) | |||
| Have children (under18) | No | 15 (100) | 204 (75) | 0.026 | 15 (71) | 130 (73) | 0.843 |
| Yes | 0 (0) | 69 (25) | 6 (29) | 47 (27) | |||
| Employment | Unemployed/other | 16 (100) | 269 (96) | 1.000 | 21 (95) | 171 (96) | 1.000 |
| Employed | 0 (0) | 10 (4) | 1 (5) | 7 (4) | |||
| Housing status | Absolutely homeless | 15 (94) | 217 (77) | 0.209 | 16 (73) | 140 (79) | 0.586 |
| Precariously housed | 1 (6) | 64 (23) | 6 (27) | 38 (21) | |||
| Lifetime homelessness (months)) | mean (sd) | 76.9 (100.5) | 61.1 (64.7) | 0.360 | 45.9 (66.1) | 58.9 (76.0) | 0.400 |
Bold values signify p < 0.05.
p values (for both high and moderate need comparisons) were based on independent sample t-test with equal or unequal variance (for continuous variables) and Pearson chi-square test or Fisher's exact test (for categorical variables) as appropriate.
Table 3B.
Comparisons of Socio-demographic characteristics by 12-month follow up completion status.
| High need (n = 23 vs. n = 274) |
Moderate need (n = 23 vs. n = 177) |
||||||
|---|---|---|---|---|---|---|---|
| 12-month visit-no n (%)/mean (sd) |
12-month visit-yes n (%)/mean (sd) |
P valuea | 12-month visit-no n (%)/mean (sd) |
12-month visit-yes n (%)/mean (sd) |
P value | ||
| Age (years) | mean (sd) | 38.3 (13.8) | 39.8 (10.9) | 0.614 | 43.5 (10.8) | 42.5 (10.5) | 0.649 |
| Gender | Female | 3 (13) | 73 (27) | 0.144 | 6 (26) | 52 (30) | 0.731 |
| Male | 20 (87) | 198 (73) | 17 (74) | 124 (70) | |||
| Ethnicity | Aboriginals | 2 (9) | 42 (15) | 0.649 | 5 (22) | 28 (16) | 0.753 |
| Caucasian | 13 (57) | 157 (57) | 12 (52) | 98 (55) | |||
| Mixed/Other | 8 (35) | 75 (27) | 6 (26) | 51 (29) | |||
| Education | High School or Higher | 9 (39) | 106 (39) | 1.000 | 13 (57) | 86 (49) | 0.474 |
| Less than high school | 14 (61) | 165 (61) | 10 (43) | 91 (51) | |||
| Marital status | Single (never married) | 19 (86) | 195 (72) | 0.143 | 13 (57) | 116 (66) | 0.395 |
| Other | 3 (14) | 76 (28) | 10 (43) | 61 (34) | |||
| Have children (under18) | No | 21 (95) | 198 (74) | 0.026 | 14 (64) | 131 (74) | 0.281 |
| Yes | 1 (5) | 68 (26) | 8 (36) | 45 (26) | |||
| Employment | Unemployed/other | 23 (100) | 262 (96) | 1.000 | 22 (96) | 170 (96) | 1.000 |
| Employed | 0 (0) | 10 (4) | 1 (4) | 7 (4) | |||
| Housing status | Absolutely homeless | 19 (83) | 213 (78) | 0.587 | 16 (70) | 140 (79) | 0.299 |
| Precariously housed | 4 (17) | 61 (22) | 7 (30) | 37 (21) | |||
| Lifetime homelessness (months) | mean (sd) | 75.9 (90.7) | 60.8 (64.6) | 0.442 | 38.5 (57.5) | 59.9 (76.7) | 0.198 |
Bold value signifies p < 0.05.
p values (for both high and moderate need comparisons) were based on independent sample t-test with equal or unequal variance (for continuous variables) and Pearson chi-square test or Fisher's exact test (for categorical variables) as appropriate.
Table 3C.
Comparisons of Socio-demographic characteristics by 18-month follow up completion status.
| High need (n = 40 vs. n = 257) |
Moderate need (n = 27 vs. n = 173) |
||||||
|---|---|---|---|---|---|---|---|
| 18-month visit-no n (%)/mean (sd) |
18-month visit-yes n (%)/mean (sd) |
P valuea | 18-month visit-no n (%)/mean (sd) |
18-month visit-yes n (%)/mean (sd) |
P value | ||
| Age (years) | mean (sd) | 40.9 (14.6) | 39.5 (10.6) | 0.557 | 42.2 (11.8) | 42.6 (10.3) | 0.851 |
| Gender | Female | 6 (15) | 70 (28) | 0.092 | 5 (19) | 53 (31) | 0.191 |
| Male | 34 (85) | 184 (72) | 22 (81) | 119 (69) | |||
| Ethnicity | Aboriginals | 6 (15) | 38 (15) | 0.946 | 4 (15) | 29 (17) | 1.000 |
| Caucasian | 22 (55) | 148 (58) | 15 (56) | 95 (55) | |||
| Mixed/Other | 12 (30) | 71 (28) | 8 (30) | 49 (28) | |||
| Education | High School or Higher | 16 (40) | 99 (39) | 0.902 | 17 (63) | 82 (47) | 0.132 |
| Less than high school | 24 (60) | 155 (61) | 10 (37) | 91 (53) | |||
| Marital status | Single (never married) | 28 (72) | 186 (73) | 0.851 | 17 (63) | 112 (65) | 0.858 |
| Other | 11 (28) | 68 (27) | 10 (37) | 61 (35) | |||
| Have children (under18) | No | 32 (86) | 187 (75) | 0.111 | 19 (73) | 126 (73) | 0.985 |
| Yes | 5 (14) | 64 (25) | 7 (27) | 46 (27) | |||
| Employment | Unemployed/other | 39 (98) | 246 (96) | 1.000 | 26 (96) | 166 (96) | 1.000 |
| Employed | 1 (3) | 9 (4) | 1 (4) | 7 (4) | |||
| Housing status | Absolutely homeless | 33 (83) | 199 (77) | 0.471 | 17 (63) | 139 (80) | 0.043 |
| Precariously housed | 7 (18) | 58 (23) | 10 (37) | 34 (20) | |||
| Lifetime homelessness (months) | mean (sd) | 60.0 (78.1) | 62.3 (65.2) | 0.845 | 59.4 (141.4) | 57.1 (58.9) | 0.884 |
Bold value signifies p < 0.05.
p values (for both high and moderate need comparisons) were based on independent sample t-test with equal or unequal variance (for continuous variables) and Pearson chi-square test or Fisher's exact test (for categorical variables) as appropriate.
Table 3D.
Comparisons of Socio-demographic characteristics by 24-month follow up completion status.
| High need (n = 47 vs. n = 250) |
Moderate need (n = 34 vs. n = 166) |
||||||
|---|---|---|---|---|---|---|---|
| 24-month visit-no n (%)/mean (sd) |
24-month visit-yes n (%)/mean (sd) |
P valuea | 24-month visit-no n (%)/mean (sd) |
24-month visit-yes n (%)/mean (sd) |
P value | ||
| Age (years) | Mean (sd) | 39.7 (14.5) | 39.7 (10.5) | 0.967 | 41.6 (12.2) | 42.8 (10.2) | 0.560 |
| Gender | Female | 10 (21) | 66 (27) | 0.435 | 7 (21) | 51 (31) | 0.228 |
| Male | 37 (79) | 181 (73) | 27 (79) | 114 (69) | |||
| Ethnicity | Aboriginals | 9 (19) | 35 (14) | 0.869 | 5 (15) | 28 (17) | 0.850 |
| Caucasian | 25 (53) | 145 (58) | 18 (53) | 92 (55) | |||
| Mixed/Other | 13 (28) | 70 (28) | 11 (32) | 46 (28) | |||
| Education | High School or Higher | 22 (47) | 93 (38) | 0.238 | 19 (56) | 80 (48) | 0.414 |
| Less than high school | 25 (53) | 154 (62) | 15 (44) | 86 (52) | |||
| Marital status | Single (never married) | 32 (70) | 182 (74) | 0.563 | 22 (65) | 107 (64) | 0.978 |
| Other | 14 (30) | 65 (26) | 12 (35) | 59 (36) | |||
| Have children (under18) | No | 36 (82) | 183 (75) | 0.329 | 23 (70) | 122 (74) | 0.615 |
| Yes | 8 (18) | 61 (25) | 10 (30) | 43 (26) | |||
| Employment | Unemployed/other | 46 (98) | 239 (96) | 1.00 | 32 (94) | 160 (96) | 0.626 |
| Employed | 1 (2) | 9 (4) | 2 (6) | 6 (4) | |||
| Housing status | Absolutely homeless | 37 (79) | 195 (78) | 0.912 | 23 (68) | 133 (80) | 0.110 |
| Precariously housed | 10 (21) | 55 (22) | 11 (32) | 33 (20) | |||
| Lifetime homelessness (months) | Mean (sd) | 46.2 (41.9) | 65.0 (70.4) | 0.016 | 61.3 (127.1) | 56.6 (59.5) | 0.740 |
Bold value signifies p < 0.05.
p values (for both high and moderate need comparisons) were based on independent sample t-test with equal or unequal variance (for continuous variables) and Pearson chi-square test or Fisher's exact test (for categorical variables) as appropriate.
Student's t-test and Fishers exact test were conducted to compare continuous and categorical variables between groups respectively. We found no significant differences with respect to socio-demographic and clinical variables, except housing status (absolutely homelessness versus precariously housed) in the 18 months follow up interview and lifetime duration of homelessness in the 24 months follow up. The following sections highlight the specific strategies our team has used to complete timely recruitment and maintain high retention rates.
4. Description of recruitment and retention strategies
4.1. Selection of and work with the field interviewers
We hired interviewers who had extensive experience working with homeless individuals and community-based organizations, and who possessed strong interpersonal skills. Given that many baseline and follow-up interviews were conducted in the community, we hired staff who were self-motivated and felt comfortable working alone and in an unstructured and often chaotic work environment. Interviewers were required to have some existing knowledge of basic research concepts including confidentiality, bias, and standardized administration of research measures. We attempted to hire individuals, who were familiar with the culture, social justice and critical thinking to articulate the needs and experiences of the participants' population. For our research work we specifically hired field interviewers who were familiar with Vancouver's homeless population, knowledgeable regarding existing services for homeless individuals, and comfortable walking through poor neighborhoods and interacting with their residents. Importantly, staff were required to have the ability to assess and manage various issues such as agitation and aggression, psychotic symptoms, intoxication, suicidal ideation, and self-harm. We hired people who had calm yet assertive manners, and were non-judgmental and supportive towards even the most marginalized individuals.
Furthermore, many homeless people have experiences such as childhood abuse or prostitution [24], [25] that are difficult, painful, or shameful to recall, and we invested effort in finding interviewers who were able to address such topics with respect and sensitivity. Retrospectively turnover rate amongst the field interviewers was low (three research assistant was hired after the training as mentioned above; one research assistant was hired to replace a research assistant who left the team to move to another city) which was facilitated by giving staff freedom, being flexible with part-time and full-time needs and providing an end-of-project bonus.
4.2. Peer interviewers
One of the objectives of the At Home Study was to engage peers who have experiences living with mental illness and/or homelessness as members of various teams within the project, including both service provision and research. Two peers were recruited and hired as field research interviewers, with the intention that they would administer research interviews, but more importantly to emotionally support the participants during the interview process. Although there was genuine commitment to include peer interviewers in the field research process, this remained challenging throughout the project. Time-constraints and lack of resources were the biggest obstacles for involving the peer interviewers in all aspects of the daily field research work. Peer interviewers were hired on an hourly contract-basis, which limited their physical presence in the field research office and as a team member. Furthermore, the support and collaboration with peer interviewers was not initially considered in the time allocation and workload of the field team leaders. However, peer interviewers proved to be invaluable in the qualitative interviews: building rapport, creating a safe atmosphere, and validating the participants' emotions that emerged during the interview process.
4.3. Staff training
-
•Standardized yet practical administration of measures/questionnaires:
-
oIntroduction of the interview and its different components; explaining – in detail yet in plain language – participants' role and rights in this process and how participants can execute these rights (e.g. whom to phone, where to go if they would like to discuss the study, withdraw from the study, or learn more about the study)
-
oTraining around potential sources of bias and how to clarify ambiguous responses
-
oHow to introduce each questionnaire (as an interview session comprised of several questionnaires that covered a variety of topics, and how to probe if a participant's reply seems vague or unreliable)
-
oHow to redirect participants' attention to the interview content when they get distracted and lose focus
-
oHow to assess if a participant is not unable to continue with a measure/interview for several reasons (e.g. due to high levels of distress, or unreliable responses and how to pause or end the session)
-
oHow to introduce measures that may be emotionally challenging (e.g. around trauma histories, or child welfare system involvement), and/or judgmental (e.g. around substance use, incarceration, psychotic symptoms).
-
oHow to debrief with participant who experienced emotional distress as a result of the interview
-
o
-
•
Extensive role-playing and debriefing
-
•
Entering data onto a secure site via laptop computer
-
•Clinical Judgment:
-
oSeveral instruments, such as the MINI and the MCAS, required some degree of clinical judgment in administration and scoring. Interviewers were trained how to interpret common response styles such as over- and under-reporting, and how to use probes in order to fully assess the presence of a symptom or behaviors. If participants did not verbally acknowledge the presence of symptoms, but behavioral symptoms such as disorganized behavior or psychotic ideation could be observed, the interviewers were instructed to code the item positively.
-
oRole-playing with each other and a professional patient actor, training videotapes, and training manuals were all used to practice the MCAS and MINI application and exercising clinical judgment. We found it specifically important to repeat practice with different role-play partners and watch several training videos available.
-
oWithout following a rigid interpreter-reliability process (which was not part of the study protocol) we included extensive group discussion about different ratings amongst interviewer when using rating scales that relied on interviewers' judgment. This enhanced clinical understanding, clinical decision-making and awareness for nuances in selecting a certain rating using a certain measure.
-
o
-
•
Staff and client safety:
-
•
A safety protocol was developed with key steps in the event of aggressive, violent, and suicidal behaviors, severe intoxication, etc.
-
•
Furthermore, interviewers were encouraged to set firm boundaries with participants and to break confidentiality if needed (e.g. imminent risk of harm to self or others).
In addition to the training week, staff members also attended a three-day cross-site training event prior to recruitment, which provided a more general overview of the study as well as - initial and ongoing - training on specific measures, interviewing techniques, and the computer system that was used for data entry and storage, scheduling, and randomization.
Finally, the consent form and specific measures that had been adapted for use with homeless populations were pre-tested with a sample of 12 individuals from local shelters and drop-in centers [26]. This allowed interviewers to gain more familiarity and practice with the materials and trial certain approaches how to interact and administer interviews in realist-informed settings.
Three interviewers were hired after the study commenced and, therefore, did not participate in the initial training. New staff were paired with experienced interviewers and observed several interviews before conducting individual modules and then entire interviews with an observer before doing an interview alone. As the study progressed, informal training was conducted in team meetings where pertinent issues such as clinical judgment and challenging participants were addressed. One of two team leaders, both with extensive professional mental health training, was always available by telephone to discuss clinical questions and decision-making rules that arose during an interview. Ongoing training and support via webinar, teleconference and site visits were also provided by the National Cross-Site Study Team.
4.4. Recruitment of participants
Establishing a trusting and collaborative working relationship with existing service providers was critical to successful recruitment. Although the study generated considerable media coverage and public attention, we found that many service providers did not have a clear understanding of the study objectives and the target population. These misunderstandings, among others, created some skepticism and reservations toward the project among some service providers. Therefore, members of the Field Research Team and the broader Project Team visited numerous service agencies to communicate the study's objectives, procedures, interventions, inclusion criteria, and to answer questions and respond to feedback.
The majority of participants were recruited via referrals from existing services. One interviewer who had extensive experience working in the community service sector was designated as the primary contact person and was responsible for answering questions from referral agents, providing telephone information and screenings, and scheduling each initial interview. This individual was also the primary contact for participants who dropped into the Field Office throughout the study. The Field Office itself was not located in the inner city but easily accessible for the participants. The importance of continuous relationship building with the community service sector as well as with participants cannot be over-emphasized.
4.5. Data collection
The initial screening interview included the collection of specific personal information to facilitate locating the individual in the future. Personal information included nick-names or aliases; services, organizations, and other places frequented on a regular basis (e.g., shops, supervised injection site, street corners, and bottle depots), phone numbers and email addresses, the names and contact information of friends or family, and a brief physical description of the person. Personal information including current address (housing) was updated with each contact. A photograph of the participant was also taken with their consent.
Given the nature of the At Home/Chez Soi project, participants in the intervention arms (High and Moderate Need) were housed through the At Home/Chez Soi project which was based on the ‘Housing First’ approach (in brief: providing fast access to non-conditional housing, not related on certain pre-requisites or behaviors e.g. abstinence from substances), therefore if participants needed to move due to any reason they would be provided with another housing option by the project. Moves and transitions were recorded as part of the project; but will not be discussed in detailed here, as participants' (mostly planned) house moves in the intervention arms did not pose any difficulties to successful follow-up.
For the administration of all measures, including tracking the due dates for follow up interviews and tracking attempts to make contact with participants, we used a database designed to support multiple staff members in conducting follow up. Any written information about participants was entered into the computer system which was password protected. Hand-written notes were shredded or secured in a locked filing cabinet. A log of tracking and contact attempts was created for each participant with date, time and staff initials. Tracking history notes included the contact method used (e.g. phone, letter), date and time of contact attempt, the result of the contact attempt, and suggestions to improve future contact strategies.
4.6. Retention of participants
Retention of participants was where the structure of the field research team and the attributes of the individual interviewers became critical to the success of the project. Given that many of the participants were recruited from community based organizations in Vancouver's Downtown Eastside, knowledge of the neighborhood and its multitude of services as well as a level of comfort in working within an often chaotic urban setting was paramount. Several of the interviewers had worked previously in Vancouver's supervised injection site, and knew many of the people in the study and a mutually trusting relationship with site staff. Furthermore, participants often recognized them, and reported feeling comfortable and accepted despite their lifestyle. Thus, we were able to reach even those participants whose movements and level of disconnect from other services would have made them challenging to retain. Often interviewers would attempt to make contact through several avenues to pass messages to difficult to reach participants, and “tracking notes” in the participant's profile page with suggestions to improve future efforts to make contact were continuously updated. Also included in these tracking notes were hints about how to make a connection in difficult cases, and avoid alienating or agitating participants. For example, we would highlight a female participant's request to only be contacted by female interviewers, or another participant's paranoia about being interviewed indoors.
In rare cases, in order to ensure interviewer safety, we would include suggestions to conduct interviews in a public arena or in a location where the participant felt at ease. In making an effort to be flexible, persistent and sensitive to the needs of our study subjects, we ensured that the participants felt comfortable, respected, and accommodated. By consciously avoiding creating a situation where a subject's difficult or aggressive behavior would have made them feel like they “burned a bridge”, that could undermine rapport and lead them to exclude themselves from further follow up, we took steps to safeguarded their continued involvement in the research. In some cases, the flexibility of research staff translated to travelling significant distances to the participant's neighborhoods or workplace, and working in the evening or early morning to ensure contact. Also critical to maintaining relationships in the community and with participants was a consistent effort to be punctual for appointments in institutions (e.g., hospital, residential treatment facilities, corrections facilities), and being respectful of the rules of different administrative environments.
In Vancouver, our intervention arms included services from several existing organizations, and relationship building with intervention staff was crucial to communication with participants. Letters and verbal messages regarding follow-up interviews were very often transmitted through case managers. To illustrate an example, at a congregate housing facility one research interviewer was appointed as the contact person for the majority of the follow-ups. The residents became accustomed to being tracked and interviewed at their home, and a consistent schedule for follow-up was established. The contact person built rapport with residents through being visible at meal times, in groups, or in the garden, and going for coffee or for a walk. The residents became aware of the schedule of the on-site interviewer, and frequently dropped in to the interview room to see when they were next due, or make contact. Communication with staff regarding upcoming interviews proved effective for those who were harder to reach. Over time, the ongoing consistent presence of the interviewer and the tight knit community created a culture of participants being proactive in scheduling their own interviews. The importance of this continuous contact and rapport with residents – but also the incentive paid to the participants - cannot be emphasized enough. In addition, building relationships with staff proved essential to effectively following those residents who had moved out for reasons ranging from incarceration to moving to their home communities, or for participants with issues of acute addiction, mental illness, or aggression. The interviewer would quickly learn from staff of any circumstances impeding follow-up, check in with case managers, and adjust plans to accommodate the participant. In many cases, when a resident had severed contact with the intervention team, the case manager and researcher would work collaboratively to track down the individual and re-establish contact. In several cases the ongoing engagement or re-engagement with the research team led to subsequent re-engagement with the clinical intervention teams and enabled them to work with the clients on their housing and health challenges.
That said, it was also crucial to maintain the distinction between research staff and intervention staff, and to uphold participant confidentiality at all times. This proved beneficial to participant retention in that the interviewer was perceived as being separate from systems of care that were not always perceived to be supportive. The perception of the interviewer as a “safe audience” also helped to ensure more honest and open feedback.
4.7. Use of technology in recruitment and retention of participants
In the ongoing process of maintaining contact and communicating upcoming interviews, we employed various technologies, sometimes several modes at once. Telephone, email, texting, social media (e.g. Facebook) and voice over IP services (e.g. Skype) were all employed to good effect, yet every effort was made to protect participants' confidentiality: as telephone contact numbers could very rapidly go out of service (e.g. phones being in and out of service depending on the available funds of the participants), and computers with internet service were available free of charge in a number of facilities and public institutions in the city, email was often the most effective method of maintaining contact. Texting proved to be the most reliable way to contact a participant whose cellphone plans offered free texts but charged by the minute for calls. In one case where a participant moved to Toronto and had no phone, his interviews were conducted via Skype. One individual, in his effort to get clean, purposefully disconnected his phone and neglected his email in an effort to avoid triggers and unhealthy contact with drug using acquaintances. The one mode of communication he maintained during this period was his social media account. In contacting him through this we experienced the unexpected benefit of enhancing rapport and trust, as he was then able to share pictures of his family, himself as an adolescent, and his estranged son with whom he was re-establishing contact. This kind of connection not only helped retain him in follow-up, but instilled a sense of rapport and trust that enabled him to broach difficult subjects that he would previously have declined to address. We were aware that contacting people through social media might interfere with confidentiality, so we did not contact participants through social media accounts on a large scale, but instead used this method as an individualized approach with one participant as per his wish. By doing so, we were very cautious in avoiding any communication around the terms ‘homelessness’ and/or ‘mental illness’, instead we would remind him of his ‘meeting’ or ‘interview’.
For the most difficult to reach individuals, we found that outreach, dropping in to homeless services, or stopping by parks or alleys that participants were known to frequent, knocking on doors of participant's rooms, and transmitting messages by word of mouth was the only hope of retention. Of particular help were the photos, which helped newer interviewers in identifying and tracking participants with whom they were not familiar and who had not been photographed at baseline. As another approach we put up posters in the areas where homeless people are known to be, with a generic reminder to call the study office and check if he/she is a participant of the At Home Study and if they are due for a follow-up appointment, as often times people might not remember that they are part of an ongoing research project that runs over several years.
4.8. Incentives
Probably the most effective tool for retaining participants' interest in the study was the use of cash incentives for continued contact. Participants were paid honoraria ranging from $20 to $40 for participating in the interviews, based on an average of $20 per hour, given the fact that some follow-up interviews took longer than others. Added incentives of being taken for coffee or lunch assisted greatly in creating an informal atmosphere, and were critical to engagement in some cases. We also offered bus tickets for those participants who were willing to travel to the field office, and light refreshments at the start of each interview to impart a sense of hospitality and appreciation.
5. Conclusion
The specific characteristics of homeless individuals necessitate the use of intensive and unconventional strategies to recruit and retain them in longitudinal scientific studies. Many of the procedures described in this paper appear quite individualized and time-consuming. Based on our experience, the intensity of the overall effort is more important than a specific procedure per se. Many study participants will be reached by the first attempt. However, researchers should be prepared to adopt multiple strategies of different intensity to retain individuals in the study.
We believe that one of the crucial factors for creating and maintaining ongoing relationships with our study participants was our staff's contextual knowledge of the environment, cultural norms, conditions, and circumstances of the population and Vancouver's Downtown Eastside. The statistical data describing our retention rates demonstrate that these culturally sensitive procedures have been highly successful. But besides the statistics, we also felt that the behaviors of our study participants indicated the appropriateness of our strategy. For example, evidence of sex work, criminal behavior, drug use or drug dealing could certainly have been an impediment to access, especially when inviting a researcher into personal space (apartment or hotel room), but because of the experience of many of the interviewers in the neighborhood and the rapport established over years of contact, they were invited into rooms where a stranger would have been turned away. Furthermore, we did not have a single incident in the Field Research Office over the course of the study; neither assaults, nor theft, nor breaking into the field office, or the like, except some occasional verbal aggression of the more marginalized participants. Our active and outreaching (rather than office-based only) approach and our good relationship with staff from other services and systems (e.g., forensic) was also essential to the success of the project. Further, our hiring strategy to have individuals on the field research team with exceptional interpersonal skills likely benefitted that we are able to engage even the most marginalized individuals, e.g. participants with absolute (street/shelter) homelessness and with prolonged lifetime homeless histories, which showed different retention characteristics and therefore are likely in need of very flexible and individual retention strategies as described above. Our main conclusion is that longitudinal experimental research projects with vulnerable and hidden populations benefit from extensive outreach work and collaborative approaches that are based on attitudes of mutual respect, contextual knowledge and trust.
Funding
This research was funded by a grant to Simon Fraser University from Health Canada and the Mental Health Commission of Canada.
Trial registration
Current Controlled Trials ISRCTN57595077 and ISRCTN66721740. ISRCTN57595077 - Vancouver at home study: housing first plus assertive community treatment versus congregate housing plus supports versus treatment as usual.
Conflict of interest
The authors declare not conflict of interest.
Acknowledgement
We thank all participants of the At Home/Chez Soi project in Vancouver who generously gave their time and shared their insights and knowledge on mental health and homelessness.
We thank all members of the Vancouver At Home/Chez Soi project team, especially Catharine Hume, Tracy Schoenfeld, Liz Evans, Melissa Stephens, Greg Richmond, Sue Baker, Mark MacDonald, and the other field research interviewers Jeff West, Kurt Lock, Bernd Mueller, Melinda Markey, Lauren Currie, Nancy Laliberte, and Arianna Kennedy as well as the many students who helped with the work at the field research office.
The national At Home/Chez Soi project team: Jayne Barker, PhD (2008–11) and Cameron Keller, MHCC National Project Lead; Paula Goering, RN, PhD, Research Lead and approximately 40 investigators from across Canada and the US. In addition there are 5 site coordinators and numerous service and housing providers as well as persons with lived experience.
References
- 1.Strehlau V., Torchalla I., Li K., Schuetz C., Krausz M. Mental health, concurrent disorders, and health care utilization in homeless women. J. Psychiatr. Pract. 2012;18(5):349–360. doi: 10.1097/01.pra.0000419819.60505.dc. [DOI] [PubMed] [Google Scholar]
- 2.Torchalla I., Strehlau V., Li K., Krausz M. Substance use and predictors of substance dependence in homeless women. Drug Alcohol Depend. 2011;118(2–3):173–179. doi: 10.1016/j.drugalcdep.2011.03.016. [DOI] [PubMed] [Google Scholar]
- 3.Breakey W.R., Fischer P.J., Kramer M., Nestadt G., Romanoski A.J., Ross A. Health and mental health problems of homeless men and women in baltimore. J. Am. Med. Assoc. 1989;262(10):1352–1357. [PubMed] [Google Scholar]
- 4.Koegel P., Sullivan G., Burnam A., Morton S.C., Wenzel S. Utilization of mental health and substance abuse services among homeless adults in Los Angeles. Med. Care. 1999 Mar;37(3):306–317. doi: 10.1097/00005650-199903000-00010. [DOI] [PubMed] [Google Scholar]
- 5.North C.S., Eyrich K.M., Pollio D.E., Spitznagel E.L. Are rates of psychiatric disorders in the homeless population changing? Am. J. Public Health. 2004 Jan;94(1):103–108. doi: 10.2105/ajph.94.1.103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Teesson M., Hodder T., Buhrich N. Psychiatric disorders in homeless men and women in inner Sydney. Aust. N. Z. J. Psychiatry. 2004 Mar;38(3):162–168. doi: 10.1080/j.1440-1614.2004.01322.x. [DOI] [PubMed] [Google Scholar]
- 7.Fichter M., Koniarczyk M., Greifenhagen A., Koegel P., Quadflieg N., Wittchen H.U. Mental illness in a representative sample of homeless men in Munich, Germany. Eur. Arch. Psychiatry Clin. Nuerosci. 1996 Jun;246(4):185–196. doi: 10.1007/BF02188952. [DOI] [PubMed] [Google Scholar]
- 8.Kovess V., Mangin Lazarus C. The prevalence of psychiatric disorders and use of care by homeless people in Paris. Soc. Psychiatry Psychiatr. Epidemiol. 1999 Nov;34(11):580–587. doi: 10.1007/s001270050178. [DOI] [PubMed] [Google Scholar]
- 9.Vázquez C., Muñoz M., Sanz J. Lifetime and 12-month prevalence of DSM-III-R mental disorders among the homeless in Madrid: a European study using the CIDI. Acta Psychiatr. Scand. 1997 Jun;95(6):523–530. doi: 10.1111/j.1600-0447.1997.tb10141.x. [DOI] [PubMed] [Google Scholar]
- 10.Hough R.L., Tarke H., Renker V., Shields P., Glatstein J. Recruitment and retention of homeless mentally ill participants in research. J. Consult. Clin. Psychol. 1996;64(5):881–891. doi: 10.1037//0022-006x.64.5.881. [DOI] [PubMed] [Google Scholar]
- 11.Dumville J.C., Torgerson D.J., Hewitt C.E. Reporting attrition in randomised controlled trials. BMJ. 2006 Apr 22;332(7547):969–971. doi: 10.1136/bmj.332.7547.969. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Odierna D.H., Schmidt L.A. The effects of failing to include hard-to-reach respondents in longitudinal surveys. Am. J. Public Health. 2009 Aug;99(8):1515–1521. doi: 10.2105/AJPH.2007.111138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Calsyn R.J., Klinkenberg W.D., Morse G.A., Miller J., Cruthis R. For the Hiv/aids Treatment Adherence. Recruitment, engagement, and retention of people living with HIV and co-occurring mental health and substance use disorders. AIDS Care. 2004 Jan;16(Suppl. 1):S56–S70. doi: 10.1080/09540120412331315286. [DOI] [PubMed] [Google Scholar]
- 14.Cepeda A., Valdez A. Ethnographic strategies in the tracking and retention of street-recruited community-based samples of substance using hidden populations in longitudinal studies. Subst. Use Misuse. 2010 Apr;45(5):700–716. doi: 10.3109/10826081003591282. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lankenau S.E., Sanders B., Hathazi D., Bloom J.J. Recruiting and retaining mobile young injection drug users in a longitudinal study. Subst. Use Misuse. 2010 Apr;45(5):684–699. doi: 10.3109/10826081003594914. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Redwood D., Leston J., Asay E., Ferucci E., Etzel R., Lanier A.P. Strategies for successful retention of Alaska Native and American Indian study participants. J. Prim. Prev. 2011 Feb;32(1):43–52. doi: 10.1007/s10935-010-0209-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Baker S.G. Gender, ethnicity, and homelessness accounting for demographic diversity on the streets. Am. Behav. Sci. 1994 Feb 1;37(4):476–504. [Google Scholar]
- 18.Hwang S.W. Homelessness and health. Can. Med. Assoc. J. 2001 Jan 23;164(2):229–233. [PMC free article] [PubMed] [Google Scholar]
- 19.Goering P.N., Streiner D.L., Adair C., Aubry T., Barker J., Distasio J. The at home/chez Soi trial protocol: a pragmatic, multi-site, randomised controlled trial of a housing first intervention for homeless individuals with mental illness in five canadian cities. BMJ Open [Internet] 2011 Jan 1;1(2) doi: 10.1136/bmjopen-2011-000323. http://bmjopen.bmj.com/content/1/2/e000323 [cited 2012 Mar 20] Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Somers J., Patterson M., Moniruzzaman A., Currie L., Rezansoff S.N., Palepu A. Vancouver at home: pragmatic randomized trials investigating housing first for people who are homeless and mentally ill. Trials. 2013;14(365) doi: 10.1186/1745-6215-14-365. https://trialsjournal.biomedcentral.com/articles/10.1186/1745-6215-14-365 Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Sheehan D.V., Lecrubier Y., Sheehan K.H., Amorim P., Janavs J., Weiller E. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J. Clin. Psychiatry. 1998;59(Suppl 20):22–33. quiz 34–57. [PubMed] [Google Scholar]
- 22.Barker S., Barron N., McFarland B.H., Bigelow D.A., Carnahan T. A community ability scale for chronically mentally ill consumers: Part II. Applications. Community Ment. Health J. 1994 Oct;30(5):459–472. doi: 10.1007/BF02189063. [DOI] [PubMed] [Google Scholar]
- 23.Barker S., Barron N., McFarland B.H., Bigelow D.A. A community ability scale for chronically mentally ill consumers: Part I. Reliability and validity. Community Ment. Health J. 1994 Aug;30(4):363–383. doi: 10.1007/BF02207489. [DOI] [PubMed] [Google Scholar]
- 24.Torchalla I., Strehlau V., Li K., Schuetz C., Krausz M. The association between childhood maltreatment subtypes and current suicide risk among homeless men and women. Child. Maltreat. 2012 May 1;17(2):132–143. doi: 10.1177/1077559512439350. [DOI] [PubMed] [Google Scholar]
- 25.Weiser S.D., Dilworth S.E., Neilands T.B., Cohen J., Bangsberg D.R., Riley E.D. Gender-specific correlates of sex trade among homeless and marginally housed individuals in San Francisco. J. Urban Health. 2006 Jul;83(4):736–740. doi: 10.1007/s11524-005-9019-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Adair C.E., Holland A.C., Patterson M.L., Mason K.S., Goering P.N., Hwang S.W. Cognitive interviewing methods for questionnaire pre-testing in homeless persons with mental disorders. J. Urban Health. 2012 Feb;89(1):36–52. doi: 10.1007/s11524-011-9632-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Veldhuizen C., Adair C.E., Methot C., Kopp B.C., O'Campo P., Bourque, Streiner D.L., Goering P.N. Patterns and predictors of attrition in a trial of a housing intervention for homeless people with mental illness. Soc. Psychiatry Psychiatr. Epidemiol. 2014 doi: 10.1007/s00127-014-0909-x. published online June 12, 2014. [DOI] [PubMed] [Google Scholar]