Does recruitment source moderate treatment effectiveness? A subgroup analysis from the EVIDENT study, a randomised controlled trial of an internet intervention for depressive symptoms

Abstract

Objective

This study aims to examine whether the effects of internet interventions for depression generalise to participants recruited in clinical settings.

Design

This study uses subgroup analysis of the results of a randomised, controlled, single-blind trial.

Setting

The study takes place in five diagnostic centres in Germany.

Participants

A total of 1013 people with mild to moderate depressive symptoms were recruited from clinical sources as well as internet forums, statutory insurance companies and other sources.

Interventions

This study uses either care-as-usual alone (control) or a 12-week internet intervention (Deprexis) plus usual care (intervention).

Main outcome measures

The primary outcome measure was self-rated depression severity (Patient Health Questionnaire-9) at 3 months and 6 months. Further measures ranged from demographic and clinical parameters to a measure of attitudes towards internet interventions (Attitudes towards Psychological Online Interventions Questionnaire).

Results

The recruitment source was only associated with very few of the examined demographic and clinical characteristics. Compared with participants recruited from clinical sources, participants recruited through insurance companies were more likely to be employed. Clinically recruited participants were as severely affected as those from other recruitment sources but more sceptical of internet interventions. The effectiveness of the intervention was not differentially associated with recruitment source (treatment by recruitment source interaction=0.28, p=0.84).

Conclusion

Our results support the hypothesis that the intervention we studied is effective across different recruitment sources including clinical settings.

Trial registration number

ClinicalTrials.gov NCT01636752.

Strengths and limitations of this study.

• Ours is the first trial to examine the effect of recruitment source on outcome.

• The large sample size makes detection of subgroup effects more likely.

• The absence of a subgroup effect does not prove that the effect applies to all subgroups.

• More randomised trials of internet interventions in clinical settings are needed.

Background

Numerous studies^1–4 and meta-analyses^5–10 have shown that internet interventions are effective in the treatment of a broad array of psychiatric disorders, ranging from depression^5–7 and anxiety^7–9 to post-traumatic stress disorder¹⁰ and schizophrenia.^{1 2} However, most participants in these studies have been recruited through media advertisements,^{7 9} so it remains unclear whether they are similar to those seeking face-to-face treatment in regular clinical settings⁷ and whether the effects for internet interventions generalise across different recruitment settings.⁵

Although some studies suggest that the promising results from efficacy studies can be transferred to routine clinical practice,^{11 12} one recent study¹³ of two internet interventions in primary care reported null findings. This is not necessarily due to the fact that these interventions are not effective in primary care but this might be explained by insufficient use of the interventions. No previous studies have directly examined whether differences in recruitment source are associated with the effectiveness of depression-focused internet interventions.

A better understanding of whether participants recruited from different sources differ in other important characteristics could help investigators avoid sampling bias or target specific clinical or demographic subgroups. Previous studies have addressed associations of recruitment source with patient characteristics in an internet clinic,¹⁴ in a trial of an internet intervention¹⁵ and in a trial of face-to-face psychotherapy for depression.¹⁶ It might also be important to know if participants from certain recruitment sources are particularly open-minded towards internet interventions. However, none of the previous studies has compared attitudes towards internet interventions across different recruitment sources.

Subgroup analyses examining associations between recruitment source and intervention effectiveness require large sample sizes.¹⁷ We have recently published one of the largest randomised trials of an internet intervention, the EVIDENT trial.³ Over 1000 participants were randomised for this trial that demonstrated the effectiveness of the intervention (Deprexis) for mild to moderate depressive symptoms. In the EVIDENT trial, we also developed a novel questionnaire measuring positive and negative attitudes towards internet interventions.¹⁸

Using the EVIDENT trial data set, the current paper has two main objectives. First, we sought to examine whether recruitment source is systematically associated with various baseline parameters, including demographic and clinical characteristics as well as attitudes towards internet interventions. Second, we aimed to examine whether recruitment source is differentially associated with the effectiveness of the intervention. We also report on our general experiences with regard to recruiting participants from clinical settings in the EVIDENT trial.

Methods

The EVIDENT study is a multicenter (diagnostic interviews were conducted in five sites in Germany), randomised controlled trial (RCT). The trial was approved by the Ethics Committee of the German Psychological Association (DGPs SM 04_2012) and is registered with ClinicalTrials.gov (NCT01636752). The full study protocol has been published.¹⁹

Participants

Participants were recruited via multiple settings and online informed consent was obtained prior to the baseline assessment. The main recruitment sources were internet forums for depression, magazines for members of statutory German health insurance companies and various inpatient and outpatient clinics, ranging from general practitioners' practices to psychiatrists' and psychotherapists' clinics, practices and hospital settings.

Recruitment source was assessed by self-report; specifically, a combination of a multiple-choice question (clinical setting, internet forums, insurance company, other) and a free-text answer was used to identify the exact source via which each patient was recruited. One of the authors (CG) cross-checked the free-text answers against the multiple-choice answer and resolved any discrepancies through discussion with her local study team (CS and JPK).

The main inclusion criterion for the RCT was the presence of self-reported mild to moderate depressive symptoms, operationalised as a score from 5 to 14 on the Patient Health Questionnaire-9 (PHQ-9).²⁰ Eligible participants were from 18 to 65 years of age, had internet access and were able to communicate in German. Participants with acute suicidality or a lifetime diagnosis of bipolar disorder or schizophrenia were excluded.

Interventions

Following a naturalistic and pragmatic design approach, care-as-usual was not influenced by the investigators. All participants were permitted to use any form of treatment, including antidepressant medication and psychotherapy. Participants were randomised equally (1:1) to the two groups (intervention or control). Participants in the control condition received only care-as-usual (hereafter referred to as the CAU group). They were offered access to the internet intervention after the last follow-up assessment. Participants in the intervention group received immediate access to the internet intervention (Deprexis) in addition to care-as-usual. Briefly, this programme consists of modules covering content that is broadly consistent with CBT (eg, cognitive restructuring, behavioural activation, acceptance and mindfulness, problem-solving).²¹ The intervention can be used with or without guidance by a clinician.²² In our trial, participants randomised to the intervention group with an initial PHQ-9 score from 10 to 14 received the guided version (e-mail support); those scoring from 5 to 9 on the PHQ-9 received the unguided version.

Outcome measures

The primary outcome for the RCT was change on PHQ-9.²⁰ The internal consistency of the PHQ-9 based on the trial data was good (Cronbach's alpha=0.83). The Mini International Neuropsychiatric Interview (MINI)²³ was used to assess the presence of a depressive disorder as well as to rule out a lifetime diagnosis of bipolar disorder or schizophrenia. Clinician-rated severity of depression was assessed with the 24-item version of the Hamilton Depression Rating Scale (HDRS-24)²⁴ (Cronbach's alpha=0.79). The MINI and the HDRS-24 were administered via telephone by trained raters.

Attitudes towards internet interventions were assessed using a questionnaire that was developed during this trial, the Attitudes towards Psychological Online Interventions Questionnaire (APOI).¹⁸ The APOI is the first questionnaire that measures both positive and negative attitudes towards internet interventions in general. It comprises four subscales with scores ranging from 4 to 20, and these are labelled ‘scepticism and perception of risks’, ‘confidence in effectiveness’, ‘technologisation threat’ and ‘anonymity benefits’. The total score ranges from 16 to 80 with higher scores reflecting a more favourable attitude towards internet interventions. When calculating the total score, the polarity of the subscale scores for ‘scepticism and perception of risks’ and ‘technologisation threat’ is reversed so that all subscales contribute equally to the total score. The internal consistency of the APOI in this sample is acceptable to good (Cronbach's alpha=0.77).

Demographic details and treatment history were assessed with non-standardised questionnaires. We also employed the following self-rating scales: a measure of health-related quality of life (Short-Form Health Survey: SF-12)²⁵ that covers physical health related quality of life (SF-12 PH) and mental health related quality of life (SF-12 MH), a broad symptom measure covering dimensions ranging from general well-being to interpersonal relationships (Questionnaire for the Evaluation of Psychotherapeutic Processes: FEP-2)²⁶ and the Web Screening Questionnaire (WSQ), an instrument screening for frequent mental disorders.²⁷

Assessments

The PHQ-9 was administered via online questionnaires along with all the other self-ratings at baseline, after 3 months (post-assessment) and after 6 months (follow-up assessment). Raters contacted participants for the MINI and the HDRS-24 at baseline and after 3 months.

Recruiter survey

We also invited the clinicians recruiting for our study to participate in an online survey. They were asked to provide demographic data and to complete two questionnaires: an unstandardised questionnaire that assessed their recruitment experience and the Attitudes towards Psychological Online Interventions Questionnaire, adapted for healthcare professionals (APOI-HP).²⁸

Statistical analysis

Statistical analyses were performed with SPSS 22 (IBM Corporation). We calculated univariate analyses of variance for continuous variables. Post hoc tests were Bonferroni corrected for multiple comparisons. For categorical variables, we calculated univariate multinomial logistic regression analyses. Effect sizes are presented as Cohen's d for continuous data and numbers needed to treat (NNT) for dichotomous data.

For the analysis of the effect of the recruitment source on treatment effectiveness, we used linear mixed models (LMM), as they have the advantage of using all available data of each subject. They also offer the opportunity to choose an appropriate covariance structure reflecting the potential dependence due to repeated measurements.²⁹ Adjustment for baseline measure was chosen as this accounts for regression to the mean.³⁰ The analysis followed the intention-to-treat principle, which included all randomised participants. No missing values were substituted as LMMs based on all observed data are valid and unbiased methods for missing at random data (MAR).³¹

The outcome was analysed as change from baseline with a random intercept for the participant. Time, treatment group, recruitment source and the interaction term treatment by recruitment source were entered as fixed effects. We chose an autoregressive covariance structure and allowed variances to vary between assessment points. The choice was based on Akaike's Information Criterion from a fixed set of candidate structures, namely a first-order autoregressive (AR1) or scaled identity structure or heterogeneous versions thereof. The hypothesis that the recruitment source influences the effect of the intervention on depressive symptoms was tested on the treatment by recruitment source interaction. Here, the effect of the intervention is defined as the mean difference between average change in outcome for the intervention group minus average change in outcome for the CAU group (the difference in differences). The subgroup analysis had been pre-specified in the study protocol.¹⁹

Results

Recruitment and participant flow

For the participant flow chart, please refer to the report of the main results of this study.¹⁹ Briefly, 2020 participants were assessed for eligibility, and 1007 (49.9%) were excluded. The most common exclusion criterion was exceeding a score of 14 on the PHQ-9 (748, 37.0%). Non-completion rates for the main outcome measure were 21.6% at post-assessment (n=219) and 24.6% at follow-up (n=259). The non-completion rate did not differ between the different recruitment sources (χ²₃=4.34, p=0.227 for the post-assessment and χ²₃=2.06, p=0.559 for the follow-up assessment).

Most participants (46%) self-identified as coming from the ‘other’ recruitment source (see table 1). The remaining participants came from statutory health insurance companies (27%), internet forums (17%) and clinical sources (10%). Inspection of the free-text answers revealed that most of the participants in the ‘other’ category learnt about the study through articles in the news media.

Table 1.

Differences in demographic data between different recruitment sources

	Recruitment source												Statistics
	(1) Clinical			(2) Internet forums			(3) Insurance			(4) Other			Between groups
Demographic characteristics	n	Mean	SD	n	Mean	SD	n	Mean	SD	n	Mean	SD	df	F	p
Age	105	41.58	11.930	171	41.89	11.239	271	43.32	9.777	466	43.23	11.327	3,1009	1.250	0.290
Marital Status	n	n	%	n	n	%	n	n	%	n	n	%	df	χ2	p
Married and cohabiting	105	36	34.3	171	57	33.3	271	131	48.3	466	201	43.1	15	30.29	0.011
Married and not cohabiting		4	3.8		5	2.9		9	3.3		10	2.1
Committed relation		24	22.9		32	18.7		44	16.2		89	19.1
Single		29	27.6		51	29.8		51	18.8		116	24.9
Divorced		10	9.5		21	12.3		35	12.9		49	10.5
Widowed		2	1.9		5	2.9		1	0.4		1	0.2
Education status	n	n	%	n	n	%	n	n	%	n	n	%	df	χ2	p
Highest secondary	105	52	49.5	171	87	50.9	271	118	43.5	466	263	56.4	18	37.21	0.005
Higher secondary		15	14.3		38	22.2		54	19.9		65	13.9
Middle secondary		27	25.7		35	20.5		84	31.0		97	20.8
Lower secondary		6	5.7		8	4.7		12	4.4		27	5.8
Still in school		1	1.0		0	0		0	0		1	0.2
No degree		1	1.0		0	0		0	0		0	0
Other		3	2.9		3	1.8		3	1.1		13	2.8
Employment status	n	n	%	n	n	%	n	n	%	n	n	%	df	χ2	p
Full-time	105	44	41.9	171	65	38.0	271	136	50.2	466	189	40.6	9	30.24	<0.001
Part time		17	16.2		32	18.7		75	27.7		103	22.1
Other		18	17.1		27	15.8		24	8.9		62	13.3
None		26	24.8		47	27.5		36	13.3		112	24.0

Participant characteristics

For descriptive and inferential statistics on the differences between the four recruitment sources, refer to table 1 (demographic data) and table 2 (clinical characteristics). Briefly, we did not find any statistically significant differences for a broad range of clinical characteristics including self-rated and clinician-rated depression severity, psychosocial functioning and self-reported comorbid symptoms. Participants recruited through online forums were slightly more likely to suffer from dysthymia and participants from clinical settings and other sources were slightly more likely to report symptoms of panic disorder, but these differences were not statistically significant. We did find statistically significant differences between the recruitment sources for measures of resource use. Compared with participants recruited through insurance companies and other sources, participants recruited in clinical settings were more likely to be in psychiatric treatment (p<0.05; OR vs insurance 2.71, OR vs other sources 1.70), psychotherapy (p<0.01; OR vs insurance 2.66, OR vs other sources 1.99) and inpatient psychiatric treatment (p<0.001; OR vs insurance 4.06, OR vs other sources 3.33, OR vs internet forums 2.30). They also reported having had significantly more sick leave days (p<0.001; d vs insurance 0.52, d vs other sources 0.41).

Table 2.

Differences in clinical characteristics between different recruitment sources

	Recruitment source												Statistics
	(1) Clinical			(2) Internet forum			(3) Insurance			(4) Other			Between groups
Clinical characteristics	n	Mean	SD	n	Mean	SD	n	Mean	SD	n	Mean	SD	df	F	p
PHQ-9	105	10.06	2.60	171	10.21	2.44	271	10.40	2.39	466	10.30	2.38	3,1009	0.57	0.635
PHQ-9 suicidality item		1.30	0.48		1.23	0.45		1.21	0.41		1.27	0.46		1.91	0.127
SF-12 PH	102	46.42	9.52	169	46.83	9.91	262	47.64	9.06	456	48.05	9.44	3,985	1.25	0.291
SF-12 MH		30.11	6.96		31.39	8.45		31.49	7.34		31.51	7.69		0.99	0.398
FEP-2	105	2.93	0.47	171	2.99	0.47	271	2.92	0.46	466	2.918	0.45	3,1009	1.08	0.357
HRSD		17.47	7.49		17.49	7.58		16.14	7.47		16.73	7.41		1.48	0.219
Diagnosis	n	n	%	n	n	%	n	n	%	n	n	%	df	χ2	p
Dysthymia	105	33	31.4	171	77	45.0	271	100	36.9	466	166	35.6	3	6.49	0.090
Depressive episode		36	34.3		47	27.5		75	27.7		132	28.3		1.90	0.594
More than five episodes		36	34.3		66	38.6		99	36.5		192	41.2		2.62	0.453
Panic d/o (WSQ)		40	38.1		47	27.5		75	27.7		159	34.1		6.68	0.083
Social phobia (WSQ)		46	43.8		80	46.8		127	46.9		221	47.4		0.45	0.930
Alcohol use d/o (WSQ)		4	3.8		6	3.5		11	4.1		31	6.7		4.16	0.245
Resource use	n	n	%	n	n	%	n	n	%	n	n	%	df	χ2	p
General practitioner	105	92	87.6	171	141	82.5	271	240	88.6	466	397	85.2	3	3.70	0.295
Psychiatrist		44	41.9		72	42.1		57	21.0		139	29.8		28.67	<0.001
Psychotherapist		52	49.5		73	42.7		73	26.9		155	33.3		22.58	<0.001
Neurologist		20	19.0		38	22.2		39	14.4		62	13.3		8.76	0.033
Inpatient psychiatry		19	18.1		15	8.8		14	5.2		29	6.2		20.35	<0.001
Sick leave days		40.13	66.68		29.99	53.54		15.72	36.41		20.15	43.40	3,1009	8.81	<0.001

FEP-2, broad self-rated symptom measure covering dimensions ranging from general well-being to interpersonal relationships; HRSD, clinician-rated depression severity; PHQ-9, self-rated depression severity; SF-12 PH, physical health related quality of life; SF-12 MH, mental health related quality of life; WSQ, web screening questionnaire for mental disorders.

We also observed differences for demographic variables. Compared with participants recruited in a clinical setting, participants recruited through statutory health insurance companies were more likely to be employed full-time (p<0.01; OR 2.23) or part-time (p<0.01; OR 3.19). Participants from the different recruitment sources also differed in their attitudes to internet interventions (table 3). Compared with participants recruited through insurance companies and other sources, those recruited in clinical settings had less favourable attitudes towards internet interventions (p<0.01; d vs insurance 0.43, d vs other sources 0.40). In particular, they scored higher on technologisation threat (p<0.01; d vs insurance companies 0.41, d vs other sources 0.34) and lower on anonymity benefits (p<0.01; d vs internet forums 0.42, d vs insurance companies 0.41, d vs other sources 0.32).

Table 3.

Differences in attitude to psychological internet intervention

APOI score	Recruitment Source												Statistics
	(1) Clinical			(2) Internet Forum			(3) Insurance			(4) Other			Between groups
	n	Mean	SD	n	Mean	SD	n	Mean	SD	n	Mean	SD	df	F	p
Scepticism and risk perception	105	9.86	2.38	169	9.88	2.25	270	9.36	2.28	460	9.30	2.31	3,1009	3.86	0.009
Confidence in effectiveness		16.43	2.15		16.44	2.14		16.66	2.03		16.74	2.15		1.19	0.314
Technologisation Threat		12.47	2.12		12.04	2.50		11.52	2.41		11.63	2.53		4.91	0.002
Anonymity benefits		11.48	2.88		12.62	2.61		12.70	3.07		12.47	3.19		4.37	0.005
Total		53.58	6.37		55.14	6.72		56.47	6.83		56.28	6.96		5.85	0.001

Intervention usage and use of other treatments

A total of 509 participants were randomised to the intervention group. The mean number of sessions of at least 10 min duration was 8.32 (SD=4.71), the mean total usage time was 429.70 (SD=294.0) min (about 7 h). Periods of inactivity of 5 min or longer were subtracted in the computation of the total usage time. Participants from the different recruitment sources did not differ with respect to the number of sessions (F_3,481=0.47, p=0.70) or the total usage time (F_3,481=0.51, p=0.70). The intervention and the CAU group did not differ with respect to the use of concomitant treatments (eg, psychotherapy, psychotropic medication) during the study period (see online supplementary table 1).

Primary and secondary outcomes

As reported previously,³ the intervention had a significant effect on the main outcome, change in PHQ-9 scores from baseline to post and follow-up. Whereas depressive symptoms decreased in both groups, changes in PHQ-9 differed significantly (main effect of treatment: F_1,829=23.05, p<0.001) between groups. In the intervention group, PHQ-9 scores decreased by 1.43 (95% CI 0.85 to 2.02) points more than in the CAU group, on average. Both the main effect of recruitment source (F_3,825=2.61, p=0.051) and the interaction term (treatment assignment by recruitment source) were not statistically significant (F_3,824=0.28, p=0.84)(table 4).

Table 4.

Main effect of treatment and treatment by recruitment source interaction on estimated mean change in self-rated depressive symptoms (PHQ-9) from baseline to post and follow-up and minimally clinically relevant change of PHQ-9 at post-assessment. The sum of participants in the treatment groups (intervention plus CAU) is smaller than the number of participants in each subgroup because some participants did not complete the post or the follow-up assessment and could therefore not be included in the main analysis. A sensitivity analysis using multiple imputation to replace missing values yielded essentially the same results as the main analysis

Recruitment source	Treatment	Estimated change			Effect Size		Minimally clinically important change of PHQ-9		Effect size
		M	SE	95% CI	d	95% CI	n	%	NNT	95% CI
Main effect of treatment		F1,823=21.84, p<0.001					B=1.18 (SE 51), p=0.021
All (n=1013)	Intervention (n=509)	2.94	0.21	2.53 to 3.35	0.30	0.01 to 0.59	143	28.1	9	6 to 15
	CAU (n=504)	1.51	0.21	1.09 to 1.92			83	16.5
Treatment by recruitment source interaction		F3,817=0.29, p=0.83					B=−0.16 (SE 16), p=0.31
Clinical (n=105)	Intervention (n=42)	3.30	0.54	2.24 to 4.37	0.27	−0.49 to 1.04	17	29.8	5	−3 to 107
	CAU (n=38)	2.35	0.58	1.21 to 2.49			8	16.7
Internet forums (n=171)	Intervention (n=63)	2.20	0.46	1.30 to 3.10	0.39	−0.23 to 1.01	23	28.0	5	3 to 15
	CAU (n=62)	0.81	0.45	−0.07 to 1.69			9	10.1
Insurance (n=271)	Intervention (n=106)	3.24	0.35	2.56 to 3.93	0.48	0.00 to 0.96	40	29.9	7	4 to 39
	CAU (n=113)	1.52	0.34	0.85 to 2.19			26	19.0
Other (n=466)	Intervention (n=184)	3.00	0.27	2.48 to 3.52	0.46	0.09 to 0.83	63	26.7	8	5 to 28
	CAU (n=186)	1.34	0.27	0.81 to 1.87			40	17.4

The analysis of the secondary outcomes (HRSD, SF-12 and FEP-2) mostly yielded the same pattern of results (table 5). The main effect of group was statistically significant for all secondary outcomes except for SF-12 PH. The main effect of recruitment source was significant for SF-12 MH (p=0.011) with patients recruited via internet forums reporting smaller improvements compared with participants recruited from clinical settings (−3.82; 95% CI −7.18 to −0.47; Bonferroni corrected p=0.016) and participants recruited from other sources (−2.57; 95% CI −4.99 to −0.15; Bonferroni corrected p=0.030). The interaction term (treatment assignment by recruitment source) was not statistically significant for any of the secondary outcomes.

Table 5.

Main effect of treatment, main effect of recruitment source and treatment by recruitment source interaction on estimated mean change in secondary outcomes

	Main effect of treatment			Main effect of recruitment source			Treatment by recruitment source interaction
	df	F	p	df	F	p	df	F	p
HRSD	1,696	11.82	0.001	3,688	1.26	0.300	3,687	32.99	0.551
FEP-2	1,819	76.17	<0.001	3,815	2.40	0.067	3,814	0.141	0.935
SF-12 PH	1,789	2.23	0.135	3,785	0.312	0.816	3,784	0.319	0.811
SF-12 MH	1,789	136.06	<0.001	3,785	3.736	0.011	3,784	0.972	0.405

FEP-2, broad self-rated symptom measure covering dimensions ranging from general well-being to interpersonal relationships; HRSD, clinician-rated depression severity; SF-12 PH, physical health related quality of life; SF-12 MH, mental health related quality of life.

Sensitivity analyses

As a sensitivity analysis for the primary outcome, we reran the analysis of the interaction with a binary subgroup definition. Here we summarised the following recruitment sources as non-clinical: statutory health insurance companies, internet forums and ‘other’ recruitment sources. Here, we replicated the significant main effect of treatment (F_1,834=7.94, p<0.01) and found a significant main effect of recruitment source on change of PHQ-9 (F_1,834=5.45, p=0.02). Symptom change was greater in those recruited from clinical sources compared with those not recruited from clinical sources (1.50; 95% CI 0.33 to 2.68). The interaction term (treatment by binary recruitment source) was not statistically significant (F_1,834=1.66, p=0.20) confirming the result of the main subgroup analysis above.

We also conducted a sensitivity analysis with the binary outcome ‘minimally clinically important change of PHQ-9’ as the dependent variable (binary logistic regression: χ²₃=19.749, p<0.001, Nagelkerke's r²=0.031). A minimally clinically important individual PHQ-9 improvement was defined as five-point reduction.³² In keeping with the results of the previously reported analyses, we found a main effect of treatment (B=1.18 (SE 51), p=0.021) and no main effect of recruitment source (B=0.08 (SE 13), p=0.53) and that the treatment by recruitment source interaction term was not statistically significant (B=−0.16 (SE 16), p=0.31).

Rerunning the primary analysis without baseline correction did not alter our results substantially (main effect of group: F_1,827=20.47, p<0.001; main effect of recruitment source on PHQ change: F_3,827=2.28, p=0.078; treatment assignment by recruitment source interaction: F_3,827=0.18, p=0.91). In a final sensitivity analysis, we used multiple imputation (50 imputations) to estimate missing scores by evaluating the relationships between observed and missing scores as well as baseline scores. The results were essentially the same as for the main analysis (main effect of treatment: F_1,1004=111.52, p<0.001; main effect of recruitment source on change of PHQ-9: F_3,1004=2.547, p=0.055; treatment assignment by recruitment source interaction: F_3,1004=0.45, p=0.72).

Recruiter characteristics

A total of 89 persons who supported our recruitment efforts in clinical settings (the recruiters) were contacted via e-mail for an online survey. Of these, 48 completed the survey (54%). They were mostly female (69%) and their mean age was 44.06 (SD 12.17) years. Almost half of them reported working in an inpatient setting (42%), mostly as psychotherapists (50%), specialists in psychosomatic medicine (33%) and specialists in psychiatry (22.9%). Recruiters could name multiple fields of work and, therefore, the total sum exceeds 100%. Recruiters also completed a questionnaire inquiring about their experiences with regard to the recruitment process. Here, 40% reported that they often forgot to talk with their patients about the study. About 25% wrote that they did not have the time to talk with their patients about the study or that their patients' symptoms were too severe to participate in the study. Only 12.5% of respondents reported inadequate computer literacy as a barrier to participating in the study. On the APOI, the recruiters had a total mean score of 51.23 (SD 12.17) and the following subscale mean scores (SD): scepticism and risk perception, 11.14 (2.55); confidence in effectiveness, 16.08 (1.92); perceived technology disadvantages, 14.50 (SD 2.34); and perception of anonymity benefits, 12.64 (2.47).

Discussion

Principal findings

This study examined associations of recruitment source with participant characteristics and effectiveness in a trial of an internet intervention for depressive symptoms. We found few demographic or clinical differences among participants recruited from different sources. To our knowledge, this is the first study that examines the association of the recruitment source with the effectiveness of an internet intervention. Here, we found no moderating influence of the recruitment source on the treatment effect. We did find an indication, however, that the recruitment source might predict course of depressive symptoms independent of treatment group assignment. Decrease of symptoms was greater in those recruited from clinical sources than in those recruited via other settings. This finding was only statistically significant in one of the sensitivity analysis though and should thus be replicated in other studies before firm conclusions can be drawn from this finding.

Comparison with other studies

Some of the findings regarding clinical characteristics contrast with results from a previous study,¹⁵ which found somewhat more severe symptoms in patients recruited through clinical settings. Participants recruited in clinical settings in our study were more likely to be on sick leave, suggesting that, despite similar current symptom severity, their symptom-related psychosocial impairment in the 6 months preceding randomisation might have been greater. Even though participants recruited in clinical settings did not differ from others in depression severity or quality of life, they were about twice as likely to be in psychiatric treatment, compared with participants recruited via health insurance companies (OR 2.71) or other sources, such as news media (OR 1.70). This might indicate that internet interventions reach people who chose not to seek treatment through more conventional means in spite of substantial symptom severity.³³

We have found that participants recruited through insurance companies were more likely to be employed. Also we observed a significant between-groups difference regarding level of education. These findings might provide some orientation for researchers wishing to recruit participants with certain demographics as it has been noted that participants in internet studies as well as outpatient treatment centres are more highly educated than the general population.¹⁴

Participants recruited through clinical settings had a less favourable view of internet interventions compared with the other groups. The recruiters working in these clinical settings viewed internet interventions less favourably than the participants. Understandably, patients engaging with psychiatric or psychotherapeutic treatment and clinical treatment providers may regard internet based treatments with somewhat greater scepticism. Interestingly, recruiters for our study had a more positive view of internet interventions than psychotherapists recruited through professional associations for psychotherapists who were surveyed in a separate study.²⁸ This might be due to sampling bias: clinicians who are sceptical of internet interventions are less likely to recruit for a study of such an intervention.

Limitations of the study

In spite of this relatively positive attitude, the 89 recruiters only recruited 105 participants for this trial that were eligible to participate and could thus be randomised. This figure must be interpreted with caution though as we could not link study participants to individual recruiters. Therefore, we do not know whether the recruiters surveyed here actually recruited the participants in this study that self-identified as clinically recruited. Still, these figures do suggest that it is more difficult to recruit for an internet intervention through clinical settings compared with recruitment through the media and the internet. These recruitment difficulties were not related to characteristics of the internet intervention but rather to more general problems with recruiting for studies in a busy clinical routine.

There are some further limitations to consider when interpreting our results. The most common recruitment source was ‘other’, and most of these participants learnt about our study through news media. The clinical recruitment sources were heterogeneous. Most of the clinical recruiters self-identified as psychotherapists. Our results may therefore have been different if we had recruited in general practice. Also, our sample reported mild to moderate depressive symptoms and it is therefore unclear if our results also extend to people with more severe depressive symptoms or other primary mental health complaints. The inclusion of only mild and moderately depressed subjects might also have limited our ability to detect baseline differences in clinical characteristics.

Furthermore, the absence of a statistically significant interaction in our subgroup analysis does not necessarily mean that the treatment effect applies to all subgroups.¹⁷ Statistical power is considerably lower for interaction analyses compared with the main effect analysis, particularly if the subgroups are not identical in size as in our study.³⁴ Inspection of table 4 suggests that a differential treatment effect might have attained statistical significance in an even larger sample. We have previously reported that the internet intervention was less effective for patients with mild to moderate depressive symptoms who received concurrent psychiatric or psychotherapeutic treatment.³ Internet interventions may therefore confer the greatest benefit for individuals who are not in specialised psychiatric or psychotherapeutic care. However, this difference may also depend on symptom severity, as we have previously observed stronger effects among severely depressed individuals who used an internet intervention and received concurrent antidepressant medication.⁴

Conclusion

We conclude that the internet intervention studied here can be regarded as an effective intervention, also when offered in a clinical setting. However, additional replications with patients recruited from clinical settings would be desirable to establish the robustness of this conclusion. In terms of their clinical and demographic characteristics, participants recruited from treatment settings are very similar to participants recruited via insurance companies, internet forums and the media. From a public health perspective, it appears justified to make this intervention available in clinical treatment settings and beyond. When deployed in clinical settings, evidence-based internet interventions could be added to the repertoire of existing treatments; when deployed outside of treatment settings, they might offer effective help for underserved people who, for various reasons, do not receive other forms of treatment.