Abstract
Introduction
Chronic pain is highly prevalent, associated with substantial personal and economic burdens, and increased risk for mental disorders. Individuals in green professions (agriculturists, horticulturists, foresters) show increased prevalence of chronic pain and other risk factors for mental disorders. Available healthcare services in rural areas are limited. Acceptance towards face-to-face therapy is low. Internet and mobile-based interventions (IMIs) based on Acceptance and Commitment Therapy (ACT) might be a promising alternative for this population and may enable effective treatment of chronic pain. The present study aims to evaluate the clinical and cost-effectiveness of an ACT-based IMI for chronic pain in green professions in comparison with enhanced treatment as usual (TAU+).
Methods and analysis
A two-armed pragmatic randomised controlled trial will be conducted. Two hundred eighty-six participants will be randomised and allocated to either an intervention or TAU+ group. Entrepreneurs in green professions, collaborating spouses, family members and pensioners with chronic pain are eligible for inclusion. The intervention group receives an internet-based intervention based on ACT (7 modules, over 7 weeks) guided by a trained e-coach to support adherence (eg, by positive reinforcement). Primary outcome is pain interference (Multidimensional Pain Interference scale; MPI) at 9 weeks post-randomisation. Secondary outcomes are depression severity (Quick Inventory Depressive Symptomology; QIDS-SR16), incidence of major depressive disorder, quality of life (Assessment of Quality of Life; AQoL-8D) and possible side effects associated with the treatment (Inventory for the Assessment of Negative Effects of Psychotherapy; INEP). Psychological flexibility (Chronic Pain Acceptance Questionnaire, Committed Action Questionnaire, Cognitive Fusion Questionnaire) will be evaluated as a potential mediator of the treatment effect. Furthermore, mediation, moderation and health-economic analyses from a societal perspective will be performed. Outcomes will be measured using online self-report questionnaires at baseline, 9-week, 6-month, 12-month, 24-month and 36-month follow-ups.
Ethics and dissemination
This study was approved by the Ethics Committee of the University of Ulm, Germany (file no. 453/17—FSt/Sta; 22 February 2018). Results will be submitted for publication in peer-reviewed journals and presented at conferences.
Trial registration number
German Clinical Trial Registration: DRKS00014619. Registered on 16 April 2018.
Keywords: internet- and mobile-based intervention, randomised controlled trial, green professions, prevention, chronic pain
Strengths and limitations of this study.
This protocol is the first large-scale pragmatic randomised controlled trial, investigating the clinical and cost-effectiveness of a guided Acceptance and Commitment Therapy–based internet and mobile-based intervention (IMI) for chronic pain in green professions.
Preventive effects of the IMI on mental health (eg, onset depression) will be investigated.
The present study evaluates short-term and long-term effects over an extensive time period of 3 years.
The study also focuses on effect moderating and mediating factors (eg, psychological flexibility, working alliance, technological alliance).
All IMI outcomes are assessed through online self-report measures only and the generalisability of the results are limited to the target population of green professions.
Introduction
Chronic pain is highly prevalent worldwide and associated with major personal and societal burden.1–5 For European countries, Breivik and colleagues1 report prevalences from 12% (Spain) to 30% (Norway). In Germany, the prevalence is estimated at 17%.1 In a more recent survey, an even higher proportion of the responders (28.3%) reported chronic pain.6 Chronic pain leads to substantial reduction in global functioning and the quality of life.2 7 Besides the negative effects of chronic pain itself, chronic pain is highly linked to and promotes comorbid mental disorders,1 8–11 which lead to further health burdens, reduced global functioning and greater expenses.12–14 Given the high disease and economic burden of chronic pain, effective treatments are highly needed.
In the treatment of chronic pain, multidimensional and interdisciplinary therapies are the gold standard.15 Elements of Cognitive-Behavioural Therapy or Acceptance and Commitment Therapy (ACT) are most often involved in those treatments.16–18 According to a meta-analysis, the ACT-based approaches seem to be effective in the treatment of chronic pain, reducing both pain-related symptoms (d=0.43, 95% CI 0.22 to 0.64) and depression severity (d=0.69, 95% CI 0.47 to 0.92).19 ACT assumes psychological flexibility as a mechanism of change. Psychological flexibility is defined as “the ability to contact the present moment more fully as a conscious human being, and to change or persist in behavior when doing so serves valued ends”20 (p. 7). Although effective treatments are available for chronic pain, many affected individuals remain untreated.1 This corresponds to one-third of the population in Europe.1 Thus, the access to adequate treatments needs to be facilitated, especially for populations with an increased prevalence for chronic pain and a limited access to healthcare.
Individuals both in green professions (agriculturists, horticulturists, foresters) and rural regions might constitute an example to this population: They show an increased prevalence for pain.7 21–23 In addition, they are exposed to risk factors for chronic pain, pain severity and mental disorders (eg, major depressive disorder (MDD)). Most of these risk factors comprise, for example, financial pressures, high administrative workload, long working hours, high stress levels, part-time jobs off the farm, health problems and preceding work accidents.22–25 Hence, this population could considerably profit from effective treatments, reducing pain-related and mental health outcomes. Consequently, the risk for mental disorders (eg, MDD) might also be lowered.1 8–11 24 25 However, agriculturalists show limited acceptance towards and utilisation of face-to-face therapeutic treatments.26 Moreover, the availability of multidimensional and interdisciplinary therapies is limited.27–29 Thus, alternative and new ways to deliver treatment might be beneficial.
Internet and mobile-based interventions (IMIs) can help overcome barriers of face-to-face treatments.30–32 For example, IMIs are temporally and spatially flexible, can reduce the waiting time for treatment and foster anonymity.31 33 IMIs have been shown to be effective in both intervention34 and prevention.32 35 In chronic pain samples, IMIs achieved effect sizes similar to their face-to-face counterparts.36 For pain interference, a meta-analytic effect of Hedge’s g=−0.42 (95% CI −0.55 to −0.28) and for pain severity of g=−0.35 (95% CI −0.54 to −0.17) favouring the intervention group was found when compared with wait-list or treatment-as-usual groups.36 Furthermore, there were effects on variables unrelated to pain, such as mood ratings (eg, depression) g=−0.27 (95% CI −0.38 to −0.16) or catastrophising g=−0.65 (95% CI −0.95 to −0.36).36
Despite strong evidence for the general effectiveness of IMIs for chronic pain, the effectiveness of ACT-based IMIs in green professions is currently unclear. This is true for the effects on both pain-related and mental health variables. Sociodemographic factors in this population such as higher age, lower educational level or lower internet affinity might lead to different effects.36–38 Moreover, it is unclear whether a reduction of pain-related symptoms and mental health burden would help prevent mental disorders (eg, MDD) in this population in the long term. This study is part of a nationwide prevention project targeting the health and well-being of individuals in green professions. The effects of an ACT-based IMI for chronic pain will be systematically investigated in this population. The following research questions will be examined in a 36-month follow-up pragmatic randomised controlled trial:
Is the IMI effective in the reduction of pain interference in comparison with an enhanced treatment as usual (TAU+) group?
Is the IMI effective in the reduction of mental health variables (eg, depression, stress, sleep quality, anxiety) compared with a TAU+ group?
Is the IMI effective in the prevention of MDD?
Which variables moderate and mediate the effect between the intervention group and TAU+group?
Is the IMI cost-effective compared with the TAU+group?
What are the levels of intervention satisfaction, adherence, acceptance and negative side effects for the intervention group?
Methods and analysis
Design
This is a two-armed randomised controlled trial comparing a guided internet-based intervention to an enhanced treatment as usual (TAU+) group. All assessments will be conducted online and will take place before the randomisation (at baseline T0), 9 weeks after the randomisation (T1), at 6-month follow-up (T2), at 12-month follow-up (T3), at 24-month follow-up (T4) and at 36-month follow-up (T5).
This trial will be conducted and reported according to the CONSORT 2010 statement39 40 and to the guidelines for execution and reporting internet intervention research.41
Inclusion and exclusion criteria
We will include (a) agriculturists, foresters and horticulturists with sufficient insurance coverage from a health insurance company (SVLFG) in Germany. The study is accessible for entrepreneurs, collaborating spouses and family members, as well as pensioners with sufficient insurance coverage who contribute to the production process (‘Altenteiler’). Participants are required (b) to be aged 18 or above, (c) to report chronic pain, following the recommendations of the International Association of Pain in which the duration of pain is stated as ≥6 months,42 (d) to experience considerable pain intensity (=at least grade II according to the Chronic Pain Grade questionnaire (CPG)43–45), (e) to provide an email address, (f) to have internet access and (g) be willing to give informed consent.
Participants are excluded if they (a) receive an on-going psychotherapeutic treatment, (b) are not able to distance themselves from suicidal ideation (not able to sign a non-suicide contract) and (c) if they are eligible for the parallel clinical trial PROD-A and would instead prefer to participate in it.46 See figure 1 for a study flow chart. Inclusion and exclusion criteria are assessed in a self-report online survey, in which applicants’ eligibility will be assessed prior to the trial inclusion and randomisation. The hand-signed informed consents and non-suicide contracts (see below) will be checked by the study team.
Figure 1.
Flowchart of the study procedure.
Suicidal ideation will be assessed with the PHQ-9 suicide item during the screening period and with the Quick Inventory Depressive Symptomology (QIDS-SR16; Rush et al 2003) suicide items during the assessments at T0–T5. If the participants score ≥1 on the PHQ-9 or QIDS-SR16 items, the BDI-II suicide item will additionally be administered since it is an efficient and more reliable screening instrument for suicidality.47 All participants with an elevated suicide risk will receive an email with urgent advice to seek help and relevant information on available help services (eg, their general practitioner, local psychiatric emergency units or the national emergency number). The wording of the information material is adapted in emphasis, depending on the severity of suicidal ideation. In case of a BDI-II item score of >1, participants will be asked for a non-suicide contract. Participants refusing the non-suicide contract will be excluded from the study. If a score of BDI-II ≥2 is reported, a more detailed clarification of self-endangerment will be performed by a psychological psychotherapist which is independent of the study inclusion status and assessment point. This procedure on suicidal ideation is adapted from prior studies45 48 and was approved by the Ethics Committee of the University of Ulm.
Recruitment and procedure
The study population will be recruited via a journal for green professions and policy-holders of SVLFG with a quarterly circulation of 1.3 million. The journal article contains key information about the study protocol and describes various ways to participate in or get further information about the study (eg, postal return, fax, email, telephone and internet links). In addition, postal mailings and newsletter articles are being used to recruit participants. The recruitment has started in February 2018 and takes place nationwide in Germany.
All recruitment ways lead to an initial online screening survey, in which the applicants’ eligibility will be assessed. If applicants are eligible for the present trial (acronym: PACT-A) or the parallel study focusing on the prevention of depression in green professions (acronym: PROD-A), they can choose between the trials as long as the recruitment for both trials is open. The participation is limited to one of the trials. The participants who are eligible for PACT-A will be asked to sign an informed consent and forwarded to the baseline assessment (T0). After the baseline assessment, the participants will be randomised and will receive either a guided internet-based intervention or an enhanced TAU. The participants will be contacted and reminded on the follow-up assessments (T1–T5).
Randomisation and blinding
The randomisation will take place at an individual level. It will be conducted by a person, blinded on all processes throughout the intervention. The blinding of the participants is not possible due to the nature of the intervention. Data collectors and data analysts are blinded regarding the participant’s group membership. Group membership is only known by the persons administering the allocated treatments to the participants. The randomisation and allocation will be performed using permuted block randomisation with randomly arranged variable block sizes of (2, 4, 6) and an allocation ratio of 1:1. The randomisation list was created using a web-based randomisation program (https://sealedenvelope.com/). The randomisation will be carried out after the baseline assessment.
Power calculation and sample size
The primary outcome of the present study is the standardised mean difference between the intervention group and the TAU+ control group in pain interference at T1. For internet interventions targeting chronic pain, Buhrman and colleagues36 reported an average effect of Hedges’ g=−0.42 (−0.55 to −0.28). An a priori power calculation, targeting an effect of d=0.42 at an alpha level of 5% (one-tailed) and a power of 90%, has yielded a total sample size of 196. A t-test was used for power calculation. The calculations were conducted using G*Power (V.3.1.9.2).49
Several reviews and studies show high drop-out rates for internet interventions.36 50 51 For instance, Melville and colleagues reported an average drop-out rate of 31% in a systematic review.50 Since the present study aims for long-term follow-up and drop-out is assumed, the estimated sample size of 196 will be increased to be able to compensate a drop-out rate of 31%. Thus, the targeted total sample size of this study is going to be 286. However, the drop-out rate or effect in this study might differ from the assumed values and thus, a post hoc power analysis will be conducted to determine the actual power at T1 and follow-ups at T2–T5.
Intervention development
The online intervention is based on an effective internet intervention against chronic pain (d=0.58) called ACTonPAIN51 and was adjusted to the targeted population by an IMI-providing company named GET.ON Institute. The present study team has not been involved in the intervention adjustments by the GET.ON Institute and are only engaged in the evaluation of this intervention.
Intervention content
The structure of the intervention follows the recommendations by Hayes and colleagues52 and the structure of ACTonPAIN (for further information about ACTonPAIN, see45 51). The intervention is structured in seven sessions. In the first session, basic information about the intervention like acute pain, chronic pain and the effects of pain are presented, and the participants will be introduced to the concept of mindfulness and key concepts of ACT. Session 2 focuses on acceptance, session 3 on defusion, session 4 on self-concept, session 5 on values, session 6 on committed actions and session 7 summarises all sessions and fosters the sustainability of the learned content. Throughout the intervention, mindfulness and mindfulness exercises are going to be part of all sessions. The mindfulness exercises are aiming to encourage participants to perceive the here-and-now without judging or trying to change and alter it. Participants are advised to work on the sessions on a weekly basis.
All sessions are interactive, containing audios, videos, illustrated examples and exercises. In addition to the exercises within the sessions, participants receive homework assignments between each session. Throughout the intervention, participants will be guided by an e-coach. All e-coaches are trained psychologists, who are supervised by a licensed psychotherapist and paid by the GET.ON Institute. The e-coaches give feedback via email or telephone, based on the participant’s preference to each session during the intervention phase (=weekly feedback, if participants adhere to the treatment protocol). The content of the feedback matches the assignments of the participants and supports treatment adherence (eg, by positive reinforcement). The e-coaches will contact the participants on a monthly basis for over a year after the end of the intervention to monitor and maintain their current state. The duration of feedback can vary across participants and will be monitored to enable post hoc analysis on guidance time.
In addition to the intervention, all participants in the intervention group will have unrestricted access to TAU.
TAU+ content
The control group has full access to TAU. Based on the strong evidence for the effectiveness of ACT and internet-based ACT, it is unethical to withhold improved treatment from the control group.36 53 54 Hence, the control group will be provided with psychoeducation as a minimal support treatment (TAU+).54 55 The participants of the control group will receive an information letter with psychoeducation about stress, depression and chronic pain (pdf via email). The control group will receive a link to a free online audio CD with further information about stress and stress reduction. In addition, they will also be provided with information about different treatment options available in routine care.
TAU may vary across individuals since they have unrestricted access to it. We obtain information about the TAU in the economic evaluation with the help of the TiC-P56 57 (see below). The TAU utilisation will be reported, and potential systematic differences will be examined.
Patient and public involvement
Patient and public involvement (PPI) representatives have provided input to the present study in several stages. First, PPI representatives were included in the intervention development by the GET.ON Institute to identify core symptoms, improve the usability, design and the overall tailoring of the intervention for the target population. Second, the PPI representatives gave feedback on the study materials to ensure the comprehensibility of the materials. The outcomes were defined according to international standards and the expertise of the research group. The PPI representatives, the SVLFG insurance company and the GET.ON Institute had no influence on the outcomes, data analysis methods or study design. However, the burden of the intervention from the patients’ perspective is a crucial outcome of the study and both quantitative and qualitative methods will be applied to capture the burden and side effects. The dissemination plan of the study results includes publications in peer-reviewed journals, conferences as well as special reports for SVLFG, the GET.ON Institute and the PPI representatives.
Outcomes
For the primary and secondary outcomes, the recommendations of the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT58 59) are considered. In addition, the ACT-related, other health-related indicators (eg, sleep quality or alcohol use), health-related quality of life, intervention costs and participants’ satisfaction will be assessed. Demographic and medical variables will be examined as potential moderating and mediating variables. For an overview of all variables and respective measures, see table 1. All questionnaires will be transferred to an online survey platform (https://www.unipark.com/) and completed online. The online administration of assessments can yield psychometric properties equivalent to their paper-and-pencil counterparts.60
Table 1.
Summary of assessments
| Instrument | Time of measurement | |||||||
| Screening | T0 | T1 | T2 | T3 | T4 | T5 | ||
| Screening instruments | ||||||||
| CPG | Chronic Pain Grade questionnaire | ✔ | ||||||
| Chronicity of Ppain | Pain longer for 6 months (yes/no) | ✔ | ||||||
| PHQ-9 | Patient Health Questionnaire | ✔ | ||||||
| Primary outcome | ||||||||
| MPI | Multidimensional Pain Inventory | ✔ | ✔* | ✔ | ✔ | ✔ | ✔ | |
| Secondary outcomes | ||||||||
| BPI | Brief Pain Inventory | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| QIDS-SR16 | Quick Inventory for Depressive Symptomatology | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| PSS-10 | Perceived Stress Scale | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| ISI | Insomnia Severity Index | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Numeric rating scale | Pain Intensity (0–10) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Perceived improvement | Improvement (1–7) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| AUDIT-10 | Alcohol Use Disorder Identification Test | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| GAD-7 | Generalized Anxiety Disorder | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Adapted items from CIDI 3.0, CIDI-SC and Epi-Q Screening Survey | Prevalence of major depression | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| AQoL-8D | Adjusted Quality of Life | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| SPE | Subjective prognosis of employment | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Intervention-related outcomes | ||||||||
| WAI-SR†, WAI-SRT‡ | Therapeutic relationship | ✔ | ✔ | |||||
| TAI-OT† | Technological alliance | ✔ | ✔ | |||||
| CSQ-I† | Patient satisfaction | ✔ | ||||||
| INEP† | Inventory of negative effects in psychotherapy | ✔ | ✔ | ✔ | ||||
| Negative effects† | Negative effects in psychotherapy | ✔ | ✔ | ✔ | ||||
| ACT-related variables | ||||||||
| CPAQ | Chronic Pain Acceptance Questionnaire | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| CFQ | Cognitive Fusion Questionnaire | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| CAQ | Committed Action Questionnaire | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Cost measurement | ||||||||
| TiC-P | Utilisation of health services, work-related productivity | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Other assessments | ||||||||
| Sociodemographics | ✔ | |||||||
| Predictors of major depression | ✔ | |||||||
| BDI-II | Suicidality item | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
T0, baseline; T1, 9 weeks after randomisation; T2, 6 months after randomisation; T3, 12 months after randomisation; T4, 24 months after randomisation; T5, 36 months after randomisation.
*The primary outcome is the standardised mean difference between intervention and control group at T1. MPI will also be assessed at T0 and T2–T5 as secondary outcome.
†To be completed by the intervention group only.
‡To be assessed by the e-coaches only.
Primary outcome
The primary outcome is the standardised mean difference in pain interference between the intervention group and the control group at 9 weeks after randomisation (T1), as measured by the Multidimensional Pain Interference scale (MPI).61 62 The MPI61 62 measures the degree in which everyday activities are hampered. The German version of the MPI consist of ten 7-point Likert items (0=“no interference/change” to 6=“extreme interference/change”). The internal consistency is reported to be excellent (α=0.94) and the retest reliability (r=0.78) as good.
Secondary outcomes
Pain interference
Pain interference will also be assessed with the MPI at T2–T5. As recommended by the IMMPACT,58 59 the Brief Pain Inventory (BPI)63 64 will additionally be used at T0–T5 for the assessment of pain interference. The BPI measures pain interference with regard to mood, sleep, social relations and enjoyment of life.63 64 On seven items, the participants have to indicate, on an 11-point scale (0=“does not interfere” to 10=“completely interferes”) to what extent the intensity of pain affects their daily functioning. The internal consistency of the scale (α=0.88) is reported to be good.65
Pain intensity
Pain intensity will be measured by an 11-point numerical rating scale. The participants have to rate their worst, least and average pain during the last week, from 0 (“no pain”) to 10 (“pain as bad as you can imagine”).
Overall improvement
The Patient Global Impression of Change scale (PGIC66) measures the participants’ global improvement after treatment on a 7-point scale from “very much improved” to “very much worse”.58 59
Depression
The German version of the Quick Inventory Depressive Symptomology (QIDS-SR1667) assesses the symptom severity of depression. The 16-item self-report inventory covers all nine DSM-5 symptom criteria of MDD. The QIDS-SR16 is characterised by high internal consistency of α=0.86.67 The items are rated on a 4-point scale ranging between 0 and 3. The total score ranges between 0 and 27, with a higher score indicating higher depressive symptom severity. The QIDS can also be analysed categorically: A comparison of QIDS-SR16 scores with current and lifetime diagnosis based on Structured Clinical Interview for DSM-IV-TR Axis I Disorders (SCID) as a measure criterion showed that QIDS-SR16 is a reliable screening instrument for the diagnosis of MDD.68 The cut-off scores of 13 and 14 yielded best results for sensitivity (76.5%) and specificity (81.8%), leading to correct classification of over 80% of the participants.68 In this study, the cut-off score in QIDS-SR16 for clinical depression is set to 13 for all measurement points within the 36-month follow-up period.
In addition, items which have been adapted from the Composite International Diagnosis Interview version 3.0 (CIDI 3.0), the Screening Scales (CIDI-SC)69 and the Epi-Q Screening Survey70 will be used to assess the prevalence of major depressive episode.
Stress
The 10-item version of the perceived stress scale (PSS-1071) measures the perception of stress in participants. The scale particularly assesses how “unpredictable, uncontrollable, and overloading respondents find their lives”.72 High values indicate a higher stress level. The German PSS-10 has a good reliability of ω=0.89.73 The PSS-10 will be modified to address the perception of stress load in the last week instead of the last month.
Anxiety
The seven-item Generalized Anxiety Disorder questionnaire (GAD-7)74 is a short self-report measure to assess the likelihood of generalised anxiety disorder. Higher values indicate higher anxiety levels. The German version has an excellent internal consistency of α=0.89.75 Overall, the GAD-7 is a valid and reliable instrument to screen for generalised anxiety disorder.74 75
Sleep quality
Sleep quality will be measured using the Insomnia Severity Index (ISI).76 The ISI is a brief self-report scale composed of seven items to identify clinical insomnia. Severe insomnia or low sleep quality is indicated by high values. The questionnaire has been validated in clinical and community samples and is characterised by high internal consistency (α=0.90 to 0.92) and adequate discriminative validity of the individual items.77 78 The German version of the ISI was validated in three different samples and shows satisfactory internal consistency across all three (α=0.76 in adolescents, α=0.77 in young adults, α=0.81 in adult workers).79
Alcohol consumption
The Alcohol Use Disorder Identification Test (AUDIT)80 will be applied to screen for hazardous alcohol use. The AUDIT has been validated in six different countries and therefore is cross-culturally applicable.81 82 The 10-item self-report questionnaire measures a unidimensional construct with adequate internal consistency ranging between α=0.80 and α=0.83.83 84 In the present study, the German Münster Version of the AUDIT following S3 guideline was applied.
Quality of life
Health-related quality of life will be assessed with the self-report questionnaire entitled Assessment of Quality of Life (AQoL-8D). This questionnaire consists of 35 items, covering the three physical dimensions ‘independent living’, ‘pain’ and ‘senses’, as well as the five psychosocial dimensions ‘mental health’, ‘happiness’, ‘coping’, ‘relationships’ and ‘self-worth’.85 Higher values indicate a lower quality of life. The AQoL-8D is characterised by a high Cronbach’s alpha of 0.96 and good psychometric properties85 and will be applied for cost-utility analyses.
Reliable change index
The procedure by Jacobson and Truax86 will be used to calculate reliable change in the primary outcome.87 Both reliable improvement and reliable deterioration will be calculated.
ACT-related variables
The German version of the Chronic Pain Acceptance Questionnaire (CPAQ88) will be used to measure psychological flexibility in respect to their activity engagement and pain willingness. Participants have to indicate their activity engagement (CPAQ-AE) and their pain willingness (CPAQ-PW) on a 20-item, 7-point Likert scale ranging from 0 (“never true”) to 6 (“always true”). The German CPAQ has a good internal consistency (α=0.84–0.87).
The German version of the Cognitive Fusion Questionnaire (CFQ-D)89 will be administered to measure cognitive fusion as a facet of psychological flexibility. The CFQ-D consists of seven items ranging from “never true (1)” to “always true (7)”. All items are positively keyed and higher scores indicate a higher level of cognitive fusion. The internal consistency is excellent (α=0.94).89
To measure the facet Committed Actions, the German version of the Committed Action Questionnaire (CAQ-D)90 will be used. The CAQ-D consists of 18 items: 9 positively keyed items and 9 negatively keyed items. Responders have to indicate to what extent an item applies to them on a scale from 0 (never true) to 6 (always true). The internal consistency of the CAQ-D is good (α=0.87).90
Work capacity
Work capacity will be measured with the German version of the Subjective Prognostic Employment Scale (SPE91). The SPE is a validated short self-report scale composed of three items with high internal consistency (Guttman scaling: rep=0.99). The SPE was developed to assess the subjective endangerment and prognosis of work capacity.91
Intervention-related variables
Intervention satisfaction will be assessed using the German version of the Client Satisfaction Questionnaire (CSQ-892; German version: ZUF-893), specifically adapted for assessing patient satisfaction with internet-based interventions (CSQ-I).94 The CSQ-8 is a self-report questionnaire consisting of eight items characterised by high internal consistency (α=0.93).92 The adapted German version CSQ-I has been validated for the assessment of patient satisfaction with internet-based interventions and is characterised by equally high internal consistency.94 95 CSQ-I will be applied to assess satisfaction with online trainings in the intervention group. An adapted version of the CSQ-I will be applied to the control group to evaluate satisfaction with information material.
The short version of the Working Alliance Inventory (WAI-SR)96 will be applied to measure the therapeutic alliance between the client and e-coach. The 12-item self-report questionnaire covers the three subscales: (a) agreement on tasks, (b) agreement on goals and (c) development of an affective bond. For the German version, internal consistencies between α=0.81 and α=0.91 were reported for the subscales and internal consistencies between α=0.90 and α=0.93 for the total score.96 97 The participants in the intervention group will complete the WAI-SR at T2 and T4. In addition, the e-coaches will be requested to complete the 10-item therapist version (WAI-SRT, developed by Adam O Horvath, http://wai.profhorvath.com/) at T1 and T3. This will allow us to compare how the therapeutic relationship is experienced by the client and e-coach, to gain a differentiated and comprehensive picture of the experienced working alliance. The WAI-SR and the WAI-SRT were adapted in wording for the current study investigating therapeutic alliance in guided internet-based interventions. The items were changed to refer to e-coaches instead of therapists and to online trainings instead of therapy. Due to a lack of suitable normative data for the interpretation of WAI, we will follow Jasper and colleagues98 for interpretation: the mean scores of WAI will be labelled as low (score: 1.00–2.44), medium (score: 2.45–3.44) and high/positive (score: 3.45–5.00).98
In addition, the Technological Alliance Inventory—Online Therapy (TAI-OT) will be administered to assess the technological alliance between the client and online intervention. The TAI-OT is a new self-report questionnaire developed by Labpsitec consisting of 12 items and measures the degree to which the online programme is perceived as helpful in achieving therapeutic goals (http://www.labpsitec.uji.es/esp/index.php). Based on the similarity between TAI and WAI, the same cut-off values for interpretation will be applied to TAI.98
Side effects of psychotherapy will be assessed with the Inventory for the Assessment of Negative Effects of Psychotherapy (INEP). The INEP records whether any negative changes, which are experienced during or after the treatment in the social and/or work environment, are attributed on the psychotherapeutic intervention.99 In this trial, an adapted 22-item version covering possible negative effects associated specifically with online trainings (eg, concerns about data protection) is applied.
In addition, an open question will be included for qualitative assessments of negative side effects of internet-based interventions. Participants will describe experienced negative events and side effects, their time of beginning, their frequency and their duration. Two further questions rate the negative impact of these events in the past and at present time.
Adherence, usage time and other usage variables will be tracked by the healthcare provider GET.ON. Under compliance with the European General Data Protection Regulation and data sharing agreements, these data will be merged with data from the assessments to explore the influence of adherence.
Cost measures
Cost evaluation will be based on the German version of the Dutch cost questionnaire entitled “Trimbos Institute and Institute of Medical Technology Questionnaire for Costs Associated with Psychiatric Illness” (TiC-P57). In this self-report questionnaire, the usage of healthcare services (eg, general practice services, sessions with psychotherapists or psychiatrists) and productivity loss (eg, hospital days, absenteeism, presenteeism) are registered. The German version of the TiC-P has been used in several studies (eg, see refs. 48 100 101). We adapted the questionnaire for the population of agriculturists, foresters and horticulturists.
Covariates
As potential moderating variables, demographic information (eg, gender, age, education), information about the agricultural farm (eg, farm size, area cultivated, number of workers) and about the situation of the entrepreneurial family (eg, financial situation, number of relatives living and working together, general work load) will be recorded at baseline. Furthermore, a variety of predictors (eg, personality, prior experience of violence and aggression, childhood experiences) will be included to assess relevant factors for development of depressive symptomology.
Screening instruments
Three questionnaires are used in the screening. The German version of the Patient Health Questionnaire (PHQ-9102) will be administered as a depression screening inventory to detect subthreshold depression (PHQ-9 ≥5). The PHQ-9 consists of nine items on a 4-point scale with a rating scale ranging from 0 to 3 (0=“not at all”, 1=“several days”, 2=“more than half the days”, 3=“nearly every day”). Each item assesses one symptom of MDD. In addition, an item is included to register severity of daily life limitations associated with depressive symptoms. The computerised version (α=0.88) of the PHQ-9 shows an equally high internal consistency as the paper-and-pencil version (α=0.89).103
In addition to the PHQ-9, the BDI-II suicide item is used,104 if applicants show elevated suicidal tendency based on PHQ-9 or QIDS-SR16 (see previous section for a more detailed procedure on suicidal tendency). On the BDI-II item, applicants have to indicate their suicidality on a 3-point scale (BDI-II item 1: “I have thoughts of killing myself, but I would not carry them out”; BDI-II item 2: “I would like to kill myself”; BDI-II item 3: “I would kill myself if I had the chance”).
During the screening phase, considerable pain intensity will be measured with the German version of the Chronic Pain Grade questionnaire (CPG).43 The CPG is a 14-item questionnaire assessing pain disability and pain intensity. Based on the disability and intensity score, four different severity grades are given (see refs. 43 44). The internal consistency of the German CPG is good (α=0.82).
Statistical analyses
All statistical analyses will be conducted on an intention-to-treat basis. In addition, for those who have adhered to the study protocol and completed most of the intervention sessions (at least 80% of the modules), per-protocol (PP) analyses will be conducted. Data will be checked regarding missingness patterns and mechanisms. Analyses will be adapted accordingly (eg, using multiple imputation by chained equations105). All analyses will be carried out in the software R.106 R packages used for analyses will be reported.
Clinical analyses
Regression analyses will be used as the primary method. The estimates will be adjusted for baseline in regression analyses. For continuous outcomes (eg, pain interference), linear regression will be carried out, and for dichotomous outcomes (eg, onset of depression), a Poisson regression will be employed. To quantify the effect of the intervention, group allocation will be inserted as a dummy-coded predictor in all regression models. All continuous variables will be z-standardised. Based on the data structure, the regression analyses will be adjusted (eg, use of robust estimation or use of multilevel regression analysis in case of substantial intraclass correlation). For all analyses, the alpha level will be set at 5%. Except for the primary outcome, two-sided tests will be conducted. Between-group differences in the primary outcome will also be tested without an adjustment for baseline using one-sided t-test (see power calculation) and the effect size will be reported as Cohen’s d.
Moderator and mediator analyses
To examine for whom the intervention is best suited, exploratory moderation analyses will be conducted. Based on previous findings,37 38 sociodemographic (eg, gender) and health-related variables (eg, baseline pain severity) will be analysed. Regression analyses will be employed wherein group will be included as a dummy-coded predictor along with the main effects of moderator variables and their interaction effects with group. Effect coding will be applied to all categorical (eg, gender) and z-standardisation to continuous (eg, baseline severity) moderator variables. Each moderator will be tested in a separate regression model. Furthermore, a holistic model including all identified moderators will be estimated.
Mediation analyses will be conducted to examine which processes might explain effects on health outcomes. The three facets of psychological flexibility (acceptance, cognitive fusion and committed action) measured within the present study as well as therapeutic alliance and technological alliance will be analysed as potential mediators. We will use time-lagged mediation models according to Cole and Maxwell.107
Economic evaluation
The health-economic evaluation will include both a cost-effectiveness analysis (CEA) and cost-utility analysis (CUA). The evaluation will be performed from a societal (eg, all relevant costs) and a public healthcare perspective (eg, only direct medical costs) within a time horizon of 12 and 36 months. Missing cost and effect data will be imputed, using multiple imputation by chained equations. The CEA will include an assessment of the value of PACT-A by calculating the difference in costs between PACT-A and TAU+ and dividing this by the difference in effectiveness of both treatment options (ie, calculating the incremental cost-effectiveness ratio (ICER)). In the CUA, the ICER will be expressed as incremental costs per quality-adjusted life year gained.108 109 We will use a SURE model (seemingly unrelated regression equations) to allow for correlated residuals of the cost and effect equations while adjusting for potential confounders (eg, baseline differences in utility scores).110 Based on these non-parametric bootstrapped simulations of the ICER (n=2500), 95% bias-corrected and accelerated CIs will be obtained for incremental costs and effects, respectively, as well as a 95% CI around the ICER based on the bootstrap acceptability method.111 The bootstrapped ICERs will be graphically displayed in a cost-effectiveness plane and will also be shown in a cost-effective acceptability curve disclosing the probability that PACT-A is cost-effective for a range of willingness-to-pay thresholds.112 To test the robustness of the base-case findings, probabilistic sensitivity analyses will be done by changing several assumptions made in the base-case scenario (eg, about cost prices and volumes). An incremental net benefit regression analysis will be performed to ascertain which sub-groups benefit more from PACT-A in terms of superior cost-effectiveness.110 113
Discussion
In this study protocol, we described the design of a pragmatic randomised controlled trial, which evaluates the (cost-)effectiveness of a guided internet-based Acceptance and Commitment Therapy for chronic pain in individuals in green professions. The effects of the intervention on pain and mental health–related outcomes will be evaluated over an extensive follow-up period of 36 months. The preventive effects on mental health (eg, depression) in individuals with chronic pain in agricultural professions will be investigated.
Some limitations of the current study should be considered. Prior studies have shown that IMIs mainly reach female and higher-educated individuals.51 114 If only female and highly educated people from the target population were reached, the generalisability of this study would be restricted. To overcome this selection bias, we are using multiple recruitment strategies to reach individuals from both sexes and different educational levels. In addition, prior studies have shown that IMIs are frequently accompanied by high drop-out rates,36 50 51 which can lead to a significant loss of power. Studies with guided IMIs show a lower drop-out rate.115 To avoid a loss of power, the sample size was increased for this study, taking into account an average drop-out rate of 31%.50 However, drop-out rates vary greatly across studies.36 50 51 Consequently, this study could also be underpowered or overpowered. To quantify the actual power, post hoc power analyses will be conducted and the limitations on the statistical power of the study will be discussed.
Within this study, categorical analyses are planned to determine the prevalence and incidence of MDD within the target population. For this purpose, cut-off scores for QIDS will be used to define MDD (eg, QIDS ≥13 indicates MDD68). In-depth diagnostic procedures (eg, SCID interviews) could be more reliable to determine such estimates, but these are rather cost and time intensive. As a result, the reliability of the prevalence and incidence rates estimated in this study might be restricted. Using a QIDS cut-off score of 13 leads to high sensitivity (76.5%), high specificity (81.8%) and a correct classification of 80.7%.68 Moreover, it has been shown that SCID interviews have a rather fair than perfect inter-rater reliability (kappa=0.66).116 Thus, instead of the SCID, we consider the use of QIDS as a more appropriate approach, especially since the results will also be verified by items adapted from the CIDI69 and Epi-Q Screening Survey.70 The absence of structured clinical interviews should also be discussed regarding the inclusion criteria of chronic pain. Chronic pain is only assessed by self-report measurements (CPG+duration greater than 6 months). Since self-reported chronic pain is not verified by a clinical diagnosis, it can be questioned whether the participants included in this study indeed suffer from chronic pain. The International Association for the Study of Pain highlighted that chronic pain is seen as a disease of its own right which can be validly assessed by self-report measures and targeted, despite the individual syndromes of a person.117 Thus, the self-report is assumed to be a suitable measurement for chronic pain. Lastly, based on the strong evidence for the effectiveness of internet-based and ACT-based interventions as well as results indicating a moderate effectiveness of a previous version of the present intervention, we strongly assume effects on pain interference. Thus, we decided on a one-sided testing of the primary outcome.36 51 53 As commonly done in psychotherapeutic research, we set the alpha level at 5% for all analyses. However, the alpha could have been set to 2.5% for the one-sided test to provide a more conservative evaluation of the primary outcome.
Despite the limitations, several strengths should be highlighted. First, to our best knowledge, the effectiveness of ACT-based IMIs on chronic pain in green professions is currently obscure. This study will give first insights into the effectiveness of ACT-based IMIs in green professions with chronic pain. Moreover, by the long-term evaluation, this study can make a valuable addition to the current evidence base. Besides positive effects (eg, decrease of symptom severity), IMIs can also have negative side effects (eg, time pressure).118 This study will also investigate the negative short-term and long-term effects of the IMI.
Another major strength of this study results from the routine care setting. Based on the setting, this study will provide information about the actual effectiveness of this IMI rather than its efficacy. It can be argued that this study is characterised by a strong external validity, regarding the generalisability of the intervention to green professions in Germany. In this vein, this study will give a robust estimation of the (cost-)effectiveness, side effects, treatment satisfaction and feasibility in the target population of green professions.
Lastly, the investigation of potential effect moderators and mediators should be highlighted. Those analyses will give us a deeper understanding of how certain processes contribute to the effectiveness of ACT-based IMIs as well as for whom this IMI is best suited for.
Overall, we consider the present study design well suited to answer the stated research questions. Based on the existing evidence,32 36 48 we expect the IMI to be effective. If proven to be so, the implementation of this IMI in the preventive routine care of green professions will have a major impact on public health.
Ethics and dissemination
Written informed consent for participation in the study will be obtained from all participants prior to their involvement. All participant information will be stored securely in locked file cabinets and/or password protected in a secured cloud storage with restricted access. All reports, data collection and administrated forms will be only identified by a coded ID number to maintain participant confidentiality. All records that contain names or other personal identifiers, such as informed consent forms, will be stored separately from the study records identified by ID number. Listings that link participant ID numbers to other identifying information will be stored in separate, password-protected files with limited access.
Trial results will be presented on international conferences and published in peer-reviewed journals. Central results will be communicated to SVLFG and can be further used by SVLFG for information of insured persons, dissemination of health offers and further improvement of the healthcare offer.
Trial status
The trial is currently ongoing. The recruitment of participants has started in January 2018.
Supplementary Material
Acknowledgments
The authors would like to thank Jiaxi Lin, Ann-Marie Küchler, Friederike Reiner and Viktoria Egle for their supporting role in the decision for and preparation of outcomes. Also, the authors thank Natalie Trzebecki, Julia Schüngel and Sofia Ring for the creation of the Unipark survey and their engagement in enrolling and supporting the participants throughout the study as well as the GET.ON staff who provided details on their intervention. Furthermore, we thank Ayca Ilgaz for proofreading the manuscript.
Footnotes
Contributors: HB and DE obtained funding for this study. HB, DE, IT, LB and YT contributed to the study design. CB contributed to the design of the economic evaluation study. LB and YT are responsible for the recruitment and coordination of the trials. JT and JF supported the recruitment. YT drafted the manuscript. IT was responsible for the development of the suicidal procedure/TAU+ psychoeducation material, and is currently supervising the recruitment process and conduct of the trials as the operational project lead. All authors provided critical revision of the article. All authors approved the final manuscript.
Funding: The insurance company SVLFG gave a financial contribution the University of Ulm and Friedrich- Alexander University of Erlangen-Nürnberg as expense allowance.
Disclaimer: The SVLFG had no role in the study design, decision to publish or preparation of this manuscript. The SVLFG will not be involved in data collection, the analyses, decision to publish or the preparation of future papers regarding this study.
Competing interests: DE reports to have received consultancy fees and served in the scientific advisory board of several companies such as Minddistrict, Lantern, Novartis, Sanofi, Schoen Kliniken and German health insurance companies. He is a stakeholder of the Institute for Online Health Training (GET.ON), which aims to implement scientific findings related to digital health interventions into routine care. HB reports to have received consultancy fees and fees for lectures/workshops from chambers of psychotherapists and training institutes for psychotherapists in the e-mental-health context. IT reports to have received fees for lectures/workshops in the e-mental-health context from educational institutions for psychotherapists. She is implementation lead and project lead for the EU-research project ImpleMentAll at the Institute for Online Health Training (GET.ON). All other authors reported no competing interests.
Patient and public involvement: Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Patient consent for publication: Not required.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1.Breivik H, Collett B, Ventafridda V, et al. Survey of chronic pain in Europe: prevalence, impact on daily life, and treatment. Eur J Pain 2006;10:287–333. 10.1016/j.ejpain.2005.06.009 [DOI] [PubMed] [Google Scholar]
- 2.Breivik H, Eisenberg E, O’Brien T, et al. The individual and societal burden of chronic pain in Europe: the case for strategic prioritisation and action to improve knowledge and availability of appropriate care. BMC Public Health 2013;13:1229. 10.1186/1471-2458-13-1229 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Demyttenaere K, Bonnewyn A, Bruffaerts R, et al. Comorbid painful physical symptoms and anxiety: prevalence, work loss and help-seeking. J Affect Disord 2008;109:264–72. 10.1016/j.jad.2007.12.231 [DOI] [PubMed] [Google Scholar]
- 4.Rayner L, Hotopf M, Petkova H, et al. Depression in patients with chronic pain attending a specialised pain treatment centre: prevalence and impact on health care costs. Pain 2016;157:1472–9. 10.1097/j.pain.0000000000000542 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.van Hecke O, Torrance N, Smith BH. Chronic pain epidemiology and its clinical relevance. Br J Anaesth 2013;111:13–18. 10.1093/bja/aet123 [DOI] [PubMed] [Google Scholar]
- 6.Häuser W, Schmutzer G, Hinz A, et al. Prävalenz chronischer Schmerzen in Deutschland: Befragung einer repräsentativen Bevölkerungsstichprobe. Schmerz 2013;27:46–55. [DOI] [PubMed] [Google Scholar]
- 7.Hoffman PK, Meier BP, Council JR. A comparison of chronic pain between an urban and rural population. J Community Health Nurs 2002;19:213–24. 10.1207/S15327655JCHN1904_02 [DOI] [PubMed] [Google Scholar]
- 8.Bair MJ, Robinson RL, Katon W, et al. Depression and pain comorbidity: a literature review. Arch Intern Med 2003;163:2433–45. 10.1001/archinte.163.20.2433 [DOI] [PubMed] [Google Scholar]
- 9.Surah A, Baranidharan G, Morley S. Chronic pain and depression. Contin Educ Anaesthesia Crit Care Pain 2014;14:85–9. 10.1093/bjaceaccp/mkt046 [DOI] [Google Scholar]
- 10.Outcalt SD, Kroenke K, Krebs EE, et al. Chronic pain and comorbid mental health conditions: independent associations of posttraumatic stress disorder and depression with pain, disability, and quality of life. J Behav Med 2015;38:535–43. 10.1007/s10865-015-9628-3 [DOI] [PubMed] [Google Scholar]
- 11.Tegethoff M, Belardi A, Stalujanis E, et al. Comorbidity of mental disorders and chronic pain: chronology of onset in adolescents of a national representative cohort. J Pain 2015;16:1054–64. 10.1016/j.jpain.2015.06.009 [DOI] [PubMed] [Google Scholar]
- 12.World Health Organization The global burden of disease: 2004 update. 2004 Updat 2008;146. [Google Scholar]
- 13.Abajobir AA, Abate KH, Abbafati C, et al. Global, regional, and national disability-adjusted life-years (DALYs) for 333 diseases and injuries and healthy life expectancy (HALE) for 195 countries and territories, 1990–2016: a systematic analysis for the global burden of disease study 2016. Lancet 2017;390:1260–344. 10.1016/S0140-6736(17)32130-X [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Trautmann S, Rehm J, Wittchen H-U. The economic costs of mental disorders: do our societies react appropriately to the burden of mental disorders? EMBO Rep 2016;17:1245–9. 10.15252/embr.201642951 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kumar N. WHO normative guidelines on pain management report of a Delphi study to determine the need for guidelines that should be developed by WHO. World Heal Organ 2007:1–50. [Google Scholar]
- 16.Kerns RD, Sellinger J, Goodin BR. Psychological treatment of chronic pain. Annu Rev Clin Psychol 2011;7:411–34. 10.1146/annurev-clinpsy-090310-120430 [DOI] [PubMed] [Google Scholar]
- 17.Traue H, Horn AB, Kessler H, et al. Psyche und Schmerz : Baron R, Koppert W, Strumpf M, et al., Praktische Schmerzmedizin. Interdisziplinäre Diagnostik - Multimodale Therapie, 2013: 117–28. [Google Scholar]
- 18.Turk DC, Wilson HD, Cahana A. Treatment of chronic non-cancer pain. Lancet 2011;377:2226–35. 10.1016/S0140-6736(11)60402-9 [DOI] [PubMed] [Google Scholar]
- 19.Veehof MM, Oskam M-J, Schreurs KMG, et al. Acceptance-based interventions for the treatment of chronic pain: a systematic review and meta-analysis. Pain 2011;152:533–42. 10.1016/j.pain.2010.11.002 [DOI] [PubMed] [Google Scholar]
- 20.Hayes SC, Luoma JB, Bond FW, et al. Acceptance and commitment therapy: model, processes and outcomes. Behav Res Ther 2006;44:1–25. 10.1016/j.brat.2005.06.006 [DOI] [PubMed] [Google Scholar]
- 21.Tripp DA, VanDenKerkhof EG, McAlister M. Prevalence and determinants of pain and pain-related disability in urban and rural settings in southeastern Ontario. Pain Res Manag 2006;11:225–33. 10.1155/2006/720895 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Min D, Baek S, Park H-W, et al. Prevalence and characteristics of musculoskeletal pain in Korean farmers. Ann Rehabil Med 2016;40:1–13. 10.5535/arm.2016.40.1.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Meucci RD, Fassa AG, Faria NMX, et al. Chronic low back pain among tobacco farmers in southern Brazil. Int J Occup Environ Health 2015;21:66–73. 10.1179/2049396714Y.0000000094 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Simkin S, Hawton K, Fagg J, et al. Stress in farmers: a survey of farmers in England and Wales. Occup Environ Med 1998;55:729–34. 10.1136/oem.55.11.729 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Onwuameze OE, Paradiso S, Peek-Asa C, et al. Modifiable risk factors for depressed mood among farmers. Ann Clin Psychiatry 2013;25:83–90. [PubMed] [Google Scholar]
- 26.Judd F, Jackson H, Fraser C, et al. Understanding suicide in Australian farmers. Soc Psychiatry Psychiatr Epidemiol 2006;41:1–10. 10.1007/s00127-005-0007-1 [DOI] [PubMed] [Google Scholar]
- 27.Prunuske JP, St. Hill CA, Hager KD, et al. Opioid prescribing patterns for non-malignant chronic pain for rural versus non-rural US adults: a population-based study using 2010 NAMCS data. BMC Health Serv Res 2014;14:1–8. 10.1186/s12913-014-0563-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Eaton LH, Langford DJ, Meins AR, et al. Use of self-management interventions for chronic pain management: a comparison between rural and nonrural residents. Pain Manag Nurs 2018;19:8–13. 10.1016/j.pmn.2017.09.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sachverständigenrat zur Begutachtung der Entwicklung im Gesundheitswesen Bedarfsferechte Versorgung - Perspektiven für ländliche Regionen und ausgewählte Leistungsbereiche Gutachten 2014. Verlag Hans Huber 2014. [Google Scholar]
- 30.Ebert DD, Van Daele T, Nordgreen T, et al. Internet- and mobile-based psychological interventions: applications, efficacy, and potential for improving mental health. Eur Psychol 2018;23:167–87. 10.1027/1016-9040/a000318 [DOI] [Google Scholar]
- 31.Lin J, Baumeister H. Internet- und mobilebasierte interventionen in der psychotherapie. Public Heal Forum 2015;23:176–9. 10.1515/pubhef-2015-0063 [DOI] [Google Scholar]
- 32.Sander L, Rausch L, Baumeister H. Effectiveness of Internet-based interventions for the prevention of mental disorders: a systematic review and meta-analysis. JMIR Ment Health 2016;3:e38 10.2196/mental.6061 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Andersson G. Internet-delivered psychological treatments. Annu Rev Clin Psychol 2016;12:157–79. 10.1146/annurev-clinpsy-021815-093006 [DOI] [PubMed] [Google Scholar]
- 34.Ebert DD, Cuijpers P, Muñoz RF, et al. Prevention of mental health disorders using internet- and mobile-based interventions: a narrative review and recommendations for future research. Front Psychiatry 2017;8:116. 10.3389/fpsyt.2017.00116 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.van Zoonen K, Buntrock C, Ebert DD, et al. Preventing the onset of major depressive disorder: a meta-analytic review of psychological interventions. Int J Epidemiol 2014;43:318–29. 10.1093/ije/dyt175 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Buhrman M, Gordh T, Andersson G. Internet interventions for chronic pain including headache: a systematic review. Internet Interv 2016;4:17–34. 10.1016/j.invent.2015.12.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.DasMahapatra P, Chiauzzi E, Pujol LM, et al. Mediators and moderators of chronic pain outcomes in an online self-management program. Clin J Pain 2015;31:404–13. 10.1097/AJP.0000000000000125 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Karyotaki E, Kleiboer A, Smit F, et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol Med 2015;45:2717–26. 10.1017/S0033291715000665 [DOI] [PubMed] [Google Scholar]
- 39.Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332. 10.1136/bmj.c332 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. 10.1136/bmj.c869 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Proudfoot J, Klein B, Barak A, et al. Establishing guidelines for executing and reporting internet intervention research. Cogn Behav Ther 2011;40:82–97. 10.1080/16506073.2011.573807 [DOI] [PubMed] [Google Scholar]
- 42.Merskey H, Bogduk N. Classification of chronic pain, 1994. [Google Scholar]
- 43.Klasen BW, Hallner D, Schaub C, et al. Validation and reliability of the German version of the chronic pain grade questionnaire in primary care back pain patients. Psychosoc Med 2004;1:Doc07. [PMC free article] [PubMed] [Google Scholar]
- 44.Von Korff M, Ormel J, Keefe FJ, et al. Grading the severity of chronic pain. Pain 1992;50:133–49. 10.1016/0304-3959(92)90154-4 [DOI] [PubMed] [Google Scholar]
- 45.Lin J, Lüking M, Ebert DD, et al. Effectiveness and cost-effectiveness of a guided and unguided internet-based acceptance and commitment therapy for chronic pain: study protocol for a three-armed randomised controlled trial. Internet Interv 2015;2:7–16. 10.1016/j.invent.2014.11.005 [DOI] [Google Scholar]
- 46.Braun L, Titzler I, Ebert DD, et al. Clinical and cost-effectiveness of guided internet-based interventions in the indicated prevention of depression in green professions (PROD-A): study protocol of a 36-month follow-up pragmatic randomized controlled trial. BMC Psychiatry 2019;19:278. 10.1186/s12888-019-2244-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Green KL, Brown GK, Jager-Hyman S, et al. The predictive validity of the Beck depression inventory suicide item. J Clin Psychiatry 2015;76:1683–6. 10.4088/JCP.14m09391 [DOI] [PubMed] [Google Scholar]
- 48.Sander L, Paganini S, Lin J, et al. Effectiveness and cost-effectiveness of a guided Internet- and mobile-based intervention for the indicated prevention of major depression in patients with chronic back pain—study protocol of the PROD-BP multicenter pragmatic RCT. BMC Psychiatry 2017;17:1–13. 10.1186/s12888-017-1193-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Faul F, Erdfelder E, Buchner A, et al. Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods 2009;41:1149–60. 10.3758/BRM.41.4.1149 [DOI] [PubMed] [Google Scholar]
- 50.Melville KM, Casey LM, Kavanagh DJ. Dropout from Internet-based treatment for psychological disorders. Br J Clin Psychol 2010;49:455–71. 10.1348/014466509X472138 [DOI] [PubMed] [Google Scholar]
- 51.Lin J, Paganini S, Sander L, et al. An internet-based intervention for chronic pain. Dtsch Aerzteblatt Online 2017;114:681–8. 10.3238/arztebl.2017.0681 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Hayes SC, Strosahl K, Wilson KG. Acceptance and commitment therapy: an experiential approach to behavior change. Guilford Press, 1999. [Google Scholar]
- 53.Veehof MM, Trompetter HR, Bohlmeijer ET, et al. Acceptance- and mindfulness-based interventions for the treatment of chronic pain: a meta-analytic review. Cogn Behav Ther 2016;45:5–31. 10.1080/16506073.2015.1098724 [DOI] [PubMed] [Google Scholar]
- 54.Gold SM, Enck P, Hasselmann H, et al. Control conditions for randomised trials of behavioural interventions in psychiatry: a decision framework. Lancet Psychiatry 2017;4:725–32. 10.1016/S2215-0366(17)30153-0 [DOI] [PubMed] [Google Scholar]
- 55.Salvetti MdeG, Cobelo A, Vernalha PdeM, et al. Effects of a psychoeducational program for chronic pain management. Rev Lat Am Enfermagem 2012;20:896–902. 10.1590/s0104-11692012000500011 [DOI] [PubMed] [Google Scholar]
- 56.Hakkaart-van Roijen L, van Straten A, Donker M, et al. Trimbos/iMTA questionnaire for costs associated with psychiatric illness (TIC-P). Rotterdam Inst Med Technol Assess 2002. [Google Scholar]
- 57.Bouwmans C, De Jong K, Timman R, et al. Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss in patients with a psychiatric disorder (TiC-P). BMC Health Serv Res 2013;13:217. 10.1186/1472-6963-13-217 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Dworkin RH, Turk DC, Wyrwich KW, et al. Interpreting the clinical importance of treatment outcomes in chronic pain clinical trials: IMMPACT recommendations. J Pain 2008;9:105–21. 10.1016/j.jpain.2007.09.005 [DOI] [PubMed] [Google Scholar]
- 59.Dworkin RH, Turk DC, Farrar JT, et al. Core outcome measures for chronic pain clinical trials: IMMPACT recommendations. Pain 2005;113:9–19. [DOI] [PubMed] [Google Scholar]
- 60.Buchanan T. Internet-based questionnaire assessment: appropriate use in clinical contexts. Cogn Behav Ther 2003;32:100–9. 10.1080/16506070310000957 [DOI] [PubMed] [Google Scholar]
- 61.Kerns RD, Turk DC, Rudy TE. The West Haven-Yale Multidimensional Pain Inventory (WHYMPI). Pain 1985;23:345–56. 10.1016/0304-3959(85)90004-1 [DOI] [PubMed] [Google Scholar]
- 62.Flor H, Rudy TE, Birbaumer N, et al. Zur Anwendbarkeit des West Haven-Yale multidimensional pain Inventory im deutschen Sprachraum - Daten zur Reliabilität und Validität des MPI-D. Der Schmerz 1990;4:82–7. [DOI] [PubMed] [Google Scholar]
- 63.Cleeland CS, Ryan KM. Pain assessment: global use of the brief pain inventory. Ann Acad Med Singapore 1994;23:129–38. [PubMed] [Google Scholar]
- 64.Radbruch L, Loick G, Kiencke P, et al. Validation of the German version of the brief pain inventory. J Pain Symptom Manage 1999;18:180–7. 10.1016/S0885-3924(99)00064-0 [DOI] [PubMed] [Google Scholar]
- 65.Keller S, Bann CM, Dodd SL, et al. Validity of the brief pain inventory for use in documenting the outcomes of patients with noncancer pain. Clin J Pain 2004;20:309–18. 10.1097/00002508-200409000-00005 [DOI] [PubMed] [Google Scholar]
- 66.Guy W. Clinical global impression scale (CGI), 1976. [Google Scholar]
- 67.Rush AJ, Trivedi MH, Ibrahim HM, et al. The 16-item quick inventory of depressive symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol Psychiatry 2003;54:573–83. 10.1016/S0006-3223(02)01866-8 [DOI] [PubMed] [Google Scholar]
- 68.Lamoureux BE, Linardatos E, Fresco DM, et al. Using the QIDS-SR16 to identify major depressive disorder in primary care medical patients. Behav Ther 2010;41:423–31. 10.1016/j.beth.2009.12.002 [DOI] [PubMed] [Google Scholar]
- 69.Kessler RC, Ustün TB. The World Mental Health (WMH) survey initiative version of the World Health Organization (WHO) composite international diagnostic interview (CIDI). Int J Methods Psychiatr Res 2004;13:93–121. 10.1002/mpr.168 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70.Kessler RC, Farley PA, Gruber M, et al. PMH72 concordance of computerized self-report measures of DSM-IV-TR mood and anxiety disorders compared to gold standard clinical assessments in primary care. Value Heal 2010;13:A118 10.1016/S1098-3015(10)72567-5 [DOI] [Google Scholar]
- 71.Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav 1983;24:385–96. 10.2307/2136404 [DOI] [PubMed] [Google Scholar]
- 72.Cohen S, Janicki-Deverts D. Who’s stressed? Distributions of psychological stress in the United States in probability samples from 1983, 2006, and 2009. J Appl Soc Psychol 2012;42:1320–34. [Google Scholar]
- 73.Reis D, Lehr D, Heber E, et al. The German version of the Perceived Stress Scale (PSS-10): evaluation of dimensionality, validity, and measurement invariance with exploratory and confirmatory bifactor modeling. Assessment 2019;26:1246–59. 10.1177/1073191117715731 [DOI] [PubMed] [Google Scholar]
- 74.Spitzer RL, Kroenke K, Williams JBW, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006;166:1092–7. 10.1001/archinte.166.10.1092 [DOI] [PubMed] [Google Scholar]
- 75.Löwe B, Decker O, Müller S, et al. Validation and standardization of the generalized anxiety disorder screener (GAD-7) in the general population. Med Care 2008;46:266–74. 10.1097/MLR.0b013e318160d093 [DOI] [PubMed] [Google Scholar]
- 76.Bastien CH, Vallières A, Morin CM. Validation of the insomnia severity index as an outcome measure for insomnia research. Sleep Med 2001;2:297–307. 10.1016/S1389-9457(00)00065-4 [DOI] [PubMed] [Google Scholar]
- 77.Morin CM, Belleville G, Bélanger L, et al. The insomnia severity index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep 2011;34:601–8. 10.1093/sleep/34.5.601 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 78.Gagnon C, Bélanger L, Ivers H, et al. Validation of the insomnia severity index in primary care. J Am Board Fam Med 2013;26:701–10. 10.3122/jabfm.2013.06.130064 [DOI] [PubMed] [Google Scholar]
- 79.Gerber M, Lang C, Lemola S, et al. Validation of the German version of the insomnia severity index in adolescents, young adults and adult workers: results from three cross-sectional studies. BMC Psychiatry 2016;16:174. 10.1186/s12888-016-0876-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 80.Babor T, Higgins-Biddle JC, Saunders JB, et al. The alcohol use disorders identification test: guidelines for use in primary care. Geneva World Heal Organ 2001:1–40. [Google Scholar]
- 81.Saunders JB, Aasland OG, Amundsen A, et al. Alcohol consumption and related problems among primary health care patients: WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption—I. Addiction 1993;88:349–62. 10.1111/j.1360-0443.1993.tb00822.x [DOI] [PubMed] [Google Scholar]
- 82.Saunders JB, Aasland OG, Babor TF, et al. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption—II. Addiction 1993;88:791–804. 10.1111/j.1360-0443.1993.tb02093.x [DOI] [PubMed] [Google Scholar]
- 83.Fleming MF, Barry KL, MacDonald R. The Alcohol Use Disorders Identification Test (AUDIT) in a college sample. Int J Addict 1991;26:1173–85. 10.3109/10826089109062153 [DOI] [PubMed] [Google Scholar]
- 84.Hays RD, Merz JF, Nicholas R. Response burden, reliability, and validity of the CAGE, Short MAST, and AUDIT alcohol screening measures. Behav Res Methods, Instruments, Comput 1995;27:277–80. 10.3758/BF03204745 [DOI] [Google Scholar]
- 85.Richardson J, Iezzi A, Khan MA, et al. Validity and reliability of the Assessment of Quality of Life (AQoL)-8D Multi-Attribute Utility Instrument. Patient - Patient-Centered Outcomes Res 2014;7:85–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86.Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12–19. 10.1037/0022-006X.59.1.12 [DOI] [PubMed] [Google Scholar]
- 87.Lambert MJ. Outcome in psychotherapy: the past and important advances. Psychotherapy 2013;50:42–51. 10.1037/a0030682 [DOI] [PubMed] [Google Scholar]
- 88.Nilges P, Köster B, Schmidt CO. Schmerzakzeptanz – Konzept und Überprüfung einer deutschen Fassung des Chronic Aain Acceptance Questionnaire. Schmerz 2007;21:57–67. 10.1007/s00482-006-0508-1 [DOI] [PubMed] [Google Scholar]
- 89.China C, Hansen LB, Gillanders DT, et al. Concept and validation of the German version of the cognitive fusion questionnaire (CFQ-D). J Contextual Behav Sci 2018;9:30–5. 10.1016/j.jcbs.2018.06.003 [DOI] [Google Scholar]
- 90.Terhorst Y, Baumeister H, McCracken LM, et al. Psychological flexibility—validation of the German Committed Action Questionnaire. Prep. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 91.Mittag O, Raspe H. Eine kurze Skala Zur Messung Der subjektiven Prognose Der Erwerbstätigkeit: Ergebnisse einer Untersuchung an 4279 Mitgliedern Der gesetzlichen Arbeiterrentenversicherung zu Reliabilität (Guttman-Skalierung) und Validität Der Skala. Rehabilitation 2003;42:169–74. [DOI] [PubMed] [Google Scholar]
- 92.Attkisson CC, Zwick R. The client satisfaction questionnaire. Eval Program Plann 1982;5:233–7. 10.1016/0149-7189(82)90074-X [DOI] [PubMed] [Google Scholar]
- 93.Schmidt J, Lamprecht F, Wittmann WW. [Satisfaction with inpatient management. Development of a questionnaire and initial validity studies]. Psychother Psychosom Med Psychol 1989;39:248–55. [PubMed] [Google Scholar]
- 94.Boß L, Lehr D, Reis D, et al. Reliability and validity of assessing user satisfaction with web-based health interventions. J Med Internet Res 2016;18:1–13. 10.2196/jmir.5952 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 95.Thiart H, Lehr D, Ebert DD, et al. Log in and breathe out: Internet-based recovery training for sleepless employees with work-related strain—results of a randomized controlled trial. Scand J Work Environ Health 2015;41:164–74. 10.5271/sjweh.3478 [DOI] [PubMed] [Google Scholar]
- 96.Wilmers F, Munder T, Leonhart R, et al. Die deutschsprachige version des working alliance inventory – short revised (WAI-SR) – ein schulenübergreifendes, ökonomisches und empirisch validiertes instrument Zur Erfassung Der therapeutischen Allianz. Klin Diagnostik und Eval 2008;1:343–58. [Google Scholar]
- 97.Munder T, Wilmers F, Leonhart R, et al. Working Alliance Inventory-Short Revised (WAI-SR): psychometric properties in outpatients and inpatients. Clin Psychol Psychother 2010;17:231–9. 10.1002/cpp.658 [DOI] [PubMed] [Google Scholar]
- 98.Jasper K, Weise C, Conrad I, et al. The working alliance in a randomized controlled trial comparing Internet-based self-help and face-to-face cognitive behavior therapy for chronic tinnitus. Internet Interv 2014;1:49–57. 10.1016/j.invent.2014.04.002 [DOI] [Google Scholar]
- 99.Ladwig I, Rief W, Nestoriuc Y. Welche Risiken und Nebenwirkungen hat Psychotherapie? - Entwicklung des Inventars zur Erfassung Negativer Effekte von Psychotherapie (INEP). Verhaltenstherapie 2014;24:252–63. 10.1159/000367928 [DOI] [Google Scholar]
- 100.Lin J, Sander L, Paganini S, et al. Effectiveness and cost-effectiveness of a guided internet- and mobile-based depression intervention for individuals with chronic back pain: protocol of a multi-centre randomised controlled trial. BMJ Open 2017;7:e015226. 10.1136/bmjopen-2016-015226 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 101.Buntrock C, Ebert DD, Lehr D, et al. Effect of a web-based guided self-help intervention for prevention of major depression in adults with subthreshold depression: a randomized clinical trial. JAMA 2016;315:1854–63. 10.1001/jama.2016.4326 [DOI] [PubMed] [Google Scholar]
- 102.Löwe B, Kroenke K, Herzog W, et al. Measuring depression outcome with a brief self-report instrument: sensitivity to change of the Patient Health Questionnaire (PHQ-9). J Affect Disord 2004;81:61–6. 10.1016/S0165-0327(03)00198-8 [DOI] [PubMed] [Google Scholar]
- 103.Erbe D, Eichert H-C, Rietz C, et al. Interformat reliability of the patient health questionnaire: validation of the computerized version of the PHQ-9. Internet Interv 2016;5:1–4. 10.1016/j.invent.2016.06.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 104.Hautzinger M, Keller F, Kühner C. Das beck depressionsinventar II. Deutsche bearbeitung und handbuch zum BDI-II. Pearson 2006. [Google Scholar]
- 105.van Buuren S, Groothuis-Oudshoorn K. mice : Multivariate Imputation by Chained Equations in R.. J Stat Softw 2011;45. [Google Scholar]
- 106.R Core Team R: a language and environment for statistical computing. Vienna, Austria: R Found. Stat. Comput, 2018. http://www.R-project.org/ [Google Scholar]
- 107.Cole DA, Maxwell SE. Testing mediational models with longitudinal data: questions and tips in the use of structural equation modeling. J Abnorm Psychol 2003;112:558–77. 10.1037/0021-843X.112.4.558 [DOI] [PubMed] [Google Scholar]
- 108.Drummond M, Sculpher M, Claxton K, et al. Methods for the economic evaluation of health care programmes. Oxford University Press, 2015. https://global.oup.com/academic/product/methods-for-the-economic-evaluation-of-health-care-programmes-9780199665877?cc=de&lang=en [Google Scholar]
- 109.Drummond M, Sculpher M. Common methodological flaws in economic evaluations. Med Care 2005;43:11–5. 10.1097/01.mlr.0000170001.10393.b7 [DOI] [PubMed] [Google Scholar]
- 110.Willan AR, Briggs AH, Hoch JS. Regression methods for covariate adjustment and subgroup analysis for non-censored cost-effectiveness data. Health Econ 2004;13:461–75. 10.1002/hec.843 [DOI] [PubMed] [Google Scholar]
- 111.Glick HA, Doshi JA, Sonnad SS, et al. Economic evaluation in clinical trials. Oxford University Press, 2014. [Google Scholar]
- 112.Fenwick E, O'Brien BJ, Briggs A. Cost-effectiveness acceptability curves—facts, fallacies and frequently asked questions. Health Econ 2004;13:405–15. 10.1002/hec.903 [DOI] [PubMed] [Google Scholar]
- 113.Hoch JS, Briggs AH, Willan AR, et al. Something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Econ 2002;11:415–30. [DOI] [PubMed] [Google Scholar]
- 114.Titov N, Andrews G, Kemp A, et al. Characteristics of adults with anxiety or depression treated at an Internet clinic: comparison with a national survey and an outpatient clinic. PLoS One 2010;5:e10885. 10.1371/journal.pone.0010885 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 115.Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev 2012;32:329–42. 10.1016/j.cpr.2012.02.004 [DOI] [PubMed] [Google Scholar]
- 116.Lobbestael J, Leurgans M, Arntz A. Inter-rater reliability of the structured clinical interview for DSM-IV axis I disorders (SCID I) and axis II disorders (SCID II). Clin Psychol Psychother 2011;18:75–9. 10.1002/cpp.693 [DOI] [PubMed] [Google Scholar]
- 117.IASP Unrelieved pain is a major global healthcare problem, 2004. [Google Scholar]
- 118.Rozental A, Boettcher J, Andersson G, et al. Negative effects of internet interventions: a qualitative content analysis of patients’ experiences with treatments delivered online. Cogn Behav Ther 2015;44:223–36. 10.1080/16506073.2015.1008033 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.

