Abstract
Introduction
Informal caregivers provide the majority of care to individuals with chronic health conditions, benefiting the care recipient and reducing use of formal care services. However, providing informal care negatively impacts the mental health of many caregivers. E-mental health interventions have emerged as a way to provide accessible mental healthcare to caregivers. Much attention has been given to reviewing the effectiveness and efficacy of such interventions, however, factors related to implementation have received less consideration. Therefore, this mixed-methods systematic review will aim to examine factors associated with the effectiveness and implementation of e-mental health interventions for caregivers.
Methods and analysis
Eligible studies published since 1 January 2007 will be searched for in several electronic databases (CINAHL Plus with Full Text, the Cochrane Library, EMBASE, PsycINFO, PubMed and Web of Science), clinical trial registries and OpenGrey, with all screening steps conducted by two independent reviewers. Studies will be included if they focus on the implementation or effectiveness of e-mental health interventions designed for informal adult caregivers of adults with cancer, heart disease, stroke, diabetes, dementia or chronic obstructive pulmonary disease. Pragmatic randomised controlled trials quantitatively reporting on caregiver anxiety, depression, psychological distress or stress will be used for a qualitative comparative analysis to identify combinations of conditions that result in effective interventions. Qualitative and quantitative data on implementation of e-mental health interventions for caregivers will be integrated in a thematic synthesis to identify barriers and facilitators to implementation. These results will inform future development and implementation planning of e-mental health interventions for caregivers.
Ethics and dissemination
Ethical approval is not required for this study as no primary data will be collected. Results will be disseminated in the form of a scientific publication and presentations at academic conferences and plain language summaries for various stakeholders.
PROSPERO registration number
CRD42020155727.
Keywords: mental health, World Wide Web technology, primary care, depression & mood disorders, anxiety disorders
Strengths and limitations of this study.
The mixed-method design of this review will ensure a wide variety of data on implementation is captured and interpretations account for both qualitative and quantitative research findings.
The peer-reviewed, comprehensive search strategy with all selection steps completed by two independent reviewers will ensure a thorough search of the literature and reduce bias in study selection.
High heterogeneity across studies in terms of implementation or intervention features is easily accommodated in a qualitative comparative analysis.
Crisp set qualitative comparative analysis produces concrete results, increasing the usability of findings for healthcare professionals and decision-makers.
However, crisp set qualitative comparative analysis dichotomises all variables including the outcome, therefore, a more detailed understanding of the strength of the effect size may be lost in this process.
Introduction
Informal caregivers (hereafter referred to as caregivers) are family members or friends who provide unpaid support and care to individuals with healthcare needs. Caregivers play a vital societal role in healthcare systems worldwide, providing up to 80% of care to individuals with long-term care needs.1 Informal care provision can include emotional support, assistance with household tasks (eg, cooking, cleaning), medical care, transportation, managing finances and advocacy on behalf of the care recipient.2 3 Demand for caregivers is expected to increase in the future as the proportion of older adults in populations around the world increases and healthcare policies favour deinstitutionalisation and outpatient care.4 5 As societal dependence on informal care continues to grow, it is becoming increasingly important to implement programmes and policies to support individuals who become caregivers.2 6
Caregivers can experience both positive and negative outcomes over the course of their time as a caregiver.3 7 8 Caregiving can lead to an improved relationship between caregiver and care recipient, feelings of personal development and a sense of accomplishment related to obtaining skills and recognising the impact of the care they provide.8 However, caregivers may also experience negative outcomes related to the caregiving role, such as financial strain and poor physical and mental health.3 7 9 Indeed, the rate of depression and anxiety among caregivers exceeds that of the general population.10 The prevalence of depressive symptoms in cancer and stroke caregivers is often above 40% and the prevalence of anxiety ranges from 21% to over 40%.11 12 Mental health problems can result in large personal and societal costs related to increased morbidity and reduced productivity.13–15 Additionally, poor caregiver mental health negatively impacts distress levels in the care recipient16 17 and the quality of care provided by the caregiver.18–21
There is a clear need to develop effective interventions and resources to prevent or reduce the mental health burden experienced by caregivers. However, caregivers have reported various barriers to accessing mental health services such as lack of knowledge regarding available services, financial barriers, stigma and prioritisation of the caregiving role over self-care.22 E-health technologies have emerged as an accessible way to provide support and information to caregivers23–25 and can be designed to achieve various goals such as, improving communication, teaching skills or reducing depression.26 27 Numerous systematic reviews and meta-analyses have examined e-health interventions for caregivers of adults with chronic health conditions, showing the potential for e-health interventions to improve caregiver well-being.23 25 28–40
E-mental health interventions, that is, mental health interventions delivered via the internet or using mobile technologies,27 41 42 represent a subset of e-health interventions. E-mental health solutions offer a means to improve mental health service access globally43 44 by eliminating many barriers to mental health service access (eg, transportation, stigma, time)25 42 45 and are often more cost-effective than traditional therapies.42 45 Meta-analyses show reductions in caregiver’s depression and anxiety in response to e-mental health interventions.37 39 However, as many reviews focus on intervention efficacy and effectiveness,23 28–32 34 36–40 46 gaps remain in our understanding of factors related to the intervention and the implementation context that make e-mental health interventions effective among caregivers.
Wider literature suggests that the implementation of e-health programmes in real-world settings often encounters many barriers,24 33 47 48 preventing effective interventions from being made available to those who need them. Few reviews have examined factors related to implementation of e-health interventions for caregivers,33 35 with no current reviews, to the best of our knowledge, focusing on implementation of e-mental health interventions for caregivers exclusively. Evaluating the implementation of an intervention is essential to gain insights into why interventions succeed or fail when put into practice. Factors influencing implementation can be related to the intervention itself, the participants, the implementation setting and wider societal factors (eg, regional policies).49 Trials with a more pragmatic design may be better suited to investigating factors potentially associated with implementation given real-world conditions are more closely reflected in pragmatic trials.50 51 However, systematic reviews and meta-analyses do not often distinguish between pragmatic and explanatory (also referred to as efficacy) trials despite the different conditions (eg, setting, recruitment methods, eligibility criteria, control of adherence to and delivery of the intervention) under which interventions are evaluated.50 52 Identifying trials with a pragmatic design may be a valuable factor to consider when interpreting results of reviews to inform implementation.
The aim of this review is to examine factors related to the effectiveness and implementation of e-mental health interventions for caregivers of adults with chronic diseases. Two approaches will be used to investigate this. First, studies with more pragmatic designs will be used exclusively to determine which combinations of intervention or implementation characteristics are associated with effectiveness using a qualitative comparative analysis. Second, reports regarding the implementation of e-mental health interventions will be thematically synthesised to establish the common barriers and facilitators to e-mental health implementation. Findings from this review can be used to guide the development of effective e-mental health interventions to support caregivers and ensure the successful implementation of these interventions within real-world healthcare settings.
Methods and analysis
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA) checklist53 (online supplementary appendix 1) and the Joanna Briggs methodology for mixed-methods systematic reviews54 were used to guide the development of this protocol. Any protocol amendments will be recorded in PROSPERO.
bmjopen-2019-035406supp001.pdf (87.4KB, pdf)
Study eligibility criteria
The eligibility criteria used to inform study inclusion and exclusion are outlined using population, interventions, comparators, outcomes and study design (PICOS).55 56
Population
Unpaid adult caregivers (aged 18 years or older) of adults with either cancer, chronic obstructive pulmonary disease (COPD), dementia, diabetes, heart disease or stroke. Care recipient’s chronic health conditions eligible for inclusion were selected as, globally, they are the largest sources of disability adjusted life years due to physical chronic diseases in adults57 and often require informal care.58 No restrictions will be placed on the frequency or amount of care provided for someone to be considered a caregiver. Studies exclusively focusing on caregivers with severe mental health conditions (eg, psychosis or bipolar disorder) will be excluded, as the focus of this review is on e-mental health interventions targeting psychological health difficulties associated with the provision of informal care, for example anxiety or depression, as opposed to targeting severe mental health conditions. Studies with interventions that solely focus on caregivers providing care to non-community dwelling care recipients will be excluded, given caregivers of individuals who do not live in the community may spend less time providing informal care59 and generally experience lower levels of depression.60 61 Additionally, studies of interventions designed specifically for caregivers of individuals at the end-of-life (eg, within a few months of death) will be excluded, as end-of-life caregiving is associated with additional needs and burdens, for example difficulties related to grief and bereavement.62
Interventions
Interventions will use internet technology, such as web-based platforms or mobile-based applications, to deliver a mental health intervention to caregivers.27 41 E-mental health interventions can encompass many types of mental health support such as screening, prevention, treatment or service delivery.41 This review will focus on interventions targeting the treatment of common caregiver psychological health difficulties (anxiety, depression, psychological distress or stress). This can include any type of mental health treatment, including psychoeducation. Psychoeducation is defined as the provision of information regarding common psychological health difficulties and can be delivered passively (eg, an information website) or actively (eg, an information website with therapist support, homework or exercises).63 The majority of therapeutic materials within the e-mental health intervention must be internet based, however, this may be supplemented with additional forms of support (such as telephone contact, face-to-face support or video conferencing). There are no restrictions on the amount of support provided within the e-mental health intervention. Interventions delivered via telephone, CD-ROM (Compact Disc-Read Only Memory) or video (including Skype) alone will be excluded.
Comparators
As it is necessary to determine effect sizes for the qualitative comparative analysis,64 only studies of pragmatic randomised controlled trials with non-active controls will be included in this analysis. Non-active controls include: no treatment, wait-list control, treatment as usual, non-specific treatment component control (eg, control for attention) or education on the care recipient’s condition.65 Studies using psychoeducation or active controls (eg, controls using specific treatment components or studies comparing two therapies) will be excluded.
For thematic synthesis of barriers and facilitators to implementation, studies of any design (eg, randomised controlled trials, process evaluations, focus groups) will be included in the analysis, regardless of the presence or absence of a control.
Outcomes
For the qualitative comparative analysis, studies must report on caregiver mental health outcomes, specifically anxiety, depression, psychological distress or stress, measured using an instrument with at least acceptable reliability (Cronbach’s alpha ≥0.7).66 Reliability of outcome measures will be assessed based on the main validation paper of the relevant measurement instrument, as this review will likely include studies with different caregiver populations, ages, genders and languages, the combination of which may not have been validated. Examples of eligible measurement instruments include the Centre for Epidemiologic Studies-Depression Scale,67 the Hospital Anxiety and Depression Scale68 or the Perceived Stress Scale.69
For the thematic synthesis, studies will report on barriers and/or facilitators to intervention implementation. This may include qualitative (eg, interviews or focus groups) or quantitative (eg, Normalisation Measure Development questionnaire70) data. Barriers or facilitators can include factors related to any aspect of the Consolidated Framework for Implementation Research49 or the implementation outcome framework developed by Proctor et al.71 The Consolidated Framework for Implementation Research consists of five domains related to implementation, namely (1) intervention characteristics (eg, adaptability, complexity); (2) outer setting (eg, external policies, patient needs and resources); (3) inner/implementation setting (eg, culture within the organisation, readiness for implementation); (4) characteristics of individuals (eg, self-efficacy, individual stage of change) and (5) process (eg, planning, engaging).49 The implementation outcome framework broadly classifies measurable implementation outcomes which includes acceptability, adoption, feasibility, fidelity, reach, appropriateness, implementation cost and sustainability.71
Study designs
Studies included for the qualitative comparative analysis must be pragmatic randomised controlled trials (also referred to as effectiveness trials). Pragmatic trials will be identified using the validated PRagmatic Explanatory Continuum Indicator Summary 2 (PRECIS-2) tool.50 PRECIS-2 was developed with input from clinicians, researchers and policy-makers to allow trialists to assess how pragmatic or explanatory their trial design is across nine domains: eligibility criteria, recruitment, setting, organisation, flexibility (delivery), flexibility (adherence), follow-up, primary outcome and primary analysis.50 Trials with a pragmatic design will be defined as any trial with a mean score of 3 or higher using the PRECIS-2 tool.50 PRECIS-2 has been used with this cut-off score to categorise studies in another systematic review,72 although to our knowledge it has not previously been used to exclude studies from a systematic review. Using a cut-off score of 3 should ensure generous inclusion of trials containing at least a mixture of pragmatic and explanatory design features.50
To assess barriers and facilitators to implementation, any study type with quantitative and/or qualitative data will be eligible for inclusion.
Search strategy
Comprehensive literature searches will be conducted in multiple electronic databases (CINAHL Plus with Full Text, the Cochrane Library, EMBASE, PsycINFO, PubMed and Web of Science). Clinical trial registries (www.clinicaltrials.gov and www.who.int/trialsearch/) will be searched for relevant completed clinical trials and the resulting publications will be found and screened for inclusion. Searches for grey literature will be performed using OpenGrey (http://www.opengrey.eu/), a database of grey literature in Europe such as research reports and conference papers.
The search strategy was developed in consultation with Agnes Kotka, a librarian at Uppsala University and was reviewed by Professor Mariët Hagedoorn and Truus van Ittersum (University Medical Centre Groningen, University of Groningen) and Dr. Nathan Davies (University College London) following the PRESS peer review guidelines73 (online supplementary appendix 2). The search was constructed using terms related to (1) caregivers; (2) the chronic health conditions of interest (cancer, COPD, dementia, diabetes, heart disease and stroke); (3) e-health/information and communication technology; (4) mental health and (5) psychological therapies (see online supplementary appendix 3). Included terms were informed by existing reviews focusing on the population and/or intervention of interest to this review.9 33 39 74–79 Search terms were refined based on feedback from the peer-review process, resulting in the addition of more truncations to search terms, elimination of repetitive search terms that did not retrieve additional records and the addition of an abbreviation missed prior to the peer-review process. The search will include relevant Medical Subject Headings when possible and terms will be searched for in the title/abstract of publications. Included studies will be restricted to those published in English, Dutch, German or Swedish. Literature produced from January 2007 onwards will be eligible for inclusion. Technologies from work published prior to 2007 may be outdated and other reviews have shown that production of publications involving e-health began to rise from 2007 onwards.33 35 Electronic searches will be rerun prior to reporting of results to ensure the search is as up to date as possible.
bmjopen-2019-035406supp002.pdf (406.2KB, pdf)
bmjopen-2019-035406supp003.pdf (144.7KB, pdf)
On final inclusion of any studies, their references, results from forward citation searches and from the first three pages of the ‘find similar’ search function in PubMed will be used to check for any additional studies of interest. Experts in the field will be contacted to identify further studies for inclusion.
Study selection
Results of database searches will be imported into EndNote for deduplication following the procedures outlined by Bramer et al.80 Remaining records will be imported into the online screening software Rayyan.81 Titles, abstracts and full texts will be screened independently by two reviewers. Conflicts will be discussed and a third reviewer will be consulted if consensus cannot be reached. Study selection will be based on the criteria outlined by the PICOS, with reasons for study exclusion being recorded at the full-text screening stage. Full texts will be checked against each sub-section of the PICOS, recording which sub-sections are or are not met by each study, with an overall reason for exclusion being reported in the PRISMA flow diagram. This will facilitate detailed discussions regarding study exclusion when conflicts arise. If studies do not contain enough information to decide on inclusion, the original authors will be contacted at most twice over a 1-month period to obtain information to determine study eligibility. If the original authors do not respond, the study will be excluded. Abstracts, theses, books, commentaries, editorials and letters to the editor will be excluded. Reviews and study protocols will also be excluded, however, the references of related reviews will be checked for additional studies of interest, published results of relevant study protocols will be obtained and if protocol results are unpublished, authors will be contacted to determine whether access to unpublished results is possible.
Records retrieved from searches of clinical trial registries and OpenGrey will be screened for eligibility by one reviewer. When relevant clinical trial registries are identified, any resulting publications will be retrieved and screened for inclusion, unless already captured by the electronic database searches. If results from relevant trial registries are unpublished, authors will be contacted to determine if they are able to share details of any available results. Authors of grey literature records that do not contain enough information to assess eligibility will also be contacted for additional study details.
Exclusion of studies on the basis of adopting a more explanatory, as opposed to pragmatic, trial design will be conducted as a final step during the full-text screening process. This screening step will only be applied to trials eligible for the qualitative comparative analysis. Studies will be scored using the PRECIS-2 tool by two independent reviewers and studies with a mean score below 3 will be excluded.50 72
Assessment of methodological quality
Methodological quality of studies included in the qualitative comparative analysis will be evaluated using the Cochrane Risk of Bias 2.0 tool for randomised controlled trials.82 83 This evaluation will facilitate the identification of selection, performance, measurement, attrition and reporting bias.83 Authors will be contacted if more information is required to complete the quality assessment. Reporting bias will be explored by comparing outcomes measures described in study protocols to the outcome measures reported in the methods and results sections of the corresponding completed trial. In response to any identified inconsistencies, authors will be contacted to determine potential causes of this. Study assessment will be conducted by two independent reviewers, followed by discussion of any discrepancies, consulting a third reviewer as needed. Studies will not be excluded based on methodological quality, however the results of the Cochrane Risk of Bias 2.0 evaluation will be reported descriptively.
Data extraction
Data from included full texts will be extracted into Microsoft Excel (2016), using a data extraction form developed for this review based on the Centre for Reviews and Dissemination guidelines.56 Extracted information will include data pertaining to study participants, study design, the intervention and relevant outcomes (full details in online supplementary appendix 4). Data used in the qualitative comparative analysis and thematic synthesis will be extracted independently by two reviewers, with resulting extractions compared for accuracy and completion. All other data will be extracted by one reviewer and verified by a second reviewer. If conflicts arise, the original publication will be referred to in order to resolve misunderstandings and a third reviewer will be consulted if necessary. Authors will be contacted at most twice to obtain additional data and/or clarification as needed. Qualitative results pertaining to implementation will be transferred into NVivo V.10 software84 for thematic synthesis.
bmjopen-2019-035406supp004.pdf (29KB, pdf)
Data synthesis
Data related to the characteristics of each included study, such as the sample (eg, sample size, participant demographics) or intervention (eg, duration, type of support provided, delivery mode) characteristics, will be reported in summary tables. Further data synthesis will involve two analysis methods. Pragmatic randomised controlled trials with quantitative mental health outcome data will be included in the qualitative comparative analysis. Publications of any study design reporting on implementation will be included in the thematic synthesis, taking an integrative approach to synthesise both qualitative and quantitative findings.
Qualitative comparative analysis
A crisp set qualitative comparative analysis will be conducted to determine sets of conditions that result in effective e-mental health interventions for caregivers.64 Crisp set qualitative comparative analysis involves dichotomising outcome data (eg, effective or not effective) and conditions (eg, present or absent) selected for inclusion in the analysis into distinct categories.64 A crisp set analysis approach was selected over a fuzzy set analysis as the results will be more clearly interpretable and easier for decision-makers to use.85
The first step of a qualitative comparative analysis is to build a data table containing information regarding the effectiveness of each study and conditions related to the intervention and its implementation (see online supplementary appendix 5).64 Conditions to include in the data table will be based on important factors related to intervention components (eg, uses goal-setting, homework), intervention delivery methods (eg, mobile app, computer) and implementation (eg, acceptability, feasibility). By restricting this analysis to pragmatic trials, which are designed to more closely reflect real-world settings, implementation conditions are more likely to be reported. Conditions selected will be adjusted given the need to ensure adequate heterogeneity is present.64 Qualitative comparative analysis requires diversity among studies in terms of conditions present and intervention effectiveness in order to determine the combination of factors sufficient for interventions to be effective.64 Therefore, adjustments to outcome classification and conditions selected for analysis will be needed after data collection is completed.
bmjopen-2019-035406supp005.pdf (34.7KB, pdf)
Intervention effectiveness will be measured as the standardised mean effect size between control and comparator groups’ mental health outcomes, calculated using Hedges’ g and the Comprehensive Meta-Analysis (V.3) software. Effect sizes will be calculated for all mental health outcomes of interest for this review (anxiety, depression, psychological distress and stress) and will be based on data collected immediately after intervention completion. If enough studies report subsequent postintervention follow-ups, these effect sizes will be calculated to explore whether different factors contribute to sustained intervention success. Effect sizes will be used to create crisp sets to categorise studies as effective (Hedges’ g≥0.3) or not effective.86 If most interventions are effective (or not effective), a different classification system will be created to ensure adequate heterogeneity for analysis,64 for example, categorising studies as highly effective (Hedges’ g≥0.5) or not highly effective.86 Proposed cut-offs were developed based on existing meta-analyses of e-mental health interventions.87–90
The main data table will use general effectiveness as the outcome measure, meaning the primary mental health outcome as identified in each study will be used to represent the effectiveness of that intervention. If studies include multiple outcomes of interest, but do not identify a primary outcome measure, the outcome most frequently measured in included studies will be used to evaluate intervention effectiveness. Secondary analyses may be conducted for anxiety, depression, psychological distress and stress separately, to explore whether different conditions are more important for different outcome measures. However, this is dependent on identification of an adequate number of studies for each outcome of interest. After completion of the data table, truth tables will be constructed and the software fs/QCA (V.3.1b) will be used to determine the sufficient conditions for effective e-mental health interventions.91 92
Thematic synthesis
Data from studies addressing implementation of e-mental health interventions for caregivers will be thematically synthesised using a deductive coding approach, to identify barriers and facilitators experienced during implementation.93 94 It will likely be necessary to integrate qualitative and quantitative data as many aspects of implementation such as acceptability, feasibility and usability, may be measured using quantitative tools.95 First, qualitative data will be thematically analysed using the Consolidated Framework for Implementation Research to guide coding.49 This framework was selected a priori as it was developed by combining multiple implementation theories into a single, comprehensive theory covering all aspects related to implementation49 and it has been used as a coding guide in other reviews on implementation.33 96 Qualitative data will be coded based on the 39 pre-defined constructs within the Consolidated Framework for Implementation Research,49 with the creation of additional codes if needed.
Quantitative data will be narratively summarised to facilitate subsequent integration of qualitative and quantitative findings. Creating narrative summaries will involve approaches such as textually describing study findings and grouping findings based on the constructs and domains of the Consolidated Framework for Implementation Research.94 Initially, 10% of full texts included in the thematic synthesis will be coded independently by two reviewers, followed by discussion of the coding process in consultation with a third reviewer. The remaining coding will be conducted by one reviewer with regular discussions with a second reviewer, involving a third reviewer as needed. Results of the initial coding of qualitative data and narrative summaries of quantitative data will be analysed together to identify barriers and facilitators to implementation. Two reviewers will independently identify barriers and facilitators, followed by discussion involving a third reviewer as needed.93 Through this discussion, more abstract, analytical themes will be developed that go beyond the initial codes and identified barriers and facilitators.93 This process will be iterative, modifying barriers and facilitators after defining initial analytical themes, followed by further refinement of analytical themes until the analytical themes fully encompass all codes and identified barriers and facilitators.93
Patient and public involvement
There was no patient or public involvement in the development of this protocol. However, caregivers and healthcare professionals will be involved in the interpretation of data for the thematic synthesis.97 98 Results of the thematic synthesis will be presented to a multidisciplinary group of stakeholders (eg, caregivers, mental health service providers) to explore whether the identified themes resonate with their experiences or if they feel important aspects related to implementation have not been captured by the synthesis. Their perspectives will be incorporated into the interpretation of the results of the thematic synthesis.
Discussion
Despite the importance of implementation planning and need for effective e-mental health interventions for caregivers in real world practice, there have been no reviews focusing on this area. Using pragmatic trials and implementation research, this review will identify both the key characteristics of effective interventions and barriers and facilitators to implementation. A qualitative comparative analysis will be employed to identify combinations of conditions resulting in effective e-mental health interventions for caregivers, a method which, to the best of our knowledge, has not yet been used in this field. The results of the qualitative comparative analysis can be used to improve the design of future e-mental health interventions by identifying intervention components and implementation factors important to intervention effectiveness in real-world settings.
Additionally, common barriers and facilitators to implementation of e-mental health interventions for caregivers identified in this review can be used to inform implementation planning for similar interventions designed to reduce the mental health burden experienced by caregivers. For example, results may highlight the importance of providing training to individuals delivering the intervention or involvement of management staff in implementation activities. Improving our understanding of factors associated with implementation will allow implementers to both account for and avoid common implementation challenges, thereby potentially increasing subsequent uptake and effectiveness of e-mental health programmes developed to support caregivers.
Ethics and dissemination
The results of this work will be disseminated in the form of a scientific publication in a peer-reviewed journal and as presentations at conferences. Plain language summaries will be prepared and provided to groups working with or supporting caregivers and healthcare organisations. Results will also be disseminated throughout the Marie Sklodowska-Curie Innovation Training Network, ENTWINE, which conducts research related to informal care and technological interventions to support caregivers.
Supplementary Material
Acknowledgments
We would like to thank Agnes Kotka, librarian at Uppsala University, for assisting in development of the search strategy. We are also grateful to Professor Mariët Hagedoorn, Truus van Ittersum (University Medical Centre Groningen, University of Groningen) and Dr. Nathan Davies (University College London), for providing valuable feedback as the peer-reviewers of the search strategy.
Footnotes
Contributors: CC contributed to the design of the study and wrote the manuscript. JW, who acts as the reviews guarantor, conceived the study, contributed to the study design and critically revised the manuscript draft. LvE and RS critically revised the study design and manuscript. All authors approved of the final manuscript.
Funding: This work was supported by the European Union’s Horizon 2020 research and innovation programme under the Marie-Sklodowska Curie grant agreement no 814 072.
Disclaimer: Funders were not involved in the creation, development or publication of this protocol, nor will they be involved in the conduct, analysis or reporting of the resulting systematic review.
Competing interests: None declared.
Patient consent for publication: Not required.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1.Hoffmann F, Rodrigues R. Informal carers: who takes care of them? Policy brief. Vienna: : European Centre for Social Welfare Policy and Research, 2010. Available: https://www.euro.centre.org/publications/detail/387 [Accessed 8 Jul 2019].
- 2.National Academies of Sciences Engineering and Medicine Family caregiving roles and impacts : Committee on Family Caregiving for Older Adults, Board on Health Care Services, Health and Medicine Division, et al., eds. Families Caring for an Aging America. Washington, DC: The National Academies Press, 2016. [PubMed] [Google Scholar]
- 3.Girgis A, Lambert S. Cost of informal caregiving in cancer care. Cancer Forum 2017;41:16–22 https://search.informit.com.au/documentSummary;dn=112312308779892;res=IELHEA [Google Scholar]
- 4.Spasova S, Baeten R, Coster S, et al. Challenges in long-term care in Europe: A study of national policies. Brussels: European Comission: European Social Policy Network (ESPN), 2018. [Google Scholar]
- 5.Colombo F, Llena-Nozal A, Mercier J, et al. Help wanted? providing and paying for long-term care. OECD Publishing 2011. [Google Scholar]
- 6.Schulz R, Czaja SJ. Family caregiving: a vision for the future. Am J Geriatr Psychiatry 2018;26:358–63. 10.1016/j.jagp.2017.06.023 [DOI] [PubMed] [Google Scholar]
- 7.Lambert SD, Levesque J, Girgis A. The impact of cancer and chronic conditions on caregivers and family members : Koczwara B, Cancer and chronic conditions. Singapore: Springer, 2016: 159–202. [Google Scholar]
- 8.Li Q, Loke AY. The positive aspects of caregiving for cancer patients: a critical review of the literature and directions for future research. Psychooncology 2013;22:2399–407. 10.1002/pon.3311 [DOI] [PubMed] [Google Scholar]
- 9.Coumoundouros C, Ould Brahim L, Lambert SD, et al. The direct and indirect financial costs of informal cancer care: a scoping review. Health Soc Care Community 2019;27:e622–36. 10.1111/hsc.12808 [DOI] [PubMed] [Google Scholar]
- 10.Pinquart M, Sörensen S. Differences between caregivers and noncaregivers in psychological health and physical health: a meta-analysis. Psychol Aging 2003;18:250–67 https://psycnet-apa-org.proxy3.library.mcgill.ca/fulltext/2003-05349-011.pdf 10.1037/0882-7974.18.2.250 [DOI] [PubMed] [Google Scholar]
- 11.Loh AZ, Tan JS, Zhang MW, et al. The global prevalence of anxiety and depressive symptoms among caregivers of stroke survivors. J Am Med Dir Assoc 2017;18:111–6. 10.1016/j.jamda.2016.08.014 [DOI] [PubMed] [Google Scholar]
- 12.Geng H-M, Chuang D-M, Yang F, et al. Prevalence and determinants of depression in caregivers of cancer patients: a systematic review and meta-analysis. Medicine 2018;97:e11863. 10.1097/MD.0000000000011863 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Birnbaum HG, Kessler RC, Kelley D, et al. Employer burden of mild, moderate, and severe major depressive disorder: mental health services utilization and costs, and work performance. Depress Anxiety 2010;27:78–89. 10.1002/da.20580 [DOI] [PubMed] [Google Scholar]
- 14.Kessler RC. The costs of depression. Psychiatr Clin North Am 2012;35:1–14. 10.1016/j.psc.2011.11.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Doran CM, Kinchin I. A review of the economic impact of mental illness. Aust Health Rev 2019;43:43–8. 10.1071/AH16115 [DOI] [PubMed] [Google Scholar]
- 16.Hagedoorn M, Sanderman R, Bolks HN, et al. Distress in couples coping with cancer: a meta-analysis and critical review of role and gender effects. Psychol Bull 2008;134:1–30 https://psycnet-apa-org.proxy3.library.mcgill.ca/fulltext/2007-19419-001.pdf 10.1037/0033-2909.134.1.1 [DOI] [PubMed] [Google Scholar]
- 17.Hodges LJ, Humphris GM, Macfarlane G. A meta-analytic investigation of the relationship between the psychological distress of cancer patients and their carers. Soc Sci Med 2005;60:1–12. 10.1016/j.socscimed.2004.04.018 [DOI] [PubMed] [Google Scholar]
- 18.Smith GR, Williamson GM, Miller LS, et al. Depression and quality of informal care: a longitudinal investigation of caregiving stressors. Psychol Aging 2011;26:584–91. 10.1037/a0022263 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Christie J, Smith GR, Williamson GM, et al. Quality of informal care is multidimensional. Rehabil Psychol 2009;54:173–81. 10.1037/a0015705 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Cooney C, Howard R, Lawlor B. Abuse of vulnerable people with dementia by their carers: can we identify those most at risk? Int J Geriatr Psychiatry 2006;21:564–71. 10.1002/gps.1525 [DOI] [PubMed] [Google Scholar]
- 21.Beach SR, Schulz R, Williamson GM, et al. Risk factors for potentially harmful informal caregiver behavior. J Am Geriatr Soc 2005;53:255–61. 10.1111/j.1532-5415.2005.53111.x [DOI] [PubMed] [Google Scholar]
- 22.Mosher CE, Given BA, Ostroff JS. Barriers to mental health service use among distressed family caregivers of lung cancer patients. Eur J Cancer Care 2015;24:50–9. 10.1111/ecc.12203 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Ploeg J, Ali MU, Markle-Reid M, et al. Caregiver-focused, web-based interventions: systematic review and meta-analysis (Part 2). J Med Internet Res 2018;20:e11247. 10.2196/11247 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Christie HL, Martin JL, Connor J, et al. eHealth interventions to support caregivers of people with dementia may be proven effective, but are they implementation-ready? Internet Interv 2019;18:100260. 10.1016/j.invent.2019.100260 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Sin J, Henderson C, Spain D, et al. eHealth interventions for family carers of people with long term illness: a promising approach? Clin Psychol Rev 2018;60:109–25. 10.1016/j.cpr.2018.01.008 [DOI] [PubMed] [Google Scholar]
- 26.Pagliari C, Sloan D, Gregor P, et al. What is eHealth (4): a scoping exercise to map the field. J Med Internet Res 2005;7:e9. 10.2196/jmir.7.1.e9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Eysenbach G. What is e-health? J Med Internet Res 2001;3:e20. 10.2196/jmir.3.2.e20 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Aldehaim AY, Alotaibi FF, Uphold CR, et al. The impact of technology-based interventions on informal caregivers of stroke survivors: a systematic review. Telemed J E Health 2016;22:223–31. 10.1089/tmj.2015.0062 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Tang WP, Chan CW, So WK, et al. Web-Based interventions for caregivers of cancer patients: a review of literatures. Asia Pac J Oncol Nurs 2014;1:9–15. 10.4103/2347-5625.135811 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Kaltenbaugh DJ, Klem ML, Hu L, et al. Using web-based interventions to support caregivers of patients with cancer: a systematic review. Oncol Nurs Forum 2015;42:156–64. 10.1188/15.ONF.156-164 [DOI] [PubMed] [Google Scholar]
- 31.McKechnie V, Barker C, Stott J. Effectiveness of computer-mediated interventions for informal carers of people with dementia-a systematic review. Int Psychogeriatr 2014;26:1619–37. 10.1017/S1041610214001045 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Wasilewski MB, Stinson JN, Cameron JI. Web-Based health interventions for family caregivers of elderly individuals: a scoping review. Int J Med Inform 2017;103:109–38. 10.1016/j.ijmedinf.2017.04.009 [DOI] [PubMed] [Google Scholar]
- 33.Christie HL, Bartels SL, Boots LMM, et al. A systematic review on the implementation of eHealth interventions for informal caregivers of people with dementia. Internet Interv 2018;13:51–9. 10.1016/j.invent.2018.07.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Guay C, Auger C, Demers L, et al. Components and outcomes of Internet-based interventions for caregivers of older adults: systematic review. J Med Internet Res 2017;19:e313. 10.2196/jmir.7896 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Heynsbergh N, Heckel L, Botti M, et al. Feasibility, useability and acceptability of technology-based interventions for informal cancer carers: a systematic review. BMC Cancer 2018;18:244. 10.1186/s12885-018-4160-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Hopwood J, Walker N, McDonagh L, et al. Internet-Based interventions aimed at supporting family caregivers of people with dementia: systematic review. J Med Internet Res 2018;20:e216. 10.2196/jmir.9548 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Sherifali D, Ali MU, Ploeg J, et al. Impact of Internet-based interventions on caregiver mental health: systematic review and meta-analysis. J Med Internet Res 2018;20:e10668. 10.2196/10668 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Ploeg J, Markle-Reid M, Valaitis R, et al. Web-Based interventions to improve mental health, general caregiving outcomes, and general health for informal caregivers of adults with chronic conditions living in the community: rapid evidence review. J Med Internet Res 2017;19:e263. 10.2196/jmir.7564 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Zhao Y, Feng H, Hu M, et al. Web-Based interventions to improve mental health in home caregivers of people with dementia: meta-analysis. J Med Internet Res 2019;21:e13415. 10.2196/13415 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Boots LMM, de Vugt ME, van Knippenberg RJM, et al. A systematic review of Internet-based supportive interventions for caregivers of patients with dementia. Int J Geriatr Psychiatry 2014;29:331–44. 10.1002/gps.4016 [DOI] [PubMed] [Google Scholar]
- 41.Riper H, Andersson G, Christensen H, et al. Theme issue on e-mental health: a growing field in Internet research. J Med Internet Res 2010;12:e74. 10.2196/jmir.1713 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Lal S. E-mental health: promising advancements in policy, research, and practice. Healthc Manage Forum 2019;32:56–62. 10.1177/0840470418818583 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Rebello TJ, Marques A, Gureje O, et al. Innovative strategies for closing the mental health treatment gap globally. Curr Opin Psychiatry 2014;27:308–14. 10.1097/YCO.0000000000000068 [DOI] [PubMed] [Google Scholar]
- 44.Naslund JA, Aschbrenner KA, Araya R, et al. Digital technology for treating and preventing mental disorders in low-income and middle-income countries: a narrative review of the literature. Lancet Psychiatry 2017;4:486–500. 10.1016/S2215-0366(17)30096-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Health Quality Ontario Internet-Delivered cognitive behavioural therapy for major depression and anxiety disorders: a health technology assessment. Ont Health Technol Assess Ser 2019;19:1–199. [PMC free article] [PubMed] [Google Scholar]
- 46.Sin J, Norman I. Psychoeducational interventions for family members of people with schizophrenia: a mixed-method systematic review. J Clin Psychiatry 2013;74:e1145–62. 10.4088/JCP.12r08308 [DOI] [PubMed] [Google Scholar]
- 47.Folker AP, Mathiasen K, Lauridsen SM, et al. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European Internet services. Internet Interv 2018;11:60–70. 10.1016/j.invent.2018.02.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Gitlin LN, Marx K, Stanley IH, et al. Translating evidence-based dementia caregiving interventions into practice: State-of-the-science and next steps. Gerontologist 2015;55:210–26. 10.1093/geront/gnu123 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50. 10.1186/1748-5908-4-50 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Loudon K, Treweek S, Sullivan F, et al. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ 2015;350:h2147. 10.1136/bmj.h2147 [DOI] [PubMed] [Google Scholar]
- 51.Rothwell PM. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet 2005;365:82–93. 10.1016/S0140-6736(04)17670-8 [DOI] [PubMed] [Google Scholar]
- 52.Aves T, Allan KS, Lawson D, et al. The role of pragmatism in explaining heterogeneity in meta-analyses of randomised trials: a protocol for a cross-sectional methodological review. BMJ Open 2017;7:e017887. 10.1136/bmjopen-2017-017887 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;350:g7647. 10.1136/bmj.g7647 [DOI] [PubMed] [Google Scholar]
- 54.Lizarondo L, Stern C, Carrier J, et al. Chapter 8: Mixed methods systematic reviews In: Aromataris E, Munn Z, eds Joanna Briggs Institute Reviewer’s Manual. The Joanna Briggs Institute, 2017. https://wiki.joannabriggs.org/display/MANUAL/Chapter+8%3A+Mixed+methods+systematic+reviews (accessed 1 Jul 2019). [Google Scholar]
- 55.Costantino G, Montano N, Casazza G. When should we change our clinical practice based on the results of a clinical study? searching for evidence: PICOS and PubMed. Intern Emerg Med 2015;10:525–7. 10.1007/s11739-015-1225-5 [DOI] [PubMed] [Google Scholar]
- 56.Centre for Reviews and Dissemination Systematic Reviews: CRD’s guidance for undertaking reviews in health care. 3rd ed. CRD, University of York, 2009. Available: www.york.ac.uk/inst/crd [Accessed 10 Jul 2019].
- 57.Institute for Health Metrics and Evaluation GBD compare data visualization. Seattle, WA: : IHME, University of Washington, 2018. Available: http://www.healthdata.org/data-visualization/gbd-compare [Accessed 10 Jul 2019].
- 58.Sinha M. Portrait of caregivers, 2012, 2013. Available: https://www150.statcan.gc.ca/n1/pub/89-652-x/89-652-x2013001-eng.htm [Accessed 11 Jul 2019].
- 59.König H-H, Leicht H, Brettschneider C, et al. The costs of dementia from the societal perspective: is care provided in the community really cheaper than nursing home care? J Am Med Dir Assoc 2014;15:117–26. 10.1016/j.jamda.2013.10.003 [DOI] [PubMed] [Google Scholar]
- 60.Bleijlevens MHC, Stolt M, Stephan A, et al. Changes in caregiver burden and health-related quality of life of informal caregivers of older people with dementia: evidence from the European RightTimePlaceCare prospective cohort study. J Adv Nurs 2015;71:1378–91. 10.1111/jan.12561 [DOI] [PubMed] [Google Scholar]
- 61.Gaugler JE, Mittelman MS, Hepburn K, et al. Clinically significant changes in burden and depression among dementia caregivers following nursing home admission. BMC Med 2010;8:85. 10.1186/1741-7015-8-85 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.DuBenske LL, Wen K-Y, Gustafson DH, et al. Caregivers' differing needs across key experiences of the advanced cancer disease trajectory. Palliat Support Care 2008;6:265–72. 10.1017/S1478951508000400 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Donker T, Griffiths KM, Cuijpers P, et al. Psychoeducation for depression, anxiety and psychological distress: a meta-analysis. BMC Med 2009;7:79. 10.1186/1741-7015-7-79 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Thomas J, O'Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev 2014;3:67. 10.1186/2046-4053-3-67 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Mohr DC, Spring B, Freedland KE, et al. The selection and design of control conditions for randomized controlled trials of psychological interventions. Psychother Psychosom 2009;78:275–84. 10.1159/000228248 [DOI] [PubMed] [Google Scholar]
- 66.Fitzpatrick R, Davey C, Buxton MJ, et al. Evaluating patient-based outcome measures for use in clinical trials. Health Technol Assess 1998;2:i-iv, 1-74. 10.3310/hta2140 [DOI] [PubMed] [Google Scholar]
- 67.Radloff LS. The CES-D scale: a self-report depression scale for research in the general population. Appl Psychol Meas 1977;1:385–401. [Google Scholar]
- 68.Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand 1983;67:361–70. 10.1111/j.1600-0447.1983.tb09716.x [DOI] [PubMed] [Google Scholar]
- 69.Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav 1983;24:385–96. 10.2307/2136404 [DOI] [PubMed] [Google Scholar]
- 70.Vis C, Ruwaard J, Finch T, et al. Toward an objective assessment of implementation processes for innovations in health care: psychometric evaluation of the normalization measure development (NoMAD) questionnaire among mental health care professionals. J Med Internet Res 2019;21:e12376. 10.2196/12376 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71.Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011;38:65–76. 10.1007/s10488-010-0319-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Sajobi TT, Li G, Awosoga O, et al. A comparison of meta-analytic methods for synthesizing evidence from explanatory and pragmatic trials. Syst Rev 2018;7:19. 10.1186/s13643-017-0668-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 73.McGowan J, Sampson M, Salzwedel DM, et al. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 2016;75:40–6. 10.1016/j.jclinepi.2016.01.021 [DOI] [PubMed] [Google Scholar]
- 74.Lambert SD, Duncan LR, Kapellas S, et al. A descriptive systematic review of physical activity interventions for caregivers: effects on caregivers' and care recipients' psychosocial outcomes, physical activity levels, and physical health. Ann Behav Med 2016;50:907–19. 10.1007/s12160-016-9819-3 [DOI] [PubMed] [Google Scholar]
- 75.Woodford J, Farrand P, Richards D, et al. Psychological treatments for common mental health problems experienced by informal carers of adults with chronic physical health conditions (protocol). Syst Rev 2013;2:9. 10.1186/2046-4053-2-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 76.Corry M, While A, Neenan K, et al. A systematic review of systematic reviews on interventions for caregivers of people with chronic conditions. J Adv Nurs 2015;71:718–34. 10.1111/jan.12523 [DOI] [PubMed] [Google Scholar]
- 77.Vis C, Mol M, Kleiboer A, et al. Improving implementation of eMental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR Ment Health 2018;5:e20. 10.2196/mental.9769 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 78.Wozney L, McGrath PJ, Gehring ND, et al. eMental healthcare technologies for anxiety and depression in childhood and adolescence: systematic review of studies reporting implementation outcomes. JMIR Ment Health 2018;5:e48. 10.2196/mental.9655 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 79.Farrand P, Matthews J, Dickens C, et al. Psychological interventions to improve psychological well-being in people with dementia or mild cognitive impairment: systematic review and meta-analysis protocol. BMJ Open 2016;6:e009713. 10.1136/bmjopen-2015-009713 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 80.Bramer WM, Giustini D, de Jonge GB, et al. De-duplication of database search results for systematic reviews in endnote. J Med Libr Assoc 2016;104:240–3. 10.3163/1536-5050.104.3.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 81.Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan-a web and mobile APP for systematic reviews. Syst Rev 2016;5:210. 10.1186/s13643-016-0384-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 82.Higgins JP, Savovic J, Page M, et al. Revised Cochrane risk-of-bias tool for randomized trials (rob 2), 2019. Available: https://drive.google.com/file/d/1xBzD1BeRrXH2DHg58fnTNKinTkX5SLEt/view [Accessed 22 Jul 2019].
- 83.Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. 10.1136/bmj.d5928 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 84.QSR International Pty Ltd NVivo qualitative data analysis software 2014.
- 85.Hill LG, Cooper BR, Parker LA. Qualitative comparative analysis: a mixed-method tool for complex implementation questions. J Prim Prev 2019;40:69–87. 10.1007/s10935-019-00536-5 [DOI] [PubMed] [Google Scholar]
- 86.Sullivan GM, Feinn R. Using effect size-or why the P value is not enough. J Grad Med Educ 2012;4:279–82. 10.4300/JGME-D-12-00156.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 87.Davies EB, Morriss R, Glazebrook C. Computer-delivered and web-based interventions to improve depression, anxiety, and psychological well-being of university students: a systematic review and meta-analysis. J Med Internet Res 2014;16:e130. 10.2196/jmir.3142 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 88.van Beugen S, Ferwerda M, Hoeve D, et al. Internet-Based cognitive behavioral therapy for patients with chronic somatic conditions: a meta-analytic review. J Med Internet Res 2014;16:e88. 10.2196/jmir.2777 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 89.Lorenzo-Luaces L, Johns E, Keefe JR. The generalizability of randomized controlled trials of self-guided Internet-based cognitive behavioral therapy for depressive symptoms: systematic review and meta-regression analysis. J Med Internet Res 2018;20:e10113. 10.2196/10113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 90.Linde K, Sigterman K, Kriston L, et al. Effectiveness of psychological treatments for depressive disorders in primary care: systematic review and meta-analysis. Ann Fam Med 2015;13:56–68. 10.1370/afm.1719 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 91.Ragin C, Davey S. Fuzzy-Set/Qualitative Comparative Analysis 3.0. Irvine, Californai: : Department of Sociology, University of California, 2016. Available: http://www.socsci.uci.edu/~cragin/fsQCA/software.shtml [Accessed 12 Jul 2019].
- 92.Ragin CC. User’s guide to fuzzy-set/qualitative comparative analysis. Irvine, California, 2017. Available: www.fsqca.com [Accessed 12 Jul 2019].
- 93.Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 2008;8:45. 10.1186/1471-2288-8-45 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 94.Popay J, Roberts H, Sowden A, et al. Guidance on the conduct of narrative synthesis in systematic reviews, 2006. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.178.3100&rep=rep1&type=pdf [Accessed 31 Jul 2019].
- 95.Lewis CC, Fischer S, Weiner BJ, et al. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci 2015;10:155. 10.1186/s13012-015-0342-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 96.Kerins C, McSharry J, Hayes C, et al. Barriers and facilitators to implementation of menu labelling interventions to support healthy food choices: a mixed methods systematic review protocol. Syst Rev 2018;7:88. 10.1186/s13643-018-0752-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 97.Pollock A, Campbell P, Struthers C, et al. Stakeholder involvement in systematic reviews: a scoping review. Syst Rev 2018;7:208. 10.1186/s13643-018-0852-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 98.Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy 2011;102:105–16. 10.1016/j.healthpol.2011.05.002 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2019-035406supp001.pdf (87.4KB, pdf)
bmjopen-2019-035406supp002.pdf (406.2KB, pdf)
bmjopen-2019-035406supp003.pdf (144.7KB, pdf)
bmjopen-2019-035406supp004.pdf (29KB, pdf)
bmjopen-2019-035406supp005.pdf (34.7KB, pdf)
