Skip to main content
NCBI home page
HRB Open Research logoLink to HRB Open Research
. 2022 Jun 9;4:125. Originally published 2021 Dec 6. [Version 2] doi: 10.12688/hrbopenres.13435.2

Implementation of telemedicine consultations for people with mental health conditions in the community: a protocol for a systematic review

Emer Galvin 1,a, Shane Desselle 2, Blánaid Gavin 3, Etain Quigley 4, Mark Flear 5, Ken Kilbride 6, Fiona McNicholas 3, Shane Cullinan 1, John Hayden 1
PMCID: PMC9627101  PMID: 36348659

Version Changes

Revised. Amendments from Version 1

Our updated manuscript includes amendments in response to the feedback and suggestions from the peer reviewers: - We have clarified that the review is focused on general mental health services, and not specifically psychiatry. We have replaced the term “telepsychiatry” with “telemental health” in the manuscript to remove this ambiguity. - We have clarified that we are interested in all mental health professionals involved in the provision of mental health services via telemedicine. - We have acknowledged the findings of previous reviews in the broader area of telemental health implementation. - We have provided further rationale for the focus on video, phone and live-messaging consultations. - We have provided additional clarification for our decision to exclude grey literature and for the time limits of our review. - A second reviewer will now review 100% of full-text articles and extract data from a random 20% sample of articles.

Abstract

Background: The COVID-19 pandemic response has led to an exponential increase in the use and spread of telemedicine internationally. In community mental health care settings, telemedicine services were implemented within a few weeks, with little time for rigorous planning. Despite the reported acceptability of telemedicine by patients and clinicians, barriers to its implementation have come to light. There is now a need to investigate these barriers, and facilitators, as telemedicine begins to show potential promise beyond the pandemic. We propose a review that aims to identify the factors affecting the implementation of telemedicine consultations for patients with mental health conditions in the community.

Methods: A systematic review will be conducted and reported according to the PRISMA guidelines. Five electronic databases will be searched using a pre-defined search strategy from 2016 to 2021. Only studies of synchronous, interactive telemedicine consultations conducted via video, phone or live messaging between patients and providers will be included. Quantitative, qualitative and mixed methods studies will be eligible for inclusion. Only studies published in the English language will be included. Titles and abstracts will be screened by two reviewers. Full text articles will be screened by two reviewers. The methodological quality of studies will be assessed using the Mixed Method Appraisal Tool (MMAT) by two reviewers. Data will be extracted and tabulated to address the aims of the review. A narrative synthesis will be conducted and reported factors will be mapped to the domains of the Consolidated Framework for Implementation Research (CFIR).

Conclusion: By identifying the factors that influence the implementation of telemedicine consultations for patients with mental conditions in the community, consideration can be given to both barriers and facilitators that could be addressed in future mental health services planning.

PROSPERO registration: CRD42021273422 (04/10/2021)

Keywords: telemedicine, remote consultations, telepsychiatry, mental health, COVID-19

Introduction

The COVID-19 pandemic response caused swift, unprecedented changes in the delivery of healthcare. One such change was the rapid and widespread expansion of telemedicine services to comply with social distancing policies and reduce the spread of the virus 1, 2 . Telemedicine is defined as the provision of healthcare at a distance through telecommunications and information technologies 3 . Worldwide, longstanding regulatory barriers to telemedicine delivery were amended to facilitate its use during the pandemic 4 . In community settings, telemedicine allowed for the continued and essential access to mental health services during the pandemic. This use of telemedicine in mental health care is often referred to as telemental health, and is defined as the use of telecommunication for the provision or support of mental health services over a distance 5 . Heterogeneous definitions of telemedicine exist in the literature, so for the purposes of this review, telemedicine refers to live, synchronous remote consultations between provider and patient, using video, audio or live messaging modalities. These telemedicine consultations were chosen as the focus of this review as these types of consultations became commonplace during the pandemic 6 , acting as a temporary replacement for in-person consultations.

In the past two decades, telemedicine in mental health care has emerged as a safe and acceptable method of improving mental health care access for those who are disenfranchised or hard-to-reach. Systematic reviews and meta-analyses have shown that treatment effects of telemental health are comparable to face-to-face mental health care 7– 10 . There is also a body of evidence supporting the equivalence of telemental health to face-to-face care with regards to patient satisfaction and therapeutic alliance 11– 13 . Despite a steady increase in use and evidence for its effectiveness, telemedicine constitutes a small portion of all mental health services prior to the pandemic 14, 15 . Previous reviews of implementation factors have attributed the under-utilisation of telemedicine in mental health services to a number of reasons including strict licensure regulations and insurance policies that limit the reimbursement of telemedicine services 16, 17 and reluctance by clinicians 18, 19 .

The rapid and highly variable adoption of telemedicine in mental health care settings during the COVID-19 pandemic has shed some light on this research- implementation gap. A number of challenges and barriers have come to light, including lack of technological infrastructure, privacy concerns, difficulty in establishing rapport and problems with conducting high quality assessments 20, 21 . Despite these challenges, patients and clinicians have reported satisfaction with, and acceptance of, telemental health services during the pandemic 22, 23 . Moreover, some patients and service users have expressed the desire to continue to use telemental health services in the future 15, 24 ; a view mirrored by policy makers and mental health professionals 25, 26 . To harness the possible potential of telemedicine in future mental health care, a systematic exploration of the factors that affect successful telemedicine implementation in community mental health services is now needed. Identifying these factors, both enabling and hindering, will help ensure its acceptable and effective use going forward.

To gain a thorough understanding of the factors that influence the implementation of telemedicine into community mental health services, a strong theoretical foundation to guide interpretation of these factors is required. Various theories, models and frameworks have been developed in the area of implementation research to understand the determinants of translating research into practice 27 . One such framework is the Consolidated Framework for Implementation Research (CFIR) 28 . The CFIR consolidates the various terms used in implementation research into five domains considered to be important moderators or mediators of implementing practice into research 28 . These five domains include 1) intervention characteristics, 2) outer setting, 3) inner setting, 4) characteristics of individuals and 5) process. This framework has been used to assess the implementation of evidence-based practices in health and mental health settings 29, 30 . The CFIR encompasses terms and concepts from numerous implementation frameworks and has therefore been selected to map the findings of this review. Its comprehensiveness allows for the categorization of various implementation factors across a variety of study designs and patient populations 28 , which is relevant to this review.

To our knowledge, no systematic review exists of the factors that affect implementation of telemedicine for patients with mental health conditions in community settings. In consideration of the swift and variable adoption of telemedicine in the COVID-19 pandemic, a number of overlooked factors may emerge as likely determinants of successful telemedicine implementation. This review is pertinent as mental health professionals and policy makers are now interested in the continued, long-term use of telemedicine in mental services beyond the COVID-19 pandemic 14, 31 . Moreover, this review is particularly relevant given the unprecedented rise in mental health problems and increased demand on mental health services arising from the pandemic 32, 33 . It is crucial that we now take stock of the available evidence regarding the challenges and successes of implementing telemedicine to identify key factors for its acceptable adoption into routine community mental health care. Subsequently, consideration can be given to solutions that address these factors by stakeholders involved in mental health services planning.

The primary aim of this review is to identify, summarise and interpret the key factors affecting the implementation of telemedicine consultations for patients with mental health conditions in the community. The secondary aim is to map these factors to the domains of the CFIR.

Protocol

This protocol has been prepared following the PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols) 2015 checklist 34, 35 (See Reporting Guidelines). The protocol is registered on the International Prospective Register of Systematic Reviews (PROSPERO) as CRD42021273422. A systematic review design was chosen to synthesise the available literature. As the scope of the research question is narrow in nature and we are interested in synthesising empirical research, a systematic review approach was deemed appropriate 36 .

Eligibility criteria

Studies will be selected for inclusion in the systematic review according to the following PEO acronym criteria (population, exposure and outcome).

Population. The population will include adults and children (aged < 18 years) with a diagnosis of a mental disorder or in receipt of care from a mental health professional (e.g. psychiatrist, psychotherapist, counsellor). Mental disorders included will be in accordance with the International Classification of Diseases (ICD-11) criteria for mental and behavioural disorders (WHO, 2019). We will include common mental disorders such as depression, generalised anxiety disorder, social anxiety disorder, obsessive-compulsive disorder and post-traumatic stress disorder. Severe mental disorders such as schizophrenia and other psychotic disorders, and bipolar disorder will also be included. Neurocognitive disorders, such as dementia, perinatal mental disorders, disorders associated with substance abuse, disorders associated with stress, eating disorders and neurodevelopmental disorders, such as ADHD, will be included. Studies that focus on people learning difficulties, intellectual disabilities and people with mental health problems secondary due to physical illness will also be included. We will include studies which include health care professionals (e.g. doctors, nurses, allied health professionals) involved in the provision of mental health care via telemedicine to patients with the above conditions or without a formal psychiatric diagnosis.

Exposure. Studies evaluating synchronous, live, interactive telemedicine consultations between patient and provider, including video-conferencing, telephone and live-messaging only will be included. Studies will be restricted to those that use synchronous (real-time) consultations between a patient and one, or more, health care professionals. Studies will be excluded if they use asynchronous methods, in which the healthcare professional and patient do not interact in real-time, such as email communications. Studies exploring telemedicine consultations between practitioners, such as when a health care professional seeks advice from another practitioner, will be excluded.

Outcome. Studies with data on the factors that affect the implementation of telemedicine consultations for people with mental health conditions in the community will be included. Studies that explore the views and experiences of patients, parents/carers of patients and/or healthcare professionals on the implementation of telemedicine consultations for patients with mental health conditions in the community are eligible for inclusion. Studies only exploring anticipated or hypothetical views will be excluded.

Study design. Qualitative, quantitative and mixed-methods studies will be included in the review. Systematic reviews, meta-analyses and study protocols will be excluded as they do not contain primary research. We will only include full-text studies reported in the English language, due to constraints on resources to translate studies. Grey literature will be excluded, including non-peer reviewed articles, conference proceedings, case reports, editorials, opinion papers and letters. We have decided to exclude grey literature because of the variability in quality, peer review, supplication in full-text reviews and the need for prompt publication of findings.

Setting. Only studies in primary care, community or outpatient settings will be included. Studies in inpatient hospital settings, residential care homes, and prisons will be excluded.

Information sources and search strategy

An initial search of the MEDLINE database will be conducted to identify articles on the topic. Five electronic databases will be systematically searched for studies published between 2016 and 2021; MEDLINE (PubMed), Embase, Web of Science, CINAHL and APA PsycINFO. This timeframe (2016 to 2021) was chosen to include studies prior to, and during, the pandemic to ensure a comprehensive review of implementation-related factors. We chose not to include studies before this time point has the use of technology has grown dramatically in the past five years and the context in which telemedicine was implemented previously may be largely different to what is relevant in today’s context. The search strategy will contain terms pertaining to mental health conditions, telemedicine, community settings and implementation. See Extended Data for the PubMed sample search strategy 35 . Forward and backward citation searches of included articles will also be conducted.

Study selection

Two reviewers (EG and JH) will independently screen titles and abstracts for all articles in order to identify studies that meet the inclusion and exclusion criteria. The full texts of all selected articles will be collected and examined by two reviewers, independent of each other (EG and JH). Any disagreements will be mediated through a third reviewer (SC). Duplicates will be excluded. A PRISMA flow chart will display the articles examined at each stage, detailing the number of papers included and excluded and reasons for exclusions.

Data extraction and management

The following data items will be extracted from all studies by one reviewer (EG):

  • 1.

    Author(s)

  • 2.

    Publication year

  • 3.

    Date of data collection

  • 4.

    Country of publication

  • 5.

    Study aims

  • 6.

    Population characteristics

  • 7.

    Sample size

  • 8.

    Study design

  • 9.

    Telemedicine consultation type (video/phone/messaging)

  • 10.

    Key relevant outcomes relating to the research question

In addition, reported factors will be extracted and categorised into the five domains of the CFIR by one reviewer (EG). A second reviewer (JH) will extract data items from a random 20% sample of studies. Microsoft Excel software will be used to organize the extracted data. Any uncertainties regarding data will be resolved by attempting to conduct study authors via email. Selected articles will be stored and managed using EndNote X9 Reference Manager Library.

Quality assessment

Two reviewers (EG and JH) will independently assess the methodological quality of the included studies using the Mixed-Methods Appraisal Tool (MMAT) (Hong et al., 2018). Any discrepancies between study assessments will be discussed and resolved. Studies will not be excluded based on quality.

Data synthesis

One reviewer (EG) will perform a thematic synthesis on the extracted qualitative findings following Thomas and Harden’s 37 guidance. This will involve coding the qualitative findings and analysing the identifying themes relating to the implementation factors. This inductive approach was chosen to ensure all relevant implementation factors were identified, including those that may not fit in an existing framework. This synthesis will be verified by a second reviewer (JH). Quantitative findings will be summarised narratively. One reviewer (EG) will map the reported factors to the five domains of the CFIR (outer-setting, inner-setting, intervention, individual or process) and present them in a table format. We will use the GRADE Confidence in the Evidence from Reviews of Qualitative Research (CERQual) to rate the overall confidence in the qualitative evidence synthesis and the narrative summaries of the quantitative data 38 . Any notable similarities and differences in implementation factors between studies conducted before, and during, the pandemic will be discussed narratively.

Study status

The database searches were completed in August 2021. Full-text screening was completed in November 2021. It is anticipated that this review will be completed in March 2022.

Discussion

The rapid and unprecedented uptake of telemedicine since the COVID-19 pandemic has brought this modality of healthcare to the forefront of health services research. The potential of telemedicine to increase access to mental health services and alleviate the mental health burden is promising. However, challenges to its implementation in current mental health services are still present, such as the potential exacerbation of inequalities and technological barriers 39 . This review will aim to shed light on the factors that may enable or hinder the implementation of telemedicine for people with mental health conditions in the community. By identifying and interpreting these factors, consideration can be given to solutions that can optimise remotely-delivered mental health care for patients in the community.

A strength of this review is the use of a determinant framework, the CFIR, to map the findings to as it aids the transferability and generalisability of findings to other implementation studies 28 . Furthermore, the CFIR incorporates concepts from multiple implementation theories which makes it less likely that important factors will be overlooked 28 . A number of steps will be undertaken to minimise the risk of meta-biases in the review. Firstly, the systematic review will be conducted and reported using the PRISMA guidelines 40 . Secondly, the risk of selection bias will be minimised by two independent reviewers performing title and abstract screening. Thirdly, two independent reviewers will appraise the quality of the included studies. A limitation of the review is that the studies will be restricted to those conducted in the English language due to financial constraints which may bias the results. In addition, excluding unpublished studies and grey literature may increase the risk of publication bias. Another potential limitation is the heterogeneity of implementation factors and patient populations reported in the included studies. This may make it difficult to synthesise and compare the findings from each study.

Despite these limitations, this review will be an important and timely contribution to understanding how to improve the implementation of telemedicine for patients with mental health conditions in the community. Findings of the review may advise policy makers and other stakeholders involved in the implementation of telemedicine services, informing their future development. The results of the systematic review will be reported in a peer-reviewed journal, presented at national and international conferences and included in a PhD thesis.

Data availability

Underlying data

No underlying data are associated with this article.

Extended data

Open Science Framework: Implementation of telemedicine consultations for people with mental health conditions in the community: a protocol for a systematic review. https:///doi.org/10.17605/OSF.IO/CKBEQ 35 .

This project contains the following extended data:

  • Sample Search Strategy PubMed.pdf (PubMed Search Strategy)

Reporting guidelines

Open Science Framework: PRISMA-P checklist for "Implementation of telemedicine consultations for people with mental health conditions in the community: a protocol for a systematic review", https:///doi.org/10.17605/OSF.IO/CKBEQ 35 .

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

The authors would like to thank Paul Murphy, information specialist at RCSI, for his help in advising on the search strategies.

Funding Statement

EG is funded by a Clement Archer PhD Scholarship from the School of Pharmacy and Biomolecular Sciences, Royal College of Surgeons in Ireland.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 2; peer review: 1 approved, 1 approved with reservations]

References

  • 1. Chen JA, Chung WJ, Young SK, et al. : COVID-19 and telepsychiatry: Early outpatient experiences and implications for the future. Gen Hosp Psychiatry. 2020;66:89–95. 10.1016/j.genhosppsych.2020.07.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Wright JH, Caudill R: Remote Treatment Delivery in Response to the COVID-19 Pandemic. Psychother Psychosom. 2020;89(3):130–132. 10.1159/000507376 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Wootton R, Craig J, Patterson V: Introduction to Telemedicine.2 ed., London: CRC Press,2017;226. 10.1201/9781315272924 [DOI] [Google Scholar]
  • 4. Kinoshita S, Cortright K, Crawford A, et al. : Changes in telepsychiatry regulations during the COVID-19 pandemic: 17 countries and regions' approaches to an evolving healthcare landscape. Psychol Med. 2020;1–8. 10.1017/S0033291720004584 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Malhotra S, Chakrabarti S, Shah R: Telepsychiatry: Promise, potential, and challenges. Indian J Psychiatry. 2013;55(1):3–11. 10.4103/0019-5545.105499 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Car J, Koh GCH, Foong PS, et al. : Video consultations in primary and specialist care during the covid-19 pandemic and beyond. BMJ. 2020;371:m3945. 10.1136/bmj.m3945 [DOI] [PubMed] [Google Scholar]
  • 7. Bashshur RL, Shannon GW, Bashshur N, et al. : The Empirical Evidence for Telemedicine Interventions in Mental Disorders. Telemed J E Health. 2016;22(2):87–113. 10.1089/tmj.2015.0206 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Hubley S, Lynch SB, Schneck C, et al. : Review of key telepsychiatry outcomes. World J Psychiatry. 2016;6(2):269–282. 10.5498/wjp.v6.i2.269 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Mohr DC, Vella L, Hart S, et al. : The Effect of Telephone-Administered Psychotherapy on Symptoms of Depression and Attrition: A Meta-Analysis. Clin Psychol (New York). 2008;15(3):243–253. 10.1111/j.1468-2850.2008.00134.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Osenbach JE, O'Brien KM, Mishkind M, et al. : Synchronous Telehealth Technologies in Psychotherapy for Depression: A Meta-Analysis. Depress Anxiety. 2013;30(11):1058–1067. 10.1002/da.22165 [DOI] [PubMed] [Google Scholar]
  • 11. Christensen LF, Moller AM, Hansen JP, et al. : Patients' and providers' experiences with video consultations used in the treatment of older patients with unipolar depression: A systematic review. J Psychiatr Ment Health Nurs. 2020;27(3):258–271. 10.1111/jpm.12574 [DOI] [PubMed] [Google Scholar]
  • 12. Hilty DM, Ferrer DC, Parish MB, et al. : The effectiveness of telemental health: a 2013 review. Telemed J E Health. 2013;19(6):444–454. 10.1089/tmj.2013.0075 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Parish MB, Fazio S, Chan S, et al. : Managing Psychiatrist-Patient Relationships in the Digital Age: a Summary Review of the Impact of Technology-enabled Care on Clinical Processes and Rapport. Curr Psychiatry Rep. 2017;19(11):90. 10.1007/s11920-017-0839-x [DOI] [PubMed] [Google Scholar]
  • 14. Barnett P, Goulding L, Casetta C, et al. : Remote working in mental health services: a rapid umbrella review of pre-COVID-19 literature. medRxiv. 2020. 10.1101/2020.11.30.20240721 [DOI] [Google Scholar]
  • 15. Nicholas J, Bell IH, Thompson A, et al. : Implementation lessons from the transition to telehealth during COVID-19: a survey of clinicians and young people from youth mental health services. Psychiatry Res. 2021;299:113848. 10.1016/j.psychres.2021.113848 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Douglas MD, Xu J, Heggs A, et al. : Assessing Telemedicine Utilization by Using Medicaid Claims Data. Psychiatr Serv. 2017;68(2):173–178. 10.1176/appi.ps.201500518 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Scott Kruse C, Karem P, Shifflett K, et al. : Evaluating barriers to adopting telemedicine worldwide: A systematic review. J Telemed Telecare. 2018;24(1):4–12. 10.1177/1357633X16674087 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Brooks E, Turvey C, Augusterfer EF: Provider Barriers to Telemental Health: Obstacles Overcome, Obstacles Remaining. Telemed J E Health. 2013;19(6):433–437. 10.1089/tmj.2013.0068 [DOI] [PubMed] [Google Scholar]
  • 19. Cowan KE, McKean AJ, Gentry MT, et al. : Barriers to Use of Telepsychiatry: Clinicians as Gatekeepers. Mayo Clin Proc. 2019;94(12): 2510–2523. 10.1016/j.mayocp.2019.04.018 [DOI] [PubMed] [Google Scholar]
  • 20. Wyler H, Liebrenz M, Ajdacic-Gross V, et al. : Treatment provision for adults with ADHD during the COVID-19 pandemic: an exploratory study on patient and therapist experience with on-site sessions using face masks vs. telepsychiatric sessions. BMC Psychiatry. 2021;21(1): 237. 10.1186/s12888-021-03236-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Khan AW, Kader N, Hammoudeh S, et al. : Combating COVID-19 pandemic with technology: Perceptions of Mental Health Professionals towards Telepsychiatry. Asian J Psychiatr. 2021;61:102677. 10.1016/j.ajp.2021.102677 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Molfenter T, Roget N, Chaple M, et al. : Use of telehealth in substance use disorder services during and after COVID-19: Online survey study. JMIR Mental Health. 2021;8(2):e25835. 10.2196/25835 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Guinart D, Marcy P, Hauser M, et al. : Patient Attitudes Toward Telepsychiatry During the COVID-19 Pandemic: A Nationwide, Multisite Survey. JMIR Mental Health. 2020;7(12):e24761. 10.2196/24761 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Severe J, Tang R, Horbatch F, et al. : Factors Influencing Patients’ Initial Decisions Regarding Telepsychiatry Participation During the COVID-19 Pandemic: Telephone-Based Survey. JMIR Form Res. 2020;4(12):e25469. 10.2196/25469 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25. Johnson S, Dalton-Locke C, Vera San Juan N, et al. : Impact on mental health care and on mental health service users of the COVID-19 pandemic: a mixed methods survey of UK mental health care staff. Soc Psychiatry Psychiatr Epidemiol. 2021;56(1):25–37. 10.1007/s00127-020-01927-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Holmes EA, O'Connor RC, Perry VH, et al. : Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry. 2020;7(6):547–560. 10.1016/S2215-0366(20)30168-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Nilsen P: Making sense of implementation theories, models and frameworks. Implementation Sci. 2015;10(1):53. 10.1186/s13012-015-0242-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Damschroder LJ, Aron DC, Keith RE, et al. : Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. 10.1186/1748-5908-4-50 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Batsis JA, McClure AC, Weintraub AB, et al. : Barriers and facilitators in implementing a pilot, pragmatic, telemedicine-delivered healthy lifestyle program for obesity management in a rural, academic obesity clinic. Implement Sci Commun. 2020;1:83. 10.1186/s43058-020-00075-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Hawley CE, Genovese N, Owsiany MT, et al. : Rapid Integration of Home Telehealth Visits Amidst COVID‐19: What Do Older Adults Need to Succeed? J Am Geriatr Soc. Wiley Online Library.2020;68(11):2431–2439. 10.1111/jgs.16845 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Moreno C, Wykes T, Galderisi S, et al. : How mental health care should change as a consequence of the COVID-19 pandemic. Lancet Psychiatry. 2020;7(9):813–824. 10.1016/S2215-0366(20)30307-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. Torous J, Myrick KJ, Rauseo-Ricupero N, et al. : Digital Mental Health and COVID-19: Using Technology Today to Accelerate the Curve on Access and Quality Tomorrow. JMIR Ment Health. 2020;7(3):e18848. 10.2196/18848 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33. Racine N, McArthur BA, Cooke JE, et al. : Global Prevalence of Depressive and Anxiety Symptoms in Children and Adolescents During COVID-19: A Meta-analysis. JAMA Pediatr. 2021;175(11):1142–1150. 10.1001/jamapediatrics.2021.2482 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Moher D, Shamseer L, Clarke M, et al. : Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. 10.1186/2046-4053-4-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Galvin E: Implementation of telemedicine consultations for people with mental health conditions in the community: a protocol for a systematic review.2021. 10.17605/OSF.IO/CKBEQ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Paré G, Trudel MC, Jaana M, et al. : Synthesizing information systems knowledge: A typology of literature reviews. Inf Manag. 2015;52(2):183–99. 10.1016/j.im.2014.08.008 [DOI] [Google Scholar]
  • 37. Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45. 10.1186/1471-2288-8-45 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Lewin S, Booth A, Glenton C, et al. : Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(Suppl 1):2. 10.1186/s13012-017-0688-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39. Stoll J, Sadler JZ, Trachsel M: The Ethical Use of Telepsychiatry in the Covid-19 Pandemic. Front Psychiatry. 2020;11:665. 10.3389/fpsyt.2020.00665 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Page MJ, McKenzie JE, Bossuyt PM, et al. : The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. 10.1136/bmj.n71 [DOI] [PMC free article] [PubMed] [Google Scholar]
HRB Open Res. 2022 Jun 23. doi: 10.21956/hrbopenres.14827.r32280

Reviewer response for version 2

Karen Jean Day 1

Thank you for making changes to your protocol in response to my review.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Telehealth, mental health and telemedicine, e-mental health, digital health, health informatics.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

HRB Open Res. 2022 Jun 21. doi: 10.21956/hrbopenres.14827.r32281

Reviewer response for version 2

Shalini Lal 1, Rossana Peredo 2

Thank you to the authors for their detailed answers. The authors have addressed the concerns we raised in our peer review report appropriately, with the exception of one comment regarding methodology: 

" Comment 5: Methods: Authors should consider piloting the data extraction process with two reviewers, and include a second reviewer to validate the data extraction to ensure consistency and comprehensiveness in the data extraction process, and to minimize biases and human error.

Response: Thank you for this comment. We agree that independent data extraction from two reviewers would help to improve consistency and comprehensiveness. Due to resource and time constraints, a second reviewer will now extract data items from a random 20% sample of studies to minimize bias. The Data extraction and management section of the manuscript has been amended to reflect this change."

The response clarifies the reason for using only one reviewer for data extraction, however, we consider that this should be acknowledged in the discussion section as part of the limitations. Also, it is unclear if the second reviewer will extract data from a random 20% sample (pg. 5, line 22) as part of a pilot or a validation process. We suggest clarifying and describing the details of this process. Will the second reviewer extract the data before or after the first reviewer completes the entire data extraction? How will the two extractions be compared? What will happen in case of disagreement? 

Finally, a minor editorial comment: there seems to be a typo: conduct vs contact, in the following phrase: “Any uncertainties regarding data will be resolved by attempting to conduct study authors via email” (pg. 5)”.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Partly

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

NA

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

HRB Open Res. 2022 Sep 23.
Emer Galvin 1

Comment:

"Comment 5: Methods: Authors should consider piloting the data extraction process with two reviewers, and include a second reviewer to validate the data extraction to ensure consistency and comprehensiveness in the data extraction process, and to minimize biases and human error.

Response: Thank you for this comment. We agree that independent data extraction from two reviewers would help to improve consistency and comprehensiveness. Due to resource and time constraints, a second reviewer will now extract data items from a random 20% sample of studies to minimize bias. The Data extraction and management section of the manuscript has been amended to reflect this change."

The response clarifies the reason for using only one reviewer for data extraction, however, we consider that this should be acknowledged in the discussion section as part of the limitations.

Response:

Sincere thanks for your comments. We have acknowledged the limitation of using a single reviewer for data extraction in the Discussion section:

A final limitation is the use of one reviewer to extract data which may increase the risk of errors at this stage.

Comment:

Also, it is unclear if the second reviewer will extract data from a random 20% sample (pg. 5, line 22) as part of a pilot or a validation process. We suggest clarifying and describing the details of this process. Will the second reviewer extract the data before or after the first reviewer completes the entire data extraction? How will the two extractions be compared? What will happen in case of disagreement?

Response:

Thank you for this comment. We have sought guidance on the data extraction process from Taylor, Mahtani, and Aronson (2021) and have clarified the details of data extraction in the manuscript. Please see changes to the Data extraction and management section below:

Due to restraints on resources that prevent independent extraction of data by two reviewers, one reviewer (EG) will extract the data while a second reviewer (JH) will review a random 20% of the extracted data for accuracy. Any disagreements between the reviewers relating to the extracted data will aimed to be resolved through consensus. If the two reviewers are unable to come to a consensus, a third reviewer (SC) will be consulted. If the third reviewer is unable to arbitrate, the study authors will be contacted to seek clarification on the issue. If this step is unsuccessful, the disagreement will be recorded and reported in the review (Taylor, Mahtani & Aronson, 2020). In addition, reported factors will be extracted and categorised into the five domains of the CFIR by one reviewer (EG).

Reference: Taylor KS, Mahtani KR, Aronson JK. Summarising good practice guidelines for data extraction for systematic reviews and meta-analysis. BMJ Evidence-Based Medicine 2021;26:88-90.

Comment:

Finally, a minor editorial comment: there seems to be a typo: conduct vs contact, in the following phrase: “Any uncertainties regarding data will be resolved by attempting to conduct study authors via email” (pg. 5)”.

Response:

Thank you for spotting this error. We have carefully reviewed the manuscript for typographical errors.

HRB Open Res. 2022 Mar 7. doi: 10.21956/hrbopenres.14639.r30996

Reviewer response for version 1

Shalini Lal 1, Rossana Peredo 2

The authors present a protocol of a systematic review that aims to identify and summarize the key factors that affect the implementation of telemedicine based on recent literature. This purpose is particularly timely given that the use of telemedicine has increased exponentially especially during the pandemic, along with the number of publications on this subject. In general, the protocol is aligned with guidelines for systematic review protocols, and the consideration of theory to guide the interpretation of the factors is a strength of the review. The following points would be important to consider to strengthen the protocol:

  1. Introduction: It would be good to provide some indication of existing reviews on the topic to help strengthen and better situate this review in relation to what is already known on the topic of implementation factors pertaining to telemedicine – especially given decades of research existing on the topic. There are a number of reviews related to the current review that could be considered, some that are focused on implementation factors pertaining to specific mental health conditions (e.g., Kruse et al. 2018 1 ), others that are focused on implementation factors in telemedicine (that include as part of the included studies those focused on mental health field), and others that review telemedicine in mental health services more broadly that include a component addressing implementation factors.

  2. Introduction: Clarify the rationale for including phone-based telemedicine, video, and messaging, as the technologies are different, with implementation factors likely to differ at least in part. Moreover, phone-based telemedicine has a longer, multi-decade research history including implementation research, and thus the rationale for the need to include this format of telemedicine in this current review is unclear.

  3. Methods: Given the focus on Covid-19 in the abstract and introduction, it would be good to elaborate the justification for choosing 2016 (vs. 2019, vs. 2015 etc.) as publication date restriction. Moreover, the objective of a systematic review is to cover all available literature, as such, any restrictions should be well justified.

  4. Methods: Involving two reviewers during screening and selection process, and piloting the process to ensure consistency taken in the study selection process is generally considered important to minimize bias and human error. Please clarify why a second reviewer will only be involved in 20% of random sample validation and this only at the stage of full text article retrieval.

  5. Methods: Authors should consider piloting the data extraction process with two reviewers, and include a second reviewer to validate the data extraction to ensure consistency and comprehensiveness in the data extraction process, and to minimize biases and human error.

  6. Method: It would be good to clarify how authors will handle the studies pre-pandemic versus during pandemic, as the implementation factors may overlap but also differ.

  7. Discussion: It would be good to reduce the repetition in the first paragraph of the discussion with what is written in the introduction section of the paper

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Partly

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

NA

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

References

  • 1. : Factors influencing the adoption of telemedicine for treatment of military veterans with post-traumatic stress disorder. J Rehabil Med .2018;50(5) : 10.2340/16501977-2302 385-392 10.2340/16501977-2302 [DOI] [PubMed] [Google Scholar]
HRB Open Res. 2022 May 31.
Emer Galvin 1

Dear Dr Shalini Lal & Dr Rossana Peredo,

Thank you very much for your comments. We appreciate your valuable comments and have responded to each one below.

Comment: Introduction: It would be good to provide some indication of existing reviews on the topic to help strengthen and better situate this review in relation to what is already known on the topic of implementation factors pertaining to telemedicine – especially given decades of research existing on the topic. There are a number of reviews related to the current review that could be considered, some that are focused on implementation factors pertaining to specific mental health conditions (e.g., Kruse et al. 20181), others that are focused on implementation factors in telemedicine (that include as part of the included studies those focused on mental health field), and others that review telemedicine in mental health services more broadly that include a component addressing implementation factors.

Response: Thank you for this comment. We acknowledge that previous reviews in the broader area of telemental health have been conducted to examine implementation barriers and facilitators. Given the increase in telemedicine adoption during the pandemic, it is likely that new implementation factors may emerge, or indeed previous challenges and benefits may be emphasised. We have amended the Introduction section to highlight existing reviews, which can be seen in bold below:

Previous reviews of implementation factors have attributed the under-utilisation of telemedicine in mental health services to a number of reasons including strict licensure regulations and insurance policies that limit the reimbursement of telemedicine services 15, 16 and reluctance by clinicians 17, 18. (pg. 3)

Comment: Introduction: Clarify the rationale for including phone-based telemedicine, video, and messaging, as the technologies are different, with implementation factors likely to differ at least in part. Moreover, phone-based telemedicine has a longer, multi-decade research history including implementation research, and thus the rationale for the need to include this format of telemedicine in this current review is unclear .

Response: Thank you for this comment. We chose to focus on live, synchronous methods of telemedicine as these were most commonly used as a replacement for in-person consultations during the pandemic. We chose to focus on these three modalities of telemedicine consultations which often include both phone and video (and sometimes live-messaging) as health care professionals and patients use these interchangeably, and sometimes in combination with each other. In other cases, video consultations may be used without visual input which would have similarities to phone consultations. We have amended the manuscript to provide additional rationale for our decision to focus on these live, synchronous modalities of telemedicine. Please see below the change in bold:

These telemedicine consultations were chosen as the focus of this review as these types of consultations became commonplace during the pandemic 1, acting as a temporary replacement for in-person consultations. (pg. 3)

Comment: Methods: Given the focus on Covid-19 in the abstract and introduction, it would be good to elaborate the justification for choosing 2016 (vs. 2019, vs. 2015 etc.) as publication date restriction. Moreover, the objective of a systematic review is to cover all available literature, as such, any restrictions should be well justified

Response: Thank you for this comment. We chose the five-year time restriction for a number of reasons. Firstly, a scoping search of the literature revealed a low number of studies on remote mental health consultations before the five-year time point (i.e. consultations that mirror those that were conducted during the pandemic). Secondly, the use of technology has increased in recent years, which means that telemental health before this time point was conducted in a different context and implementation factors may not be relevant to the current technology and health landscape. We have amended the manuscript to include further rationale for this time restriction. Please see the amendment in bold below:

We chose not to include studies before this time point has the use of technology has grown dramatically in the past five years and the context in which telemedicine was implemented previously may be largely different to what is relevant in today’s context. (pg. 4)

Comment: Methods: Involving two reviewers during screening and selection process, and piloting the process to ensure consistency taken in the study selection process is generally considered important to minimize bias and human error. Please clarify why a second reviewer will only be involved in 20% of random sample validation and this only at the stage of full text article retrieval.

Response: Thank you for this comment. The second reviewer will now review 100% of full-text articles, independent to the first reviewer. The Study selection section of the manuscript has been amended to include this change. Please see the change in bold below:

The full texts of all selected articles will be collected and examined by two reviewers, independent of each other (EG and JH). (pg. 4)

Comment: Methods: Authors should consider piloting the data extraction process with two reviewers, and include a second reviewer to validate the data extraction to ensure consistency and comprehensiveness in the data extraction process, and to minimize biases and human error.

Response: Thank you for this comment.  We agree that independent data extraction from two reviewers would help to improve consistency and comprehensiveness. Due to resource and time constraints, a second reviewer will now extract data items from a random 20% sample of studies to minimize bias. The Data extraction and management section of the manuscript has been amended to reflect this change.

Comment: Method: It would be good to clarify how authors will handle the studies pre-pandemic versus during pandemic, as the implementation factors may overlap but also differ.

Response: Thank you for this comment. We plan to synthesis all included studies in one analysis. However, we plan to highlight similarities and differences narratively in the Results and Discussion section of the review. We have amended the Data synthesis section of manuscript to clarify this. Please see the change in bold below:

Any notable similarities and differences in implementation factors between studies conducted before, and during, the pandemic will be discussed narratively. (pg. 5)

Comment: Discussion: It would be good to reduce the repetition in the first paragraph of the discussion with what is written in the introduction section of the paper

Response: Thanks for this comment. We have made some amendments in the Discussion section to reduce repetition from the Introduction section.

HRB Open Res. 2022 Feb 14. doi: 10.21956/hrbopenres.14639.r31312

Reviewer response for version 1

Karen Jean Day 1

Thank you for the opportunity to review this literature review protocol about telemedicine consultations for people with mental health conditions in the community. Overall, it is very well-written, clearly stating the review purpose, what is planned, how it will be done, and what outcomes are expected. It is good to see the use of frameworks to strengthen the literature review. The protocol could be improved in the following ways.

In the introduction, first paragraph, the authors indicate that the study is about telemedicine, and they provide an excellent definition of the concept. The link between mental health, psychiatry, and mental health professionals is not as clear. Is this review about general mental health and all those healthcare professionals who provide services, or is it focused on psychiatry? It appears that the review is about the former, but the boundaries are blurred in ways that may create confusion. Who are the mental health professionals who are not psychiatrists, e.g., do psychiatric nurses, psychologists, counselors, social workers qualify as ‘mental health professionals’? I ask this question because you later say (page 4, section on population), “ We will include studies which include health care professionals (e.g. doctors, nurses, allied health professionals) involved in the provision of telemedicine to patients with the above conditions.” This statement introduces ambiguity about who the healthcare professionals are who you assume to provide mental health care via telemedicine. It would be good if you could remove this ambiguity throughout the paper.

The choice of a systematic review appears to be a good one. However, one cannot judge this without some discussion about alternative types of reviews that you may have considered and rejected for various reasons. The article by Pare et al. (2015) 1 can help you briefly write about alternative approaches that you have rejected to strengthen your decision for a systematic review.

Your choice to exclude conference proceedings may not work in your favour. There are many useful digital health- and telehealth-related conference proceedings that are peer-reviewed and published and you may end up with an unhelpful bias. Many digital health practitioners attend these conferences and are influenced in their practice by what they learn from the conferences. Conversely, many of them don’t have access to journal publications that are behind a subscription paywall. IEEE is a highly regarded conference and their proceedings are often given the same value as journal publications. Telehealth and digital health conferences, such as MedInfo, are published in “Studies in Health Technology and Informatics”. It would be a shame to exclude these peer-reviewed papers that influence practice and policy. You plan to do a quality review of the included publications to protect the quality of your review, so you can still report on conference proceedings without compromising the review. If you choose to not take this advice, please indicate why in your final published protocol.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Telehealth, mental health and telemedicine, e-mental health, digital health, health informatics.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References

  • 1. : Synthesizing information systems knowledge: A typology of literature reviews. Information & Management .2015;52(2) : 10.1016/j.im.2014.08.008 183-199 10.1016/j.im.2014.08.008 [DOI] [Google Scholar]
HRB Open Res. 2022 May 31.
Emer Galvin 1

Dear Dr Karen Day,

Sincere thanks for your comments. We appreciate your valuable input and have responded to your comments below. Please see the amendments in the revised paper.

Comment: In the introduction, first paragraph, the authors indicate that the study is about telemedicine, and they provide an excellent definition of the concept. The link between mental health, psychiatry, and mental health professionals is not as clear. Is this review about general mental health and all those healthcare professionals who provide services, or is it focused on psychiatry? It appears that the review is about the former, but the boundaries are blurred in ways that may create confusion.

Response: Thanks for this comment. This review is about general mental health and all those healthcare professionals who provide mental health services. We acknowledge that the various terms (i.e. telepsychiatry) may cause some confusion so we have changed any mention of the term “telepsychiatry” to “telemental health” in the manuscript in an attempt to overcome this ambiguity.

Comment: Who are the mental health professionals who are not psychiatrists, e.g., do psychiatric nurses, psychologists, counselors, social workers qualify as ‘mental health professionals’? I ask this question because you later say (page 4, section on population), “We will include studies which include health care professionals (e.g. doctors, nurses, allied health professionals) involved in the provision of telemedicine to patients with the above conditions.” This statement introduces ambiguity about who the healthcare professionals are who you assume to provide mental health care via telemedicine. It would be good if you could remove this ambiguity throughout the paper.

Response: Thank you for this comment. We will include studies of healthcare professionals involved in the provision of mental health services via telemedicine. We have amended the Population section of the manuscript to attempt to clarify any ambiguity regarding this inclusion criteria. We have also amended the Population section to include mental health professionals who provide mental health services to people without a formal diagnosis of a mental disorder. Please see amendments to the Population section in bold below:

The population will include adults and children (aged < 18 years) with a diagnosis of a mental disorder or in receipt of care from a mental health professional (e.g. psychiatrist, psychotherapist, counsellor). (pg. 4)

We will include studies which include healthcare professionals (e.g. doctors, nurses, psychologists, counsellors, psychotherapists) involved in the provision of mental health care via telemedicine to patients with the above conditions or without a formal psychiatric diagnosis. (pg. 4)

Comment: The choice of a systematic review appears to be a good one. However, one cannot judge this without some discussion about alternative types of reviews that you may have considered and rejected for various reasons. The article by Pare et al. (2015)1 can help you briefly write about alternative approaches that you have rejected to strengthen your decision for a systematic review.

Response: Thank you for this comment and for attaching that reference. We chose not to do a scoping review as the scope of the questions is quite narrow, which is more suited to a systematic review. Furthermore, a scoping review is relevant when attempting to understand the initial size and nature of the available literature. This type of methodology is not suited to this review where previous reviews have been conducted in the broader area of telemental health implementation, prior to the pandemic. We also intended to include only empirical research which is more suited to a systematic review design, than a scoping review design. We decided not to conduct a meta-analysis as we are interested in narratively presenting our findings. It is unlikely that a meta-analysis would be possible as the outcome of interest (e.g. implementation factors) does not have standardised outcome measure that would be comparable across studies. We have included additional sentences in the Protocol section of the manuscript outlining our decision. Please see changes in bold below:

A systematic review design was chosen to synthesise the available literature. As the scope of the research question is narrow in nature and we are interested in synthesising empirical research, a systematic review approach was deemed appropriate 2. (pg. 3)

Comment: Your choice to exclude conference proceedings may not work in your favour. There are many useful digital health- and telehealth-related conference proceedings that are peer-reviewed and published and you may end up with an unhelpful bias. Many digital health practitioners attend these conferences and are influenced in their practice by what they learn from the conferences. Conversely, many of them don’t have access to journal publications that are behind a subscription paywall. IEEE is a highly regarded conference and their proceedings are often given the same value as journal publications. Telehealth and digital health conferences, such as MedInfo, are published in “Studies in Health Technology and Informatics”. It would be a shame to exclude these peer-reviewed papers that influence practice and policy. You plan to do a quality review of the included publications to protect the quality of your review, so you can still report on conference proceedings without compromising the review. If you choose to not take this advice, please indicate why in your final published protocol.

Response: Thank you very much for your comment. We acknowledge that IEEE is a useful resource, and includes peer-reviews papers. We will use this resource to help shape recommendations in the Discussion section of the final review. We acknowledge that excluding conference proceedings may run the risk of excluding potentially relevant studies. We have decided to exclude this “grey literature” because of the variability in quality, peer review, supplication in full-text reviews, and the need for prompt publication of findings. We recognise the potential limitation of excluding these studies and this is reflected in the Discussion section. We will also acknowledge this as a limitation on the final review. We have amended the Study design section of the protocol to clarify why we have excluded grey literature. Please see the amendment in bold below.

We have decided to exclude grey literature because of the variability in quality, peer review, supplication in full-text reviews, and the need for prompt publication of findings. (pg. 4).

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Availability Statement

    Underlying data

    No underlying data are associated with this article.

    Extended data

    Open Science Framework: Implementation of telemedicine consultations for people with mental health conditions in the community: a protocol for a systematic review. https:///doi.org/10.17605/OSF.IO/CKBEQ 35 .

    This project contains the following extended data:

    • Sample Search Strategy PubMed.pdf (PubMed Search Strategy)

    Reporting guidelines

    Open Science Framework: PRISMA-P checklist for "Implementation of telemedicine consultations for people with mental health conditions in the community: a protocol for a systematic review", https:///doi.org/10.17605/OSF.IO/CKBEQ 35 .

    Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

    Articles from HRB Open Research are provided here courtesy of Health Research Board Ireland

    RESOURCES