Abstract
Introduction
Physicians often suffer from burnout and stress, not only affecting themselves, but also their patients and the healthcare system in general. An increasing number of studies suggest that mindfulness-based interventions improve physicians’ well-being as well as the quality of care they deliver. However, the evidence is scattered, and a systematic review and meta-analysis is lacking. To the best of our knowledge, this systematic review and meta-analysis will be the first to assess the effectiveness of mindfulness-based interventions in reducing burnout and stress among physicians. Further, it aims to uncover potential moderators of intervention effectiveness.
Methods and analysis
MEDLINE, Embase, PsycINFO, PSYINDEX, Web of Science, CINAHL and the Cochrane Central Register of Controlled Trials will be screened without language or publication date restrictions. In addition, backward and forward citation searches of included studies and relevant reviews will be conducted. Studies examining the effect of interventions for physicians explicitly based on mindfulness will be included. Primary outcomes will be pre-post changes in burnout and stress if assessed with validated measures. Two reviewers independently search, select and extract data, and rate the methodological quality of the studies. Both controlled and uncontrolled studies will be included. Randomised controlled trails will be meta-analysed separately using between-group effect. In addition, non-randomised trials including non-controlled before-after studies will be meta-analysed using within-group effect. Potential moderators and sources of between-study heterogeneity will be tested using meta-regression and subgroup analyses. Futhermore, a narrative synthesis will be pursued. The Grading of Recommendations Assessment, Development and Evaluation system (GRADE) will be used to assess the quality of the cumulated evidence.
Ethics and dissemination
Ethical approval is not required. Results will be published in a peer-reviewed journal and presented at international conferences.
PROSPERO registration number
CRD42019133077
Keywords: mindfulness-based intervention, physician, burnout, stress, systematic review, meta-analysis
Strengths and limitations of this study.
We conduct the first systematic review and meta-analysis on the effectiveness of mindfulness-based interventions in reducing burnout and stress among physicians, using a fine-meshed yet comprehensive literature search.
We follow the Preferred Reporting Items for Systematic Review and Meta-analysis Protocols guidelines.
Limitations of the quality of evidence will be assessed using the Grading of Recommendations Assessment, Development and Evaluation system (GRADE).
We consider all relevant evidence by separately providing effect estimates for randomised controlled trails and non-randomised trials including non-controlled before-after studies.
The diversity of intervention formats and designs of included studies might lead to considerable heterogeneity among studies.
Introduction
Rationale
Medicine is a rewarding and at the same time highly demanding and stressful profession. Physicians are exposed to human suffering, need to take on tremendous responsibility and face expectations of faultless performance. They need to deal with excessive workloads and long working hours and often struggle to balance professional and personal life.1 Among the stages of a physician’s career, medical residency is a particularly demanding period. At the beginning of their career, resident physicians often experience role transition and relocation, resulting in fewer available support systems and feelings of isolation.2 A lack of supervisory support, restricted autonomy and the frequent confrontation with unfamiliar and difficult job demands have deleterious impact on resident physicians’ well-being.3 4 Consequently, in comparison to the general population and other healthcare professions, physicians and especially resident physicians have higher prevalences of burnout and stress.5–7
Burnout is a work-related syndrome characterised by emotional exhaustion, often complemented by depersonalisation and a low sense of personal accomplishment.8 9 Burnout has negative personal consequences, as physicians suffering from burnout are more at risk of substance abuse, relationship trouble, depression and suicide.10 Furthermore, burnout endangers quality of care and patient safety, because affected physicians commit more medical errors,11 12 adhere less to practice and safety standards13 and are more likely to provide suboptimal patient care.14 Consequently, patients of physicians suffering from burnout are less satisfied with the care they receive and take longer to recover.15 On a societal level, burnout in physicians causes increased costs in healthcare system through higher levels of absenteeism, job turnover and early retirement as well as reduced productivity and less patient access to physician care.15 These consequences, in turn, increase the work-related stressors that account for burnout and stress, as they lead to even more workload for the remaining physicians, thus creating a vicious circle. In light of these findings, it is important to improve physicians’ well-being, for the sake of themselves, their patients and the healthcare system.
A promising means of improving well-being is to cultivate mindfulness. Mindfulness can be described as non-judgmental and calm awareness of the present moment with an attitude of curiosity, openness and acceptance.16 It is often taught in mindfulness-based interventions (MBIs), which have been shown to be effective within a wide range of clinical and nonclinical settings.17 18 Mindfulness is flexibly implemented, portable, self-directed and noninvasive; all of which are attractive properties to busy practitioners such as physicians.19
MBIs for physicians have recently become the subject of extensive research. Krasner et al20 found that an 8-week mindfulness programme for primary care physicians reduced burnout and increased empathy.20 Importantly, improvements in mindfulness predicted improvements in the emotional exhaustion and personal accomplishment burnout subscales. In a randomised controlled trial (RCT), Verweij et al21 did not find significant between-group effects of an MBI on the emotional exhaustion values of Dutch resident physicians.21 Nevertheless, the baseline values had a moderating effect, suggesting that MBIs might be especially beneficial to those with greater emotional exhaustion.
Up to now, there is no meta-analytical assessment of the effectiveness of MBIs among physicians. Nevertheless, several reviews and meta-analyses within broader research questions hint at the usefulness of MBIs to reduce physicians’ burnout and stress.22–26 However, these previous reviews and meta-analyses did either not focus on MBIs,22–25 or not on physicians,26 despite the fact that physicians are more affected by work-related stress than other healthcare professionals.7 Moreover, in recent years, a growing number of trials on MBIs for physicians have been published of which none is covered by any of the available reviews.19 21 27 28
In summary, MBIs seem to be promising in reducing burnout and stress in physicians. Nevertheless, the evidence is scattered, and a systematic summary of the increasing number of studies is missing.
Objectives
The aim of this systematic review and meta-analysis is to examine the effectiveness of MBIs in reducing burnout and stress among physicians. It further aims to explore potential moderators of the intervention effectiveness and sources of between-study heterogeneity such as characteristics of the specific intervention, the tested population or the study design.
Methods
We prepared this protocol in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols.29 The report of the systematic review and meta-analysis will follow the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines.30 We registered this study protocol at the International Prospective Register of Systematic Reviews platform (PROSPERO).
Eligibility criteria
Population
We seek to draw conclusions on the effectiveness of MBIs in reducing burnout and stress in the population of physicians. In consequence, eligible study populations include practicing physicians or resident physicians regardless of their work setting (hospital or primary care), specialty or country. By contrast, we will exclude investigations on medical students or healthcare providers other than physicians. Criteria for inclusion and exclusion are given in table 1.
Table 1.
Inclusion and exclusion criteria
Criterion | Inclusion | Exclusion |
Population | Practicing physicians and resident physicians | Medical students, healthcare providers other than physicians, mixed samples |
Intervention | Interventions explicitly based on mindfulness | Interventions without explicit focus on mindfulness |
Comparator | Randomised controlled trials, non-randomised trials, non-controlled before-after studies | Case-control studies, systematic reviews, meta-analyses, clinical case studies, qualitative studies, editors’ letters |
Outcome | Burnout or stress measured using validated self-report questionnaires, pre and post intervention | Self-report questionnaires without validation |
Language | All languages | None |
Publication date | All dates | None |
Interventions
Due to its standardised structure, the mindfulness-based stress reduction programme (MBSR31) is one of the most popular and most evaluated MBIs.17 Nevertheless, as we seek to provide a summary of MBIs effectiveness in their practical application, we must not neglect the extensive use of various forms of MBIs other than MBSR. Hence, we do not restrict our analyses to MBSR, but include all eligible references that explicitly base their intervention on mindfulness. However, we will exclude interventions that might integrate mindfulness elements but without explicitly stating to be based on mindfulness. As we will assess the impact of the dosage of treatment on the effectiveness of MBIs, no minimum dosage of treatment is set for inclusion.
Study design and comparators
Randomized controlled trials (RCTs) will be included as well as non-randomised trials (NRTs) including non-controlled before-after studies (NCBAs). With RCTs, both active or waitlist control conditions will be considered. While an exclusion of NRTs may lead to neglecting certain evidence,32 the combination of NRTs with RCTs may over-estimate the treatment effect.33 Consequently, RCTs and NRTs, including NCBAs, will be analysed in separate subgroup meta-analyses to evaluate the characteristics of the study design as an influence on the intervention effects.
Outcome measures
The main outcomes of this systematic review and meta-analysis will be changes in burnout and stress from pre to post intervention. Outcomes must have been assessed using validated self-report questionnaires. For studies measuring burnout with Maslach Burnout Inventory we will only consider the emotional exhaustion subscale, as it is recommended against aggregating the different facets of the inventory,9 and as did other relevant meta-analyses22 34 and primary studies.21 35 In addition, as MBIs are supposed to unfold their positive effects through the training of mindfulness,31 we will extract validated self-report measures of mindfulness as a secondary outcome.
Search strategy
To ensure a comprehensive and multidisciplinary literature search we will screen seven electronic bibliographic databases: MEDLINE, PsycINFO, PSYINDEX, Web of Science, CINAHL, Embase and the Cochrane Central Register of Controlled Trials using a search strategy optimised for sensitivity. To find all potentially relevant studies we will not apply language or publication date restrictions. Search terms are related to (1) mindfulness, (2) interventions and (3) terms concerning the professional title of physicians. Exemplary searches for PsycINFO, Web of Science and Medline are available in the online supplementary material. To find studies in the grey literature, we will contact experts in the field of mindfulness research in healthcare, including authors of eligible studies, authors of article and conference abstracts identified through the database search as well as authors of significant reviews.22 25 26 Furthermore, we will conduct backward citation searches of all included studies and relevant reviews, as well as forward citation searches in Google Scholar, in order to find hitherto unidentified references. Finally, if the full text of certain references is not accessible, we will contact the authors of these studies.
bmjopen-2019-032295supp001.pdf (72.4KB, pdf)
Study selection
We will use Ryyan36 and Zotero37 to manage identified studies. Two reviewers (JCF and JJB) will independently screen titles and abstracts. Full texts will be obtained if at least one reviewer judges an article to meet inclusion criteria. Subsequently, the two reviewers will verify the eligibility of individual references through independent full-text screening. If discrepancies cannot be resolved through discussion, a third reviewer (ASG) will be consulted. Cohen’s kappa will be calculated to determine the agreement between reviewers.38 Reasons for exclusions will be recorded. According to the PRISMA guidelines, we will illustrate the process of literature search and study selection in a flow diagram.30
Data extraction
Information from eligible studies will be extracted independently by two reviewers (JCF and JJB) using a standardised form. Extracted data will include information on (1) the study: authors, publication date, country, experimental design, type of control, (2) the population: sample size (treatment/control), mean age, sex proportion, dropout, career stage (resident/practicing physician), (3) the intervention: delivery format (online/offline/mixed), duration of an average single session, number of sessions, home practice, exposure (time under professional guidance), theoretical background (MBSR/adapted MBSR/other MBIs), group size and (4) the outcomes: means and SD for burnout, stress and self-reported mindfulness for all conditions (pre, post and follow-up). Extracted data will be entered into Cochrane Collaboration’s Review Manager (RevMan V.5.3) and statistical software R. If insufficient data are provided to calculate effect sizes or to determine the methodological quality of the original study, we will contact the authors of the respective studies for clarification.
Risk of bias assessment in individual studies
Two reviewers (JCF and JJB) will independently perform risk of bias assessment. If discrepancies cannot be resolved through discussion, a third reviewer (ASG) will be consulted. For RCTs, we will use the Cochrane risk of bias tool for randomised trials V.2.0 (ROB V.2.0).39 ROB V.2.0 is a domain-based evaluation considering bias arising from (1) the randomisation process, (2) deviations from intended interventions, (3) missing outcome data, (4) measurement of the outcome and (5) selection of the reported result. The risk of bias in each domain will be rated as either ‘low risk of bias’, ‘some concerns’ or ‘high risk of bias’. Furthermore, a corresponding overall risk-of-bias judgement for each study will be made. For uncontrolled or NRTs we will use the effective public health practice project quality assessment tool for quantitative studies.40 It rates study quality in eight sections: (1) selection bias, (2) study design, (3) confounders, (4) blinding, (5) data collection methods, (6) withdrawals and drop-outs, (7) intervention integrity and (8) quantitative analyses of single studies. In each section, the evidence will be rated as ‘strong’, ‘moderate’ or ‘weak’. A corresponding global rating will be determined for each study. If the number of included studies is sufficient, we will perform sensitivity analyses by excluding high-risk studies. Cohen’s kappa will be calculated to determine the agreement on the quality rating of each study between reviewers.38
Risk of bias across studies
If available, we will retrieve study protocols and trial registrations to identify potential bias due to selective reporting of results and selective publication. To assess potential publication bias, we will examine the funnel plot for asymmetry,41 and perform sensitivity analyses with different publication bias tests.42 43 To assess the overall quality of evidence, we will use the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, containing the five dimensions: (1) risk of bias, (2) inconsistency of results, (3) indirectness of evidence, (4) imprecision of effect size and (5) publication bias.44 Quality of evidence will be categorised into ‘high’, ‘moderate’, ‘low’ or ‘very low’. The quality of evidence assessment will influence data synthesis when considering the strength of the evidence.
Data synthesis
We will use I² statistics and forest plots to assess heterogeneity of included studies.45 According to the Cochrane handbook, I² values will be interpreted as unimportant (I²<40%), moderate (30%–60%), substantial (50%–90%) or considerable heterogeneity (75%–100%).46 If the included studies are sufficiently homogenous (I²≤60%) or heterogeneity is sufficiently reduced by the prespecified differentiation into subgroups, we will use a random effects model to meta-analyse the primary studies. We will calculate effect sizes for both burnout and stress using the standardised intervention-control difference (between-group comparison) with RCTs and the standardised post/pre-intervention difference (within-group comparison) with NRTs. Accordingly, two separate meta-analyses will be calculated. The first meta-analysis will summarise all RCTs using the between-group effect size. The second meta-analysis will summarise all non-randomised and uncontrolled trials using the within-group effect size. Forest plots will be produced for each of the outcomes for within-group and between-group effect sizes. The forest plots for the within-group and between-group effects will be stratified for physician’s career stage (residents and practicing physicians).
We prespecify two subgroup analyses. We will contrast RCTs and NRTs to evaluate potential differences in the estimation of MBIs effectiveness due to different study designs. In addition, we will compare residents and practicing physicians to consider career stage of a physician as a potential moderator of the effectiveness of interventions. As an extension to subgroup analyses, we will conduct a meta-regression including self-reported mindfulness, intervention dosage and amount of home practice as potential moderators of the effect.
Furthermore, a comprehensive narrative synthesis will be conducted and relevant characteristics will be described qualitatively, especially if the heterogeneity of the included studies is large. Following the GRADE approach, we will provide a comprehensive display of results on relevant domains using ‘summary of findings’ tables.
Patient and public involvement
Patients and the public were not involved in the design or planning of the study.
Ethics and dissemination
Ethical approval is not required. Results will be published in a peer-reviewed journal and presented at international conferences.
Supplementary Material
Acknowledgments
The authors are thankful for the valuable advice and support provided by Stefan Schmidt, University Medical Center Freiburg, who critically reviewed the final manuscript.
Footnotes
JCF and JJB contributed equally.
Contributors: JCF and JJB drafted the work and made substantial contributions to the conception. They wrote and approved the submitted version of the protocol and account for accuracy and integrity of any part of the work. ASG read, edited and approved several versions of the manuscript and consulted in all stages of the conception.
Funding: This work represents independent research supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Project number 197396619—SFB 1015. The DFG also funded the article processing charge together with the University of Freiburg in the funding programme Open Access Publishing.
Competing interests: None declared.
Patient consent for publication: Not required.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1. Shanafelt TD, Hasan O, Dyrbye LN, et al. Changes in burnout and satisfaction with work-life balance in physicians and the general us working population between 2011 and 2014. Mayo Clin Proc 2015;90:1600–13. 10.1016/j.mayocp.2015.08.023 [DOI] [PubMed] [Google Scholar]
- 2. Goldman ML, Shah RN, Bernstein CA. Depression and suicide among physician trainees: recommendations for a national response. JAMA Psychiatry 2015;72:411–2. 10.1001/jamapsychiatry.2014.3050 [DOI] [PubMed] [Google Scholar]
- 3. Prins JT, Gazendam-Donofrio SM, Tubben BJ, et al. Burnout in medical residents: a review. Med Educ 2007;41:788–800. 10.1111/j.1365-2923.2007.02797.x [DOI] [PubMed] [Google Scholar]
- 4. Prins JT, Hoekstra-Weebers JEHM, Gazendam-Donofrio SM, et al. The role of social support in burnout among Dutch medical residents. Psychol Health Med 2007;12:1–6. 10.1080/13548500600782214 [DOI] [PubMed] [Google Scholar]
- 5. Dyrbye LN, West CP, Satele D, et al. Burnout among U.S. medical students, residents, and early career physicians relative to the general U.S. population. Acad Med 2014;89:443–51. 10.1097/ACM.0000000000000134 [DOI] [PubMed] [Google Scholar]
- 6. Dyrbye L, Shanafelt T. A narrative review on burnout experienced by medical students and residents. Med Educ 2016;50:132–49. 10.1111/medu.12927 [DOI] [PubMed] [Google Scholar]
- 7. Rutledge T, Stucky E, Dollarhide A, et al. A real-time assessment of work stress in physicians and nurses. Health Psychol 2009;28:194–200. 10.1037/a0013145 [DOI] [PubMed] [Google Scholar]
- 8. Kristensen TS, Borritz M, Villadsen E, et al. The Copenhagen burnout inventory: a new tool for the assessment of burnout. Work Stress 2005;19:192–207. 10.1080/02678370500297720 [DOI] [Google Scholar]
- 9. Maslach C, Jackson SE, Leiter MP. Maslach burnout inventory manual. Palo Alto, CA: Consulting Psychologists Press, 1996. [Google Scholar]
- 10. Wallace JE, Lemaire JB, Ghali WA. Physician wellness: a missing quality indicator. The Lancet 2009;374:1714–21. 10.1016/S0140-6736(09)61424-0 [DOI] [PubMed] [Google Scholar]
- 11. Shanafelt TD, Balch CM, Bechamps G, et al. Burnout and medical errors among American surgeons. Ann Surg 2010;251:995–1000. 10.1097/SLA.0b013e3181bfdab3 [DOI] [PubMed] [Google Scholar]
- 12. West CP, et al. Association of resident fatigue and distress with perceived medical errors. JAMA 2009;302:1294–300. 10.1001/jama.2009.1389 [DOI] [PubMed] [Google Scholar]
- 13. de Oliveira GS, Chang R, Fitzgerald PC, et al. The prevalence of burnout and depression and their association with adherence to safety and practice standards: a survey of United States anesthesiology trainees. Anesth Analg 2013;117:182–93. 10.1213/ANE.0b013e3182917da9 [DOI] [PubMed] [Google Scholar]
- 14. Shanafelt TD, Bradley KA, Wipf JE, et al. Burnout and self-reported patient care in an internal medicine residency program. Ann Intern Med 2002;136:358–67. 10.7326/0003-4819-136-5-200203050-00008 [DOI] [PubMed] [Google Scholar]
- 15. West CP, Dyrbye LN, Shanafelt TD. Physician burnout: contributors, consequences and solutions. J Intern Med 2018;283:516–29. 10.1111/joim.12752 [DOI] [PubMed] [Google Scholar]
- 16. Bishop SR, Lau M, Shapiro S, et al. Mindfulness: a proposed operational definition. Clin Psychol Sci Pract 2004;11:230–41. 10.1093/clipsy.bph077 [DOI] [Google Scholar]
- 17. Khoury B, Sharma M, Rush SE, et al. Mindfulness-based stress reduction for healthy individuals: a meta-analysis. J Psychosom Res 2015;78:519–28. 10.1016/j.jpsychores.2015.03.009 [DOI] [PubMed] [Google Scholar]
- 18. Khoury B, Lecomte T, Fortin G, et al. Mindfulness-based therapy: a comprehensive meta-analysis. Clin Psychol Rev 2013;33:763–71. 10.1016/j.cpr.2013.05.005 [DOI] [PubMed] [Google Scholar]
- 19. Ireland MJ, Clough B, Gill K, et al. A randomized controlled trial of mindfulness to reduce stress and burnout among intern medical practitioners. Med Teach 2017;39:409–14. 10.1080/0142159X.2017.1294749 [DOI] [PubMed] [Google Scholar]
- 20. Krasner MS, Epstein RM, Beckman H, et al. Association of an educational program in mindful communication with burnout, empathy, and attitudes among primary care physicians. JAMA 2009;302:1284–93. 10.1001/jama.2009.1384 [DOI] [PubMed] [Google Scholar]
- 21. Verweij H, van Ravesteijn H, van Hooff MLM, et al. Mindfulness-based stress reduction for residents: a randomized controlled trial. J Gen Intern Med 2018;33:429–36. 10.1007/s11606-017-4249-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Panagioti M, Panagopoulou E, Bower P, et al. Controlled interventions to reduce burnout in physicians: a systematic review and meta-analysis. JAMA Intern Med 2017;177:195–205. 10.1001/jamainternmed.2016.7674 [DOI] [PubMed] [Google Scholar]
- 23. Regehr C, Glancy D, Pitts A, et al. Interventions to reduce the consequences of stress in physicians: a review and meta-analysis. J Nerv Ment Dis 2014;202:353–9. 10.1097/NMD.0000000000000130 [DOI] [PubMed] [Google Scholar]
- 24. Clough BA, March S, Chan RJ, et al. Psychosocial interventions for managing occupational stress and burnout among medical doctors: a systematic review. Syst Rev 2017;6:2272–81. 10.1186/s13643-017-0526-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. West CP, Dyrbye LN, Erwin PJ, et al. Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis. The Lancet 2016;388:2272–81. 10.1016/S0140-6736(16)31279-X [DOI] [PubMed] [Google Scholar]
- 26. Lomas T, Medina JC, Ivtzan I, et al. A systematic review of the impact of mindfulness on the well-being of healthcare professionals. J Clin Psychol 2018;74:319–55. 10.1002/jclp.22515 [DOI] [PubMed] [Google Scholar]
- 27. Lases SS, Lombarts MJMH, Slootweg IA, et al. Evaluating mind fitness training and its potential effects on surgical Residents’ well-being: A mixed methods pilot study. World J Surg 2016;40:29–37. 10.1007/s00268-015-3278-z [DOI] [PubMed] [Google Scholar]
- 28. Schroeder DA, Stephens E, Colgan D, et al. A brief mindfulness-based intervention for primary care physicians: a pilot randomized controlled trial. Am J Lifestyle Med 2018;12:83–91. 10.1177/1559827616629121 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:g7647 10.1136/bmj.g7647 [DOI] [PubMed] [Google Scholar]
- 30. Moher D, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151:264–9. 10.7326/0003-4819-151-4-200908180-00135 [DOI] [PubMed] [Google Scholar]
- 31. Kabat-Zinn J, Hanh TN. Full catastrophe living: using the wisdom of your body and mind to face stress, pain, and illness. New York, NY: Delta Trade Paperbacks, 2009. [Google Scholar]
- 32. Sharma M, Nazareth I, Petersen I. Observational studies of treatment effectiveness: worthwhile or worthless? Clin Epidemiol 2019;11:35–42. 10.2147/CLEP.S178723 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Parker LA, Saez NG, Porta M, et al. The impact of including different study designs in meta-analyses of diagnostic accuracy studies. Eur J Epidemiol 2013;28:713–20. 10.1007/s10654-012-9756-9 [DOI] [PubMed] [Google Scholar]
- 34. Ruotsalainen JH, Verbeek JH, Mariné A, et al. Preventing occupational stress in healthcare workers. Cochrane Database Syst Rev 2014;11. [DOI] [PubMed] [Google Scholar]
- 35. Linzer M, Poplau S, Grossman E, et al. A cluster randomized trial of interventions to improve work conditions and clinician burnout in primary care: results from the healthy work place (HWP) study. J Gen Intern Med 2015;30:1105–11. 10.1007/s11606-015-3235-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan-a web and mobile APP for systematic reviews. Syst Rev 2016;5:210 10.1186/s13643-016-0384-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Rosenzweig R. Roy Rosenzweig center for history and new media, 2019. Available: www.zotero.org [Accessed 25 Apr 2019].
- 38. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46. 10.1177/001316446002000104 [DOI] [Google Scholar]
- 39. Higgins JPT, Sterne JAC, Hróbjartsson A. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, eds Cochrane methods. Cochrane database of systematic reviews 2016, 2016. [Google Scholar]
- 40. Armijo-Olivo S, Stiles CR, Hagen NA, et al. Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract 2012;18:12–18. 10.1111/j.1365-2753.2010.01516.x [DOI] [PubMed] [Google Scholar]
- 41. Peters JL, Sutton AJ, Jones DR, et al. Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. J R Stat Soc Ser A Stat Soc 2010;173:575–91. 10.1111/j.1467-985X.2009.00629.x [DOI] [Google Scholar]
- 42. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000;56:455–63. 10.1111/j.0006-341X.2000.00455.x [DOI] [PubMed] [Google Scholar]
- 43. Simonsohn U, Nelson LD, Simmons JP. P-curve: a key to the file-drawer. J Exp Psychol 2014;143:534–47. 10.1037/a0033242 [DOI] [PubMed] [Google Scholar]
- 44. Guyatt GH, Oxman AD, Vist GE, et al. Grade: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6. 10.1136/bmj.39489.470347.AD [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45. Higgins JPT, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses. BMJ 2003;327:557–60. 10.1136/bmj.327.7414.557 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Higgins JPT, Green S, Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration, 2011. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2019-032295supp001.pdf (72.4KB, pdf)