Abstract
Introduction
While access to antiretroviral therapy (ART) for people living with HIV has expanded in recent years, additional efforts are required to support adherence to medication and retention in care. Interventions should be applicable in real-world settings and amenable to widespread use. The objectives of this overview are to identify effective pragmatic interventions that increase adherence to ART and retention in care for people living with HIV at high risk for suboptimal adherence and retention in high-income countries.
Methods and analysis
We will conduct an overview of systematic reviews of studies on interventions which target improved adherence to medication and retention in care among high-risk people living with HIV in high-income countries (men who have sex with men, African, Caribbean and black people, sex workers, people who inject drugs, indigenous people and other socially marginalised groups). We will search the following databases: PubMed, EMBASE (Exerpta Medica Database), CINAHL (Cumulative Index to Nursing and Allied Health Literature), PsycINFO, Web of Science and the Cochrane Library. We will conduct screening, data extraction and assessment of methodological quality of the systematic reviews. Analysis will be narrative. Our findings will be interpreted in light of the certainty of the evidence, level of pragmatism, setting and population of interest.
Ethics and dissemination
Only published secondary data will be used in this study, and therefore ethics approval is not required. Our findings will be disseminated as peer-reviewed manuscripts, conference abstracts and through community activities. The findings from this overview will inform a mixed-methods study among people living with HIV and health workers in Ontario, Canada.
Keywords: antiretroviral therapy, Hiv, adherence, retention, pragmatic
Strengths and limitations of this study.
An exhaustive and comprehensive search strategy.
Findings will be interpreted in the light of levels of pragmatism and quality of evidence.
Trials not yet included in systematic reviews will not be evaluated.
Introduction
More than 37 million people are living with HIV worldwide as of 2017.1 Even though the number of new infections is decreasing, the number of people living with HIV is on the rise.2 This is because individuals are living longer and healthier lives, mostly as a result of effective antiretroviral therapy (ART).2 When taken as prescribed, ART reduces viral load and facilitates immune reconstitution.3 However, adherence to ART is often suboptimal.4 This leads to worse and costly treatment outcomes (treatment switches due to development of resistance to first-line agents, more hospitalisations and death).3 5 6 The increased longevity of people living with HIV implies that they would have to take medication for longer, and therefore strategies should be put in place to support adherence and retention in care over the life-time of the individual. Even though more recent evidence suggests that lower levels of adherence (~85% of pills) may still lead to viral suppression,7 8 the highest levels of adherence are recommended to ensure optimal clinical and biological outcomes, as well as to prevent development of resistance and onward transmission of the virus9 10
Adherence to medication, defined as the ‘extent to which patients take medications as prescribed by their healthcare provider’ or more broadly as ‘the extent to which a person’s behaviour—taking medication, following a diet, or executing lifestyle changes—corresponds with agreed recommendations from a healthcare provider’ is a complex phenomenon.11 12 The first definition only describes compliance to specifications from the provider, whereas the later includes all recommendations jointly agreed on by both parties. Adherence is known to be linked to patient factors such as age, depression, level of education, social factors (such as level of social support, stigma), medication factors (such as pill burden, type of drug, side-effects), provider-related factors (quality of provider-patient relationship, trust, satisfaction with care), disease characteristics (stage of disease), clinical setting including where care is located, and other health system factors, such as funding for treatments.13–16
Retention in care (or continued care) is essential to adherence. However, retention in care is more challenging to define as there is no gold standard.17 Some authors have proposed ‘remaining connected to medical care, once entered’ as a working definition. With regard to HIV, retention has been defined as ‘patients known to be alive and receiving treatment,’ or based on the frequency of clinic visits (varying from 2 weeks to 1 year),18 or the number of viral load tests conducted each year.19
In high-income countries like Canada, injection drug use, homelessness and sex work are factors associated with suboptimal adherence.20 In addition, low self-efficacy, comorbid psychiatric conditions and female gender are also linked to lower rates of adherence,21 22 as are younger age (<40 years), drug use, indigenous ethnicity and hazardous drinking or smoking status.19 23 These findings indicate that adherence to ART and retention in care in Canada are profoundly shaped by the social contexts of patients’ lives.
This is contrary to low-income and middle-income countries, where in addition to these social factors, poverty, access to medication, comorbid diseases and health system factors play a larger role.24 For example, out-of-pocket payment for healthcare, poor transportation infrastructure and stock-outs may hamper regular consumption of medication. Retention in care is a necessary element for adherence to medication, as patients must be connected to medical care to receive clinical, pharmaceutical and laboratory care. In a Canadian cohort of people living with HIV, only 7.5% of people living with HIV had a gap in care (no care for up to 1 year) over a period of 2 years, but up to 20% had suboptimal levels of retention (only one visit per year instead of two).19
If up to 20% of the 37 million people living with HIV have suboptimal or discontinuous care, more than 7 million people in the world may be more likely to fail treatment, develop resistant strains, transmit the virus, and experience poor clinical outcomes and reduced quality and quantity of life.25
The need to rethink HIV care strategies to improve adherence and retention in care was recently highlighted in a cost-effectiveness model indicating that novel approaches to engage and retain patients in care are critical. The authors estimate that improved retention will reduce HIV incidence by 54% and mortality by 64% with a cost-effectiveness ratio of $45 300 per quality-adjusted life-year (QALY) gained.26
In the field of adherence to ART and retention in care among people living with HIV, a number of effective interventions have been identified that improve adherence or clinical outcomes and retention in care.18 27 However, scaling up such interventions has been challenging due to the levels of complexity (multiple interconnecting parts), the resources required and challenges in teasing out the ‘essential’ ingredient of multicomponent interventions.28 29 In addition, these interventions did not often address the needs of subpopulations at high risk of poor adherence or discontinued care, and were often designed for implementation (and tested) in low-income and middle-income settings only, and may therefore be less applicable to high-risk populations in high-income country contexts.27
The purpose of this overview is to inform policies in high-income countries on strategies to improve adherence to ART and retention in care. Our objectives are to:
Summarise the evidence on pragmatic (applicable in broad routine clinical practice as opposed to controlled research settings) and effective adherence or retention enhancing interventions among priority populations (men who have sex with men (MSM), African, Caribbean and black people (ACB), women at risk (including sex workers), people who inject drugs, indigenous peoples) and other socially marginalised groups (immigrants, refugees and people with precarious immigration status) of people living with HIV. These are groups identified to be at high risk of discontinuing treatment in Ontario.
Identify knowledge gaps in intervention research about medication adherence and retention for the different populations of interest.
This overview will be guided by the following questions:
What interventions have been demonstrated to improve adherence to therapy or retention in care for people (adults and children) living with HIV?
How pragmatic are these interventions in terms of participant characteristics, trial setting, flexibility of interventions and clinical relevance of interventions?
Which interventions are adapted for subpopulations such as MSM, ACB populations, women at risk, people who use drugs (injection and non-injection), indigenous populations and other marginalised groups (immigrants, refugees and people with precarious immigration status)?
What resources (people, equipment, money) are needed for implementation of these interventions?
There is uncertainty in the most effective interventions for adherence and retention in HIV care in high-income settings. For example, a recent systematic review in WHO stratum A (a list of countries including Canada and the USA with very low child mortality and low adult mortality) found that most interventions had no effect or did not improve both adherence and clinical outcomes.30 On the other hand, a US-based study identified a number of effective interventions (for adherence) including interactive discussions, pager messages and home visits.31 Further, 10 best practices for improving linkage and retention to care including case management and use of motivational interviewing were identified in a 2016 systematic review, although the authors noted that more rigorous study designs were needed to evaluate their effectiveness.32 A recent scoping review reported that integrating HIV services with other services is beneficial to care, retention and adherence.33 However, it is unclear which interventions are most effective, applicable to specific settings or populations and can be implemented on a large scale.
To answer these questions, we will conduct an overview of systematic reviews to compile evidence from multiple systematic reviews into an accessible document in order to guide and add power to decision-making.34 Therefore, in this review we will critically appraise the current literature, explore how to incorporate our findings into usual practice and support health worker and policy decisions regarding choice of intervention.
Methods
This work is an overview of systematic reviews and will be guided by standard Cochrane methods.35
Patient and public involvement
This research question was formulated based on HIV management priorities identified by the Ontario HIV Treatment Network (OHTN) which specifically aims to close gaps in the care cascade by improving adherence to medication and retention in care. The author team includes patients, representatives of community-based organisations, care providers and researchers who are involved in the design and implementation of this project.
Criteria for considering systematic reviews for inclusion
We will include systematic reviews (with predetermined objectives, eligibility criteria, at least two databases searched, data extraction and quality assessment of included studies) that include at least one study that reports on a randomised comparison of an intervention designed to improve adherence to ART or retention in care as defined by the investigators. We will exclude abstracts, non-systematic reviews and other overviews.
Search methods for identification of systematic reviews
We will conduct an exhaustive and comprehensive search of the following databases: PubMed, EMBASE (Exerpta Medica Database), CINAHL (Cumulative Index to Nursing and Allied Health Literature), PsycINFO, Web of Science and the Cochrane Library. These databases will be searched from 1995 (when combination ART was introduced) to present.36 No language restrictions will be set. A search strategy will be developed in collaboration with a librarian from the Health Sciences Centre Library at McMaster University and a Cochrane Trial Search Coordinator which will be adapted for each database. The search strategy will be appraised by an independent librarian using Peer Review of Electronic Search Strategies guidelines.37 The following terms in various combinations (and forms) will be used for the search:
‘Systematic review OR meta-analysis; Adherence OR compliance OR retention OR dropouts OR loss to follow-up OR attrition OR nonadheren* OR uncompliant* OR treatment refusal OR persistence OR non-persistence; HIV OR human immune-deficiency virus OR human immuno-deficiency virus; Antiretroviral therapy OR antiretrovirals OR antiretroviral treatment OR Highly Active Antiretroviral Therapy OR ART or HAART’
The bibliographies of identified reviews will be searched. In order to find grey literature, we will search institutional websites such as the WHO, the National Institute for Health and Care Excellence and the Joint United Nations Programme on HIV/AIDS. Experts in the field of HIV adherence and retention will be approached to help identify other relevant articles.
Systematic review selection and data collection
The results of the search will be collated in Endnote reference manager.38 Duplicate citations will be removed, then the remaining references will be screened for relevance based on their titles and abstracts by at least two reviewers working independently (LM, DL). Full text copies of the potentially relevant titles will be screened against our inclusion criteria. Data from the included studies will be extracted using a piloted data-extraction form. The following types of information will be extracted: date of publication, number and type of included studies, number of participants, type of participants, study setting (country and clinic type), type of intervention(s), resources used, measures of adherence and retention used (as defined by the authors), and other effectiveness measures. Country income level will be defined as per World Bank criteria.39 Article screening and data extraction will be conducted in duplicate. Agreement on screening and inclusion will be measured using the Kappa statistic,40 and discrepancies will be resolved by discussion or arbitration (LT). Non-English studies will be screened by colleagues at McMaster University who speak and read French, Chinese, Spanish, German and Italian, or by crowdsourcing with the global Cochrane community. As needed, authors of systematic review or trials will be contacted for clarifications or missing information. The key characteristics of included studies will be reported in a table of included studies. The excluded studies and the reasons for exclusion will also be reported in a table. We will assess the risk of bias in the included systematic reviews using the risk of bias in systematic reviews (ROBIS) tool.41 This tool can be used to appraise systematic reviews in three phases: assessing the relevance of the question, identifying concerns with the review process and judging risk of bias. As such a systematic review may be judged to be at high, low or unclear risk of bias.41 The included systematic reviews will be checked for overlap of studies, since one study may appear in more than one review. The review will be reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.42 Findings will be reported on the systematic review level (aggregated findings) and on the individual study level. Only data from randomised comparisons will be used.
Analysis and interpretation
We will conduct narrative analyses. We will create a list of the most effective interventions, including their settings, target populations, category of adherence/retention issue addressed (patient, medication, provider-related, disease, clinical setting and other health system factors) the relative and absolute measures of effect. For an intervention to be considered effective, it should improve either adherence or retention measures and at least one clinical or laboratory outcome (eg, viral load). Studies that do not include clinical measures will be included, but considered as indirect evidence. The certainty of the evidence will be assessed using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach which categorises each outcome by how confident we are that the effect estimate is close to the quantity of interest.43 Using this approach, the certainty rating across studies can be high, moderate, low or very low. Then we will use the Rating of Included Trials on the Efficacy-Effectiveness Spectrum (RITES) tool to appraise how pragmatic these interventions are.44 With this tool, trials can be rated on a five-point Likert scale in four domains: (1) participant characteristics, (2) trial setting, (3) flexibility of interventions and (4) clinical relevance of interventions. For each of these domains, a score of 1 (very explanatory—strong emphasis on efficacy) to 5 (very pragmatic—strong emphasis on effectiveness) can be allocated.44 This tool is specifically designed for post-hoc appraisal of clinical trials. The resources requirements and ‘burden’ of the intervention will be noted (eg, number of staff, skill set, hours per week, cost, number of patient visits, internet use, literacy level of users). GRADE and RITES tools will be used in duplicate by data extractors working independently.
The findings will be interpreted with consideration of the populations that are at high risk of poor adherence and retention: MSM, ACB men and women, indigenous men and women, individuals who use drugs, women who face systemic risk and other groups known to have challenges with engagement in care. Notes will be made as to whether the interventions were tested in these populations and demonstrated to be effective.
Ethics and dissemination
We plan to publish at least two peer-reviewed manuscripts, and to submit results to international and national conferences such as the Canadian Association for HIV Research, the OHTN Research Conference and the International AIDS Society conferences. The findings from this overviews will inform a mixed-methods study among healthcare workers and people living with HIV on the challenges and facilitators to implementing adherence and retention enhancing strategies. These findings will respond to identified HIV management concerns in Ontario and will provide insights into how to support adherence to medication and retention in care among high-risk populations in high-income countries, especially Canada.
Supplementary Material
Footnotes
Contributors: LM developed the first draft of the manuscript. EA, DOL, BR, MS, DM, LPR, CL, SM, ACB, WH and LT revised several versions of the manuscript and approved the final version. LM is the guarantor of the protocol.
Funding: This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests: None declared.
Patient consent: Not required.
Ethics approval: Only published secondary data will be used in this study and therefore ethics approval is not required.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1. UNAIDS, 2017. Fact sheet - Latest statistics on the status of the AIDS epidemic. Secondary Fact sheet - Latest statistics on the status of the AIDS epidemic. http://www.unaids.org/en/resources/fact-sheet
- 2. UNAIDS, 2016. AIDS by the numbers 2015. Secondary AIDS by the numbers 2015. http://www.unaids.org/sites/default/files/media_asset/AIDS-by-the-numbers-2016_en.pdf
- 3. Bangsberg DR, Perry S, Charlebois ED, et al. Non-adherence to highly active antiretroviral therapy predicts progression to AIDS. AIDS 2001;15:1181–3. [DOI] [PubMed] [Google Scholar]
- 4. Ortego C, Huedo-Medina TB, Llorca J, et al. Adherence to highly active antiretroviral therapy (HAART): a meta-analysis. AIDS Behav 2011;15:1381–96. 10.1007/s10461-011-9942-x [DOI] [PubMed] [Google Scholar]
- 5. Paterson DL, Swindells S, Mohr J, et al. Adherence to protease inhibitor therapy and outcomes in patients with HIV infection. Ann Intern Med 2000;133:13321–30. [DOI] [PubMed] [Google Scholar]
- 6. García de Olalla P, Knobel H, Carmona A, et al. Impact of adherence and highly active antiretroviral therapy on survival in HIV-infected patients. J Acquir Immune Defic Syndr 2002;30:105–10. [DOI] [PubMed] [Google Scholar]
- 7. Bangsberg DR. Less than 95% adherence to nonnucleoside reverse-transcriptase inhibitor therapy can lead to viral suppression. Clin Infect Dis 2006;43:939–41. 10.1086/507526 [DOI] [PubMed] [Google Scholar]
- 8. Apisarnthanarak A, Mundy LM. Long-term outcomes of HIV-infected patients with <95% rates of adherence to nonnucleoside reverse-transcriptase inhibitors. Clin Infect Dis 2010;51:115–7. 10.1086/653445 [DOI] [PubMed] [Google Scholar]
- 9. Haubrich RH, Little SJ, Currier JS, et al. The value of patient-reported adherence to antiretroviral therapy in predicting virologic and immunologic response. California Collaborative Treatment Group. AIDS 1999;13:1099–107. [DOI] [PubMed] [Google Scholar]
- 10. Mannheimer S, Friedland G, Matts J, et al. The consistency of adherence to antiretroviral therapy predicts biologic outcomes for human immunodeficiency virus-infected persons in clinical trials. Clin Infect Dis 2002;34:1115–21. 10.1086/339074 [DOI] [PubMed] [Google Scholar]
- 11. Blaschke TF, Osterberg L, Vrijens B, et al. Adherence to medications: insights arising from studies on the unreliable link between prescribed and actual drug dosing histories. Annu Rev Pharmacol Toxicol 2012;52:275–301. 10.1146/annurev-pharmtox-011711-113247 [DOI] [PubMed] [Google Scholar]
- 12. Osterberg L, Blaschke T. Adherence to medication. N Engl J Med 2005;353:487–97. 10.1056/NEJMra050100 [DOI] [PubMed] [Google Scholar]
- 13. WHO, 2003. Adherence to long-term therapies: Evidence for action. Secondary Adherence to long-term therapies: Evidence for action. http://www.who.int/chp/knowledge/publications/adherence_report/en/
- 14. Logie CH, Lacombe-Duncan A, Wang Y, et al. Pathways from HIV-Related stigma to antiretroviral therapy measures in the HIV care cascade for women living with HIV in Canada. J Acquir Immune Defic Syndr 2018;77:144–53. 10.1097/QAI.0000000000001589 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Hodgson I, Plummer ML, Konopka SN, et al. A systematic review of individual and contextual factors affecting ART initiation, adherence, and retention for HIV-infected pregnant and postpartum women. PLoS One 2014;9:e111421 10.1371/journal.pone.0111421 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Katz IT, Ryu AE, Onuegbu AG, et al. Impact of HIV-related stigma on treatment adherence: systematic review and meta-synthesis. J Int AIDS Soc 2013;16(3 Suppl 2):18640. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Mugavero MJ, Westfall AO, Zinski A, et al. Measuring retention in HIV care: the elusive gold standard. J Acquir Immune Defic Syndr 2012;61:574–80. 10.1097/QAI.0b013e318273762f [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Geng EH, Nash D, Kambugu A, et al. Retention in care among HIV-infected patients in resource-limited settings: emerging insights and new directions. Curr HIV/AIDS Rep 2010;7:234–44. 10.1007/s11904-010-0061-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Rachlis B, Burchell AN, Gardner S, et al. Social determinants of health and retention in HIV care in a clinical cohort in Ontario, Canada. AIDS Care 2017;29:828–37. 10.1080/09540121.2016.1271389 [DOI] [PubMed] [Google Scholar]
- 20. Joseph B, Kerr T, Puskas CM, et al. Factors linked to transitions in adherence to antiretroviral therapy among HIV-infected illicit drug users in a Canadian setting. AIDS Care 2015;27:1128–36. 10.1080/09540121.2015.1032205 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Kerr T, Palepu A, Barness G, et al. Psychosocial determinants of adherence to highly active antiretroviral therapy among injection drug users in Vancouver. Antivir Ther 2004;9:407–14. [PubMed] [Google Scholar]
- 22. Tapp C, Milloy M-J, Kerr T, et al. Female gender predicts lower access and adherence to antiretroviral therapy in a setting of free healthcare. BMC Infectious Diseases 2011;11:1–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Lourenço L, Nohpal A, Shopin D, et al. Non-HIV-related health care utilization, demographic, clinical and laboratory factors associated with time to initial retention in HIV care among HIV-positive individuals linked to HIV care. HIV Med 2016;17:269–79. 10.1111/hiv.12297 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Mbuagbaw L, Thabane L, Ongolo-Zogo P, et al. Trends and determining factors associated with adherence to antiretroviral therapy (ART) in Cameroon: a systematic review and analysis of the CAMPS trial. AIDS Res Ther 2012;9:37 10.1186/1742-6405-9-37 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Skarbinski J, Rosenberg E, Paz-Bailey G, et al. Human immunodeficiency virus transmission at each step of the care continuum in the United States. JAMA Intern Med 2015;175:588–96. 10.1001/jamainternmed.2014.8180 [DOI] [PubMed] [Google Scholar]
- 26. Shah M, Risher K, Berry SA, et al. The epidemiologic and economic impact of improving HIV testing, linkage, and retention in care in the United States. Clin Infect Dis 2016;62:220–9. 10.1093/cid/civ801 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Mbuagbaw L, Sivaramalingam B, Navarro T, et al. Interventions for enhancing adherence to antiretroviral therapy (ART): a systematic review of high quality studies. AIDS Patient Care STDS 2015;29 10.1089/apc.2014.0308 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Barker PM, Reid A, Schall MW. A framework for scaling up health interventions: lessons from large-scale improvement initiatives in Africa. Implement Sci 2016;11:12 10.1186/s13012-016-0374-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Willis CD, Riley BL, Stockton L, et al. Scaling up complex interventions: insights from a realist synthesis. Health Res Policy Syst 2016;14:88 10.1186/s12961-016-0158-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Mathes T, Pieper D, Antoine SL, et al. Adherence-enhancing interventions for highly active antiretroviral therapy in HIV-infected patients - a systematic review. HIV Med 2013;14:583–95. 10.1111/hiv.12051 [DOI] [PubMed] [Google Scholar]
- 31. Charania MR, Marshall KJ, Lyles CM, et al. Identification of evidence-based interventions for promoting HIV medication adherence: findings from a systematic review of U.S.-Based Studies, 1996-2011. AIDS Behav 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Higa DH, Crepaz N, Mullins MM. Prevention Research Synthesis Project. Identifying best practices for increasing linkage to, retention, and re-engagement in HIV Medical care: findings from a systematic review, 1996-2014. AIDS Behav 2016;20:951–66. 10.1007/s10461-015-1204-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Heard A, Peterson K, Modi S, et al. Integrating HIV services with other health services to improve care, retention and adherence. 3ie scoping report 7. New Delhi: International Initiative for Impact Evaluation, 2017. [Google Scholar]
- 34. Silva V, Grande AJ, Carvalho AP, et al. Overview of systematic reviews - a new type of study. Part II. Sao Paulo Med J 2015;133:206–17. 10.1590/1516-3180.2013.8150015 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Becker LA, Oxman AD. Overviews of reviews. Cochrane handbook for systematic reviews of interventions: cochrane book series, 2008:607–31. [Google Scholar]
- 36. Palmisano L, Vella S. A brief history of antiretroviral therapy of HIV infection: success and challenges. Ann Ist Super Sanita 2011;47:44–8. 10.4415/ANN_11_01_10 [DOI] [PubMed] [Google Scholar]
- 37. McGowan J, Sampson M, Salzwedel DM, et al. PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement. J Clin Epidemiol 2016;75:40–6. 10.1016/j.jclinepi.2016.01.021 [DOI] [PubMed] [Google Scholar]
- 38. Reuters T. EndNote X7. Philadelphia, PA, USA: Thomson Reuters:, 2013. [Google Scholar]
- 39. The World Bank, 2018. World bank open data. Secondary world bank open data. https://data.worldbank.org/
- 40. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med 2005;37:360–3. [PubMed] [Google Scholar]
- 41. Whiting P, Savović J, Higgins JP, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016;69:225–34. 10.1016/j.jclinepi.2015.06.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010;8:336–41. 10.1016/j.ijsu.2010.02.007 [DOI] [PubMed] [Google Scholar]
- 43. Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64:383–94. 10.1016/j.jclinepi.2010.04.026 [DOI] [PubMed] [Google Scholar]
- 44. Wieland LS, Berman BM, Altman DG, et al. Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews. J Clin Epidemiol 2017;84:95–104. 10.1016/j.jclinepi.2017.01.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.