Economic evaluations of scaling up strategies of evidence-based health interventions: a systematic review protocol

Abstract

Introduction

Scaling science aims to help roll out evidence-based research results on a wide scale to benefit more individuals. Yet, little is known on how to evaluate economic aspects of scaling up strategies of evidence-based health interventions.

Methods and analysis

Using the Joanna Briggs Institute guidance on systematic reviews, we will conduct a systematic review of characteristics and methods applied in economic evaluations in scaling up strategies. To be eligible for inclusion, studies must include a scaling up strategy of an evidence-based health intervention delivered and received by any individual or organisation in any country and setting. They must report costs and cost-effectiveness outcomes. We will consider full or partial economic evaluations, modelling and methodological studies. We searched peer-reviewed publications in Medline, Web of Science, Embase, Cochrane Library Database, PEDE, EconLIT, INHATA from their inception onwards. We will search grey literature from international organisations, bilateral agencies, non-governmental organisations, consultancy firms websites and region-specific databases. Two independent reviewers will screen the records against the eligibility criteria and extract data using a pretested extraction form. We will extract data on study characteristics, scaling up strategies, economic evaluation methods and their components. We will appraise the methodological quality of included studies using the BMJ Checklist. We will narratively summarise the studies’ descriptive characteristics, methodological strengths/weaknesses and the main drivers of cost-effectiveness outcomes. This study will help identify what are the trade-offs of scaling up evidence-based interventions to allocate resources efficiently.

Ethics and dissemination

No ethics approval is required as no primary data will be collected. The results will be published in a peer-reviewed, international journal and presented at national and international conferences.

Strengths and limitations of this study.

• This is the first systematic review to provide evidence on economic evaluation approaches for the scaling up strategies of evidence-based interventions.

• We plan a strong, rigorous and reproducible methodology for conducting our systematic reviews of economic evaluations.

• We follow the Joanna Briggs Institute guidance for conducting systematic reviews of economic evaluations.

• A comprehensive search strategy will be employed to retrieve both peer-reviewed and grey publications.

• The review may face some limitations to generalisability due to the highly context-specific nature of economic evaluations.

Introduction

Researchers, healthcare professionals and decision-makers are increasingly focusing on filling the gap between knowledge and practice. In recent years, growing efforts to bridge this gap have produced a vast body of knowledge on the efficacy and effectiveness of health interventions and their implementation in practice.^1–3 Most of this evidence derives from experimental studies in which interventions are delivered under optimal, or at least ‘best practice’ conditions, generally conducted on relatively small populations and from projects done in given settings. To date, these efforts have produced a wide set of well-documented effective health interventions.1 2 4 5 However, health decision-makers are still not systematically implementing such evidence to benefit more people on a wider scale.1 2 4–8 One way to fill this gap is to develop and implement strategies to scale up effective evidence-based interventions (EBIs) in health.^{7 9}

While both efficacy and effectiveness are key to the roll out of EBIs on a large scale, other factors—such as costs and cost-effectiveness—are central to the successful scale up of EBIs.^{8 10–14} As health systems face continuous strains and limited resource availability, economic evaluations can play an important role in informing health decision-makers on the trade-offs in costs health benefits of choosing and defining a scaling up strategy.10 12 14–21 Economic evaluations are a means to both assess the value for money and inform resource allocation decision-making.²² To do so, economic evaluations compare alternative choices in terms of both costs and consequences.²² Alternative choices refer to the different ways in which healthcare resources can be used to improve health. The type of economic evaluations are generally defined by the number of alternatives compared, whether both costs and consequences are examined, and how the consequences are expressed.²²

Little is known on what these evaluations should include to analyse the cost-effectiveness of scaling up strategies, as the cost-effectiveness of EBIs does not necessarily reflect the cost-effectiveness of the scaling up effort.8 13 15–19 21 23 While not many, a small number of studies synthesised the costs and cost-effectiveness of scaling up strategies of EBIs in health. Mostly conducted in low and middle income countries, these reviews show that included studies generally focus, among other interventions, on national immunisation programmes,^{21 24 25} maternal, infant and children health programmes²⁰ and HIV/AIDS prevention and care interventions.^{16 26} Despite being conducted in specific geographical areas and having a narrow focus on scaling up strategies of certain health interventions, these reviews provide insights into the economic evaluation research production of scaling up strategies. These reviews reveal a great variability among the included economic evaluation studies. When included, these studies vary in perspectives, scope, approaches, assumptions, cost categories and are often not presented in a way that can be easily comparable and generalised across settings and countries.19–21 26–28

Oftentimes, the lack of complete availability of scaling up cost data or the use of models leads economic analysts to rely on assumptions that may not reflect the complexity of implementing scaling up strategies.8 16–19 21 26 29 30 For example, economic evaluations of scaling up strategies may posit that scaling up implementation costs are a fixed part of the intervention costs.^{19 30 31} In reality, scaling up strategies may present additional costs to that of the intervention that can greatly vary across interventions and settings, potentially leading to both economies and/or diseconomies of scale.²⁹ Costs and cost-effectiveness estimates may change according to the type of intervention being expanded, the size of the targeted population, the prevalence/incidence of the disease, the relevant efficacy level of the intervention, the geography and the financial resources available and needed.8 13 15–17 19 29 32 Specific to scaling up strategies, costs and estimates related to infrastructure and available human resources can vary based on the different scaling up strategy operationalisation and management, the cost impacts of change, including the excess cost of service delivery as uptake changes and the opportunity costs to providers and patients participating in the activities.8 13 15–17 19 29 32 Finally, implementation and scale-up theoretical frameworks—that support thinking and interpretation of ‘real world’ complex data—consider economic constructs in scaling up strategies in different ways. For example, some frameworks consider cost (and resource) mobilisation as a key objective,^{33 34} yet implementation frameworks consider costs as an implementation outcome.³⁵ Frameworks vary also in the ways they consider potential benefit or effectiveness (‘cost–benefit’).³⁶ This variability then results in a wide heterogeneity of studies and approaches when it comes to economically evaluating scaling up strategies. Costs and cost-effectiveness estimates may also vary according to different modelling approaches. For example, ex-ante economic evaluations are often used for informing preimplementation decision-making using available evidence and modelling to simulate the costs and consequences of alternatives.¹⁵

We argue then that little is known on how to evaluate the economic aspects of these strategies to understand what constitutes the trade-offs of scaling up EBIs to allocate resources efficiently. Thus, we seek to identify and describe the methods and issues related to economic evaluations aimed at assessing scaling up strategies of EBIs in health.

Objectives

Our goals are to:

• Identify and describe which economic evaluations methods are used to assess scaling up strategies of EBIs in health.

• Identify and describe the costs and cost elements adopted in such economic evaluations.

• Identify and describe environmental factors accounted for in such economic evaluations.

• Discuss the strengths and limitations of each approach and explain reasons for variation in the reporting of economic evaluations of scaling up strategies of EBIs in health.

Methods

Study design

We are conducting a systematic review following Joanna Briggs Institute (JBI) guidance for conducting systematic review of evidence from all (ie, partial and full) economic evaluations addressing a question(s) about scaling up health intervention strategies’ cost-effectiveness.^{37 38} We adopted Preferred Reporting Items for Systematic Reviews and Meta-Analyses protocols (PRISMA-P) guidelines for reporting of systematic reviews protocols³⁹ (online supplemental additional file 1). We registered the protocol on Open Science Framework database (registration number osf.io/fsq84).

Eligibility criteria

Studies included in the review must adhere to the eligibility criteria described below following the Population, Intervention, Comparison, Outcomes (PICOS) as outlined in the PRISMA-P guidelines.³⁹

Population: We will include studies in which the population of interest is any individual, organisation, or system—directly or indirectly—involved in the delivery or receipt of any health services that was the target of the scale-up.

Intervention: We will include research studies that investigate strategies for scaling up. Included studies must evaluate a scaling up strategy of an EBI (and not the evidence-based health intervention itself). For the purposes of this systematic review, we consider:

• A health intervention to be a health service or a package of health services aimed at improving, maintaining, promoting or restoring health.^{40 41}

• EBIs in health as health interventions that are effective, efficacious and ready for dissemination.⁴²

• A strategy as one or more initiatives, approaches or activities that directly aim to change the supply or demand of EBIs in health to improve reach, adoption and sustainability of an EBI.

• Scaling up in healthcare as the ‘deliberate efforts to increase the impact of successfully tested health interventions so as to benefit more people and to foster policy and program development on a lasting basis’.^{12 34 43} In other words, scaling up strategies are systematic courses of action that aim to roll out successful local health interventions to regional, national or international levels to reach broader populations and settings over time.^{34 43}

No restrictions will be made on the type of EBI or impact (effectiveness) metric chosen. The scaling up of an EBI can be implemented as a standalone intervention, or as an addition in combination with other interventions.

Comparator: There are no restrictions on the type of comparator. Included studies may report economic evaluations that compare the studied scaling up strategy to current practice (ie, no scaling up), or to alternative scaling up strategies.

Outcomes: All reported partial or full economic evaluation outcomes are of interest. Outcomes will include measures related to costs and cost-effectiveness. Partial evaluations focused only on costs will include cost outcomes reported as monetary amounts. Full economic evaluations cost-effectiveness outcomes will include incremental cost-effectiveness ratio, incremental cost-utility ratio, net benefit, cost–benefit ratio. The metric chosen to report the health gain (effectiveness) used in the economic evaluations will not be an inclusion criterion. It can include (but not restricted to) for instance cost/illness averted, cost/quality-adjusted life year gained, or cost/disability-adjusted life year averted. All viewpoints/analytic perspectives will be considered with no restrictions. We expect that a variety of outcomes are used in studies to report on the cost-effectiveness of scaling up EBIs. Studies in which only scaling up strategy’s effectiveness, adoption or health gain was reported will not be included.

Study design: Any study design that includes any type of empirical economic evaluation, as well as any modelling and methodological considerations will be included. We will include both full economic evaluation designs, such as cost-effectiveness analysis, cost-utility analysis and cost–benefit analysis and partial economic evaluation designs, such as cost minimisation analysis, cost comparison/cost analysis, cost outcome descriptions, cost descriptions and budget impact analysis. Additionally, included modelling studies can be based on a meta-analysis of data from randomised trials or using secondary data from literature and those based on observational studies or analysis of large administrative databases. Both published and unpublished grey literature will be included. We will exclude the following studies: reviews, systematic reviews, qualitative studies, clinical effectiveness studies, critical reviews, editorials, commentaries, abstracts, protocols and academic theses.

Settings: We will review studies independently of the settings, thus, including any healthcare setting (ie, public health, primary care clinic, hospital, etc.) in both rural and urban areas. We will not restrict the inclusion criteria based on geography. Economic evaluations undertaken within any country context will be included.

Information sources

The information sources include a search of the following electronic bibliographic databases from their inception onwards: Medline, Web of Science, Embase, Cochrane Library Database, PEDE, EconLIT and INHATA. Additionally, since economic evaluation studies are often conducted for the government or by government agencies, we will systematically perform an internet search as this has been shown to regularly capture eligible studies not identified by other databases.²⁵ We will perform an extensive search strategy using free text, with no restrictions on date and year of publication. A search of web pages of international organisations, bilateral agencies, non-governmental organisations and consultancy firms involved in the delivery, funding or evaluation of scaling up EBIs. Reports found to have a matching publication in the published literature will be excluded. We will search the following Internet search databases and data sources: Google, Google Scholar, INESSS (Institut national d'excellence en santé et en services sociaux), OpenGrey, Grey Literature Report, GreyNet, Canadian Evaluation Society, EuroScan, databases included in the ‘Grey Matters—A Practical Deep web Search Tool for Evidence based Medicine’ (CADTH) Checklist and region-specific databases (African Index Medicus, Eastern Mediterranean Literature (WHO), Index Medicus for South-East Asia Region, LILACS for Latin America). We will then conduct a webpage search of following organisations/agency/governmental websites: UNICEF, WHO, GAVI Alliance, Programme for Appropriate Technology in Health, Johns Hopkins School of Public Health, World Bank, Global Affairs Canada, UK Department for International Development and US Agency for International Development.

Search strategy

Our information specialist (NR) developed a Medline strategy with input from the project team. An iterative process of revision was conducted by the members of the research team. Comments will be integrated for a final version of the search strategy. This final version was approved by the team members. Once validated, the information specialist (NR) translated this search strategy for each electronic database mentioned above. The present protocol only includes the search strategy conducted in Medline on 14 October 2020 (see online supplemental file 2). A hand search will also be performed, and citations and bibliographies of included primary studies and relevant literature reviews will be reviewed for additional relevant articles.

The search will include a combination of the following two concepts: (1) scaling and (2) Economic Evaluation basic terms. No language restrictions will be applied. The search strategy in Ovid Medline is in the online supplemental materials.

The following sources were used to find the search terms: (1) previous reviews who used the concept of scaling up^{7 20} and the concept of economic evaluation;^{20 21 44} (2) the knowledge of the experts of our multidisciplinary team in scaling up and (3) the thesaurus of the consulted bibliographic databases. All words and expressions found were tested and evaluated by the information specialist before to be integrated or rejected in the search strategy. The search strategy was commented via an iterative process by the other members of the team for the production of the final version.

The concept Scaling was created to retrieve all the potential expressions for designed the idea of the spreading of an innovation. It is designed to retrieve very used expression like ‘scaling up’, ‘scale up’, ‘spread of technologies’, but also many variations like ‘widespread adoption of the technology’ or ‘rolling out the model of care’. The concept of Economic Evaluation integrated all synonyms like ‘cost evaluation’, ‘economic analysis’ and ‘net benefit’.

Study records

Data management

In this ongoing study, we exported all citations identified from the electronic databases into Endnote X9 (citation manager software). We used EndNote X9 to remove duplicates in addition to manual checking to identify unique citations for the study selection process. Unique records were then exported into Covidence (internet-based screening and data extraction tool).

Selection process

All stages of the selection process will be performed independently by two reviewers. One reviewer (FB) developed and tested (after team validation) together with the second reviewer a pilot screening form against the eligibility criteria on a 7.5% random sample of the retrieved citations (title and abstracts) to validate the process of inclusion of articles in the review (see the data extraction codebook form template in the online supplemental file 3). This piloting stage ensured reviewers shared a common understanding of the eligibility criteria. At the title and abstract stage, the reviewers will independently screen the titles and abstracts with regard to the inclusion/exclusion criteria using Covidence. Studies not fulfilling the eligibility criteria will be excluded, and the full texts of the remaining studies will be retrieved for further assessment. Articles with abstracts that do not appear to meet the criteria for exclusion or are ambiguous, or that have a missing abstract, will be retained and reviewed in full. The full text of retained studies will be independently assessed for exclusion against inclusion/exclusion criteria by both reviewers. To resolve eligibility questions, we will contact the authors of the included studies to seek additional information. Discrepancies between reviewers will be solved through discussion, and—if needed—a consultation with a third reviewer. Any reasons for exclusion will be recorded in Covidence at the full-text stage. The results of the identification, screening and inclusion process will be displayed using the PRISMA flowchart.³⁹

Data collection process

A standardised data extraction template form will be piloted in duplicate by the reviewers. The extraction form will be informed by the study objectives, eligibility criteria and the JBI-ACTUARI (Joanna Briggs Institute-Analysis of Cost, Technology and Utilisation Assessment and Review Instrument) tool.³⁷ This template form will allow to extract from each study information on the key characteristics, the results for the outcomes of interest and the author conclusions.³⁸ The form will be tested on a 10% random sample of the included studies for data collection. This pilot test will help to identify extraction items that are missing from the template, or likely to be confusing or unnecessary. Authors’ consensus will be required before the form can be modified if deemed appropriate. The investigators will use the finalised revised and agreed on version of the data extraction form to extract data independently.

Data items

The data extracted will cover: first descriptive data about (i) the study general characteristics (eg, title, short name, corresponding author name, funding source and conflict of interest), study type (published or grey literature), study population/participants, type of scaled up intervention and authors’ description of intervention (including whether it was a standalone intervention or a combination of interventions), type of scaling up strategy (including scaling up level of implementation) and authors’ description of strategy, its comparator(s) and outcomes; (ii) study methods including evaluation design type, analytic viewpoint(s), prices and currency used for costing, time period of analysis; sensitivity testing; source of effectiveness data, measures of resource use, cost and health effect/clinical and cost effectiveness and (iii) study context (geographical, healthcare and broader service delivery setting); second reported results for the resource use and/or cost and/or cost effectiveness measures; third, when possible author conclusions about factors that promote and limit the cost-effectiveness of scaling up EBIs strategies.

Quality appraisal

There is still no consensus among health economic experts on which guidelines to follow when conducting systematic reviews of economic evaluations.^45–48 We will be using Drummond and Jefferson checklist, also known as the British Medical Journal (BMJ) checklist, as it was designed for full economic evaluations but also applicable to partial economic evaluations, report and commentaries on economic evaluations, thus aligned to our broad inclusion criteria.⁴⁹ The BMJ tool is a yes/no, 35 items checklist organised in three sections: study design, data collection and analysis and interpretation of results.⁴⁹ If items are not applicable to a specific study, a ‘not appropriate’ response can be stated. Critical appraisal will be undertaken independently by two individuals. If any disagreements arise, they will be discussed between the two reviewers and if need be resolved by team consensus or by a third reviewer.

Data synthesis

We will use descriptive structured narratives, statistics and tables to identify and summarise the key features of the included economic evaluations of scaling-up strategies and the elements considered in such evaluations. Narrative synthesis will be used to summarise the methods, highlighting important characteristics of the studies when relevant, focusing on differences/similarities and methodological weaknesses and where possible identifying the main drivers of cost-effectiveness outcomes. In particular, the synthesis will focus on:

• The assumed key theoretical trade-offs (between levels and types of resources, and levels and types of outcome) of scaling up strategies used in the included economic evaluations.

• The level and configuration of scaling up resources examined in the economic evaluations, how they are related to the levels and types of outcomes observed, and the contextual/environmental factors accounted for in these relationships.

• The conclusions regarding the relationship between the cost-effectiveness of the scaling up strategy under examination and the economic evaluation approach.

• Strengths and weaknesses of each approach for evaluating scaling up strategies of EBIs.

We expect to include a plurality of economic evaluation studies assessing scaling up strategies of EBIs with diverse interventions, populations and settings, thus we anticipate that there will be heterogeneity making difficult to perform a meta-analysis with interpretable results. We will explore this heterogeneity by narratively synthesising the differences, and if possible, the similarities in settings, participants, intervention, comparison and outcomes characteristics across studies. For example, we will perform the data synthesis of economic evaluation methods according to the economic evaluation parameters reported.

Patient and public involvement

Patients and the public were not involved in the design of this study.

Discussion

The identification and description of the methods and issues related to the economic evaluations for the scaling up strategies of EBIs in health will help understand what constitutes the trade-offs of scaling up EBIs to allocate resources efficiently. It will contribute to both health economic evaluation research in scaling science and its implementation in policy and practice. Large-scale health intervention implementation warrants governmental investment, this will also require demonstrable benefits for the patients, providers and society at large. As our world is currently hitting rock bottom by an unseen pandemic—that is, COVID-19—healthcare systems are in more need than ever to understand how to best reduce waste⁵⁰ and increase the roll out of what has more benefits than harms at the lowest cost. If deliberate efforts are not taken to efficiently allocate resources on a wide scale, healthcare systems will collapse.

To the best of our knowledge, this will be the first review that will systematically outline and summarise different economic evaluation approaches used in scaling up strategies of EBIs in health. The science of scale is young and has been too often either completely undermined or clustered with that of sustainability.⁵¹ This study will offer a valuable picture of the advancements and gaps in the application of economic evaluation methods in the scaling science. Earlier reviews of economic evaluations considering scaling up strategies were narrower and focused only on scaling up strategies of specific health interventions. As such, we believe that the findings of this study will point to identify valid recommendations for action for future research and decision-makers. First, this study can help guide future research aimed at defining costing tools and models that can be easily used in scaling up frameworks and plans. It will contribute to define the nature and selection of costs that are integral to the successful roll out of EBIs on large scale, as well as the benefits and disadvantages of each economic methodological approaches aimed at evaluating strategies identified in the literature. Second, as scaling science is becoming an increasingly relevant area for research, policy and practice, clarifying how underlying methodological assumptions are based on evidence and on the multifactorial complexity of real-world scaling strategies will advance the quantity and quality of the information extractable from the evidence to inform both research and practice.⁸ We believe this review will then offer opportunities for improvement in the quality, production, reporting and application in practice of health economic evaluative methods to scaling up strategies.

Second, we hope that this work will support the use of economic evaluations in policies that aim to successfully implement EBIs on a large scale. While health economic evaluations are a well-established component of health technology assessments, their use in implementation science, and in particular scaling science, remains limited.^{15 32} Yet, unless there are sufficient resources, not all possible scaling up strategies can be implemented. Health decision-makers need to have a clear, evidence-based understanding of the financial implications of scaling up EBIs to make an informed choice to use resources efficiently. Without systematically examining and reporting cost and cost-effectiveness evidence, the allocation of financial resources to scaling up strategies may be too high or too low. Economic evidence is then crucial for decision-makers to design scaling up strategies that are affordable and that represent an efficient use of current available resources.

Ethics and dissemination

Our research project is a systematic review based on existing primary studies and methodological papers and as such it will not be necessary to request ethics approval. Additionally, we follow the Canadian Institute for Health Research (CIHR) Ethics Guidance for Developing Partnerships with Patients and Researchers to guide the active dissemination of our findings.⁵² As per CIHR guidelines, no ethical approval is required when engaging patients and public for actively disseminating research findings.

We plan to use passive and active dissemination strategies to disseminate our findings. First, we will publish this study’s protocol and later the results of this project in leading peer-reviewed journals in health implementation and services research. We will also share our findings at local, national and international conferences addressing audiences interested in implementation science, scaling science and health economics. Second, findings from this project will be relevant for health administrators, decision-makers, health professionals and patients. To reach these audiences, we will use our networks with health organisations and health research groups (such as the Quebec Strategy for Patient-Oriented Research (SPOR) Unit). We will tailor the dissemination message to fit each audience and select champions to disseminate our results. Finally, we will use different communication channels, such as newsletters, organisation websites and webinars, to reach all relevant audiences.

Ethics statements

Patient consent for publication

Not applicable.