Abstract
Objective
To analyse and compare the cost-effectiveness of different interventions to reduce salt consumption.
Design
A systematic review of published cost-effectiveness analyses (CEA) and cost-utility analyses (CUA) was undertaken in the databases EMBASE, MEDLINE (PubMed), Cochrane and others until July 2016. Study selection was limited to CEA and CUA conducted in member countries of the Organisation for Economic Co-operation and Development (OECD) in English, German or French, without time limit. Outcomes measures were life years gained (LYG), disability-adjusted life years (DALY) and quality-adjusted life years (QALY). Relevant aspects in modelling were analysed and compared. Quality assessments were conducted using the Drummond and Jefferson/British Medical Journal checklist.
Setting
OECD member countries.
Subjects
Mainly adults.
Results
Fourteen CEA and CUA were included in the review which analysed different strategies: salt reduction or substitution in processed foods, taxes, labelling, awareness campaigns and targeted dietary advice. Fifty-nine out of sixty-two scenarios were cost-saving. The incremental cost-effectiveness ratio in international dollars (Intl.$; 2015) was particularly low for taxes, a salt reduction by food manufacturers and labelling (<−3072 Intl.$/QALY, −6187 Intl.$/LYG and <584 Intl.$/DALY over the time horizon compared with the status quo or no intervention). Targeted dietary advice was rather not cost-effective (24 600 Intl.$/QALY and >303 900 Intl.$/DALY). However, only six studies analysed cost-effectiveness from a societal perspective and quality assessments showed flaws in conducting and a lack of transparency in reporting.
Conclusions
A population-wide salt reduction could be cost-effective in prevention of hypertension and CVD in OECD member countries. However, comparability between study results is limited due to differences in modelling, applied perspectives and considered data.
Keywords: Salt, Cost-effectiveness, Systematic review, CVD, Prevention
Hypertension is the most important risk factor for CVD and contributes more than other diseases to the global burden of disease( 1 , 2 ). Changes of lifestyle, especially weight reduction, physical activity, high potassium consumption and the reduction of alcohol and salt intakes, could stop the high prevalence of suboptimal blood pressure and the incidence of hypertension( 3 , 4 ). There is convincing evidence that high sodium intake is directly associated with suboptimal blood pressure and hypertension( 5 – 8 ) and consequently with increased risk of developing CVD( 5 , 7 , 9 ). Therefore, the WHO recommends a restriction of salt intake in adults to 5 g/d and of sodium consumption to <2 g/d. For children, the maximum level for adults should be adjusted downwards respecting their energy requirements( 10 ). However, the results of recently published studies and meta-analyses show that salt and sodium intakes are considerably higher than recommended in most countries worldwide( 5 , 11 ).
In addition to the high morbidity and mortality, CVD also leads to a high economic burden. The global cost of CVD was estimated at $US 863 billion in 2010 and is projected to rise by 21 % to $US 1044 billion in 2030( 12 ). It is hypothesized that a considerable amount of these expenses could be saved by a reduction of salt intake( 13 , 14 ).
Two strategies to prevent CVD are possible: (i) the high-risk approach, targeting individuals with hypertension; or (ii) population-wide interventions, targeting the entire population( 15 , 16 ). To date, several targeted or population-wide interventions aimed at reducing salt intake have been introduced in various countries worldwide, including salt reduction in processed foods by the food industry (either voluntary or mandatory), taxes on salty foods, use of salt substitutes, health education campaigns via mass media or in schools, declaration of salt content on food packaging as a label or traffic light, and dietary advice for people of a certain age and/or blood pressure( 13 , 17 – 20 ).
Both the high incidence and prevalence of CVD worldwide and the knowledge that risk factors like salt intake are modifiable indicate the relevance of this public health nutrition topic. However, economic efficiency – additionally to effectiveness – should be a major issue within decision making in public health nutrition. Thereby, an optimal allocation of scarce resources can be secured and a comparison with other, including therapeutic options, is possible( 21 – 23 ).
Different study types exist to analyse economic consequences: full economic evaluations consider both cost and benefits and compare these among at least two alternative interventions, such as cost-effectiveness analysis (CEA) and cost-utility analysis (CUA); while partial evaluations examine either cost or benefits and do not compare different alternatives( 21 , 24 ). So far, some efforts to structure and analyse the cost-effectiveness of salt reduction activities have been undertaken( 25 – 27 ). However, these studies included both partial and full economic evaluations worldwide that reported cost-effectiveness or potential cost savings due to a salt reduction. Consequently, the in-depth comparison of results between the different study types and countries was not possible. Therefore, by including only full economic evaluations conducted in member countries of the Organisation for Economic Co-operation and Development (OECD), the present systematic literature review aimed to examine whether different approaches of population-wide and targeted salt reduction interventions are cost-effective.
Methods
Search strategy and study selection criteria
The systematic literature search was undertaken in July 2016 in the databases EMBASE, MEDLINE (PubMed), Cochrane Library, National Health Service Economic Evaluation Database, Health Technology Assessment Database and web search engines. Search items were ‘salt’ or ‘sodium’ with ‘intake’, ‘restriction’ or ‘reduction’, ‘cardiovascular disease’, ‘hypertension’, ‘blood pressure’ or ‘cardiovascular risk’. To limit results to economic studies, ‘economic evaluation’, ‘economic/cost-analysis’, ‘cost–benefit’, ‘cost-effectiveness’, ‘health care cost’, ‘resource allocation’ and ‘quality adjusted life year’ were added to the search string (see online supplementary material, Table S1). In addition, reference lists of full-text articles meeting the study selection criteria were reviewed. Studies were restricted to English, German or French publications. There was no time limit placed on the literature search. Two of the authors (E.S., D.N.) independently reviewed each article and studies were selected by eligibility criteria set a priori. Inclusion was limited to CEA and CUA reporting findings of population-wide or targeted salt reduction interventions. Only studies from OECD member countries were included to enable better comparability. Outcomes were measured as life years gained (LYG), disability-adjusted life years (DALY) or quality-adjusted life years (QALY). Cost-effectiveness was shown in an incremental cost-effectiveness ratio (ICER). All comparators were eligible for inclusion, but usually ICER were reported against the alternative ‘status quo’ or ‘no intervention’.
Data extraction, synthesis and study quality assessment
Data were extracted with a data extraction form based on the Centre for Reviews and Dissemination( 28 ). Results relating to cost-effectiveness were synthesized by comparing the given ICER.
Additionally, incremental costs were converted to international dollars (Intl.$) in 2015 by using the OECD purchasing power parity coefficient( 29 ) adjusted by the OECD deflator( 30 ). Quality assessments were based on the guidelines for economic submissions to the British Medical Journal by Drummond and Jefferson( 31 ) to detect common flaws. One item relating to generalizability issues was added, as recommended( 28 ). Differences in methodology were discussed.
Results
A total of 1351 articles were identified. After screening and applying the eligibility criteria, fourteen full economic analyses( 32 – 45 ) describing cost-effectiveness of salt reduction in Australia, Europe, New Zealand and the USA were included. Figure 1 shows the study selection process.
Basic characteristics of the included studies
The included studies analysed one to ten different salt-specific interventions to prevent hypertension and CVD with the same model (Table 1). Altogether, sixty-two simulations are presented, which can be classified into seven population-wide or targeted interventions to reduce salt intake: (i) voluntary and (ii) mandatory salt reductions in processed foods in general or in particular food groups by the food industry through the setting of maximum salt levels or the regulation of the use of salt substitutes; (iii) a sodium or salt tax; (iv) targeted dietary advice for people of a certain age or with certain blood pressure; (v) declaration of salt content on food packaging (labelling); (vi) salt awareness campaigns via mass media; and (vii) a cap-and-trade approach, setting a maximum amount of food-grade salt released to the market. Additionally, in three studies a combination of different approaches was modelled and in further two studies unspecified interventions assuming a certain salt reduction were analysed. The methods applied in the included studies differed in model, time horizon, perspective, outcome measurement, assumption of effectiveness and types of considered cost (Table 1). The choice of perspective (e.g. of society, the health system or patients) influenced which types of cost and which outcomes were considered in the analysis( 24 ). A societal perspective for judging the cost-effectiveness of salt reduction interventions was taken by six studies. Regarding the effectiveness of the individual interventions, a wide span of achieved salt reduction (0·0 to 4·6 g/d) was assumed. Depending on this reduction, the modelled decrease of systolic blood pressure showed a similar range from 0·0 up to 10 mmHg. The different cost types covered the health cost (i.e. potential cost savings due to reduced incidence and prevalence of CVD as well as additional health costs associated with the gain of life years), the cost of implementing and maintaining the salt-specific intervention, and productivity costs reflecting potential savings in productivity losses due to absenteeism caused by CVD (Table 1).
Table 1.
Outcome | Costs | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Time | Outcome | Salt reduction* | Reduction of SBP* | ||||||||||
Reference | Population | Model | horizon | Perspective | (cost/...) | Intervention | g/d | % | mmHg | % | HC | IC | PC |
Nghiem et al. (2016)( 32 ) † | New Zealand, >35 years | Markov, not dynamic | LL | HS | QALY | Mandatory salt substitution in processed foods (2 simulations) | 2·0–4·6 | 21·8–51·5 | 1·8–8·2** | Y | Y | N | |
Mandatory salt reduction in bread (2 simulations) | 0·2–0·7 | 2·3–7·9 | 0·2–1·3** | ||||||||||
Wilson et al. (2016)( 33 ) † | New Zealand, >35 years | Markov, not dynamic | LL | HS | QALY | Salt reduction in packaged foods and in fast foods/restaurants (either mandatory or voluntary, 2 simulations) | 3·0–4·6 | 35 | 2·8–8·1** | Y | Y | N | |
Salt reduction in packaged foods, fast foods/restaurants or different food categories (either mandatory or voluntary, 18 simulations) | 0·8–1·6 | 0·8–2·8** | |||||||||||
Nghiem et al. (2015)( 34 ) † | New Zealand, >35 years | Markov, not dynamic | LL | HS | QALY | Dietary advice | –§ | 0·4–0·9** | Y | Y | N | ||
Labelling | 0·1║ | 0·1–0·2** | |||||||||||
Mandatory salt reduction (2 simulations) | 0·7–1·5║ | 0·6–2·7** | |||||||||||
Combination of salt awareness campaign, voluntary salt reduction and labelling | 1·2–1·5║ | 1·1–2·7** | |||||||||||
Salt awareness campaign | 0·4–0·5║ | 0·3–0·8** | |||||||||||
Salt tax | 2·1–4·4║ | 1·9–7·7** | |||||||||||
Cap-and-trade approach | 2·1–4·4║ | 1·9–7·7** | |||||||||||
Collins et al. (2014)( 35 ) | England, >25 years | Spreadsheet | 10 years | SC | LYG | Labelling | 0·2 | 2·0 | 0·1–0·2†† | Y | Y | N | |
Voluntary salt reduction | 1·2 | 15·0 | 0·6–1·3†† | ||||||||||
Mandatory salt reduction (2 simulations) | 1·6 | 20·0 | 0·7–1·8†† | ||||||||||
Mason et al. (2014)( 36 ) | Turkey | Spreadsheet | 10 years | SC | LYG | Salt awareness campaign | 5·0 | 0·4–1·0†† | Y | Y | N | ||
Labelling | 10·0 | 0·8–2·0†† | |||||||||||
Mandatory salt reduction | 10·0 | 0·8–2·0†† | |||||||||||
Combination of mandatory salt reduction and/or labelling and/or salt awareness campaign (3 simulations) | 15·0–30·0 | 1·2–5·9†† | |||||||||||
Cobiac et al. (2012)( 37 ) | Australia, 35–84 years | Markov, not dynamic | LL | HS | DALY; QALY | Mandatory salt reduction | 0·4–0·6║ | 0·4–1·1║,** | Y | Y | N | ||
Dodhia et al. (2012)( 38 ) | England, >16 years | Spreadsheet | 10 years | HS | DALY | Voluntary salt reduction (3 simulations) | 3·0 | 2·0–5·0‡‡ | Y | N | N | ||
Dietary advice | 3·8 | 10·0 | |||||||||||
Barton et al. (2011)( 39 ) | England/Wales, 40–79 years | Spreadsheet | 10 years | HS | QALY; LYG | Mandatory salt reduction | 3·0 | 2·5 | Y | N | N | ||
Martikainen et al. 2011( 40 ) | Finland, 30–74 years | Markov, dynamic | 20 years | SC | QALY | Not specified | 1·0 | 0·6–1·2†† | Y | N | Y | ||
Bibbins-Domingo et al. (2010)( 41 ) | USA, 35–85 years | Markov, dynamic | 10 years | HS | QALY | Not specified (4 simulations) | 3·0 | 1·8–9·1║,** | Y | Y | N | ||
Cobiac et al. (2010)( 42 ) | Australia, 30–100 years | Markov, not dynamic | LL | HS | DALY | Voluntary salt reduction | 0·02–0·03║ | 0·0–0·1║,** | Y | Y | N | ||
Mandatory salt reduction | 0·4–0·6║ | 0·4–1·1║,** | |||||||||||
Dietary advice (2 simulations) | 0·0–2·9¶ | 0·0–5·0║,** | |||||||||||
Smith-Spangler et al. (2010)( 43 ) | USA, 40–85 years | Markov, dynamic | LL | SC | QALY; LYG | Voluntary salt reduction | 9·5 | mean: 1·3║ | Y | Y | N | ||
Sodium tax | 6·0 | mean: 0·9║ | |||||||||||
Murray et al. (2003)( 44 ) | Europe‡ | Markov, dynamic | 100 years | SC | DALY | Voluntary salt reduction | 15·0 | 0·9–2·6║,** | N | Y | N | ||
Mandatory salt reduction | 30·0 | 1·9–5·2 ║,** | |||||||||||
Selmer et al. (2000)( 45 ) | Norway, >40 year | Markov, dynamic | LL | SC | LYG | Combination of health promotion, salt reduction in processed foods, labelling and taxes | 0·6 | 2·0 | Y | Y | Y |
SBP, systolic blood pressure; HC, health cost; IC, intervention cost; PC, productivity cost; LL, lifelong; HS, health system; SC, society; QALY, quality-adjusted life years; LYG, life years gained; DALY, disability-adjusted life years; Y, yes; N, no.
In cases where reductions of salt and SBP were not given explicitly in the studies, estimates were recalculated using the indicated sources.
Nghiem et al. (2016), Wilson et al. (2016) and Nghiem et al. (2015) are based on the same model.
European sub-region with very low mortality rates.
0·4–0·5 g/h (gender-specific) with 4600 h/year.
Gender-specific.
Depending on elapsed time after intervention’s uptake.
Age-specific.
Depending on current blood pressure and ethnicity.
Depending on reference.
Cost-effectiveness of reducing salt
The gain in QALY over the time horizon compared with no intervention or the status quo ranged from 200 by dietary advice in New Zealand( 34 ) to 2·06 million by a voluntary salt reduction in the USA( 43 ). Studies that analysed the outcome as LYG reported an increase of 1970 by a declaration of salt content on food packaging (labelling) in England( 35 ) up to 1·3 million by the voluntary salt reduction in the USA mentioned before( 43 ). DALY averted amounted to 1700 by dietary advice in Australia( 42 ) up to 1·3 million by a mandatory salt reduction in Europe( 44 ).
As an additional benefit cost savings could be achieved. The incremental cost comprised the health-care cost in the case of no intervention or the status quo minus the sum of the intervention cost and potential cost savings in health care and/or productivity due to a salt reduction over the time horizon. This total incremental cost in international dollars (in 2015) ranged from an actual cost of 517 million Intl.$ over a lifelong time horizon by dietary advice in Australia( 42 ) to a cost saving of −109 billion Intl.$ over 10 years by an unspecified intervention in the USA( 41 ). The majority of simulations (fifty-seven out of sixty-two) resulted in net cost savings.
The total incremental benefit and cost of each simulation are shown in Fig. 2(a). To provide a more detailed view on the fifty-one scenarios showing similar incremental benefits and costs, an enlarged section of this cost-effectiveness plane is displayed in Fig. 2(b). Most estimates are in the fourth quadrant, meaning they are associated with higher benefit at lower cost and therefore are considered cost-effective. Five simulations are in the first quadrant, so they bring higher benefit at higher cost. The combined consideration of costs and benefits by means of the ICER (Table 2) shows that the five simulations with a positive ICER are simulations of voluntary and mandatory salt reduction in Europe as well as dietary advice in New Zealand and Australia. According to the authors( 44 ), the two simulations of a salt reduction in processed foods are considered highly cost-effective according to the standard of the WHO (ICER<per capita gross domestic product/DALY).
Table 2.
Intervention | ICER (Intl.$; 2015)* | Time horizon | Perspective | Population | Reference |
---|---|---|---|---|---|
Outcome measurement as cost per QALY | |||||
Not specified | −109 324 to −63 119/QALY† | 10 years | HS | USA | ( 41 ) |
Sodium tax | −19 000/QALY | LL | SC | USA | ( 43 ) |
Voluntary salt reduction | −17 243/QALY | LL | SC | USA | ( 43 ) |
Mandatory salt reduction | −9370/QALY | LL | HS | Australia | ( 37 ) |
Mandatory salt reduction | −4534/QALY | 10 years | HS | England/Wales | ( 39 ) |
Mandatory salt reduction | −3933 to −3893/QALY‡ | LL | HS | New Zealand | ( 34 ) |
Mandatory salt reduction in packaged foods, fast foods/restaurants or different food categories | −3861 to −3545/QALY | LL | HS | New Zealand | ( 33 ) |
Mandatory salt reduction in bread | −3798 to −3610/QALY | LL | HS | New Zealand | ( 32 ) |
Voluntary salt reduction in packaged foods, fast foods/restaurants or different food categories | −3774 to −2707/QALY | LL | HS | New Zealand | ( 33 ) |
Cap-and-trade approach | −3755/QALY | LL | HS | New Zealand | ( 34 ) |
Combination of salt awareness campaign, voluntary salt reduction and labelling | −3690/QALY | LL | HS | New Zealand | ( 34 ) |
Salt tax | −3660/QALY | LL | HS | New Zealand | ( 34 ) |
Salt substitution | −3657 to −3642/QALY | LL | HS | New Zealand | ( 32 ) |
Salt awareness campaign | −3399/QALY | LL | HS | New Zealand | ( 34 ) |
Labelling | −3072/QALY | LL | HS | New Zealand | ( 34 ) |
Not specified | −7/QALY | 20 years | SC | Finland | ( 40 ) |
Dietary advice | 24 625/QALY | LL | HS | New Zealand | ( 34 ) |
Outcome measurement as cost per LYG | |||||
Labelling | −316 870/LYG | 10 years | SC | England | ( 35 ) |
Voluntary salt reduction | −62 896/LYG | 10 years | SC | England | ( 35 ) |
Mandatory salt reduction | −54 270/LYG | 10 years | SC | England | ( 35 ) |
Sodium tax | −29 702/LYG | LL | SC | USA | ( 43 ) |
Voluntary salt reduction | −26 966/LYG | LL | SC | USA | ( 43 ) |
Salt awareness campaign | −14 967/LYG | 10 years | SC | Turkey | ( 36 ) |
Labelling | −8375/LYG | 10 years | SC | Turkey | ( 36 ) |
Mandatory salt reduction | −7747/LYG | 10 years | SC | Turkey | ( 36 ) |
Mandatory salt reduction | −6187/LYG | 10 years | HS | England/Wale | ( 39 ) |
Combination of mandatory salt reduction, salt awareness campaign and/or labelling | −5879/LYG to −3797/LYG§ | 10 years | SC | Turkey | ( 36 ) |
Combination of health promotion, salt reduction in processed foods, labelling and taxes | −3250/LYG | LL | SC | Norway | ( 45 ) |
Outcome measurement as cost per DALY | |||||
Voluntary salt reduction | −10 274/DALY | LL | HS | Australia | ( 42 ) |
Mandatory salt reduction | −9798/DALY | LL | HS | Australia | ( 42 ) |
Mandatory salt reduction | −7965/DALY | LL | HS | Australia | ( 37 ) |
Voluntary salt reduction | −6042 to −6017/DALY† | 10 years | HS | England | ( 38 ) |
Dietary advice (>55 years) | −3226/DALY | 10 years | HS | England | ( 38 ) |
Mandatory salt reduction | 315/DALY | 100 years | SC | Europe¶ | ( 44 ) |
Voluntary salt reduction | 584/DALY | 100 years | SC | Europe¶ | ( 44 ) |
Dietary advice (>115 and >140 mmHg) | 77 632 to 303 918/DALY║ | LL | HS | Australia | ( 42 ) |
QALY, quality-adjusted life years; LYG, life years gained; DALY, disability-adjusted life years; LL, lifelong; HS, health system; SC, society.
Transforming ICER into Intl.$ 2015 is based on point estimates of base scenarios in original studies; confidence intervals or results of sensitivity analyses were not considered.
Depending on assumed effectiveness.
Depending on included types of processed foods.
Depending on included measures.
Depending on target group.
European sub-region with very low mortality rates.
Altogether, a particularly low ICER of <−3072 Intl.$/QALY gained, −6187 Intl.$/LYG and <584 Intl.$/DALY saved, respectively, was assessed for population-wide salt reductions, like a tax on sodium or salt, voluntary or legislative collaboration with industry to reduce salt content in processed foods and the declaration of salt content on food packaging (Table 2). Combinations of different population-wide interventions (e.g. salt reduction in processed foods, salt tax, labelling and/or awareness campaigns) also were cost-saving (−3690 Intl.$/QALY and <−3250 Intl.$/LYG, respectively). Dietary advice for a high-risk group was a rather cost-ineffective intervention (24 600 Intl.$/QALY and up to 303 900 Intl.$/DALY) compared with population-wide approaches. As can be seen from Table 2, the estimates of five scenarios in the USA and England show considerably lower ICER with up to −109 324 Intl.$/QALY gained by a not further specified population-wide salt reduction of 3 g/d and −316 870 Intl.$/LYG by labelling over 10 years compared with the status quo. While the scenarios in the USA are associated with both high savings in medical costs and a high increase of QALY, the low ICER of the labelling in England is attributable to high cost savings and comparatively low LYG of 1970.
In nine studies the cost-effectiveness of different interventions to reduce salt intake were evaluated and compared with the same model. Here, the salt reduction in processed foods in general or in particular food groups was dominant – that is, higher benefit at lower costs or cost savings – compared with dietary advice( 32 , 34 , 38 , 42 ), and a salt awareness campaign was dominant compared with labelling( 34 , 36 ). Other results were inconsistent; for example, the comparison of salt reduction in processed foods showed both lower( 35 , 42 ) and higher( 33 , 44 ) ICER for a voluntary compared with a mandatory approach, and a salt/sodium tax was both dominant( 43 ) to a salt reduction by the food industry and dominated( 34 ) by it (Table 2).
Methodological quality of included studies
The quality assessment showed flaws in the conduct of nearly each study. Compared with the Drummond and Jefferson/British Medical Journal checklist( 31 ), no study fulfilled all thirty-six items (Table 3). The most frequent flaws were missing details of input data (items 8 to 13), a missing justification of choice of economic evaluation (item 7) and no discussion of the generalizability of the results (item 36). Instead of giving details of the design and results of the effectiveness studies (items 9 and 10) or the methods to value health states (items 12 and 13), most studies stated the reference of the original study. Altogether, four out of fourteen studies presented insufficient details of the methods for the estimation of costs (item 17) or the model used (item 20), and eight studies missed details regarding the execution of sensitivity analysis (items 26 to 29). Nine studies from New Zealand, England, Turkey, Finland, USA and Europe used validated models.
Table 3.
Number of studies | |||||
---|---|---|---|---|---|
Item | Yes | Partly | No | ? | NA |
Study design | |||||
1. The research question is stated | 12 | 2 | |||
2. The economic importance of the research question is stated | 6 | 8 | |||
3. The viewpoint/s of the analysis is/are clearly stated and justified | 10 | 4 | |||
4. The rationale for choosing the alternative programmes or interventions compared is stated | 13 | 1 | |||
5. The alternatives being compared are clearly described | 14 | ||||
6. The form of economic evaluation used is stated | 12 | 2 | |||
7. The choice of form of economic evaluation is justified in relation to the questions addressed | 2 | 12 | |||
Data collection | |||||
8. The source/s of effectiveness estimates used is/are stated | 13 | 1 | |||
9. Details of the design and results of effectiveness study are given (if based on a single study) | 12 | 2 | |||
10. Details of the method of synthesis or meta-analysis of estimates are given (if based on an overview of a number of effectiveness studies) | 13 | 1 | |||
11. The primary outcome measure/s for the economic evaluation is/are clearly stated | 11 | 3 | |||
12. Methods to value health states and other benefits are stated | 1 | 10 | 3 | ||
13. Details of the subjects from whom valuations were obtained are given | 1 | 10 | 3 | ||
14. Productivity changes (if included) are reported separately | 2 | 12 | |||
15. The relevance of productivity changes to the study question is discussed | 2 | 12 | |||
16. Quantities of resources are reported separately from their unit costs | 6 | 4 | 4 | ||
17. Methods for the estimation of quantities and unit costs are described | 10 | 4 | |||
18. Currency and price data are recorded | 14 | ||||
19. Details of currency of price adjustments for inflation or currency conversion are given | 12 | 1 | 1 | ||
20. Details of any model used are given | 10 | 4 | |||
21. The choice of model used and the key parameters on which it is based are justified | 8 | 6 | |||
Analysis and interpretation of results | |||||
22. Time horizon of costs and benefits is stated | 14 | ||||
23. The discount rate/s is/are stated | 13 | 1 | |||
24. The choice of rate/s is/are justified | 7 | 6 | 1 | ||
25. An explanation is given if costs or benefits are not discounted | 1 | 13 | |||
26. Details of statistical tests and confidence intervals are given for stochastic data | 9 | 4 | 1 | ||
27. The approach to sensitivity analysis is given | 11 | 2 | 1 | ||
28. The choice of variables for sensitivity analysis is justified | 8 | 3 | 3 | ||
29. The ranges over which the variables are varied are stated | 10 | 2 | 2 | ||
30. Relevant alternatives are compared | 14 | ||||
31. Incremental analysis is reported | 14 | ||||
32. Major outcomes are presented in a disaggregated as well as aggregated form | 9 | 2 | 3 | ||
33. The answer to the study question is given | 14 | ||||
34. Conclusions follow from the data reported | 12 | 2 | |||
35. Conclusions are accompanied by the appropriate caveats | 10 | 2 | 2 | ||
36. Generalizability issues are addressed | 2 | 2 | 9 | 1 |
?, not clear; NA, not appropriate.
Discussion
The present systematic review of fourteen full economic evaluations showed that the vast majority (95 %) of modelled interventions aiming at a salt reduction were cost-effective or even cost-saving. Especially low ICER were found for an unspecified intervention, salt/sodium taxes, voluntary or mandatory salt reductions in processed foods by the food industry and the declaration of salt content on food packaging (labelling). Interestingly, salt-specific health education like salt awareness campaigns via mass media or labelling had a surprisingly low ICER( 35 , 36 ), although a combination of the provision of information and structural changes might be the most effective strategy to facilitate behavioural modifications( 46 , 47 ). The low ICER of salt awareness campaigns can be attributed to the wide reach of this intervention and its comparatively low cost per subject. Targeted dietary advice for people of a certain age or with certain blood pressure was a rather cost-ineffective approach. Thus, the results demonstrate that especially population-wide salt reductions might be a cost-effective way to prevent hypertension and CVD in OECD member countries. As the salt intake through processed foods is high in OECD member countries( 11 ), a reduction of salt content by the food industry and gastronomy seems to be effective, for example collaborating with food manufacturers and sellers, by setting maximum levels of salt content, by regulating salt substitution or by implementing taxes on salt. Especially a salt reduction in staple foods like bread could effectively lower salt intake, probably without influencing quality( 48 ) or even being noticed by consumers( 49 ). However, comprehensive analyses of effectiveness and cost-effectiveness of salt reduction in gastronomy are lacking until now.
A published review( 26 ), recently updated( 25 ), concluded there is economic evidence that interventions aiming at a reduction of salt intake in the whole population, like salt reduction in processed foods, mass media campaigns, taxes and not further specified approaches, are very cost-effective in the prevention of hypertension. According to the authors, these kinds of interventions are associated with a gain in QALY, provide savings in medical costs and productivity costs due to averted hypertension cases, and/or are low in cost of implementation. Altogether the authors narratively described the results of seventeen studies, of which twelve analysed costs and benefits of salt reduction in OECD member countries: six partial and six full economic evaluations. Furthermore, the results of these reviews demonstrate that salt reductions – either population-wide or targeted – in non-OECD member countries are cost-effective as well, suggesting that they are one of the ‘best buys’ globally( 50 ). To achieve a salt reduction a combined strategy of food reformulation by the food industry and gastronomy, public awareness campaigns and changes in the environment is recommended by the WHO( 46 ) and EU( 47 ). Interestingly, such a combination of different approaches was associated with higher ICER( 34 , 36 ). Therefore, this approach might be most effective to achieve a salt reduction, but it might not be the most cost-effective approach.
A comparison of ICER of different studies has to be done carefully due to the different methods and assumptions applied in each model. Consequently, the comparison of results in the studies that analysed different interventions with the same model gives the best hint on the most cost-effective strategy. Unfortunately, the results are not consistent, showing that the assumptions made might be more important than the chosen intervention in influencing the resulting cost-effectiveness.
Measurement of cost-effectiveness
Therefore, the results are highly influenced by the methods applied and have to be seen in the context of the simplifications and assumptions made. Especially the chosen perspective for an economic analysis has an important impact on the resulting cost-effectiveness( 24 ). Of the different possible viewpoints, the societal perspective is the broadest one and might be the most appropriate for addressing public health topics. Nevertheless, only half of the included studies applied this perspective. Furthermore, the quality assessment showed flaws in all included studies. Particularly the missing details of the input data led to a lack of transparency and comprehensibility of the composition of the ICER. Similarly, the authors of earlier reviews( 25 ) stated that until now economic evaluations of population-wide salt reductions lack consideration of all key aspects to provide convincing and robust data on cost-effectiveness. This is partly attributed to imprecise model development or lack of transparency and partly to general problems assessing the costs and the benefits in public health nutrition.
Assessment of the cost of a salt reduction
Measuring the total costs from a societal perspective means to consider almost all occurring expenditures and savings independently of any stakeholder group( 21 , 24 ). Beside direct and indirect health and intervention costs arising at the health system, for patients or society, this includes changes in the consumption of resources in other sectors; for example, in the food industry due to demanding costly food reformulations or in the salt industry by decreasing sales. The assessment of these aspects is complex or even impossible, and sometimes also negligible( 21 , 24 ). For example, as products are reformulated within the natural product life cycle, it is often assumed that consideration of cost for the reformulation might lead to an overestimation of costs of implementation( 35 , 43 ). Similarly, there are various methods and recommendations for the measurement of indirect costs like productivity losses( 24 ).
Assessment of the health benefits of a salt reduction
The measurement of benefits, on the other hand, is complicated by the assessment of the effectiveness of a salt reduction owing to inconsistent data on the resulting systolic blood pressures and cardiovascular events. Although several meta-analyses show the effectiveness of a salt reduction in decreasing blood pressure( 6 , 7 ), the impact on cardiovascular mortality is less obvious, showing both significant reductions of CVD and stroke deaths( 7 , 9 ) and the denying of positive effects( 51 ). Additionally, due to a rather small benefit for individuals from population-wide salt reductions, this small benefit can easily be compensated by a small risk of negative effects( 15 ). That is why there are still some critics questioning the effectiveness and appropriateness of population-wide reductions of salt intake, especially pronouncing the so-called J-curve phenomenon with a higher cardiovascular risk for sodium under 3 g/d and over 7 g/d (<7·5 and >18 g salt daily) compared with a sodium consumption of 4–6 g/d( 52 ). The discussion about flawed results, reverse causality, insufficient sample size and appropriateness of the measurement of sodium and salt intakes following the publication of new results on the association between salt intake and cardiovascular events illustrates the fixed positions of salt proponents and opponents. However, this leads to inconsistent data and eventually to confusion of and refusal to act by policy decision makers and public health practitioners.
Unfortunately, until now there are only few analyses of the effectiveness of real-life interventions aiming at a salt reduction. Only five studies in the present review( 34 , 35 , 37 , 42 , 43 ) used realistic assumptions based on experiences made in New Zealand and Great Britain. Therefore, the chosen interventions to achieve a certain salt reduction differed in the included CEA, as well as the assumed effectiveness – depending on compliance, reach or duration. Table 1 shows that these input parameters vary in the studies and gives an explanation for the wide range of results. This is emphasized by the results of the sensitivity analyses of one research team( 33 , 34 ), showing that the uncertainty in the assumed salt reduction and the effect on the CVD risk contributed the most to uncertainty in the resulting incremental cost and benefit.
Other relevant aspects
There are other relevant factors influencing the cost-effectiveness of a salt reduction, but which are neglected in most CEA and CUA until now. One example is the delay in benefit accumulation after the implementation of the intervention, which occurs first due to a lag in blood pressure reduction( 53 ) and second as a stepwise salt reduction seems the most accepted method( 54 ).
Furthermore, most studies concentrate on the adult population, although the long-term health gains of a salt reduction probably would be the highest among the youngest. In the current review only one study( 44 ) explicitly considered children in its analyses. Unintended changes of the eating pattern following a population-wide salt reduction, such as due to a negative cross-price elasticity after introducing a salt tax( 55 ), could diminish the overall benefit of this intervention. On the other hand, preventive effects regarding cancer and other diseases( 56 ) suggest additional benefits of a salt reduction and therefore a higher cost-effectiveness. The exclusion of these aspects reduces the complexity of the model but might also lead to biased estimates about the overall cost-effectiveness of salt reductions. But this is a general problem of complex public health (nutrition) interventions and not unique for salt reduction strategies.
Limitations
Some limitations of the present review have to be mentioned. First, publication bias was not investigated here. Second, it might be possible that suitable analyses, for example unpublished grey literature by governments or non-profit organizations, were not detected in our search. In addition, some methodological aspects impeded the summarizing of economic evidence. The heterogeneity in the applied modelling techniques, the study perspective and the considered cost types made it harder to establish comparable results. Additionally, direct medical costs are country-specific as they are influenced by national treatment standards and the medical compensation system of each country( 57 ) and the effectiveness of a salt reduction depends on the current salt consumption of the population: the reduction of high salt intakes is rather easy while the further reduction of moderate salt levels seems to be more difficult – as seen in the UK( 58 ). Therefore, the results show the current state of research, which is in favour of population-wide salt reductions. Nevertheless, the precise results might not be generalized or transferred to other settings and countries easily due to the methodological aspects mentioned before. More likely, the major benefit of the current systematic review of the cost-effectiveness of salt reduction strategies might be the identification of the different methods, models and assumptions applied, as proposed elsewhere( 59 ). The strengths and weaknesses as well as the key variables shown might help to develop further models of salt reduction in different countries and settings.
Conclusion
In summary, the present systematic review of full economic evaluations highlights that salt reductions could be cost-effective in the prevention of hypertension and CVD. Especially single population-wide approaches had a favourable ICER, whereas combinations of different (population-wide) strategies were associated with lower cost savings and targeted interventions mostly were not cost-effective. Although the results might be highly influenced by the specific context, the chosen models and perspectives, the assumptions made and the considered costs, the consistent results regarding cost-effectiveness in 95 % of the modelled scenarios emphasize the introduction of salt reductions in OECD member countries.
Nevertheless, so far, published CEA of targeted and population-based salt reductions have insufficiently included all relevant aspects in their models, due to a lack of data, difficulties in assessing the cost and benefits, and simplified assumptions to reduce the complex models. Of particular note are the inconsistent and sometimes confusing results concerning a realistic estimation of the effectiveness and potential negative effects of a salt reduction. Reflecting the quote of G. Box, ‘All models are wrong, but some are useful’, it should at least be tried to obtain better model estimates and an improved comparability between studies and settings, enabling conclusions about the economic effects of salt reduction to be derived.
Acknowledgements
Financial support: This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. Conflict of interest: None. Authorship: E.S. was responsible for study concept and design, data analysis and interpretation, manuscript drafting. D.N. was responsible for study concept and design, data analysis, supervision, manuscript revision. A.K. was responsible for study concept and design, critical revision of the manuscript for important intellectual content. Ethics of human subject participation: Not applicable.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/S1368980017000593.
References
- 1. Lim SS, Vos T, Flaxman AD et al. (2012) A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet 380, 2224–2260. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. World Health Organization (2009) Global Health Risks: Mortality and Burden of Disease Attributable to Selected Major Risks. Geneva: WHO. [Google Scholar]
- 3. Appel LJ (2006) Dietary approaches to prevent and treat hypertension: a scientific statement from the American Heart Association. Hypertension 47, 296–308. [DOI] [PubMed] [Google Scholar]
- 4. Dickinson HO, Mason JM, Nicolson DJ et al. (2006) Lifestyle interventions to reduce raised blood pressure: a systematic review of randomized controlled trials. J Hypertens 24, 215–233. [DOI] [PubMed] [Google Scholar]
- 5. Mozaffarian D, Fahimi S, Singh GM et al. (2014) Global sodium consumption and death from cardiovascular causes. N Engl J Med 371, 624–634. [DOI] [PubMed] [Google Scholar]
- 6. He FJ, Li J & MacGregor GA (2013) Effect of longer-term modest salt reduction on blood pressure. Cochrane Database Syst Rev issue 4, CD004937. [DOI] [PubMed] [Google Scholar]
- 7. Aburto NJ, Ziolkovska A, Hooper L et al. (2013) Effect of lower sodium intake on health: systematic review and meta-analyses. BMJ 346, f1326. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Graudal NA, Hubeck-Graudal T & Jurgens G (2011) Effects of low sodium diet versus high sodium diet on blood pressure, renin, aldosterone, catecholamines, cholesterol, and triglyceride. Cochrane Database Syst Rev issue 11, CD004022. [DOI] [PubMed] [Google Scholar]
- 9. Strazzullo P, D’Elia L, Kandala N et al. (2009) Salt intake, stroke, and cardiovascular disease: meta-analysis of prospective studies. BMJ 339, b4567. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. World Health Organization (2012) Guideline: Sodium Intake for Adults and Children. Geneva: WHO. [PubMed] [Google Scholar]
- 11. Brown IJ, Tzoulaki I, Candeias V et al. (2009) Salt intakes around the world: implications for public health. Int J Epidemiol 38, 791–813. [DOI] [PubMed] [Google Scholar]
- 12. Bloom DE, Cafiero ET, Jané-Llopis E et al. (2011) The Global Economic Burden of Noncommunicable Diseases. Geneva: World Economic Forum. [Google Scholar]
- 13. McLaren L (2012) Policy Options for Reducing Dietary Sodium Intake. SPP Research Papers vol. 5, no. 20. Calgary: University of Calgary. [Google Scholar]
- 14. Cappuccio FP, Capewell S, Lincoln P et al. (2011) Policy options to reduce population salt intake. BMJ 343, d4995. [DOI] [PubMed] [Google Scholar]
- 15. Rose G (1985) Sick individuals and sick populations. Int J Epidemiol 14, 32–38. [DOI] [PubMed] [Google Scholar]
- 16. Knorpp L & Kroke A (2010) Salzreduktion als bevölkerungsbezogene Präventionsmaßnahme. Teil 3: Vorteile des Ansatzes und kritische Betrachtung möglicher adverser Effekte. Ernährungs Umschau 57, 410–415. [Google Scholar]
- 17. World Health Organization (2013) Mapping Salt Reduction Initiatives in the WHO European Region. Copenhagen: WHO Regional Office for Europe. [Google Scholar]
- 18. European Commission (2012) Survey on Members States’ Implementation of the EU Salt Reduction Framework. Brussels: European Union. [Google Scholar]
- 19. Webster JL, Dunford EK, Hawkes C et al. (2011) Salt reduction initiatives around the world. J Hypertens 29, 1043–1050. [DOI] [PubMed] [Google Scholar]
- 20. Karppanen H & Mervaala E (2006) Sodium intake and hypertension. Prog Cardiovasc Dis 49, 59–75. [DOI] [PubMed] [Google Scholar]
- 21. Hughes R & Margetts BM (2011) Practical Public Health Nutrition. Oxford: Wiley-Blackwell. [Google Scholar]
- 22. Appel LJ, Angell SY, Cobb LK et al. (2012) Population-wide sodium reduction: the bumpy road from evidence to policy. Ann Epidemiol 22, 417–425. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Knorpp L & Kroke A (2012) Evidence-based public health nutrition: what constitutes good evidence? CAB Rev 7, 1–22. [Google Scholar]
- 24. Drummond MF, Sculpher MJ, Torrance GW et al. (2005) Methods for the Economic Evaluation of Health Care Programmes, 3rd ed. Oxford: Oxford University Press. [Google Scholar]
- 25. Wang G & Bowman BA (2013) Recent economic evaluations of interventions to prevent cardiovascular disease by reducing sodium intake. Curr Atheroscler Rep 15, 349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Wang G & Labarthe D (2011) The cost-effectiveness of interventions designed to reduce sodium intake. J Hypertens 29, 1693–1699. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Neal B (2007) The Effectiveness and Costs of Population Interventions to Reduce Salt Consumption. Background paper to the WHO Forum and Technical Meeting on ‘Reducing Salt Intake in Populations’, 5–7 October 2006, Paris, France. Geneva: WHO. [Google Scholar]
- 28. Centre for Reviews and Dissemination (2009) Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. York: CRD, University of York. [Google Scholar]
- 29. Organisation for Economic Co-operation and Development (2016) Purchasing power parities (PPP) for GDP. http://stats.oecd.org/index.aspx?DatasetCode=SNA_TABLE4# (accessed May 2016).
- 30. Organisation for Economic Co-operation and Development (2016) Price deflator. http://stats.oecd.org/index.aspx?DatasetCode=QNA (accessed May 2016).
- 31. Drummond MF & Jefferson TO (1996) Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 313, 275–283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Nghiem N, Blakely T, Cobiac LJ et al. (2016) The health gains and cost savings of dietary salt reduction interventions, with equity and age distributional aspects. BMC Public Health 16, 423. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Wilson N, Nghiem N, Eyles H et al. (2016) Modeling health gains and cost savings for ten dietary salt reduction targets. Nutr J 15, 44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Nghiem N, Blakely T, Cobiac LJ et al. (2015) Health and economic impacts of eight different dietary salt reduction interventions. PLoS One 10, e0123915. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Collins M, Mason H, O’Flaherty M et al. (2014) An economic evaluation of salt reduction policies to reduce coronary heart disease in England: a policy modeling study. Value Health 17, 517–524. [DOI] [PubMed] [Google Scholar]
- 36. Mason H, Shoaibi A, Ghandour R et al. (2014) A cost effectiveness analysis of salt reduction policies to reduce coronary heart disease in four Eastern Mediterranean countries. PLoS One 9, e84445. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Cobiac LJ, Magnus A, Lim S et al. (2012) Which interventions offer best value for money in primary prevention of cardiovascular disease? PLoS One 7, e41842. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38. Dodhia H, Phillips K, Zannou M et al. (2012) Modelling the impact on avoidable cardiovascular disease burden and costs of interventions to lower SBP in the England population. J Hypertens 30, 217–226. [DOI] [PubMed] [Google Scholar]
- 39. Barton P, Andronis L, Briggs A et al. (2011) Effectiveness and cost effectiveness of cardiovascular disease prevention in whole populations: modelling study. BMJ 343, d4044. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Martikainen JA, Soini EJO, Laaksonen DE et al. (2011) Health economic consequences of reducing salt intake and replacing saturated fat with polyunsaturated fat in the adult Finnish population: estimates based on the FINRISK and FINDIET studies. Eur J Clin Nutr 65, 1148–1155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Bibbins-Domingo K, Chertow GM, Coxson PG et al. (2010) Projected effect of dietary salt reductions on future cardiovascular disease. N Engl J Med 362, 590–599. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Cobiac LJ, Vos T & Veerman JL (2010) Cost-effectiveness of interventions to reduce dietary salt intake. Heart 96, 1920–1925. [DOI] [PubMed] [Google Scholar]
- 43. Smith-Spangler CM, Juusola JL, Enns EA et al. (2010) Population strategies to decrease sodium intake and the burden of cardiovascular disease: a cost-effectiveness analysis. Ann Intern Med 152, 481–487. [DOI] [PubMed] [Google Scholar]
- 44. Murray CJL, Lauer JA, Hutubessy RCW et al. (2003) Effectiveness and costs of interventions to lower systolic blood pressure and cholesterol: a global and regional analysis on reduction of cardiovascular-disease risk. Lancet 361, 717–725. [DOI] [PubMed] [Google Scholar]
- 45. Selmer RM, Kristiansen I, Haglerød A et al. (2000) Cost and health consequences of reducing the population intake of salt. J Epidemiol Community Health 54, 697–702. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. World Health Organization (2007) Reducing Salt Intake in Populations: Report of a WHO Forum and Technical Meeting. Geneva: WHO. [Google Scholar]
- 47. High Level Group on Diet Physical Activity and Health (2008) EU Framework for national salt initiatives. http://ec.europa.eu/health/archive/ph_determinants/life_style/nutrition/documents/salt_initiative.pdf (accessed March 2016).
- 48. Kanzler S, Hartmann C, Gruber A et al. (2014) Salt as a public health challenge in continental European convenience and ready meals. Public Health Nutr 17, 2459–2466. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49. Girgis S, Neal B, Prescott J et al. (2003) A one-quarter reduction in the salt content of bread can be made without detection. Eur J Clin Nutr 57, 616–620. [DOI] [PubMed] [Google Scholar]
- 50. World Health Organization (2011) Prevention and Control of NCDs: Priorities for Investment: Discussion Paper. First Global Ministerial Conference on Healthy Lifestyles and Noncommunicable Disease Control (Moscow, 28–29 April 2011). Geneva: WHO.
- 51. Taylor RS, Ashton KE, Moxham T et al. (2011) Reduced dietary salt for the prevention of cardiovascular disease: a meta-analysis of randomized controlled trials (Cochrane review). Am J Hypertens 24, 843–853. [DOI] [PubMed] [Google Scholar]
- 52. O’Donnell M, Mente A, Rangarajan S et al. (2014) Urinary sodium and potassium excretion, mortality, and cardiovascular events. N Engl J Med 371, 612–623. [DOI] [PubMed] [Google Scholar]
- 53. Lewington S, Clarke R, Qizilbash N et al. (2002) Age-specific relevance of usual blood pressure to vascular mortality: a meta-analysis of individual data for one million adults in 61 prospective studies. Lancet 360, 1903–1913. [DOI] [PubMed] [Google Scholar]
- 54. Penney S (2009) Dropping the Salt: Practical Steps Countries are Taking to Prevent Chronic Non-Communicable Diseases through Population-Wide Dietary Salt Reduction. Ottawa: Public Health Agency of Canada. [Google Scholar]
- 55. Mytton O, Gray A, Rayner M et al. (2007) Could targeted food taxes improve health? J Epidemiol Community Health 61, 689–694. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56. He FJ & MacGregor GA (2009) A comprehensive review on salt and health and current experience of worldwide salt reduction programmes. J Hum Hypertens 23, 363–384. [DOI] [PubMed] [Google Scholar]
- 57. Reinhold T, Brüggenjürgen B, Schlander M et al. (2010) Economic analysis based on multinational studies: methods for adapting findings to national contexts. J Public Health 18, 327–335. [Google Scholar]
- 58. National Institute for Health and Clinical Excellence (2010) Prevention of Cardiovascular Disease. NICE Public Health Guidance 25. Manchester: NICE. [Google Scholar]
- 59. Anderson R (2010) Systematic reviews of economic evaluations: utility or futility? Health Econ 19, 350–364. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
For supplementary material accompanying this paper visit https://doi.org/10.1017/S1368980017000593.