Improving economic evaluations in stroke: A report from the ESO Health Economics Working Group

Abstract

Introduction

Approaches to economic evaluations of stroke therapies are varied and inconsistently described. An objective of the European Stroke Organisation (ESO) Health Economics Working Group is to standardise and improve the economic evaluations of interventions for stroke.

Methods

The ESO Health Economics Working Group and additional experts were contacted to develop a protocol and a guidance document for data collection for economic evaluations of stroke therapies. A modified Delphi approach, including a survey and consensus processes, was used to agree on content. We also asked the participants about resources that could be shared to improve economic evaluations of interventions for stroke.

Results

Of 28 experts invited, 16 (57%) completed the initial survey, with representation from universities, government, and industry. More than half of the survey respondents endorsed 13 specific items to include in a standard resource use questionnaire. Preferred functional/quality of life outcome measures to use for economic evaluations were the modified Rankin Scale (14 respondents, 88%) and the EQ-5D instrument (11 respondents, 69%). Of the 12 respondents who had access to data used in economic evaluations, 10 (83%) indicated a willingness to share data. A protocol template and a guidance document for data collection were developed and are presented in this article.

Conclusion

The protocol template and guidance document for data collection will support a more standardised and transparent approach for economic evaluations of stroke care.

Introduction

Interventions for stroke need to be evaluated for their cost-effectiveness, as well as their clinical effectiveness. Several countries now incorporate results of health-economic analyses as part of their national clinical guidelines for stroke.^1,2 A paucity of published cost-effectiveness studies means that very few clinical recommendations in these guidelines have information about the value of treatments. Comparability and translation of economic evaluation results are complicated by variability of study methods and differences in health care system organisation and expenditure across national and regional settings.³ Another limitation is that the quality of studies varies between studies.⁴ Therefore, with the growing number of interventions becoming available to prevent or treat stroke, it is important to standardise and improve the methods for conducting cost-effectiveness studies in stroke.

While checklists exist for the reporting of economic evaluations, no guidance is provided regarding collection of data on resource use or costs. In addition, existing recommendations for health-related economic evaluations are generic and do not provide guidance specifically for research related to patients with stroke. The European Stroke Organisation (ESO) Health Economics Working Group had its first meeting in 2015 to discuss the standardisation of health economic methods for future clinical trials,³ and was formally established in 2016 with broad aims of compiling and developing resources to facilitate economic evaluations of stroke therapies (Table 1). In this article, we give recommendations for a more standardised and transparent method for economic evaluations of stroke care.

Table 1.

Actions suggested for the working group.

Compiling existing resources for economic evaluation

1. Develop a directory of health-economic models, protocols and questionnaires.

2. Investigate processes required to identify and share such resources.

3. Identify manuals for health technology assessment in each country.

Development of resources for the standardisation of economic evaluations

1. Develop a protocol template for health-economic evaluations in stroke.

2. Develop a common model.

3. Develop a data collection questionnaire template with recommendations for essential, recommended and elective categories of variables.

4. Develop recommendations on how data should be systematically collected.

Methods

The ESO Health Economics Working Group was established in 2015 after discussion among 53 ESO members who had experience with economic evaluations of stroke therapies. Of the 53 experts, 10 were nominated as members of the executive group, while a further 16 (including two coordinators: JK and AW) were retained as corresponding members.

The working group used a modified Delphi technique for the present project.⁵ At the annual meeting at the ESO Conference in Prague in 2017, the participants agreed on a survey, which was distributed to 28 experts in economic evaluations of stroke therapies, including the members of the ESO Health Economics Working Group. The core questions in the survey focused on protocols for economic evaluations and the data collected for economic evaluations (see Online Supplement). The survey also included questions about resources that could be shared to improve economic evaluations in stroke research, including access to existing datasets and models used for economic evaluations.

Based on the responses from the survey, a protocol template and a guidance document for data collection were developed. These materials were further refined prior to presentation at the annual meeting at the ESO Conference in Gothenburg in 2018, where the materials were reviewed and consensus on the content was reached by the working group.

Results

The survey was sent to the 26 working group members and two other researchers nominated by the working group for their specific expertise. Of the 28 people invited to participate in the survey, 16 responded, of which seven worked at universities or hospitals, two in government, two in industry and five did not provide their affiliations or occupation.

Use of standard protocols

Of the 16 respondents, 13 did not use a standard template for economic evaluation protocols (81%). However, five (31%) used a checklist to guide the development of their protocols. Respondents used the Consolidated Health Economic Evaluation Reporting Standards (CHEERS),⁶ the Drummond checklist⁷ and the National Institute for Health and Care Excellence (NICE) guidelines for England to guide development of protocols.⁸ The protocol items suggested by the respondents are outlined in Figure 1.

Figure 1.

Support for the inclusion of items in a standard health-economic evaluation protocol in stroke.

Use of questionnaires to capture resource use

Fifteen respondents did not have a standard resource use questionnaire for economic evaluations (94%). Respondents indicated their support for items to be included in a standard questionnaire to capture resource use (Figure 2). At least 50% of respondents supported inclusion of 13 suggested items for a standard resource use questionnaire. Other suggested items compiled from the open text responses were: outpatient procedures, diagnostic tests, transport between hospitals, distance to care provider and income. When asked about the level of detail required for the data collected, it was suggested that when collecting information about consultations/services provided by health professionals, the number of consultations, type of provider, duration and out-of-pocket costs were important. When collecting information about medications, it was suggested that researchers could collect the broader categories of medications used by patients (e.g. antihypertensive, antithrombotic). The importance of tailoring the data collection was emphasised, and it was suggested that the amount of detail collected should depend on whatever helps to quantify the important drivers of cost relevant to that study.

Figure 2.

Support for the inclusion of suggested items on a standard resource use questionnaire in stroke.

Estimating resource use from routinely collected data

The majority of respondents indicated that they estimated resource use based on stroke type (n = 11, 69%), discharge destination (n = 10, 63%) or by first ever or recurrent stroke (n = 9, 56%). Other clinical or demographic data that could be used to estimate resource use included time since stroke, modified Rankin Scale⁹ (mRS) at discharge and 90 days, age, sex, comorbidities (e.g. atrial fibrillation) and the National Institute of Health Stroke Scale¹⁰ (NIHSS) score.

Patient outcomes

The most popular outcome measure for economic evaluations of stroke therapies were the mRS^9,11 (n = 14, 88%) followed by the EQ-5D instrument¹² (n = 11, 69%). Other outcome measures included the Stroke Impact Scale,¹³ Assessment of Quality of Life¹⁴ (AQoL) and the Barthel Index.^15,16 All respondents indicated that they collected the mRS as an ordinal scale. An advantage of the mRS was having published literature on the direct costs for each category which is useful for studies where it has not been possible to collect data on resource utilisation/costs directly from participants. An advantage of health-related quality of life measures, like the EQ-5D and the AQoL, is that utility values can be generated that can then be used in calculations of quality-adjusted life years (QALYs). The mRS and Barthel Index can be converted to utility values to serve this purpose, although the assigned utility scores are less granular than with direct quality of life measures.^17,18

Sharing available resources and data for economic evaluations

There were 11 respondents (69%) who indicated that they had access to datasets used for economic evaluations. The scope of data was broad and included data from clinical trials (acute, subacute and community-based intervention studies), administrative data, national registry data or cost data.

Of the 11 respondents who had access to existing economic evaluations data, 9 (82%) replied that they were willing to share data and 9 (90%, 1 missing response) indicated that permission to use the data would be needed, as well as funding to cover any administrative expenses (e.g. formatting the data). The issue of needing ethical approvals for secondary use of the data and the importance of acknowledging the original source of the data was expressed.

Models for economic evaluations

The short duration of clinical trials and the long-term consequences of stroke renders modelling almost inevitable since economic evaluations based upon trial results would not capture fully the benefits or harms of interventions. Eight of the respondents (50%) indicated that they had previously used models for economic evaluations, including Markov models, partition survival modelling and discrete event simulation. Several were developed in Microsoft Excel with and without add-on software and with different levels of sophistication (Visual Basic for Applications coding). The use of software such as TreeAge, SAS, R and Stata for developing models was also mentioned.

Final protocol template and guidance document for collecting resource use and cost data

Table 2 includes the items that were agreed for a protocol template for economic evaluations of stroke therapies, and Table 3 includes information to guide data collection on resource use. In Table 3 we highlight the importance of estimating the additional costs of the intervention, which would include items such as the cost of therapists and support staff, training and education, equipment, medication or facility costs. Estimating the costs of the intervention may be complex, and this must be considered in the data collection. For example, the costs of a novel treatment with a large capital outlay may be more obvious than subtle adaptations to existing care pathways. It was noted that evidence from process evaluations would assist with informing the costs to include.

Table 2.

Protocol template.

Item	Detail required or examples
Population and setting	Country/region
	Sub-groups of patients with stroke
	Organisational structure (private/public)
	Care pathways
Research question	E.g. To determine/estimate the cost-effectiveness of intervention compared to comparator/control group
Outcomes	The outcome that is used for the cost-effectiveness measure
	Questionnaire used to estimate quality of life
Treatment groups	Intervention groups
	Comparator/control groups
Perspective	Health service
	Patient
	Societal
	Direct/indirect
Study design / data source	Alongside RCT
	Model-based economic evaluations using data from multiple sources
Economic study design	Cost benefit
	Cost effectiveness
	Cost utility
	Budget impact analysis
Economic model	E.g. Model assumptions, model name and reference in literature
Reference year	E.g. Year and inflation/deflation. Source for adjusting costs
Time horizon	Assumptions made for modelling longer-term costs and outcomes
Discounting	Nationally recommended – usually 3% or 5%
Resource use collected	E.g. Hospital readmissions, family physician contacts• Delivery of the intervention and justification for inclusion• If used, the method of estimating resource use based on clinical or demographic details of participants and the reference in literature
Data collection methods	Administrative data
	Patient self-report via survey
Sub-group analysis	Stroke type
	Hospital type (private/public)
Sensitivity analysis	Monte Carlo simulation (multivariable)
	One-way sensitivity
	Scenario analysis
Funder	Government
	Industry
	Private insurance

Table 3.

Resource use data collection guidance.

Variables	Data collection guidance
Additional costs of treatment	This should be the costs of delivering the new treatment that are above and beyond a comparator treatment or usual care
Data collection to be considered for studies using a health care sector perspective a
Transport	Transport between hospital (e.g. transfer to a centre providing reperfusion) or from home, type of vehicle
Rehabilitation	Inpatient or outpatient rehabilitation, number of sessions, length of stay, specific services and procedures provided while in rehabilitation (e.g. assessment of impairment)
Hospital presentations	Number of presentations, type of presentation (e.g. emergency department or admission), dates or length of stay, specific services and procedures provided while in hospital (e.g. reperfusion)
Respite care	Number of times used, length of stay
Medications	Type of medications (e.g. antihypertensive medications), number of medications, dose, time on medication
Change in residence and living arrangements	Information to capture changes in residence as this is an indicator of independence that affects costs (e.g. costs of moving to an aged care facility may be applied)
Home modifications	Type of home modifications and out-of-pocket costs (e.g. for the installation of ramp to home)
Aids and equipment	Type of aids and equipment and out-of-pocket costs (e.g. for a walking frame)
Community services	Type of service, number of times provided and out-of-pocket costs
Family physician contacts	Number of contacts, other associated services (e.g. practice nurse) and out-of-pocket costs
Specialist contacts	Type of specialist, number of contacts and out-of-pocket costs
Private therapy	Type of therapy, number of contacts and out-of-pocket costs
Diagnostic tests	Type of tests, number of tests and out-of-pocket costs
Data collection to be considered for studies using a societal perspective
Employment/volunteer work	Type of work and hours, income and change since stroke
Carer employment/volunteer work	Type of work and hours, income and change since stroke
Household productivity	Type of activity and hours and change since stroke (e.g. cleaning, cooking, gardening, caring for family members)
Leisure time	Type of activity and hours and change since stroke
Additional items to consider
Clinical assessments at baseline	These should be clinical assessments that can be used to estimate costs (e.g. modified Rankin Scale)
Clinical outcomes after treatment	These should be clinical assessments that can be used to estimate costs (e.g. modified Rankin Scale)

^aCollecting data on the utilisation of health services can be labour intensive. Limiting data collection to certain categories of resource use and types of resources should be justified.

Discussion

In this article, we have presented a consensus-based protocol template and a guidance document for the collection of resource use data that can be used for economic evaluations of stroke therapies internationally. We recommend using these tools in addition to the generic guidelines for conducting and reporting economic evaluations. To support use in practice, two examples of economic evaluations of stroke therapies that have been summarised using our protocol template have been provided in the online supplement and Supplemental Table I.

In addition to the resources we have developed to improve the quality of economic evaluations of stroke interventions, participants supported the collection of information that will enable comparison of studies. Providing information on case mix and stroke severity of participants (e.g. the NIHSS scores) would also assist with comparisons between economic evaluations. Researchers should also consider reporting information on the structure of healthcare systems and hospitals, particularly for multi-country studies. In intensive care studies, the Therapeutic Intervention Scoring System (e.g. TISS-28) is recommended as a way to standardise costs between countries.¹⁹ A similar tool for stroke care could be developed for stroke as an extension of our current work. In the A Very Early Rehabilitation Trial (AVERT), the data collection instruments were tailored to different study centres in Australia, Asia and the United Kingdom.²⁰ Reporting cost base years, currencies, inflation indices/rates and currency exchange indices/rates is recommended in generic guidelines for reporting economic evaluations. Adhering to this recommendation would permit researchers to assess the comparability and generalisability of economic evaluations across settings.

Standardising the time horizons and perspectives would also assist with comparison of results. When conducting studies using long-term time horizons, results for shorter-term time horizons could also be reported. The perspectives of studies are often limited to direct inpatient costs (hospital or rehabilitation) although post-hospital healthcare (community) and societal costs (broader than just the health sector impacts) are as important. The costs of long-term care and support are also important, especially in evaluations of interventions that affect disability after stroke. For example, early after stroke, hospitalisation and rehabilitation would be considered as essential since these are major contributors to costs in the first year after stroke.^21,22 In the longer term, there is evidence that residential aged care facilities and informal care comprise the majority of costs.^23,24 Therefore, we recommend researchers report the type of costs incurred (e.g. hospital, community health services, gains/losses to productivity) and the time point at which these costs were incurred. For longer term economic evaluations societal costs must be captured to have meaningful results. Effects on household productivity (e.g. cooking, cleaning, gardening and caring for family members) may also be considerable in older cohorts or for women,²⁵ but this is typically overlooked in health technology assessments or economic evaluations. Reporting informal care quantities, valuation approaches and costs are recommended to enable alternative valuations to be estimated, if necessary. In other fields, questionnaires that can be used to collect indirect costs (carers’ time and indirect consequences on carers’ health) have been validated.²⁶

Participants acknowledged that a comprehensive economic evaluation may require multiple overlapping data collection methods to be used. This could include direct measurement of healthcare resource use from registries or hospital billing systems to allow for standardised capture of all care provided, a detailed evaluation of workforce time spent with the patient to add accuracy to measuring a specific care component, and the administration of patient/caregiver questionnaires to collect data on informal care, lost productivity and any health or social care resource use (societal costs) which are not available from routine records. The possibility of data linkage for the purposes of economic evaluations should be explored given there are inaccuracies with self-reported data and the potential for recall bias especially when there are long delays between follow-up assessments.²² Ideally, data collected to estimate costs should be traceable to routinely collected information in registries and administrative databases. Information about resource use prior to stroke can also be obtained through data linkage in order to quantify stroke-specific costs (i.e. increase in resource use after stroke). In addition, data linkages with clinical quality registries can make it possible to obtain patient reported outcomes at routine follow-up assessments. For example, by linking administrative or study specific data to the Australian Stroke Clinical Registry the mRS and health-related quality of life using the EuroQol-5 dimension-3 level questionnaire collected between 90 and 180 days after stroke would be available.²⁷ However, the time delays in obtaining linked data and the complexity to analyse these data needs careful consideration when planning studies.²⁸

The participants recognised the potential value of having data repositories or directories of data custodians and existing protocols, datasets, questionnaires and models that might be shared and adapted. Once available, having access to these resources will expedite economic evaluations of stroke therapies and facilitate comparability between studies. Processes to seek permission to access these resources, in compliance with relevant information governance legislation and frameworks, remain to be developed. However, improved accessibility is likely to emerge over time from wider movements towards open access to research data.

Our process for achieving the outcomes of this work in seeking to improve economic evaluations undertaken within the field of stroke may be an exemplar for other speciality fields within health. We acknowledge that in the final review we took a pragmatic approach to finalise the outstanding decisions within the executive committee, and this may be considered a limitation of the consensus process.

Summary

The ESO Health Economic Working Group aims to standardise and improve the methods of health-economic evaluations of stroke therapies. The resources that were developed and presented in this paper will facilitate these aims and ultimately contribute to the development of evidence-based clinical guidelines to improve patient care.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: DAC has received funding from the National Health and Medical Research Institute (SRF #1154273), and educational grants unrelated to this work from Boehringer Ingelheim, Shire, Medtronic, Allergan, Pfizer paid to her institution. AP has received funding for stroke research from the National Institute for Health Research (UK) and the Stroke Association (UK). MA the NMAHP Research Unit is funded by the Chief Scientist Office, (CSO) Scottish Government’s Health and Social Care Directorate, Scotland. The views expressed here are those of the authors and not necessarily those of the CSO. JLS is an employee of the University of California, which has patent rights in retrieval devices for stroke; has received contracted hourly payments for services as a scientific consultant advising on rigorous trial design and conduct to Medtronic, Stryker, Johnson and Johnson, BrainsGate, Boehringer Ingelheim (prevention only), Diffusion Medical and Abbott; has received contracted stock options for services as a scientific consultant advising on rigorous trial design and conduct to Rapid Medical. HC has received speaker honoraria, travel expenses and/or consulting fees from Boehringer-Ingelheim, MSD, Bayer, and Medtronic. MC works for Medtronic International, neurovascular division. OW has received honorarium and/or consultancy fees from Bayer and Lupin, as well as research funding from Novo Nordisk. PM has received payments for consultancy and/or educational work from Medtronic. IDZ has received speaker honoraria, travel expenses and/or consulting fees from Boehringer-Ingelheim, Sanofi, MSD, BMS, Abbvie, Medtronic. GAF has received payments for consultancy and/or educational work from Amgen, Daiichi Sankyo, Medtronic, Pfizer and Stryker. He is a National Institute of Health Research Senior Investigator. NM is an employee of Nestle Health Science, Nestec S. A. PMB has received consulting fees, speaker honoraria and/or travel expenses from DiaMedica, Nestle, Moleac, Phagenesis, ReNeuron, Sanofi. He is Stroke Association Professor of Stroke Medicine and is a National Institute of Health Research Senior Investigator. KSS has performed studies for the Swedish national board of health and welfare. Has received speaker honoraria from Allergan. AMe has received speaker honoraria, travel expenses, and/or consulting fees from Boehringer-Ingelheim, Stryker, MSD, Nestec and Phagenesis. VT has received speaker honoraria, travel expenses and consulting fees from Bayer, Boehringer Ingelheim, Amgen, Shire and Pfizer. CW has received honoraria for advisory boards or lectures from Alexion, Amgen, Bayer-Schering as well as research funding from Boehringer Ingelheim. KRL has received fees and expenses from ACI Clinical, American Heart Association, Boehringer Ingelheim, EVER NeuroPharma, Hilicon and Parexel.

Informed consent

This project was pragmatic and relied on the implied consent of the experts who are the contributing authors as part of the consensus-based processes we undertook. Therefore, we did not seek ethical approval for this work.

Ethical approval

We did not seek ethical approval for this work. This is a negligible risk project that relied on the implied consent of the participants and ESO Economic Evaluation Working Group members contributing their opinions as part of the consensus processes used.

Guarantor

Dominique Cadilhac.

Contributorship

Dominique Cadilhac, Katharina Sunnerhagen, Gary Ford, Jeffrey Saver, Christian Weimar, Anita Patel, Hanne Christensen, Vincent Thijs, Anna Ranta and Kennedy Lees conceived the study. All members of the authorship group designed the survey and/or contributed to the development outputs for this article. Dominique Cadilhac, Alastair Wilson and Joosup Kim were involved disseminating the survey and preparing meetings where outputs for the article were developed. Dominique Cadilhac and Joosup Kim wrote the first draft of the article. All authors reviewed and edited the article and approved the final version of the article.

ORCID iDs

Dominique Cadilhac https://orcid.org/0000-0001-8162-682X

Joosup Kim https://orcid.org/0000-0002-4079-0428

Peter McMeekin https://orcid.org/0000-0003-0946-7224

Isabelle Durand-Zaleski https://orcid.org/0000-0002-4078-1476

Azmil Abdul-Rahim https://orcid.org/0000-0002-1318-4027

Supplemental Material

Supplementary material for this article is available online.