Skip to main content
PMC home page
Internet Interventions logoLink to Internet Interventions
. 2018 Feb 27;16:12–19. doi: 10.1016/j.invent.2018.02.009

Assessing the costs and cost-effectiveness of ICare internet-based interventions (protocol)

Jennifer Beecham a,b,⁎, Eva-Maria Bonin a, Dennis Görlich c, Rosa Baños d,e, Ina Beintner f, Claudia Buntrock g, Felix Bolinski h,i, Cristina Botella e,j, David Daniel Ebert g, Rocio Herrero j, Rachel Potterton k, Heleen Riper h,i, Ulrike Schmidt k, Karin Waldherr l, Kiona Weisel g, Anna-Carlotta Zarski g,m, Michael Zeiler n, Corinna Jacobi f
PMCID: PMC6364355  PMID: 30775260

Abstract

Background

Mental health problems are common and place a burden on the individual as well as on societal resources. Despite the existence of evidence-based treatments, access to treatment is often prevented or delayed due to insufficient health care resources. Effective internet-based self-help interventions have the potential to reduce the risk for mental health problems, to successfully bridge waiting time for face-to-face treatment and to address inequities in access. However, little is known about the cost-effectiveness of such interventions. This paper describes the study protocol for the economic evaluation of the studies that form the ICare programme of internet-based interventions for the prevention and treatment of a range of mental health problems.

Methods

An overarching work package within the ICare programme was developed to assess the cost-effectiveness of the internet-based interventions alongside the clinical trials. There are two underlying tasks in the ICare economic evaluation. First, to develop schedules that generate equivalent and comparable information on use of services and supports across seven countries taking part in clinical trials of different interventions and second, to estimate unit costs for each service and support used. From these data the cost per person will be estimated by multiplying each participant's use of each service by the unit cost for that service. Additionally, productivity losses will be estimated. This individual level of cost data matches the level of outcome data used in the clinical trials. Following the analyses of service use and costs data, joint analysis of costs and outcomes will be undertaken to provide findings on the relative cost-effectiveness of the interventions, taking both a public sector and a societal perspective. These analyses use a well-established framework, the Production of Welfare approach, and standard methods and techniques underpinned by economic theory.

Discussion/conclusion

Existing research tends to support the effectiveness of internet-based interventions, but there is little information on their cost-effectiveness compared to ‘treatment as usual’. The economic evaluation of ICare interventions will add considerably to this evidence base.

Keywords: Cost-effectiveness, Cost-utility, Service use, Economic evaluation, Internet-based interventions, Mental health

1. Introduction

The ICare programme is both new and wide ranging, aiming to implement a common technology platform to deliver mental health care across different EU countries. It has the potential to increase the availability of internet-based interventions, and the accompanying evaluations will provide an important addition to the evidence base on how well this delivery mode works and whether the interventions are cost-effective, and under what conditions. This paper describes the economic evaluations undertaken alongside the clinical trials that form an important part of ICare activities.

Economic evaluation provides a means of assessing the costs of an intervention and analysing costs and effectiveness data together to establish whether one intervention is cost-effective relative to its comparator(s). The discipline of economics is predicated on the fact that there are never enough resources to meet all needs and wants. The current financial situation for health care suggests that budgets are becoming even tighter (OECD/EU, 2016). Within constrained health care budgets decision-makers must make choices about the best way of spending their limited resources to improve population health outcomes. Commissioners and providers must take into account a range of information as they make these decisions; among others local circumstances, clinical preferences, residents' needs, and historical spend. By applying the discipline of economics to the topic of health and health care, findings from economic evaluations can further inform these decisions.

There are many reasons why internet-based interventions may be helpful in the context of a comprehensive (mental) health care system. One review found that cost was a key reason for developing such services (Griffiths et al., 2006). Some implementers were interested in reducing treatment costs to users of health care. This is particularly important in health care systems where supporting people who live in rural areas, or where there are low levels of provision, are key issues. Long travel distances, inconvenient appointment times for workers or those with family commitments can impose user costs and be strong barriers to accessing treatment – in addition to direct monetary costs associated with accessing health care in systems where it is not free at the point of demand. Where implementers took a health service perspective, key issues were found to be around treating more people with the same resources (for example, increasing the number of people a therapist could treat) and in ensuring speedier treatment. Reaching a wider population with treatments, improving access to treatments, reducing health inequalities and promoting a shift to interventions that may prevent problems developing or prevent problems from getting worse, are important aspects of European health care policy (European Commission, 2012).

1.1. What we know

Other papers in this Special Issue point to a growing body of evidence on the effectiveness of internet based interventions. However, little is known about their associated costs or cost-effectiveness. Four recent reviews identify cost-effectiveness evaluations occurring alongside randomised trials, Hedman et al. (2012), Arnberg et al. (2014), Donker et al. (2015) and Paganini et al. (2018). Tate et al. (2009) also reviewed literature on the cost-effectiveness of internet interventions but used wider methods criteria for their search. Of the eight studies they identified (published 1995–2008) two did not use a monetary valuation, and four were cost savings analyses or return on investment studies. Only two reported incremental cost-effectiveness findings (McConnon et al., 2007; Mihalopoulos et al., 2005).

Two reviews focused on interventions for mental health (Arnberg et al., 2014; Donker et al., 2015). Literature searches up to March 2013 revealed five cost-effectiveness studies of internet-based interventions for mood and anxiety disorders (139 papers screened), of which three were excluded due to a high risk of bias (Arnberg et al., 2014). The remaining two papers were rated as having a moderate risk of bias (Hedman et al., 2011; Hollinghurst et al., 2010). Overall the authors rated the quality of the evidence on cost-effectiveness as low to very low.

Donker et al. (2015) found 16 economic evaluations occurring alongside randomised controlled trials (RCTs) of internet interventions for any mental health disorders (236 papers screened). They included studies of interventions for depression (4 studies), smoking (3), social phobia (3 studies on two trials), harmful alcohol use (2), panic disorder (1), health anxiety (1), anxiety (1), and substance abuse (1). Given the dispersed conditions and the different types of comparison groups, the analysis was mainly descriptive but the authors suggest that overall, guided internet interventions for mental health appear to be more cost-effective than the comparator. However, unguided internet interventions were found not to be more cost-effective than their comparator. The perspective taken (social, health care providers and third-party payers) did not appear to be related to decisions around cost-effectiveness. Based on the ‘Drummond Checklist’ (Drummond et al., 1996), the quality of the economic evaluations was found to be variable but 10 studies were considered of good quality as they adhered to at least 75% of the guideline criteria.

Most of the papers reviewed by Donker et al. (2015) evaluated an internet-based cognitive behavioural therapy (ICBT) so there is an overlap of six studies with the earlier review by Hedman et al. (2012) which only focussed on ICBT interventions. Hedman et al. (2012) identified two further papers of ICBT for irritable bowel syndrome.

Most recently, Paganini et al. (2018) identified seven economic evaluations of internet- and mobile-based interventions for major depression, four for less severe depression, and one preventative intervention. Of these, six interventions were considered cost-effective, with cost-utility ratios ranging from €3088-€22,609. The authors note that all of these interventions were guided, supporting earlier evidence that adherence to web-based interventions is positively linked to increased interaction with counsellors and that other 'persuasive technology elements' increase adherence (Kelders et al., 2012), and positive association between the amount of therapist input and improvement in outcome (Palmqvist et al., 2007).

An individual-level meta-analysis of guided internet-based interventions for depression (Kolovos et al., 2018) found no statistically significant differences between intervention participants and controls in terms of costs or outcomes at 12 months, and therefore a low probability that the intervention would be considered cost-effective.

1.2. What this paper adds

Mental health problems are expensive to treat, and can have costly impacts on the health and social care sector and to wider society in adulthood (Gustavsson et al., 2011). The overarching aim of this part of the ICare programme is to increase availability of information about the costs of treating people with internet-based interventions for policy-makers, insurance companies and providers, and to add to the evidence base for the cost-effectiveness of such interventions.

More specifically, our objectives are to use data from the clinical trials to

  • •

    Identify the costs of treating people with the new technology-based interventions compared to more traditional delivery methods;

  • •

    Undertake economic evaluations of the ICare internet-based interventions compared to care/treatment as usual (C/TAU; including ‘prevention as usual’) or other comparison groups within, and where feasible across, the randomised controlled trials.

Rather than consider each clinical trial separately, this paper identifies the overarching approach to the economic evaluation of the ICare interventions. Each of the seven trials is briefly described but fuller details can be found in other articles in this Special Issue. The principles underlying economic evaluation are described first, including different types of economic evaluation available (see also, Drummond et al., 2015; Knapp, 1995). The subsequent section details the methods and materials used within the ICare programme to apply these evaluative principles to the ICare clinical studies. In the Discussion section, we consider some of the challenges and limitations of undertaking economic evaluation in this cross-national context.

2. Principles of economic evaluation

2.1. Economic costing

2.1.1. Perspective of analysis

When analysing the economic impact of social policies, a key consideration concerns the perspective adopted for the analysis (Drummond et al., 2015). Deciding on a perspective in this context means deciding whose views need to be considered when it comes to decision making about investment of public funds. A common distinction is between public sector costs, focussing on costs accruing to public sector budgets, and societal or social costs, which include the perspective of everyone who bears a ‘cost’ as a result of illness, such as the individual affected, their carers and families, and wider society such as employers.

In the economic evaluation of ICare, we consider both a public-sector perspective – focussing on health and social care costs – and a societal perspective that incorporates productivity losses.

2.1.2. Types of economic costs

Economic costs fall into three categories: direct, indirect and human costs. While direct costs are tangible, monetary costs, that can be determined by observing immediate expenditure on health services, social care and other services, indirect costs are opportunity costs that represent societal output forgone as a consequence of a disease or illness, and measured in terms of lost productivity. Human costs are wider impacts associated with premature mortality or pain and suffering. Each is discussed in more detail below.

2.1.2.1. Direct costs

To determine direct costs, a unit cost for each service is multiplied by the amount of service received by the population under study. Thus, estimating unit costs of interventions and other services and supports received by participants is a central task. Beecham (2000, pp. 12–15) sets out the principles of unit costing, reflecting economic theory:

  • •

    Unit costs should be inclusive, i.e. include resources needed to provide all components of a service. This includes both fixed and variable costs, or put differently: salary costs, on-costs and overheads, regardless of the budget or source providing these resources.

  • •

    Unit costs should be developed in such a way that they match service use and allow for accurate costing of services received. For example, an outpatient service may provide a variety of sessions, ranging from brief visits with a nurse practitioner to one-to-one therapy sessions with a consultant. These are very different in terms of resource implications, and providing an overall unit cost for a generic intervention with this service would not reflect actual resources received by the client.

  • •

    Unit costs should be based on the principle of long-run marginal opportunity costs. Economic costs include both ‘accounting costs’ – costs that might, for example, be reflected in a public sector budget – and ‘opportunity costs’ – the benefit forgone from not investing in the next best alternative. In other words, not only the immediate monetary expense needs to be considered, but also the value of the benefit forgone by choosing one option over another ('opportunity cost'). ‘Marginal cost’ refers to the cost of supporting an additional patient in the service, whereas the focus on ‘long-run’ costs such as capital investment in buildings highlights the need to consider the implications of service expansion.

Beecham and Knapp (2001) highlight another important factor: Estimating costs in a manner that recognises the variations in costs between different service users, different facilities or different locations. The implication for the economic evaluation of ICare interventions is that country-specific unit costs need to be developed.

We can further identify two broad approaches to estimating unit costs.

  • •

    In the ‘top-down’ approach, all relevant expenditure is added and then divided by the corresponding unit of activity. This approach is comparatively simple to apply, often using routinely-collected data, and is most appropriate where an average cost – such as the average cost per person receiving treatment through an eating disorder service – is required. However, this approach does not allow analysis of variation in costs, for example for patients requiring additional support beyond a standard intervention or variation by patient characteristics.

  • •

    In contrast, in the bottom-up approach all resources required to provide a specific intervention or service are described and costed. The monetary value of those resources is then linked to the service-specific unit of activity. This approach tends to be more accurate and versatile, as it can be linked to individuals, thus retaining variability between patients and between sites.

These principles identified above guide our cost estimations for the economic evaluation of ICare interventions. The heterogeneity of the ICare interventions, the users and their needs, as well as the locations means that the bottom-up approach is best suited to estimating intervention costs and the costs associated with individual-level service use data collected within the clinical trials.

2.1.2.2. Indirect costs: productivity losses

Productivity is based on the theoretical model of the production function, where output is produced by combining capital, labour and technology (inputs). Productivity is a measure of output per unit of input and therefore productivity loss is the value of lost output (Zhang et al., 2011).

Absenteeism is a measure of reduction in output due to days not worked (work days lost). In the ICare trials absenteeism may be due to acute illness or hospitalisation and medical appointments. Presenteeism also reflects a reduction in productivity, but occurs while a person is at work. It represents the difference in output in the presence of a condition compared to output in the absence of the condition. Given the complexity of many modern jobs, especially in non-manual, non-industrial settings where outputs cannot be easily defined nor counted, and the complexity of an impairment resulting from a mental health problem, assessments of presenteeism generally rely on self-report measures rather than routinely-collected data (Schultz and Edington, 2007).

Productivity losses from absenteeism and presenteeism are often valued using a human capital approach. In short, the human capital method places a value on lost output by calculating the sum of discounted expected future income. The theoretical justification for this approach is that the market wage is equal to the marginal product of labour in a competitive market (Zhang et al., 2011), and therefore represents the opportunity cost of lost output to society.

One criticism of the human capital approach is that it disregards unpaid work, such as housework and caregiving, and leisure activities. To determine a ‘shadow price’ for these activities, the opportunity cost of lost time spent on unpaid work needs to be developed. Several options have been proposed for values based on opportunity costs (for an accessible summary, see Francis and McDaid, 2009): market wage forgone based on an individual's likely earnings; average wage; or minimum wage. An alternative is to use a ‘replacement cost’. This approach values the (lost) output produced as the cost of purchasing an equivalent service on the market such as the cost of child care or the cost of hiring a nurse or caregiver. Valuing leisure time presents a number of additional challenges, usually requiring the use of proxy measures to achieve a valuation. For example, the UK Department for Transport has calculated an opportunity cost for leisure time based on travel time avoided (Department for Transport, 2009).

2.2. Economic evaluation

2.2.1. Theoretical background: production of welfare approach

Underpinning each evaluative mode must be a clear understanding of how costs and outcomes can be linked. The Production of Welfare (PoW) approach provides this framework (Knapp, 2001). Based in traditional micro-economics, its predecessor is the classic theory of production, or the assumption that inputs (labour and capital) are combined to produce outputs (cars, for example). In a person-to-person service such as health care, these relationships are more complex as people form the basis of both inputs and outputs so their attributes have a large effect on the quantity and quality of outcomes.

The four main PoW elements are inter-related. Resource inputs are the labour and capital used to create a service, which can be summarised in monetary terms as cost. Service outputs (intermediate outcomes) are the levels or volume of services produced or used. Non-resource inputs are less easy to measure but can help explain outcome differences. They include the social features of the care environment and the characteristics, experiences, personalities and attitudes of the main actors (staff and users) in the system. Non-resource inputs, therefore, include potential mediators and moderators of outcomes (see Knapp, 2001). In health care, the aim is not to produce services but to produce better health and welfare; the fourth PoW component is final outcomes – changes in people's health and welfare.

Thus, the Production of Welfare framework can help structure research, explain, justify and clarify reasons why data are collected and analyses undertaken, and can help interpret results sensibly. Such frameworks are particularly important when undertaking analyses on a new topic. The model suggests a causal link between resources inputs (summarised by costs) and final outcomes, mediated by different combinations of non-resources inputs and service outputs. By clearly locating measures used to assess ICare interventions within the PoW model, the specific links – and their strength – between outcomes, costs and savings to the health care system will be identified.

2.2.2. Types of economic evaluation

According to Drummond et al. (2015), economic evaluation requires

  • •

    A comprehensive assessment of all costs and savings from the intervention;

  • •

    Assessment of outcomes;

  • •

    A relevant metric to allow comparison of one or more alternatives.

Where one or several of these elements are missing, a partial economic evaluation can be provided. A cost of illness study, for example is a partial economic evaluation as only cost data are available. So too is an analysis of efficacy, as this assesses outcomes using a comparative design, but does not incorporate any cost data.

The following are examples of full economic evaluation types that will be applied in the ICare programme:

  • 1)

    Cost-consequence analyses set out the costs to the stakeholders alongside outcomes achieved. Such analyses are useful when an intervention has several desirable outcomes; perhaps the primary clinical outcome but also factors such as quality of life or satisfaction with the intervention.

  • 2)

    Cost-effectiveness analyses (CEA) combine costs with a clinical outcome measure or other ‘natural unit’ and compare the cost-per-outcome gained for each option. For example, if two options are of equal cost, which provides greater benefit to the patient or population?

  • 3)

    Often considered a special case of CEA, the outcome used in cost-utility analysis (CUA) is health-related quality of life, which is valued in line with population preferences (utility). Commonly a cost per quality adjusted life year (QALY) gained is calculated. QALYs are a highly conflated measure and not fully relevant for all conditions, but their widespread use means comparisons can be made across conditions.

These approaches should be distinguished from (and are often confused with) cost-benefit analysis, which values both outcomes and costs in the same metric, usually in monetary terms.

3. Materials and methods

3.1. Participants and interventions

The participants for the ICare economic evaluations are those recruited to the individual ICare clinical trials. Box 1 provides a brief overview of the interventions, study design, and target recruitment numbers for the intervention and control groups and more information can be found in other papers in this Special Issue. Participants providing data on outcome measures, demographics and service use at baseline will be included in the main analysis. Sensitivity analysis will be performed using complete-case analysis.

Box 1. Trials and participants.

The ICare trials are described in more detail in other papers in this issue and only brief details are given here.

Study 1 aims to bridge waiting time for people waiting for outpatient treatment in Germany and the UK. Two hundred and seventy-five women will be randomly allocated to the ‘everyBody Plus’ intervention or to a waiting list control group.

Study 2 uses a tailored prevention intervention to help women with negative body image and symptoms of disordered eating. It is a non-randomised parallel intervention study, with 4,160 participants from the general population in German-speaking countries directed to one of five variants of the ‘everyBody’ intervention.

Study 3 evaluates an unguided web-based programme (CORE) to promote resilience and coping skills, decrease symptoms of depression and anxiety, and increase overall wellbeing in young people. This RCT will compare CORE with care as usual among a minimum of 454 university students in Spain, Germany and Switzerland.

Study 4 evaluates a prevention programme for common mental health disorders across 954 participants with sub-clinical symptoms of depression or anxiety per group, recruited from Germany, Switzerland, Spain, and the Netherlands. The RCT will compare two active treatments (guided and unguided ICare Prevent) and treatment as usual (TAU). Participants allocated to the TAU arm will receive the intervention after 12 months.

Study 5 aims to evaluate an intervention to promote a healthy lifestyle and to reduce problematic eating behaviour, eating disorder and obesity risk among students aged 14 to 19 years old. An RCT, clustered by school, will recruit at least 430 young people to participate in the intervention programmes (Weight Management and Healthy Habits) or a no-intervention control group, from school settings in Austria and Spain. Participants allocated to the control group arm will receive the intervention after 12 months.

Study 6 evaluates the web-based PLUS trans-diagnostic mental health problem prevention programme using a parallel group RCT across universities in the UK, the Republic of Ireland, Austria, and Germany. Students (N = 1,100) at high risk of developing common mental disorders will be randomly allocated to either PLUS or an intervention providing practical support for issues commonly experienced at University.

Study 7 is an RCT evaluating three forms of the We Can web-based intervention for carers of individuals with anorexia nervosa; with clinician email support (We Can-Ind), with moderated carer chatroom support (We Can-Chat), and with online forum only (We Can-Forum). The study will recruit 303 carers (and, where possible, the care recipient) through specialist eating disorder services and carer support services in the UK and Germany.

Alt-text: Box 1

3.2. Assessment and data management

Data management and monitoring will be provided by the Westfälische Wilhelms-Universität Münster, Institute of Biostatistics and Clinical Research. Support is provided for the whole consortium in order to maintain comparable high quality in the conduct of the ICare research projects in trial planning, data management, online monitoring, and analysis.

Within the ICare project a harmonised data management plan is implemented to provide high quality data with respect to accuracy, composition and organisation, completeness, transparency of processes, and timeliness. During the active phase of the trial data (i) completeness, (ii) timeliness and (iii) internal validity will be monitored. Internal validity will be checked by plausibility rules. Data will be collected on the Minddistrict platform. After export from the platform data will be processed in a unified manner for all ICare studies, using programming scripts.

Data security and confidentiality will be ensured; all relevant EU legislation and international guidelines on privacy will be observed and respected. Regarding regulation at international level, starting from the OECD guidelines including the “Guidelines on the protection of privacy and trans-border flow of personal data” (1981) and “Guidelines for the security of information systems” (1991/92), the ICare consortium in particular acknowledges heterogeneity in international data protection jurisdiction.

3.3. Measures for economic evaluation

For ICare economic evaluations there are three data collection tasks over and above data associated with characteristics of participants or their needs and outcomes. These relate to the way resources are used to deliver the ICare interventions, and to the services and supports study participants received from other parts of the health and social care system. Information to calculate productivity losses also needs to be collected. There is an addition layer of work around harmonizing the service use data to read across different countries' health care systems and service arrays.

3.3.1. Service use

A specially adapted version of the Client Service Receipt Inventory (CSRI; Beecham and Knapp, 2001) was developed to record use of services and medication at baseline and at the six-month and 12-month follow-up, covering a retrospective six-month period. Participants will be asked to provide the number of contacts with primary care, hospital services, specialist mental health services, social work and other community services.

From the original English version, the CSRI was translated for each country and language (with minor adjustments to wording to reflect the clinical focus of each individual trial) with the help of the local ICare research teams. The translation focussed not on a word for word translation, but rather on extracting equivalent information that appropriately reflects each country's health and social care system, and that uses language appropriate to the country context. In addition, for each country, questions and response options were structured so that relevant unit costs can be estimated and applied.

3.3.2. Productivity losses

A measure of productivity loss and impairment of other activities was incorporated into the CSRI, based on an adaptation of the Work Productivity and Activity Impairment Questionnaire – General Health (WPAI-GH; Reilly et al., 1993). This questionnaire was translated into the target languages with small modifications in the wording to reflect the clinical focus of each trial.

3.3.3. Outcome measures

Full details of primary outcome measures can be found in the protocol paper of each individual clinical study. In addition to the primary outcome, there are three outcome measures used in multiple trials that, where appropriate, will be used for cost-effectiveness analysis across trials.

  • •

    Severity of depression will be assessed using the Patient Health Questionnaire (PHQ-9, Kroenke et al., 2001). This self-report measure includes nine items describing depressive symptoms. PHQ-9 will only be assessed in adult populations.

  • •

    Severity of anxiety will be assessed using the Generalised Anxiety Disorder scale (GAD-7, Spitzer et al., 2006). It consists of seven items relating to symptoms of generalised anxiety. GAD-7 will only be assessed in adult populations.

  • •

    The Eating Disorder Examination – Questionnaire (EDE-Q, Fairburn and Beglin, 1994) is a self-report version of the well-established interview version of the EDE (Fairburn and Cooper, 1993), consisting of a series of questions regarding eating disordered behaviours, and concerns over shape and weight. EDE-Q will only be collected for interventions targeting disordered eating behaviours.

In addition to these outcome measures, we will explore the potential of using measures of quality of life (QoL) to conduct cost-utility analysis. Two QoL measures are used as part of different ICare trials:

  • •

    The WHOQOL-BREF (The Whoqol Group, 1998) is a self-report measure that assesses quality of life using 26 items on the domains of physical health, psychological health, social relationships, and environment.

  • •

    The Assessment of Quality of Life-8D (AQOL-8D, Richardson et al., 2014) is a 35 item instrument for the assessment of quality of life and comprises eight dimensions – independent living, pain, senses, relationships, mental health, happiness, coping, and self-worth.

  • •

    For trials employing the AQOL-8D, we will calculate quality-adjusted life years using previously developed algorithms (Dakin, 2013).

3.4. Unit costs

There is some debate in the literature about the scope of data to collect around the costs of internet-based interventions. McNamee et al. (2016) discuss this at some length reminding the reader that most costs are incurred at the development stage; their review found costs ranging from £20,000 to £500,000 (p. 856) and that maintenance costs (such as hosting costs) can be very low. Other costs specific to internet interventions may be for regular updating to ensure the intervention ‘remains the same’ (e.g. providing up-to-date information, or for software development), or where evolution over time occurs and is desirable and/or planned (McNamee et al., 2016). In the short-term evaluations that form part of the ICare programme, we exclude these ‘sunk costs’ and include only costs that will have to be incurred should the intervention be used again (see also, Donker et al., 2015).

The main recurring resource for ICare interventions are therapists and moderators providing moderated forums, weekly chats and 1-to-1 guidance through the Minddistrict online platform. Information on salaries, oncosts and overheads will be collected from intervention providers, and a cost of staff calculated in line with the theoretical approach outlined above (Beecham, 2000).

A Service Information Schedule (SIS) will be developed to collect information on salaries, on-costs and any relevant overheads associated with the therapists and moderators. This will allow us to calculate the cost associated with staff time required to provide the interventions. The cost of general resources (moderated forums and chats) will be divided by the number of registered participants to provide a per-participant cost. In addition, moderators will be asked to provide information on time spent per typical week moderating forums and moderating weekly chats to capture resources available to all intervention participants, and time spent on providing individual feedback and sending individual reminders designed to improve adherence. The cost associated with individual feedback will be calculated for each participant to capture individual-level variations in the intensity of support. To balance the need for accurate data with the need to reduce the burden on providers of information, this will be achieved by obtaining data on a) the average time it takes therapists to provide feedback and b) identifying which individual participants required time input that is higher than the average.

The second set of tasks related to unit cost estimations is to place a monetary value on the resources (services and supports) used by trial participants. Informed by the principles set out above, unit costs will be estimated for each trial drawing on national compilations (see for example Curtis, 2016; Department of Health, 2016; Hakkaart-van Roijen et al., 2016; Krauth et al., 2005; Bock et al., 2015), other local or European data, and other research.

3.5. Analyses

3.5.1. Service use and costs

Service use will be described at each time point, by intervention group, showing the number and proportion of participants reporting use of any given service. The unit cost for each service or support will be multiplied by the amount of service use each person has recorded on the CSRI for each time period. To ensure that costs are equivalent across countries, they will be expressed in Euro converted to a constant unit using each countries consumer price index and purchasing power parities (PPP; http://www.oecd.org/std/prices-ppp/) for total consumption, i.e. gross domestic product (GDP; Gustavsson et al., 2011). Means and standard deviations for costs by category (e.g. primary care, specialist mental health care, total costs) will be presented by treatment group for each time point.

Productivity losses will be calculated using a human capital approach. Loss from absence from paid employment (absenteeism) will be calculated as wages forgone, either based on self-reported wage or median earnings for a participant's occupation (where available). Losses associated with reduced productivity while at work (presenteeism) will be estimated by multiplying the daily wage (derived as above) by the average percentage reduction in productivity due to depression, sadness or mental illness (15.3%, Goetzel et al., 2004). Opportunity costs from the impact on daily routines will be valued using the relevant minimum wage or median wage for the lowest skilled group in the labour market where no minimum wage is in place. The impact on leisure time will be valued by first calculating the average number of hours of leisure time for all participants based on time use surveys for their respective country or – where unavailable – a country with a similar labour market structure. This will then be valued at the rate for the value of leisure time, such as the value of travel time avoided (Department for Transport, 2009). Where such a value is unavailable, the rate will be estimated as a proportion of the minimum wage corresponding to the average of countries where this information is available.

3.5.2. Cost-consequence and cost-effectiveness analyses

Cost-consequence analysis contrasts differential costs with differential outcomes, adjusted for salient socio-economic and baseline characteristics. Given that cost data are often skewed, with many participants incurring low or zero costs and a few incurring very high costs, appropriate statistical methods will be chosen after assessing their distribution. Common approaches include non-parametric bootstraps (Efron and Tibshirani, 1993), or Generalised Linear Models (GLMs) with an appropriate distribution and link function, such as a gamma distribution with a log link (e.g. Kilian et al., 2002).

Cost-effectiveness analysis will be conducted for clinical outcome variables and – where available – a measure of QoL (cost-utility analysis) for each trial. Results will be presented as incremental cost-effectiveness ratios (ICERs) with their corresponding 95% confidence intervals and cost-effectiveness acceptability curves (CEACs). All analyses will adjust for relevant participant characteristics and baseline scores. The ICER represents the additional cost for an incremental improvement in outcome when comparing the experimental (ex) to the control group (con):

ICER=Costex−CostconEffectex−Effectcon

However, summarizing this information in one figure is problematic, as a positive ratio can represent both higher costs and better outcomes in the experimental group, and lower costs and less favourable outcomes (compared to the control group). Moreover, a negative ratio can indicate higher costs coupled with worse outcomes (TAU dominates the intervention) as well as lower costs and better outcomes (intervention dominates TAU). Scatter plots of at least 1000 bootstrap replications of the ICER will therefore be presented to aid the interpretation of the ICER and the assessment of uncertainty surrounding it.

CEACs address the problems associated with the ICER by offering a way to examine the probability of the intervention being cost effective, given various values for society's willingness to pay (WTP) for a unit improvement in outcome. CEACs are often based on the concept of net monetary benefit (NMB). Net benefit variables are calculated for a range of possible values for WTP for all outcome variables by multiplying WTP by the outcome, and subtracting costs (Fenwick and Byford, 2005; Glick et al., 2007; O'Brien and Briggs, 2002). The difference in NMB between treatment groups for each value of WTP is estimated using a net benefit regression framework.

Where participant characteristics are associated with clinical outcomes and costs, respectively, or where the goal is to implement sub-group analysis, the assessment of cost-effectiveness and cost-utility of the ICare interventions will use Seemingly Unrelated Regression (Zellner, 1962). Here, separate regression models are fitted for a) costs at follow-up and b) each of the outcome measures considered in the economic evaluation as the dependent variable (Hoch et al., 2002). The goal is again to estimate group differences.

Within these analytical frameworks it is possible to control for baseline scores, baseline costs and confounding variables (Glick et al., 2007; Hoch et al., 2002). Bootstrapping will be used to obtain multiple estimates for the group difference, allowing a probabilistic interpretation of results: The percentage of estimates of the group difference greater than one – indicating better results for the experimental group – is plotted for each value of WTP, resulting in the CEAC (Byford et al., 2003).

All models will take into account clustering within countries and – where analysis across several trials is performed – within trials as appropriate. Analyses will be shown for a) public sector costs only and b) costs including public sector costs and productivity losses. Missing data will be addressed using multiple imputation (see Little and Rubin, 2002 for handling bootstrap in the context of multiple imputation). A blinded data review will inform the imputation strategy and the selection of multivariable models. To assess the effect of missing data, the analysis will be performed on the imputed data, and on complete cases.

4. Discussion and conclusion

This paper describes the activities to be undertaken in the economic evaluation of the interventions forming the ICare programme. The economic evaluations form an overarching “work-package” within the EU grant funding which covers cost and cost-effectiveness studies across seven clinical trials recruiting from six countries. There are between 250 and 4000 participants in each trial and while some programmes are targeted on school children or university students, others provide interventions for adults who have, or are at risk of developing, (mental) health problems.

Having a strong theoretical underpinning is key to the success of any research endeavour. The Production of Welfare (PoW) provides a clear framework for how costs and outcomes can be linked, derived from the classic theory of production (Knapp, 2001). Clearly delineating the relationships between resources inputs, non-resource inputs, intermediate outputs, and final outcomes helps disentangle the complexity of the relationships and can help interpret results sensibly. By locating the evaluative elements of the ICare interventions within that framework, the specific links between outcomes, costs and savings to the health care system will be identified.

There are a number of challenges. For example, each of the trials occur in at least two countries so operate across health care cultures where differences in ostensibly similar services, or combination of services, have to be acknowledged. Cross-national studies also require attention to be paid to the financing system and the monetary valuation of these services and supports. Complexity arises here not only because the similar professionals may undertake a different role and be supported by their employing organisation in a different way, but also because the relative value of goods is different in each country. The aim is to ensure equivalence.

Our tasks include the development of data collection schedules, estimation of unit costs and productivity losses, analysis of service use information and costs, as well as the joint analysis of costs and outcomes. There are few existing research studies to inform or compare the findings. Around half of the economic evaluations that have been reviewed were considered to be good quality, but often the perspective is narrow. The evidence tends to favour guided internet interventions as being more cost-effective than the selected comparator but the evidence base is much weaker when specific conditions are considered. The existing published papers consider a range of health problems but many of the interventions studied are based on just one treatment (ICBT). The ICare programme also includes interventions for people with eating disorders (including carers) and obesity, common mental health disorders (depression, anxiety, harmful or hazardous alcohol use and comorbid problems), and adjustment disorder. The interventions also include mental health promotion and prevention programmes, as well as interventions for at risk populations. While this is a strength of the ICare programme and can broaden the reach of internet intervention, it brings challenges to the evaluation by reducing comparability between studies.

Research evidence tends to support use of internet-based intervention of the grounds of greater effectiveness but is less well supported in terms of the evidence on costs and cost-effectiveness. Not only is the body of evidence on cost-effectiveness much smaller but neither does it cover the breadth of health conditions and interventions proposed by ICare. However, information on costs and cost-effectiveness are an important consideration for policy-makers and providers as they make decisions about how to spend their limited resources. This paper identifies the theory-based approach to economic evaluation that will underpin the analyses to produce major findings around the cost-effectiveness of internet interventions for treatment, prevention and health promotion.

Competing interests

The authors declare that they have no competing interests.

Funding

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 634757.

Author contributions

JB designed the economic evaluation in the first place. EMB contributed significantly to the design of the economic evaluation. All co-authors supported the adaptation of the CSRI for use in individual trials and countries. JB and EMB co-wrote the manuscript. JB, EMB, DG, IB, CB, DE, MZ, KW, RP and CJ contributed revisions to the manuscript. All authors read and approved the final manuscript.

Ethical issues

There will be no additional ethical issues linked to the economic evaluation of ICare interventions over and above those identified for the individual RCTs. Primary data will be collected within these trials or through the platform and will be anonymised before sent to the economics team.

Acknowledgements

The activities described form part of the European research project: Integrating Technology into Mental Health Care Delivery in Europe (ICare, N° 634757, www.icare-online.eu). The main objective of the ICare project is to develop an integrated model of health promotion and risk prevention as well as detection of the most prevalent disorders in Europe (i.e., depression, anxiety, adjustment disorder, eating disorders, and substance abuse), guiding people and professionals in the selection and application of online evidence-based interventions. US receives salary support from the National Institute for Health Research (NIHR) Biomedical Research Centre for Mental Health, South London and Maudsley NHS Foundation Trust and Institute of Psychiatry, Psychology and Neuroscience, King's College London. She is also supported by an NIHR Senior Investigator Award. The views expressed in this publication are those of the authors and not necessarily those of the National Health Service, the NIHR or the UK Department of Health.

References

  1. Arnberg F.K., Linton S.J., Hultcrantz M., Heintz E., Jonsson U. Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness. PLoS ONE. 2014;9(5):e98118. doi: 10.1371/journal.pone.0098118. [Electronic Resource] [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Beecham J. Department of Health, Dartington Social Research Unit & PSSRU; Dartington: 2000. Unit Costs - Not Exactly Child’s Play. A Guide to Estimating Unit Costs for Children’s Social Care.http://www.pssru.ac.uk/pdf/B062.pdf Retrieved from. [Google Scholar]
  3. Beecham Jennifer, Knapp Martin. Costing psychiatric interventions. In: Thornicroft G., editor. Measuring Mental Health Needs. Second Ed. Royal College of Psychiatrists; London: 2001. pp. 200–224. [Google Scholar]
  4. Bock J.O., Brettschneider C., Seidl H., Bowles D., Holle R., Greiner W., König H.H. Ermittlung Standardisierter Bewertungssätze Aus Gesellschaftlicher Perspektive Für Die Gesundheitsökonomische Evaluation. Gesundheitswesen. 2015;77(1):53–61. doi: 10.1055/s-0034-1374621. [DOI] [PubMed] [Google Scholar]
  5. Byford S., Knapp M., Greenshields J., Ukoumunne O.C., Jones V., Thompson S., Tyrer P., Schmidt U., Davidson K. Cost-effectiveness of brief cognitive behaviour therapy versus treatment as usual in recurrent deliberate self-harm: a decision-making approach. Psychol. Med. 2003;33(6):977–986. doi: 10.1017/s0033291703008183. [DOI] [PubMed] [Google Scholar]
  6. Curtis L. Personal Social Services Research Unit, University of Kent; Canterbury: 2016. Unit Costs of Health and Social Care 2016. [Google Scholar]
  7. Dakin Helen. Review of studies mapping from quality of life or clinical measures to EQ-5D: an online database. Health Qual. Life Outcomes. 2013;11(1):151. doi: 10.1186/1477-7525-11-151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Department for Transport . Transport Analysis Guidance. Department for Transport; London: 2009. Values of Time and Operating Costs. TAG Unit 3.5.6. [Google Scholar]
  9. Department of Health . Department of Health; London: 2016. NHS reference costs 2015–16. [Google Scholar]
  10. Donker T., Blankers M., Hedman E., Ljótsson B., Petrie K., Christensen H. Economic evaluations of internet interventions for mental health: a systematic review. Psychol. Med. 2015;45(16):3357–3376. doi: 10.1017/S0033291715001427. [DOI] [PubMed] [Google Scholar]
  11. Drummond M.F., Jefferson T.O., Gerard K., Adams M.E., McCall N.T., Gray D.T., Orza M.J. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ economic evaluation working party. BMJ (Clin. Res. Ed.) 1996;313(7052):275–283. doi: 10.1136/bmj.313.7052.275. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Drummond M., Stoddard G.L., Torrance G.W. 4th ed. University Press; Oxford: 2015. Methods for the Economic Evaluation of Health Care Programmes. [Google Scholar]
  13. Efron B., Tibshirani R.J. Monographs on Statistics and Applied Probability. vol. 57. Chapman and Hall; New York: 1993. An introduction to the bootstrap. [Google Scholar]
  14. European Commission eHealth Action Plan 2012–2020: Innovative Healthcare for the 21st Century. 2012. http://ec.europa.eu/digital-agenda/en/news/ehealth-action-plan-2012-2020-innovative-healthcare-21st-century Retrieved 26/10/2017.
  15. Fairburn Christopher G., Beglin Sarah J. Assessment of eating disorders: interview or self-report questionnaire? Int. J. Eat. Disord. 1994;16(4):363–370. [PubMed] [Google Scholar]
  16. Fairburn C., Cooper Z. Nature, Assessment and Treatment. 1993. The Eating Disorder Examination; pp. 317–360. [Google Scholar]
  17. Fenwick E., Byford S. A guide to cost-effectiveness acceptability curves. Br. J. Psychiatry. 2005;187:106–108. doi: 10.1192/bjp.187.2.106. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=16055820 [DOI] [PubMed] [Google Scholar]
  18. Francis J., McDaid D. SCIE’s work on economics and the importance of informal care. In: Curtis L., editor. Unit costs of health and social care 2009. Personal Social Services Research Unit, University of Kent; Canterbury: 2009. pp. 27–33.http://eprints.lse.ac.uk/32770/ Retrieved from. [Google Scholar]
  19. Glick H.A., Doshi J.A., Sonnad S.S., Polsky D. Oxford University Press Catalogue; 2007. Economic Evaluation in Clinical Trials. [Google Scholar]
  20. Goetzel Ron Z., Long Stacey R., Ozminkowski Ronald J., Hawkins Kevin, Wang Shaohung, Lynch Wendy. Health, absence, disability, and presenteeism cost estimates of certain physical and mental health conditions affecting U.S. employers. J. Occup. Environ. Med. 2004;46(4):398–412. doi: 10.1097/01.jom.0000121151.40413.bd. [DOI] [PubMed] [Google Scholar]
  21. Griffiths Frances, Lindenmeyer Antje, Powell John, Lowe Pam, Thorogood Margaret. Why are health care interventions delivered over the internet? A systematic review of the published literature. J. Med. Internet Res. 2006 doi: 10.2196/jmir.8.2.e10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Gustavsson Anders, Svensson Mikael, Jacobi Frank, Allgulander Christer, Alonso Jordi, Beghi Ettore, Dodel Richard. Cost of disorders of the brain in Europe 2010. Eur. Neuropsychopharmacol. 2011;21(10):718–779. doi: 10.1016/j.euroneuro.2011.08.008. [DOI] [PubMed] [Google Scholar]
  23. Hakkaart-van Roijen L., van der Linden N., Bouwmans C., Kanters T., Swan Tan S. Kostenhandleiding: Methodologie van kostenonderzoek en referentieprijzen voor economische evaluaties in de gezondheidszorg. Zorginstituut Nederland. 2016:1–73. www.zorginstituutnederland.nl/publicaties/publicatie/2016/02/29/richtlijn-voor-het-uitvoeren-van-economische-evaluaties-in-de-gezondheidszorg Retrieved from. [Google Scholar]
  24. Hedman Erik, Andersson Erik, Ljótsson Brjánn, Andersson Gerhard, Rück Christian, Lindefors Nils. Cost-effectiveness of internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: results from a randomized controlled trial. Behav. Res. Ther. 2011;49(11):729–736. doi: 10.1016/j.brat.2011.07.009. [DOI] [PubMed] [Google Scholar]
  25. Hedman Erik, Ljótsson Brjánn, Lindefors Nils. Cognitive behavior therapy via the internet: a systematic review of applications, clinical efficacy and cost–effectiveness. Expert Rev. Pharmacoecon. Outcomes Res. 2012;12(6):745–764. doi: 10.1586/erp.12.67. [DOI] [PubMed] [Google Scholar]
  26. Hoch J.S., Briggs A.H., Willan A.R. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Econ. 2002;11(5):415–430. doi: 10.1002/hec.678. [DOI] [PubMed] [Google Scholar]
  27. Hollinghurst Sandra, Peters Tim J., Kaur Surinder, Wiles Nicola, Lewis Glyn, Kessler David. Cost-effectiveness of therapist-delivered online cognitive-behavioural therapy for depression: randomised controlled trial. Br. J. Psychiatry. 2010;197(4):297–304. doi: 10.1192/bjp.bp.109.073080. [DOI] [PubMed] [Google Scholar]
  28. Kelders Saskia M., Kok Robin N., Ossebaard Hans C., Van Gemert-Pijnen Julia E.W.C. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J. Med. Internet Res. 2012 doi: 10.2196/jmir.2104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Kilian Reinhold, Matschinger Herbert, Loeffler W., Roick Christiane, Angermeyer Matthias C. A comparison of methods to handle skew distributed cost variables in the analysis of the resource consumption in schizophrenia treatment. J. Ment. Health Policy Econ. 2002;5(1):21–31. [PubMed] [Google Scholar]
  30. Knapp M. Contemporary Psychiatry. Springer Berlin Heidelberg; Berlin, Heidelberg: 2001. Economic evaluation of mental health care; pp. 259–277. [Google Scholar]
  31. Knapp M., editor. Ashgate Publishing; Farnham UK: 1995. The Economic Evaluation of Mental Health Care. [Google Scholar]
  32. Kolovos S., van Dongen J.M., Riper H., Buntrock C., Cuijpers P.…Ebert D.D.Bosmans. Cost effectiveness of guided Internet-based interventions for depression in comparison with control conditions: an individual-participant data meta-analysis. Depress. Anxiety. 2018:209–219. doi: 10.1002/da.22714. (November 2017) [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Krauth Christian, Hessel F., Hansmeier T., Wasem J., Seitz R., Schweikert B. Empirische Bewertungssätze in Der Gesundheitsökonomischen Evaluation - Ein Vorschlag Der AG Methoden Der Gesundheitsökonomischen Evaluation (AG MEG) Gesundheitswesen. 2005;67(10):736–746. doi: 10.1055/s-2005-858698. [DOI] [PubMed] [Google Scholar]
  34. Kroenke K., Spitzer R.L., Williams J.B. The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Little R.J., Rubin D.B. Wiley-Interscience; Hoboken, N: 2002. Statistical Analysis With Missing Data. [Google Scholar]
  36. McConnon Áine, Kirk Sara F.L., Cockroft Jennie E., Harvey Emma L., Greenwood Darren C., Thomas James D., Ransley Joan K., Bojke Laura. The internet for weight control in an obese sample: results of a randomised controlled trial. BMC Health Serv. Res. 2007;7(1):206. doi: 10.1186/1472-6963-7-206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. McNamee Paul, Murray Elizabeth, Kelly Michael P., Bojke Laura, Chilcott Jim, Fischer Alastair, West Robert, Yardley Lucy. Designing and undertaking a health economics study of digital health interventions. Am. J. Prev. Med. 2016;51(5):852–860. doi: 10.1016/j.amepre.2016.05.007. [DOI] [PubMed] [Google Scholar]
  38. Mihalopoulos C., Kiropoulos L., Shih S.T., Gunn J., Blashki G., Meadows G. Exploratory economic analyses of two primary care mental health projects: implications for sustainability. Med. J. Aust. 2005;183(10 Suppl):S73–6. doi: 10.5694/j.1326-5377.2005.tb07184.x. (doi:mih10602_fm [pii) [DOI] [PubMed] [Google Scholar]
  39. O'Brien B.J., Briggs A.H. Analysis of uncertainty in health care cost-effectiveness studies: an introduction to statistical issues and methods. Stat. Methods Med. Res. 2002;11(6):455–468. doi: 10.1191/0962280202sm304ra. [DOI] [PubMed] [Google Scholar]
  40. OECD/EU . OECD Publishing; Paris: 2016. Health at a Glance: Europe 2016: State of Health in the EU Cycle. [Google Scholar]
  41. Paganini S., Teigelkötter W., Buntrock C., Baumeister H. Economic evaluations of internet- and mobile-based interventions for the treatment and prevention of depression: a systematic review. J. Affect. Disord. 2018;225:733–755. doi: 10.1016/j.jad.2017.07.018. [DOI] [PubMed] [Google Scholar]
  42. Palmqvist Björn, Carlbring Per, Andersson Gerhard. Internet-delivered treatments with or without therapist input: does the therapist factor have implications for efficacy and cost? Expert Rev. Pharmacoecon. Outcomes Res. 2007;7(3):291–297. doi: 10.1586/14737167.7.3.291. [DOI] [PubMed] [Google Scholar]
  43. Reilly M.C., Zbrozek A.S., Dukes E.M. The validity and reproducibility of a work productivity and activity impairment instrument. PharmacoEconomics. 1993;4(5):353–365. doi: 10.2165/00019053-199304050-00006. [DOI] [PubMed] [Google Scholar]
  44. Richardson Jeff, Iezzi Angelo, Khan Munir A., Maxwell Aimee. Validity and reliability of the assessment of quality of life (AQoL)-8D multi-attribute utility instrument. The Patient. 2014;7(1):85–96. doi: 10.1007/s40271-013-0036-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Schultz A.B., Edington D.W. Employee health and presenteeism: a systematic review. J. Occup. Rehabil. 2007;17(3):547–579. doi: 10.1007/s10926-007-9096-x. [DOI] [PubMed] [Google Scholar]
  46. Spitzer Robert L., Kroenke Kurt, Williams Janet B.W., Löwe Bernd. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch. Intern. Med. 2006;166:1092–1097. doi: 10.1001/archinte.166.10.1092. [DOI] [PubMed] [Google Scholar]
  47. Tate Deborah F., Finkelstein Eric A., Khavjou Olga, Gustafson Alison. Cost effectiveness of internet interventions: review and recommendations. Ann. Behav. Med. 2009;38(1):40–45. doi: 10.1007/s12160-009-9131-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. The Whoqol Group Development of the world health organization WHOQOL-BREF quality of life assessment. The WHOQOL group. Psychol. Med. 1998;28:551–558. doi: 10.1017/s0033291798006667. [DOI] [PubMed] [Google Scholar]
  49. Zellner A. An efficient method of estimating seemingly unrelated regression equations and tests for aggregation bias. J. Am. Stat. Assoc. 1962;57:348–368. [Google Scholar]
  50. Zhang W., Bansback N., Anis A.H. Measuring and valuing productivity loss due to poor health: a critical review. Soc. Sci. Med. 2011;72(2):185–192. doi: 10.1016/j.socscimed.2010.10.026. [DOI] [PubMed] [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES