Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers

Abstract

Background

We developed a method to estimate the expected cost-effectiveness of a service intervention at the design stage and ‘road-tested’ the method on an intervention to improve patient handover of care between hospital and community.

Method

The development of a nine-step evaluation framework:

1. Identification of multiple endpoints and arranging them into manageable groups;

2. Estimation of baseline overall and preventable risk;

3. Bayesian elicitation of expected effectiveness of the planned intervention;

4. Assigning utilities to groups of endpoints;

5. Costing the intervention;

6. Estimating health service costs associated with preventable adverse events;

7. Calculating health benefits;

8. Cost-effectiveness calculation;

9. Sensitivity and headroom analysis.

Results

    Literature review suggested that adverse events follow 19% of patient discharges, and that one-third are preventable by improved handover (ie, 6.3% of all discharges). The intervention to improve handover would reduce the incidence of adverse events by 21% (ie, from 6.3% to 4.7%) according to the elicitation exercise. Potentially preventable adverse events were classified by severity and duration. Utilities were assigned to each category of adverse event. The costs associated with each category of event were obtained from the literature. The unit cost of the intervention was €16.6, which would yield a Quality Adjusted Life Year (QALY) gain per discharge of 0.010. The resulting cost saving was €14.3 per discharge. The intervention is cost-effective at approximately €214 per QALY under the base case, and remains cost-effective while the effectiveness is greater than 1.6%.

Conclusions

We offer a usable framework to assist in ex ante health economic evaluations of health service interventions.

Introduction

The decision to adopt a service level intervention turns on the evidence of its effectiveness. However, decisions also have to be made to develop an intervention in the first place, and on whether to continue development once started.^{1 2} This paper is concerned with estimating the potential cost-effectiveness of service interventions at the design and development stages, that is, at the formative stage before the intervention is rolled out in practice. This type of early economic evaluation would take place behind company doors in the case of pharmaceuticals. However, it can take place in the health services in the case of most service delivery interventions, which are typically developed ‘in the service, by the service, for the service’.³

Service delivery interventions have been classified as targeted (near patient) and generic (far patient).⁴ Targeted interventions (eg, a guideline to use thromboprophylaxis before surgery, or a forced function to prevent misconnecting the oxygen supply) are relatively easy to evaluate for effectiveness and cost-effectiveness.⁵ However, generic interventions (such as an improved nurse-to-patient ratio, or changing safety culture) have diffuse effects that spread out to affect many clinical processes and outcomes, as shown in figure 1. A recent systematic review⁶ demonstrates that most economic evaluations of such interventions are based on cost minimisation only. These studies do not capture the non-monetary value of improved outcomes for which cost utility/benefit analyses are necessary. However, our recent review (Bramley and Lilford—in preparation) revealed no articles discussing the methodological implications of this type of study for generic service interventions. We found no articles dealing with preimplementation assessments of service delivery interventions. In this paper, we describe a method for prospective evaluation of generic service delivery interventions. We demonstrate how it can be applied using a particular generic intervention—the ‘HANDOVER’ project (http://www.handover.eu), as a case study. Briefly, this was a European Union funded project to develop an educational intervention to improve patient care at the point of discharge from hospital to the community; that is, a service delivery/educational intervention to improve clinical handover of the patient from one clinical microsystem to the next.⁷ The intervention was based on earlier (exploratory) studies conducted within the HANDOVER project itself, and also knowledge of the clinical and behavioural literature dealing, for example, with the importance of clinical skills⁸ and teamwork in improving clinical transitions.⁹ The HANDOVER project is described elsewhere in this issue of the journal, and in a report to the European Commission.¹⁰ We deal here with the most intensive form of the intervention, which consists of classroom instruction, supported by a number of internet-based educational resources known as the ‘Handover Toolbox’.

Figure 1.

Representation of the widespread effects of a generic intervention. Endpoints such as mortality, or those measuring satisfaction partially measure the effect of the intervention. A sensible grouping of adverse events allows the measurements of different dimensions of effectiveness.

Methods

Framework for the evaluation of a generic service delivery intervention at the design phases

A framework for evaluation of a generic service delivery intervention was developed based on nine steps. The first step was concerned with identifying and classifying the endpoints for the evaluation. Steps 2–6 were used to derive input data. Steps 7–9 were concerned with the calculations of health benefit, net costs and cost-effectiveness.

Step 1: identification and classification of suitable endpoints

We selected endpoints on the basis of expert opinion informed by an inventory of endpoints identified from systematic reviews of the topic. As described above (and demonstrated in figure 1), the number of endpoints that might be affected by a generic service intervention is large when compared with those affected by targeted service or clinical interventions.¹¹ Ensuring that the range of salient endpoints is captured requires expert knowledge. In the case of the HANDOVER project this was acquired from the scientific management committee, informed by an inventory of endpoints used in previous reviews.^12–15 Service interventions are often designed to improve the quality and safety of care, that is, reduce the number of adverse events. Each of these events is associated with a possible contingent cost saving. In the case of a cost utility/benefit evaluation, each of these events has to be valued (in the sense that a preference weighting/utility must be assigned). It would be difficult to assign cost savings or utilities to each endpoint because there are so many. Yet, assigning one utility to cover all endpoints seems too coarse given the very different nature of the salient endpoints. Our solution was to categorise the endpoints, on the basis of the severity and duration of the associated morbidity, in order to provide a manageable number of categories.

Step 2: estimation of the baseline risk associated with each endpoint

The identification of suitable endpoints (or clusters of endpoints) is followed by an estimation of the baseline risk related to each. There are two kinds of baseline risk to consider: overall baseline risk for that endpoint, and the reversible component of that risk. This is analogous to the concepts of risk and attributable risk in classical epidemiology.¹⁶ In the service context, we are interested in the distinction between the bad outcome in question and the proportion of that bad outcome that is amenable to improved care (ie, that is preventable). For example, if the death rate is 10%, and 20% of those deaths are preventable, then the preventable risk is 2%.¹⁷ This establishes a plausible upper bound for the overall effectiveness of the proposed intervention.

Step 3: elicitation of expected effectiveness

Summative (demand-side) evaluation of interventions is based on direct empirical evidence, such as that derived from head-to-head comparisons. At the design and early development stages, however, such empirical evidence is absent or scant. Expert opinion must therefore suffice for formative (supply-side) evaluations. Such expert opinion must be quantified if it is to serve as the basis for a cost-effectiveness analysis. The necessary estimates of effectiveness should be elicited from domain experts who do not have a psychological or material personal stake in the outcome.^{18 19} Before they make their judgements, the experts should be party to a detailed description of the proposed intervention, and their knowledge should be brought up to date by drawing their attention to recent relevant reviews.¹⁹

The elicitation is typically used to construct a Bayesian ‘prior’ in the form of a ‘density’ of prior probabilities over a range of parameter values. The methods we used for eliciting a Bayesian prior have been described elsewhere.¹⁹ The centre of the probability distribution represents an expert's best guess for the effectiveness of an intervention. The priors from multiple experts can be amalgamated mathematically to yield a consolidated prior, the centre of which is a collective best guess on an effectiveness parameter. The statistical aspects of Bayesian inference are described elsewhere.^19–22 Ideally, the elicitation should be repeated for each endpoint or endpoint type. Arguably, this requirement can be relaxed if the endpoints are all highly correlated and thought to belong to the same ‘latent class’.²³

Step 4: estimating the utility values

The utility measures of disability in the case of clinical interventions are required for only one (or a small number of) health states. For example, patients with multiple sclerosis can be evaluated with respect to a disease-specific quality-of-life measure that has been mapped onto a utility scale. However, in the case of a generic service delivery intervention, such as a clinical handover intervention, a very large number of adverse events may be affected. Each event is associated with a loss of utility, but only a few of these have been calibrated. To circumvent the logistic problem of eliciting utilities for each and every adverse event, we recommended (above) that the adverse events be grouped into similar categories with respect to severity and duration, thereby limiting the number of states for which utility estimates must be obtained. These utilities may be estimated in one of two ways:

1. The disability category is assigned a ‘typical’ score on a generic quality-of-life scale (eg, EuroQol) that can be converted into a utility by means of a standard tariff;

2. A specific elicitation is made for the category, for example, by means of a standard gamble or a conjoint analysis.

Step 5: estimation of the costs of the intervention

Intervention costs are built up from fixed costs and variable costs in the usual manner. The fixed costs refer to the costs of the development, while the variable costs include the salary costs of the staff and materials deployed in delivering the intervention.

Step 6: healthcare costs associated with adverse events

Adverse patient events lead to increased health expenditure. The intervention costs are standard, but the costs generated through adverse events depend on the category of event; the more serious the event, the greater the costs associated with it on average. The estimates of costs associated with various types of endpoints may be available in the literature, and we were fortunate to find such a source of data for categories similar to those used in the HANDOVER study. In the absence of such data, we would have had to model cost-savings contingent on effectiveness (step 3) using national reference costs.

Step 7: expected health benefits and net costs

The expected health benefit (EHB) is a function of the expected effectiveness of an intervention. Resource allocation decisions require that the benefits resulting from changes in patient outcomes are captured on the same scale, for example, a Quality Adjusted Life Year (QALY) scale. Health states, each weighted by their corresponding utility value, are combined to produce the QALY. It is often convenient, for reasons we will explain below, to express the expected health gain as an expected monetary benefit (EMB). In order to convert a QALY gain to an EMB an assumption of societal willingness to pay (λ) is necessary:

The net costs are calculated by subtracting the health service costs saved from the intervention costs. While this is the standard method, it was necessary to repeat it for each of the severity and duration categories identified. The results were then combined, and weighted by the relative frequencies of the various categories.

Step 8: estimation of cost-effectiveness and cost-benefit analysis

Cost-effectiveness analysis is based on combining the costs and benefits to calculate an incremental cost-effectiveness ratio (ICER) in the usual way. Suppose an intervention incurs additional costs (or saving) over usual practice (at ΔCOSTs), and the improved effectiveness measured on the QALY scale is ΔQALYs. Then, the ICER can be expressed as

When the expected benefit is expressed as a monetary value, we can estimate the expected net benefit (ENB) from an intervention as follows:

Where λ is the society willingness to pay for a QALY gained.

Step 9: sensitivity analysis and headroom calculation

The above calculation can be repeated under various estimates of the effectiveness parameter drawn from the different points on the consolidated Bayesian prior. The headroom method is a way to explore the minimum health benefits that are needed to make an intervention cost-effective at a given level of societal willingness-to-pay value (_λ). It can be calculated by the following equation:

If the minimum necessary health benefit lies outside the bounds (say 95% credible interval) of the collective prior, then the intervention is unlikely to offer good value for money.

Results—application of the framework to the empirical study

The HANDOVER project

The HANDOVER project described in the introduction and elsewhere in this issue was used as a case study for the nine-step framework described above.

Step 1: endpoints

The literature demonstrates that the patient readmissions and adverse events are the most frequent targets for interventions to improve patient handover.^{12 14 15} However, they are not independent, since preventable readmissions are contingent on preventable adverse events. The scientific management group for the study, therefore, selected adverse events as the salient endpoint, both to model benefits to patients, and to estimate the healthcare savings that result from the reduced hospital admissions.

There are hundreds of possible adverse events, and it was therefore decided to categorise them into a manageable number. The best known classification was that used by Brennan and colleagues in the famous Harvard Medical Practice Study.²⁴ This system gives the relative frequencies in six categories of adverse events. Forster et al¹³ have produced an alternative system based on 400 patient discharges, specifically to assess adverse events following patient discharge, again with six severity levels. These two classification systems are placed side by side in table 1. We had to use expert judgement to ‘reconcile’ the two systems since they are different. For example, the proportion of deaths among patients who suffered adverse events, as cited by Brennan et al, seemed rather high at 13% for a discharge population; while no deaths, as found in the limited sample used by Forster et al, seemed optimistic. We therefore compromised at 5% (of the adverse events).

Table 1.

Comparison of two adverse event classification systems and the ‘reconciled’ version taken forward in this study

Forster et al13 (patients experiencing adverse events after discharge)		Brennan et al24 (patients experiencing adverse events in hospital)		Reconciled
Health Outcome	Proportion with the outcome	Health states	Proportion in each state	Health states	Proportion in each state
Death	0	Death	0.136	Death	0.05
Permanent disability*	0.03	Permanent disability >50%	0.026	Permanent disability >50%	0.02
		Permanent disability ≤50%	0.039	Permanent disability ≤50%	0.03
Readmission	0.21	Moderate impairment, recovery >6 months	0.028	Moderate impairment, recovery >6 months	0.10
A & E visit	0.11	Moderate impairment, recovery 1–6 months	0.137	Moderate impairment, recovery 1–6 months	0.30
Physician visit	0.14
No extra use of health service	0.51	Minimal impairment, recovery <1 month†	0.634	Minimal impairment, recovery <1 month	0.50

*These patients were also assumed to be readmitted to hospital (the observed readmission in the study was 24%).

†Brennan et al included an ‘unclassified’ category for 6.5% of adverse events—we have subsumed this into the ‘minimal impairment’ category.

Step 2: estimation of the overall risk and preventable risk

We searched the literature and identified one paper by Forster et al¹³ in which the risk of adverse events following hospital discharge was estimated. The study found that at least one adverse event occurs in 19% of discharged patients within 1 month of discharge. These adverse events were preventable (ie, the result of a failure in handover practice) in approximately one-third of cases (ie, 6.3% of all discharges).¹³ The timeframe was considered appropriate as it has been shown that the risk of preventable patient readmissions following hospital discharge is much reduced after 1 month.²⁵

Step 3: the expected effectiveness of the intervention

The plausible estimates for expected effectiveness were elicited from ‘experts’. The elicitation process is described elsewhere.²⁶ Elicitation involved experts from Europe and the USA who attended a HANDOVER project conference. The intervention (see Introduction) was described by researchers leading the workstream responsible for development of the intervention. Usable estimates of the effectiveness of the intervention were given by 24 experts. The individual probability densities were consolidated statistically.²⁷ The midpoint of this pooled prior indicates a 21% reduction in incidence of adverse events (figure 2). This represented a reduction from 6.3% to 4.7% for the risk of a preventable adverse event (an absolute reduction in risk of 1.3%). However, as shown in figure 2, there was considerable variation over the central estimate between experts. The experts also expressed considerable individual uncertainties. We made the assumption that the proportional risk reduction would be the same over various categories of adverse events.

Figure 2.

Results of the exercise to elicit experts’ estimates on expected effectiveness. Above the x-axis: pooled expert opinion on the relative reduction of the attributable risk of adverse events as a result of the implementation of a proposed intervention to improve the quality of handover (mean −0.21 (21% reduction) and 95% Higher Posterior Density Interval (−0.492 to 0.048)). Below the x-axis: Best estimate (×), lowest (•), and upper (▴) plausible value of the relative reduction of the attributable risk of adverse events as a result of the implementation of a proposed intervention to improve the quality of handover, as elicited from each of the 24 experts.

Step 4: utility loss due to adverse events

The next step was to assign a utility function to the adverse events. First, each category of event was allocated to a ‘typical’ (indicative) state on a generic quality-of-life tool, the EQ5D. The result is shown in table 2. An additional health state, ‘minimal impairment,’ is not included, as generic quality-of-life instruments are not sufficiently sensitive to detect such a change.²⁸ We therefore assumed a disutility of 0.05 for this health state (in our warfarin example below, this might correspond to a patient whose international normalised ratio (INR) is out of range, requiring an additional visit to the physician).

Table 2.

‘Typical’ EQ5D states associated with adverse events in each severity class, as defined by Brennan et al²⁴

Health state	Example: Warfarin complication	Mobility	Self-care	Usual activities	Pain/discomfort	Anxiety/depression	Index	Utility	Disutility
Moderate Impairment, recovery 1–6 months	Readmission for minor bleeding	1	1	2	1	1	11211	0.88	0.12
Moderate Impairment, recovery >6 months	Major Gastrointestinal bleeding	2	1	2	1	1	21211	0.81	0.19
Permanent Impairment	Stroke, leaving residual moderate hemiplegia	2	2	2	1	2	22212	0.64	0.36
Permanent—Severe	Stroke, leaving dense hemiplegia	2	2	3	1	2	22312	0.31	0.69

For each dimension 1 represents no handicap, 2 represents a moderate handicap, and #3 represents a severe handicap.

The utility associated with each health state identified in this way was obtained from the EQ5D manual, and then multiplied by the mean duration of the state in years (or fractions of a year) to provide an estimate of the QALY loss associated with each adverse event. It was assumed that a person can have only one adverse event.

Step 5: unit costs of the intervention

In this study, we considered a HANDOVER programme based on a facilitated classroom instruction supported by a suite of internet materials, as described in this issue and elsewhere.¹⁰ The unit cost (ie, cost averaged over discharges) for this resource-intensive form of the HANDOVER intervention has been explicated elsewhere for five European countries.²⁹ The Netherlands study sample was selected as it was the country where the costs resulting from patient admissions due to adverse events were derived—see the next step. The unit cost for The Netherlands was €16.6.

Step 6: costs associated with each adverse event

We must also consider the possible savings contingent on reducing adverse events in each category. A recent European study by Hoonhout et al³⁰ reported the mean cost of admissions associated with preventable adverse events by disability categories in 2003 to be €2979 in cases of no disability; €5973 in cases of minor disability; €6649 in cases of permanent disability; and €3831 in cases of death. Note that the study of Hoonhout and colleagues is relevant to our model since, like our model, it is based on preventable (not overall) adverse events. However, they used four categories, as opposed to Brennan et al who subdivided the permanent category into two. Moreover, Hoonhout et al deals only with cases requiring hospital admission, whereas Forster et al (see above), found that only one quarter of adverse events associated with handover resulted in patient readmission. The costs of attendance at the emergency department or doctor's office were provided by Tan et al³¹ We used our judgement to map all the above costs onto the adverse event categories identified above in step 1, as shown in table 3. All costs have been expressed in euros at 2010/2011 prices. An annual inflation rate of 3.5% was applied to convert the unit costs of adverse event categories from 2003 to 2011 values, thus corresponding to the year when the intervention was costed.

Table 3.

Calculation of QALY loss and healthcare costs associated with each patient who experiences an adverse event

Health states	Proportion of patients with adverse events in each state (a)	Average duration in state (years) (b)	Disutility of state* (c)	Cost per adverse event (€) (d)	QALY loss (a×b×c)	Cost (€) (a×d)
Death	0.05	10	1	5045	0.5	252
Permanent >50%	0.02	10	0.69	8755	0.138	175
Permanent ≤50%	0.03	10	0.36	7865	0.108	236
Moderate impairment, recovery >6 months	0.10	1	0.19	3923	0.019	392
Moderate impairment, recovery >1 to 6 months	0.30	0.25	0.12	82†	0.009	25
Minimal impairment, recovery <1 month	0.50	0.08	0.05	0	0.002	0
Expected total per person with adverse event					0.775	1080

*From table 2.

†Assuming 44% of patients incurred a visit to the emergency department (costed at €151 per visit) and in 56% of patients it involved a visit to their primary care physician (costed at €28 per visit). Based on data from Forster et al,¹³ and cost estimates from Tan et al.³¹

QALY, Quality Adjusted Life Year 2012.

Step 7: calculation of health benefit and cost savings

In table 3, we calculate the average QALY loss and the healthcare-associated costs for each patient experiencing an adverse event. The QALY loss is calculated over each category of adverse event, weighted by its probability among people with an adverse event. Patients who die or become permanently disabled are assumed to have a mean life expectancy of 10 years (see above) given that the incidence of adverse events associated with permanent disability and deaths is higher among patients aged >65 years.³² The estimated QALY loss per person that experiences an adverse event is 0.775, and the estimated associated cost is €1080 per person.

The QALY loss across all discharges is 0.15 (0.775×0.19). The QALY loss for all patient discharges due to preventable adverse events is 0.049 (0.775×0.063). If the intervention effectiveness is 21% (base-case scenario), then the QALY gain per discharged patient is 0.010 (0.21×0.049)—equivalent to one healthy life-year per 100 patient discharges.

The costs associated with adverse events (step 6 above) were mapped onto the severity categories, as shown in table 3. Avoiding adverse events can defray costs apart from those associated with readmission, especially in the case of events resulting in permanent impairment and lifelong rehabilitation. Our cost estimates are liable to underestimate total savings for this reason. The healthcare costs associated with preventable adverse events is €68 (1080×0.063) when averaged over all types of patient discharges. This amounts to €14.3 (68×0.21) given the base-case assumption on the effectiveness of the intervention. Since the unit cost of the intervention is €16.6, the net cost of the HANDOVER intervention is €2.3 (16.6–14.3).

Step 8: calculation of cost-effectiveness

The estimated results were based on the assumption of a large European (ie, The Netherlands) hospital with 50 000 discharges. The calculations for costs and benefits are given in table 4. In a hospital with 50 000 discharges each year, ineffective patient handovers result in a loss equivalent to over 7000 healthy life-years. It also results in nearly €3.5 million in health service costs that could be saved if all preventable adverse events were prevented. Under the base case, 21% of the above QALY loss can be averted (table 5). The expected cost of the intervention included the implementation cost of the HANDOVER project intervention and the cost released by the intervention. The unit cost per discharge for such a programme in The Netherlands is €16.6, and the cost for a hospital with 50 000 discharges is, therefore, €827 900 per year. This cost is subtracted from the cost savings from the reduced adverse events to give the overall net cost of the interventions. These results are given in table 5.

Table 4.

Calculation of cost-effectiveness measures (EMB and ICERs) for the intervention in a hospital with 50 000 discharges per year: the estimated total cost, QALYs, and the cost-effectiveness results

Total discharge	50000	Number of adverse events	QALY loss	Costs of adverse events (€)
Unit cost of the intervention per discharge (€)	16.6
Cost of the intervention (€)	827900
Willingness to pay (€) per QALY	20000
Rate of adverse event	0.19	9500	*0.775×9500=7365.1	*1080×9500=10261714†
Attributable to handover errors	0.333	3164	0.775×3164=2452.6	1080×3164=3417151

*From table 3.

†This assumes that the costs of non-preventable adverse events is the same as that used in the calculations for preventable adverse events—there is some evidence that preventable adverse events are more costly.³³

EMB, monetary benefit; ICER, incremental cost-effectiveness ratio; QALY, Quality Adjusted Life Year.

Table 5.

Calculation of cost-effectiveness measures (EMB and ICERs) for the intervention in a hospital with 50 000 discharges per year: Estimated costs per QALY at different levels of effectiveness

Intervention effectiveness	Proportion of discharges where AE avoided	Number of AEs avoided	QALY gained	Costs saved (€)	Net cost of intervention (€)*
100%	0.063	3164	2452.6	3417151	−2589251
21% (base case)	0.013	664	515	717602	110298
24.3%	0.015	769	596.0	830368	−2468
1.6%	0.001	51	39.2	54674	773226

*Costs of intervention, minus costs saved.

EMB, monetary benefit; ICER, incremental cost-effectiveness ratio; QALY, Quality Adjusted Life Year.

Last, the net costs and benefits are reconciled in table 6 to provide ICERs and ENBs. The proposed intervention under the base case (21% effectiveness) is highly cost-effective at only about €214 per QALY gain.

Table 6.

Calculation of cost-effectiveness measures (EMB and ICERs) for the intervention in a hospital with 50 000 discharges per year: Estimated cost-effectiveness measured in ICER and EMB

Intervention effectiveness (%)	EMB	ICER	ENB*
100	49051649	−1055.72	51640900
21	10300846	214.15	10190548
24.3	11919551	−4.14	11922018
1.6	784826	19704.37	11601

*ENB=EMB−Cost.

EMB, monetary benefit; ENB, expected net benefit; ICER, incremental cost-effectiveness ratio; QALY, Quality Adjusted Life Year.

Stage 9: headroom and sensitivity analysis result

If the intervention were 100% effective in reducing preventable adverse events, it would ‘dominate’ (table 6)—it would be both effective and cost-saving, placing it in the southeast quadrant of the cost-effectiveness plane. Dominance is lost when effectiveness drops below 24.3% (table 5).

At the societal willingness to pay of €20 000 per QALY (a conservative assumption), the minimum effect the intervention must achieve to make it cost-effective is less than 1.6%. This reflects the low unit cost of the intervention (€16.6). Nevertheless, this represents a considerable outlay (€827 900) for a large (50 000 discharge) hospital. Under the base-case effectiveness estimate (21%), the intervention cost can increase to €220.36 before it becomes cost-ineffective at the given willingness-to-pay threshold.

We also subjected the life expectancy for people with permanent disability and who died, to one-way sensitivity analysis, assuming five- and 20-year time intervals. This resulted in ICERs of €412.67 and €109.14 per QALY, respectively.

Discussion and recommendation

This paper describes the economic evaluation methodology for generic service interventions. It goes beyond most economic studies by valuing the non-monetary benefits of a complex, social intervention. It is novel in that it deals with the methodology for such studies, and focuses on the formative (preimplementation) stage of a new service. At this stage, it is necessary to estimate the effectiveness of the intervention on the basis of expert opinion. We considered it desirable to describe baseline rates for selected endpoints, and to identify the preventable (attributable) component (that is, the proportion of endpoints that result from failure of the process at which the intervention is targeted). We believe this is necessary to inject realism into the process and attenuate overoptimistic estimates by focussing respondents’ minds on the component of the endpoint that is amenable to service change.

For the HANDOVER intervention, the magnitude of the expected effects of the proposed intervention is modest (a 1.7 percentage point (ie, absolute) reduction in all adverse events). However, the model demonstrates that such an effect would, nevertheless, be highly cost-effective. The analysis shows that even small benefits (less than a 2% relative risk reduction or 0.3% absolute risk reduction in adverse events of all types) can be cost-effective for an intervention that will cost a hospital over €800 000. The small effect sizes that may be cost-effective for generic service delivery interventions have been pointed out before.⁴ However, this previous study did not take into account the health service savings that are achieved when adverse events are avoided. The result is that health service interventions will frequently be cost-effective at low magnitudes of effect, and may dominate. Although the intervention was cost-effective at an effect size of less than 2%, it had to be nearly 25% effective (ie, 1.7% absolute risk reduction) to be cost-releasing.

An intervention that might sound very expensive at the institutional level is modest when costs were hypothecated for individual patients. Under the base case, the HANDOVER intervention would be cost-effective up to approximately €220 per patient—modest when compared with clinical interventions, but a ruinous €11 million when applied to a hospital with 50 000 discharges per year. These costs may be even harder to bear when savings do not accrue to the institution that bears the cost.

A corollary of these findings is that studies to detect minimum magnitude of effects that justify their cost need to be very large to avoid type 2 error—that is, a false null result. Studies demonstrating that adverse events can be reduced from a baseline of 19% to a rate of 17.7% (ie, the 1.3% reduction posited in the base case) would require 14 060 observations, both before and after the intervention at an alpha of 0.05 and a beta of 0.8. A less biased study with contemporaneous controls would require a much larger sample size,³⁴ and would be difficult to organise given the clustered nature of the data. Likewise, detecting the less than 0.3% absolute improvement level at which the intervention becomes cost-effective would be a totally quixotic undertaking. Even after full implementation, the cost-effectiveness of inexpensive interventions may have to be modelled, rather than estimated directly from observations of improved outcome for patients. The model developed at the outset can be repopulated as development proceeds and information of various sorts gradually accumulates.³ This evidence may be surrogate evidence for patient outcomes. Such evidence is collated at the system level, upstream of the patient, as described elsewhere.⁴ While evidence on effectiveness may be very difficult or impossible to obtain, other potential inputs to economic models, such as prevalence rates of adverse events and costs, can be obtained at lower cost and/or with less difficulty.

Limitations of this study

This study is an ex ante evaluation and, therefore, by definition, encompasses many unavoidable uncertainties. This is the first study we know of to attempt an ex ante health economic assessment of a proposed service delivery intervention. It builds on methods we have developed for the device industry.^{3 35 36} The main problem we have encountered is the large number of potential adverse events, and many of the limitations stem from this conundrum. Since there is little literature to guide us in this area, we had to break new ground, and we do not claim to have provided the definitive solution. The approach we have taken is to group adverse events by severity and duration. This approach is associated with many challenges:

1. Baseline rates of preventable adverse events have to be apportioned across the categories, and this had to be induced from indirect evidence as described.

2. Ideally, separate Bayesian densities for effectiveness estimates should be elicited for each group—we assumed a common effect across groups. Severe and minor errors are not well correlated,³⁷ and adverse event categories may be affected differently by a given intervention.²³

3. Different utilities are needed for each group, and these do not appear to be available in the literature—for example, they are not available in the Tufts database.³⁸ We imputed EQ5D scores in an ad hoc way and translated these into utilities by applying the standard conversion factor (tariff). Clearly, it would have been preferable to obtain these estimates from a more representative sample of respondents. If the idea of categorising endpoints catches on and a classification system can be agreed upon, then a formal study of utilities associated with different types of adverse events (and according to how they arise) would be needed.³⁹

4. Cost savings are very different for each category. Hoonhout et al's study hypothecated costs on categories of adverse events, but the categories he used did not map precisely onto the categories we used (based on the Harvard Medical Practice Study). Furthermore, the costings in Hoonhout et al's study omitted costs of long-term care, loss of earnings, and litigation.

5. The average life expectancy of people who survive with permanent disability was difficult to ascertain, and there will always be a problem in estimating life-years lost in people who die following adverse events.

At the very least, our paper demonstrates the need to develop a consensus on the intellectual framework for assessing the outcomes of complex service and safety interventions, and then to set up epidemiological studies to collect data in the requisite form.

Conclusion

We have argued for the utility of a predevelopment economic evaluation of service interventions, and described a method for achieving this. We applied the methodology to a clinical handover intervention aimed at improving transfer of patients from hospital to the community. Adverse events constitute the outcome of interest, but our major challenge was to deal with the wide range of such events. We think that the method we have proposed shows promise, but it is in its infancy and much more work is necessary to develop supply-side health economic modelling for service delivery interventions. However, refinements in the method will not overturn one stark conclusion—service delivery interventions are cost-effective at small absolute effect sizes and at costs that cannot be borne under current thresholds used to ration new treatments.