Problems with UK government's risk sharing scheme for assessing drugs for multiple sclerosis

The government plans to make interferon beta and glatiramer available to patients with multiple sclerosis through a risk sharing scheme, despite lack of evidence of cost effectiveness. Sudlow and colleagues argue that the money would be better spent on independent research

The National Institute for Clinical Excellence (NICE) recently announced that interferon beta and glatiramer acetate were not cost effective treatments for multiple sclerosis and could not be recommended for NHS funding.¹ As a result, the Department of Health and the manufacturers developed a “risk sharing scheme” aimed at providing these drugs more cost effectively.²,³ Treatment will be provided to ambulating patients with two or more disabling relapses in the past two years (about 15% of all patients with multiple sclerosis)⁴ and their progress monitored over 10 years. However, the scheme has several scientific and practical problems that we believe limit its ability to improve the care of patients in the long term. In this paper, we review the quality of the evidence on which NICE and the Department of Health reached their decisions, consider some of the problems of the risk sharing scheme, and suggest an alternative approach.

Summary points

• NICE has announced that neither interferon beta nor glatiramer can be recommended for multiple sclerosis in the NHS

• The Department of Health plans to make these drugs available through a risk sharing scheme that is scientifically unsound and impractical

• Randomised trials suggest that azathioprine (which is 20 times cheaper) may be just as effective

• The long term effectiveness of these drugs is unknown

• Government money would be better spent on a long term randomised trial comparing interferon beta or glatiramer with azathioprine and no treatment

Methods

We identified randomised trials of disease modifying drugs in patients with multiple sclerosis from systematic reviews,^5–7 the Cochrane controlled trials register, and the treatment guidelines of the Association of British Neurologists.⁴ We also got information from discussion with colleagues, including several international experts in multiple sclerosis. We used Cochrane RevMan software to produce summary relative risks.

What is the evidence that the drugs are effective?

NICE considered data from placebo controlled trials of interferon beta and glatiramer acetate but did not assess azathioprine, which has also been widely tested in multiple sclerosis.^5–31 The three drugs produce a similar reduction (15-30%) in the relative risk of a relapse at two years (fig 1). Interferon beta and glatiramer may also reduce disability (fig 2), but appropriate data were not available for azathioprine. Although interferon beta, glatiramer, and azathioprine were all associated with more patient withdrawals than placebo, the side effects were generally mild. Azathioprine may be associated with a small increased risk of neoplasia after 10 years of treatment,³²,³³ but there is not enough long term experience with either interferon beta or glatiramer to exclude an increased risk of cancer.

Figure 1.

Risk of relapse of multiple sclerosis at about two years. Numbers were obtained from a systematic review of interferon beta in relapsing-remitting multiple sclerosis⁵ as well as individual trials. There was no statistical heterogeneity between individual trials contributing to summary statistics

Figure 2.

Risk of progression of disability at about two years. Numbers were obtained from a systematic review of interferon beta in relapsing-remitting multiple sclerosis⁵ as well as individual trials. There was no statistical heterogeneity between individual trials contributing to summary statistics

These results, although promising, are based on limited, short term data (a few hundred patients for each drug, usually followed up for no more than two years). We therefore do not know whether the effects are sustained over the long term. Several other previously noted methodological problems⁷ also limit the interpretation of the results and may have biased them in favour of active treatment. These include uncertainty about the adequacy of randomisation,³⁴ which is not always clearly described; unavoidable patient unblinding³⁵; difficulty interpreting the outcome of confirmed progression of disability, which was generally based on the widely criticised expanded disability status score³⁶,³⁷; substantial losses to follow up in a few trials; publication bias (the largest randomised trial assessing interferon beta in secondary progressive multiple sclerosis showed no overall effect on progression of disability but remains unpublished¹⁷); and funding of the trials by pharmaceutical companies, which own the data and were involved in the trials' design, conduct, analysis, and reporting.^9–24Although many trials had independent data monitoring committees, it is concerning that some data from some trials have not been placed in the public domain.

Limitations of NICE's assessment of cost effectiveness

NICE's conclusions on cost effectiveness were based mainly on an analysis commissioned from the Sheffield University School of Health and Related Research.¹,³⁸ The researchers calculated the cost per quality adjusted life year (QALY) gained at 5, 10, and 20 years after starting treatment (table). The model suggests that the threshold of £36 000 per QALY (set by the Department of Health for the risk sharing scheme) is approached only after 20 years.

Although this economic model is probably the best available for multiple sclerosis, it has several unavoidable flaws. Firstly, it depends on the quality of the evidence for effectiveness of treatment, which, as highlighted above, has major deficiencies. Treatment effects were estimated mainly from published reports. Two companies provided some additional confidential data, one refused, and one withdrew its additional data after seeing its effects on the results.¹,³⁸

Secondly, because of the lack of long term placebo controlled randomised trials, the model compares the effects of treatment with the experience of a cohort of 1000 Canadian patients with multiple sclerosis recruited in the 1970s and 1980s and followed for an average of 25 years.³⁹,⁴⁰ It assumes that treatment remains effective for as long as the patient takes it and that the benefit accrued is maintained after treatment stops. NICE acknowledges that this extrapolation of treatment effects becomes increasingly unreliable as the time horizon is increased.¹,³⁸

Thirdly, the model is heavily influenced by assumptions about future discounting of costs and benefits, the proportion of patients who stop treatment prematurely and what happens to them, and the way in which the costs of disability related to multiple sclerosis are estimated (table).³⁸ Finally, it does not consider azathioprine, which has an annual treatment cost of only about £300 a patient.

Scientific flaws of risk sharing scheme

The risk sharing scheme plans to use the Sheffield model as a basis for assessing and adjusting the real life cost effectiveness of interferon beta and glatiramer. Azathioprine has been ignored. Patients meeting the Association of British Neurologists treatment criteria⁴ will be assessed annually for 10 years to determine the rate of progression from no disability (expanded disability status score <4), through mild (4-5.5) and moderate (6-6.5), to severe disability (⩾7). The effects of each treatment will be determined every two years by comparison with the expected progression without treatment derived from the Canadian cohort.³,³⁹,⁴⁰ Target treatment effects have been agreed with the drug companies, and if these are not achieved, the drug costs will be reduced to maintain cost effectiveness at a threshold of £36 000 per QALY over 20 years.³ Unfortunately, the scheme has several major problems.

Non-randomised comparisons

The Department of Health circular states that the scheme is not a further trial of clinical effectiveness but a study to establish long term cost effectiveness.³ However, a reliable estimate of long term cost effectiveness first requires a reliable estimate of long term clinical effectiveness. This will not be achieved by comparing a modern cohort of patients treated in the United Kingdom with a historical cohort of Canadian patients since non-randomised comparisons give unreliable, biased results.⁴¹,⁴²

Lack of power calculations

The scheme will include about 7000 patients in England and Wales (plus more patients from Scotland), but the circular gives no power calculations to justify this number.³ It recognises that the non-randomised comparison will be biased and that chance may lead to imprecise measures of treatment effect. However, rather than randomising large numbers of patients, the Department of Health proposes to incorporate a tolerance margin of 10-20% in the comparison between the treated and untreated cohorts.³ It does not explain how this margin was chosen; nor is it clear whether the margin represents a relative or absolute difference in outcome.

Other biases

The risk sharing scheme is subject to several other biases. Firstly, patients already receiving treatment at the start of the scheme will be included if they fulfilled the inclusion criteria at the start of treatment and their pre-treatment disability score and other prognostic data are available.³ This will bias the comparison in favour of treatment because patients who started and then stopped treatment before the scheme because of adverse effects or perceived lack of effectiveness will not be included.

Secondly, the circular states that patients who stop taking treatment during the scheme are “to be monitored as far as possible.”³ This is not good enough. It is essential to follow up such patients because we do not know how patients respond once they stop treatment. This information is critical to the cost effectiveness calculations.

Thirdly, the scheme does not intend to have blinded assessment of outcome. Unblinded assessment of outcome in multiple sclerosis trials can result in overestimates of the effect of treatment on progression of disease.³⁵ Hence, an apparent treatment benefit may simply be due to the expectation bias of patients and their neurologists or specialist nurses. An additional competing interest bias may be introduced by unblinded assessment of outcome being done by specialist nurses whose salaries are paid for by pharmaceutical companies.

Calculation of cost effectiveness

The estimates of cost effectiveness depend critically on the various assumptions used in the modelling process, but the actual assumptions to be used are not mentioned in the circular. These will have to be made explicit and justified before the scheme starts.³

Other issues

The circular does not state what will happen if a treatment seems ineffective. Neither does it tell us which patients will be included in the analyses or whether these will be on a truly intention to treat basis. Ideally, a proper intention to treat analysis would be ensured by information on patients giving their consent being telephoned or faxed immediately to a central site. This would avoid the loss or non-registration of patients who do not do well on their chosen treatment. However, no details of this sort have been provided.

Practical problems with risk sharing scheme

The Department of Health proposed that patient recruitment would start on 6 May 2002. However, the national coordinating team was not appointed until July 2002, ethical approval has had to be sought, and many neurologists have yet to see a detailed protocol.

The cost of the drugs (more than £50m a year) for the scheme will have to be met from existing NHS budgets. In addition, collecting data is likely to put further strain on NHS resources. The circular states that “the scheme should as far as possible build on normal clinical practice without requiring elaborate additional infrastructure” and that “data entry should be as simple as possible and arise out of normal patient contacts.”³ However, neurological services are already extremely stretched (median outpatient waiting times are about 26 weeks), and many potentially eligible patients do not have regular contact with a neurologist.⁴³ Many additional consultant neurology and specialist nurse sessions (with appropriate administrative support) will be needed to evaluate patients who may be eligible and to follow up those who join the scheme. Normal patient contacts do not include assessment of the expanded disability status score, and so appointments will have to be longer to allow for this. It is not clear how all these additional sessions can be provided without seriously compromising the existing service, although the pharmaceutical companies will fund some. Local staff (probably specialist nurses) will need to be trained to collect, store, and transfer the additional data.

Alternative proposal

We believe that the government could spend the extra resources for patients with multiple sclerosis more effectively. Firstly, it should commission an independent, individual patient data overview of all relevant published and unpublished randomised trials of disease modifying drugs for multiple sclerosis. The overview would address unanswered questions about the trials and may go some way towards resolving the uncertainties about the effects of interferon beta, glatiramer, and azathioprine.

Secondly, the risk sharing scheme should be modified to include a concurrent, randomised control group rather than a historical cohort. Given that the Department of Health is committed to providing resources for the assessment, long term follow up, and drug costs for several thousand patients with multiple sclerosis, a long term randomised trial, run independently of the pharmaceutical industry, would be a far more scientifically (and so ethically) justifiable use of this money. Patients could be randomised three ways (interferon beta or glatiramer versus azathioprine versus no treatment) and followed up in the same manner and with the same outcomes as the existing scheme. Additional resources would be required for blinded outcome assessment and perhaps inclusion of a quality of life outcome, but a trial would probably be less expensive than the present scheme because only one third of patients would be taking an expensive drug. Careful explanation would have to be given to patient groups about why a randomised trial is the best way forward as fewer patients would be receiving active treatment. However, it is the patients who have most to gain by reliably establishing the long term clinical effectiveness as well as cost effectiveness of these treatments.

Conclusions

Any additional resources for patients with multiple sclerosis are welcome. However, these should be used to provide services that we know benefit patients and to support further, properly designed research into interventions about which there is still uncertainty. Uncertainty remains about the effectiveness and cost effectiveness of interferon beta and glatiramer, and the risk sharing scheme will neither resolve these nor determine the possible role of promising but far less expensive drugs such as azathioprine. All patients with multiple sclerosis, whether eligible for treatment under the terms of the scheme or not, deserve much better than this. The government should consider a more appropriate use of this large amount of public money.

Table.

Base case analysis for estimates of cost per QALY from Sheffield model^*

Trial	Type of multiple sclerosis	Current cost/patient/year (£)†	Estimates of treatment effect from published data			Cost per QALY gained (£)
			Relative relapse rate	Relative hazard rate for disability progression		Published data 20 years	Published plus confidential commercial data‡
							20 years	10 years	5 years
Intramuscular interferon beta-1-a11 12	Relapsing-remitting	9 061	0.82	0.54		48 000	106 000	618 000	783 000
Subcutaneous interferon beta-1-a 22 μg13	Relapsing-remitting	9 088	0.71	0.64		59 000	59 000	325 000	511 000
Subcutaneous interferon beta-1-a 44 μg13	Relapsing-remitting	12 068	0.67	0.62		79 000	79 000	406 000	638 000
Subcutaneous interferon beta-1-b 8 MIU910	Relapsing- remitting	7 259	0.71	0.71		52 000	39 000	226 000	365 000
Subcutaneous interferon beta-1-b 8 MIU15	Relapsing-remitting, secondary progressive	7 259	0.69	0.72		41 000	43 000	230 000	380 000
Subcutaneous glatiramer21 22	Relapsing-remitting	6 650	0.71	0.76		98 000	98 000	442 000	443 000

^* This assumes that patients are treated according to the Association of British Neurologists criteria for starting and stopping treatment; the mean age of patients at the start of treatment is 30 years; costs are discounted at 6% per year and quality of life benefits at 1.5% per year; 10% of patients will withdraw from treatment during each of the first two years and 3% per year thereafter. The model allows for variation in these assumptions, in the costs and treatment effects shown in the table above and in other factors (costs of multiple sclerosis related disability and relapses, and the utilities associated with various disability states, relapses, and treatment side effects). Some important examples are: (a) if the discount rate for future benefits is increased from 1.5% to 6% annually, the costs per QALY gained roughly double; (b) if the model assumes that nobody stops treatment prematurely, the costs per QALY gained increase further by up to a third. 

^†Costs are those quoted in the NICE and Sheffield reports and used in the cost effectiveness analyses summarised here. Note that for the risk sharing scheme, the Department of Health has stated that the annual costs are interferon beta-1-a £8502, interferon beta-1-a 22 μg £7513, Interferon beta-1-a 44 μg £8942, interferon beta-1-b 8 MIU £7259, glatiramer £5823. 

^‡Additional data were available from only Biogen^{11 12} and Schering.^{9 10 15} The nature of these data is not clear from the Sheffield report.