Participatory Workshops are Not Enough to Prevent Policy Implementation Failures: An Example of a Policy Development Process Concerning the Drug Interferon-beta for Multiple Sclerosis

Abstract

A possible explanation for policy implementation failure is that the views of the policy’s target groups are insufficiently taken into account during policy development. It has been argued that involving these groups in an interactive process of policy development could improve this. We analysed a project in which several target populations participated in workshops aimed to optimise the utilisation of an expensive novel drug (interferon beta) for patients with Multiple Sclerosis. All participants seemed to agree on the appropriateness of establishing a central registry of Multiple Sclerosis patients and developing guidelines. Nevertheless, these policy measures were not implemented. Possible explanations include (1) the subject no longer had high priority when the costs appeared lower than expected, (2) the organisers had paid insufficient attention to the perceived problems of parties involved, and (3) changes within the socio-political context. The workshops in which representatives of the policy’s target populations participated did not provide enough interactivity to prevent policy implementation failure.

Introduction

There are numerous examples of health policy measures that have failed in their implementation. A review from the Netherlands Court of Audit indicated that no evidence of a full policy implementation can be found in previous audits [20]. For example, the applied measures for reducing greenhouse gas emissions have not met pre-determined targets, the legal rules for maintaining food safety have not been able to prevent targeted risks, and policy measures for cost control of medicines have not resulted in more structural cost control [19].

The theory of argumentative policy analysis offers a possible explanation for such policy implementation failures [8, 15]. The basic idea of this theory is that actors’ behaviours can be explained by different views on a problem and the argumentation behind these views. According to the argumentative approach to policy analysis [7, 8], action is driven by processes of problem setting [27] in which actors define coherent sets of problems and solutions that correspond to these actors’ normative and empirical background theories. The way in which problems are defined depends on the assumptions the actors make about the situation and their beliefs regarding what is good practice (normative values).

Combining these findings with those of classical implementation theory, Grin and Van de Graaf [10] have argued that a policy will only be effective if both implementers and target populations consider the proposed policy measure meaningful. (Target populations consist of persons who will experience the consequences of a policy when implemented.) This means that the proposed policy should (a) make sense in the light of problems perceived by the target populations and (b) be consistent with their normative and empirical background theories.

However, the fact that policy problems and associated solutions tend to shift over time, different actors with different background theories being involved successively, renders this rather complicated. In other words, a policy measure is not invented at a specific moment in time, but develops over time. Nevertheless, the challenge is to identify policy measures that cohere with the views of all actors involved.

If the argumentative policy theory is correct, then it is advisable to identify the policy’s target populations and involve them in the process of policy development. An interactive process of policy making could thus ensure that policy coheres with and prevent that it diverges from the views of target groups.

The objective of this paper is to analyse a specific instance where target populations were involved in the process of policy development. The policy institution in this example was the Health Care Insurance Board in the Netherlands (HCIB). It is an advisory board to the Ministry of Health, particularly with respect to coverage and reimbursement issues. The Department of Policy Analysis of Medicines (PAM) is responsible for identifying developments that may jeopardise optimal medical care, analysing the nature and size of such threats and conducting further research that may provide a basis for policy decisions. The annual work programme in which topics are prioritised is submitted to the Ministry of Health for approval.

The policy objective discussed in this paper was to promote the appropriate use of a recently introduced drug for patients with multiple sclerosis. Multiple Sclerosis (MS) is a neurodegenerative disease characterised by neurological dysfunction. The drug interferon-beta (IFNβ) appeared to be a promising treatment. However, its costs were high (€ 12,000 per patient annually) and the evidence of its long-term effectiveness was limited. Therefore, policy measures to guide the prescription of this medicine were considered necessary. Representatives of prescribing physicians (neurologists), health insurance companies, and patients were invited to participate in two workshops. During these workshops, participants discussed policy measures that could promote the appropriate use of the new drug. Although two concrete policy measures had been proposed, these have never been implemented.

In this paper we evaluated whether the theory of the argumentative policy analysis could explain the proceedings in this case study. It offers a description of the process of the policy development that included the two workshops, as well as an analysis of the views of policy makers and target populations in order to assess whether the proposed policy measures fitted the perceived problems and underlying background theories.

Methods

Relevant documents were analysed and semi-structured interviews were held with the various stakeholders. These documents included correspondence, reports from the Board, internal memos on this subject, a report from Health Council on interferon-beta, reports from meetings, research proposals, and research reports. Interviews were held with three HCIB employees, two employees from the Ministry of Health, the organisers of the workshops, two neurologists, a patient, and a medical advisor from a health insurance company. Participants of the workshops were contacted and interviewed to reconstruct the proceedings of the workshops. MS patients were contacted via a Dutch MS patient organisation. Representatives of target populations were interviewed to assess their problem definitions and underlying argumentation.

Interviews were transcribed verbatim. A summary from the interview and a concept report were sent to respondents for verification and literature was used to check the findings from interviews or documents. In line with the theory of argumentative policy analysis, we used the method of reconstructing interpretative frames to analyse target populations’ argumentation [10, 11, 18]. The idea is to reconstruct (1) how problems are defined, (2) how solutions or policy measures are judged, (3) what theoretical and normative assumptions shaped them, and (4) what normative preferences underlie this all. Together, these four ‘layers’ of evaluation entail an individual’s interpretative frame.

Results

Proceedings of Project

The content of the workshops was, to a large extent, determined by two reports concerning IFNβ. These had been issued by the HCIB and by the Health Council, respectively. In July, 1995, the Ministry of Health asked both institutions to provide advice regarding the introduction of IFNβ on the Dutch market. The Ministry asked for recommendations with respect to whether or not restrictions could be imposed on the reimbursement of the drug.

In May 1996, a committee from the Health Council reported to the Ministry of Health [9]. This committee was comprised of three neurologists, a chemical technologist, and a HCIB staff member. It concluded that IFNβ could be a promising new drug, but also cautioned against unrealistic expectations. Clinical trials had shown that IFNβ could decrease the rate and severity of exacerbations, but there was no evidence of IFNβ preventing the onset of disability. The Health Council emphasised that the drug should be prescribed only to patients who meet eligibility criteria for the trials. These criteria were: (a) clinically definite Relapsing Remitting MS; (b) at least two exacerbations in the two previous years; (c) mild to moderate disability; and (d) age of 18 years or older. The Council also proposed the following policy measures: (1) develop a guideline for treatment of MS patients with IFNβ; (2) ensure that the drug is prescribed by neurologists who have sufficient experience with the diagnosis and treatment of MS patients; (3) properly instruct and guide MS patients; (4) conduct a systematic follow up of patients using IFNβ in order to evaluate side effects; (5) provide clear indications on which patients should be treated with IFNβ; (6) establish a national registry of MS patients in order to conduct further research on the effectiveness of IFNβ; and (7) conduct a re-appraisal after a number of years.

June 1996, the HCIB issued recommendations that corresponded partially with the Health Council’s report [30]. The HCIB recommended that: (a) patients fulfil the criteria described in the Health Council’s report; (b) a treatment protocol be developed; (c) health insurers approve reimbursement requests before starting treatment; (d) a prospective registration be established; and (e) the use of IFNβ be re-assessed after 3 years. Restricting prescription to a limited number of experienced neurologists was considered impossible, because IFNβ had already been included in health care packages.

In 1999, the department of Policy Analysis of Medicines (PAM) started a project on IFNβ. Initially it they considered evaluating other, less expensive interventions that could be effective, but are of no interest to the industry. Indications had been received from the field that a much cheaper drug, namely methotrexate (used in oncology and rheumatoid arthritis), could be as effective as interferon. Furthermore, PAM considered developing a national database of MS patients. PAM later decided that conducting a clinical trial was not its responsibility. PAM staff, in collaboration with neurologists, decided to initiate the development of a treatment protocol and a national registry of MS patients. They proposed the organisation of two workshops.

Workshops

In 2000, these workshops were organised by an external institute for policy research. The first workshop aimed to provide an overview of the criteria that are used for prescribing IFNβ. This workshop was preceded by an inventory study amongst health insurance companies into current reimbursement practices with respect to IFNβ. This inventory study was performed by a HCIB department. The results from the inventory study on current reimbursement practices indicated that health insurance companies did not assess reimbursement requests against medical content. Assessments were limited to an administrative review of the completeness of data [4]. Nevertheless, health insurance companies considered the pre-utilisation approval effective, because it functioned as an administrative barrier. The authors of this report recommended the following: (a) assess whether a pre-utilisation approval by a central committee could improve the assessment of reimbursement requests; (b) develop a clear protocol and design application forms.

Neurologists, medical advisors from health insurance companies, policy makers, and a representative from a patient organisation participated in the workshop. During the workshop, vignettes with patient descriptions were used to discuss which patients should be treated with IFNβ. The organisers concluded that only a few problems were perceived with respect to criteria for IFNβ use. Health insurers questioned whether the situation was indeed problematic given the small size of the patient population and IFNβ’s status as an essential medicine. Neurologists argued that problems arose incidentally. Most often, these problems were related to reimbursement requests for continued use of IFNβ. All participants agreed that subjectivity in decisions on prescription should be minimised. The participants claimed that clear distinctions should be made between the criteria for initialising treatment and the criteria for continuing treatment. The development of a guideline by neurologists was considered relevant. According to the participants, small adjustments of the current criteria would be sufficient [2].

The second workshop aimed to obtain advice on how existing databases could be improved so that the effects of IFNβ or other new drugs for MS patients could be evaluated. In preparation for this workshop, a neurologist was asked to make an inventory of available databases that contained data on IFNβ users. This inventory showed that data on MS patients were no longer being collected systematically in the Netherlands [2]. Former local databases were no longer up-to-date. The participants in the workshops agreed that a national database could be relevant and could serve the following goals: (a) policy making (such as financial surveys, planning health care capacity); (b) research (effects of treatment; monitoring for side effects); and (c) clinical practice support for neurologists (reflection on clinical practice, improvement of expertise). Participants agreed on the inclusion of general static data in the database, such as demographic details, diagnosis (type MS), the criteria used to make the diagnosis, and the initial treatment. Participants disagreed on the kinds of dynamic data, such as treatment details and information on physical functioning. A database including a broad range of data could be useful for research, but this was considered expensive. A small database that includes static data only could be used as a sample frame for selecting potential participants in further studies. In their report, the organisers concluded that additional research was needed to ascertain which variables should be included in the database.

Proposed Policy Measures

Based on the organisers’ report, the HCIB proposed the following policy measures to the Ministry of Health [5]:

• Have neurologists develop an evidence-based guideline for the treatment of patients with multiple sclerosis.

• Have reimbursement requests for IFNβ appraised on medical grounds. Possibly, implement a central pre-utilisation approval by experts (neurologists).

• Define clear criteria for (dis)continuing IFNβ reimbursement.

• Establish a national registry of MS patients. The database should enable scientific research, the improvement of treatments, monitoring, and the evaluation of efficiency and therapeutic value of new interventions.

In 2003, a medical advisor from a health insurance company mentioned that some small changes had been made to the procedures established for judging reimbursement requests (requests for continued use). A neurologist mentioned working on a guideline for diagnosis and treatment of MS patients. According to a PAM staff member, neither a central pre-utilisation approval nor a national database to prospectively register patient data had been established.

Reconstructed Interpretative Frames

In June 2003, we conducted interviews to reconstruct interpretative frames of policy makers and policy’s target populations. The actors’ views are summarised in Table 1 (IFNβ guideline) and Table 2 (national database).

Table 1.

Views towards guideline concerning IFNβ use

Actor	Judgement of solutiona	Problem definition	Background theories	Normative values
Ministry of health (1996)	+ Clear criteria for reimbursement IFNβ needed	IFNβ is an expensive medicine; uncertainty on meaning of effects Possibly inappropriate use because high expectations	Only if evidence of effectiveness is available, a medicine can be at the expense of the community	Cost control Efficiency
PAM staff	+ A central pre-utilisation approval for IFNβ use A clear treatment protocol on IFNβ should be available	Health insurance companies judged reimbursement requests differently Individual health insurer lack expertise to judge reimbursement requests	Differences in judgement result in differences in care Situation IFNβ resembles situation growth hormones, for which a central pre-utilisation approval in combination with central registration was set up Central pre-utilisation approval prevents inappropriate use	Cost control; efficiency Equal accessibility
PAM staff (2003)	− Central pre-utilisation approval no longer needed	Situation concerning IFNβ was relatively stable; no longer problems concerning reimbursement requests Costs committee higher than possible savings IFNβ use	Criteria for central pre-utilisation approval: – medicine is expensive – inappropriate use expected – decisions on treatment are complex	Cost control Efficiency
Medical advisor health insurance company	+ Guidelines on IFNβ needed for decisions reimbursement	Unclear when IFNβ is (still) meaningful	Pharmaceutical companies benefit from extending criteria Expensive drugs are financed from public funds, on a solidarity base	Cost control
Neurologist	+ Not guideline on IFNβ solely but guideline on diagnosis & treatment MS	Increase possibilities for diagnosis and therapy; variance in MS care	As in other fields increasing attention to evidence based medicine	Provide clarity for neurologists on how to care patients
Patient	+ National guideline might be useful	Differences between hospitals care (IFNβ not offered in every hospital; contradictory advises on lifestyle)	No common knowledge on MS Prognosis differs between individuals	Improve or maintain quality of life

^a‘+’ actor defines the intervention meaningful; ‘−’ intervention is not meaningful

Table 2.

Views towards a national registry of MS patients or IFNβ users

Actor	Judgement of solutiona	Problem definition	Background theories	Normative values
Ministry of health (1996)	+ For evaluation long-term effects	Uncertainty on (meaning of) effects of IFNβ	Only medicines of which evidence of effectiveness is available at the expense of the community If no evidence on effectiveness, use is inappropriate	Efficiency
PAM Staff	+ Database with only limited patient data to take samples for research or Database with all relevant data for research	For each new medicine a study on effectiveness is inefficient Health insurance companies judged reimbursement requests differently, leading to difference in health care	Aim of national database to optimise IFNβ prescription: to monitor use of IFNβ in practice (which patients (indications), volume, costs) and to evaluate effectiveness IFNβ	Efficiency
PAM Staff (2003)	− Registration without central judgement committee difficult	Central committee no longer relevant No problems perceived	Central pre-utilisation judgement provides a minimal registration of patient data	Efficiency
Medical advisor of health insurance company	+ Prospective database only way to assess long-term effectiveness	Judgement of relative value as compared to alternative interventions not systematically but only in the neurologists head	Expensive drugs are financed from public funds Pharmaceutical industry’s power is underestimated	Clear vision from the ministry of health on pharmacy Quality of care
Neurologist	+ Meaningful for evaluating effectiveness of medicines and knowledge on MS National coverage might be difficult	Clinical trials have limited follow up period Distrust amongst neurologists	MS is a chronic disease, invalidating on long term MS neurologists have their own “kingdoms”	Quality of care Increase knowledge on disease and treatment

^a‘+’ actor defines intervention as meaningful; ‘−’ intervention is not meaningful

Ministry of Health

Initially, the Ministry contended that the long-term effectiveness of the drug was not established sufficiently, while the acquisition costs were high. Trials had shown that IFNβ decreased the number of exacerbations, but it was unknown whether IFNβ would prevent disability. The potential target population, namely patients with MS, is large and the expectations of both patients and physicians may also have been high. As a result, the risk that the drug could be used inaccurately was considered to be high. Consequently, additional policy measures to control IFNβ use in clinical practice were requested.

According to employees from the Ministry of Health, the HCIB report [5] had revealed that only health insurers perceived few problems with respect to judging reimbursement requests. The costs of a committee for a central pre-utilisation approval were considered relatively high, while the costs of the interferon-beta prescription were less than expected, namely €18 million (in 1999) versus the €90-180 million estimated by the Ministry. Therefore, both interventions were considered to be no longer relevant.

PAM Staff

According to the PAM staff, the main problem was that the use of IFNβ was expected to extend to other subgroups of patients. A central pre-utilisation approval in combination with a national registry of MS patients could provide a solution. This contention was made because the situation concerning IFNβ resembled the situation concerning other drugs for which a central pre-utilisation approval had already been established. In that case, requests for reimbursement of drug use were judged by a central committee of medical experts. For this purpose, a clear protocol including criteria for IFNβ use is needed. Simultaneously, patient data were recorded in a national registry.

At the time that PAM staff started their project, it was undesirable to evaluate why the proposed policy measures had not been implemented until then. Reason was a change in the relationship between the HCIB and the Ministry of Health. In 1999, the HCIB, an independent advisory board, was established as the successor to the Sickness Funds, which was a politically involved advisory board comprised of actors from the field.

After completion of the project, the PAM staff considered the option of having reimbursement requests judged by a central committee no longer meaningful. They considered the implementation of policy measures difficult because the drug had already been introduced several years earlier. Limiting or discontinuing the reimbursement of IFNβ would have been practically impossible. Furthermore, the scale of problems relating to the prescription and/or reimbursement of IFNβ and the costs related to IFNβ use turned out to be much smaller than expected. PAM considered the initiation of a national MS patient registry without the central pre-utilisation judgment to be unfeasible.

Medical Advisor Health Insurance Company

According to a medical advisor from a health insurance company, the problem was attributable to a lack of clarity on exactly what kind of patients benefit from IFNβ. IFNβ is an expensive drug and the costs are either carried by the community (public health insurance) or reimbursed on an individual basis (private health insurance). Preferably, physicians should develop a guideline on the relative position of IFNβ in relation to alternative interventions. At the time of IFNβ’s introduction, a central pre-utilisation approval could have been relevant. However, time had passed and the prescription of IFNβ had become common practice. Unfortunately, new drugs are often introduced and included in the health care package long before all medical specialists agree on criteria for treatment. Professionals cannot develop guidelines quickly. In contrast, the time that passes between a drug being introduced into the market and its inclusion in the health care package is regulated by law and, in most cases, quite short. As a temporary solution, the drug could be included into the health care package under certain conditions, such as the registration of patient data.

Neurologist

A neurologist mentioned working on a guideline for diagnosing and treating MS patients as part of an initiative from the Dutch Society for Neurology. A guideline for diagnosis and treatment was considered relevant as it could support evidence-based practice. As in other medical fields, knowledge and treatment options have increased. The development of a guideline is a time-consuming endeavour as, often, it has to be done alongside the professional’s usual activities. A central pre-utilisation approval was no longer relevant. Prescribing IFNβ was considered common practice, also in small general hospitals. Most neurologists were already familiar with the indications for treatment. IFNβ was proven to be effective in one type of MS that is characterised by invalidating exacerbations (relapsing remitting MS) through clinical trials that demonstrated IFNβ’s ability to decrease the severity and frequency of these exacerbations. A national database could still be useful. However, its purpose must be clear. An appropriate goal could be to evaluate the long-term effectiveness of IFNβ. The neurologist questioned whether participants would have come to agreement on specific recommendations, such as which data should be collected and who should become the owner of the database. Most neurologists do have “their own kingdoms” and do not want to share these with others.

Patient

The patient claimed that agreement amongst neurologists about MS treatment could be useful. For him, the main problem was that MS care differs between hospitals. He conveyed his experiences with receiving contradictory answers to questions from numerous health professionals and the option of using IFNβ had not been discussed in the hospital where he was initially treated. For this patient, the exacerbations are highly invalidating. Obviously, his preference is to lead a normal life, in so far as that is possible. MS has a high impact on his life.

Discussion

Although results from the workshops showed that all respondents agreed that developing a guideline and a national database could be meaningful, these policy measures have never been implemented. A number of factors can provide an explanation for the proceedings in this project.

Firstly, the costs of IFNβ appeared to be lower than expected. Perhaps the information campaign on IFNβ had had this effect [3]. It may also have been that the calculation by the Ministry of Health, which differed from calculations made by the Health Council and rested on the assumption that all MS patients are treated with IFNβ, were unrealistic. In any case, the subject of IFNβ no longer had high priority. From the point of view of policy makers, the expected high costs of IFNβ in combination with its uncertain effectiveness were the main problems behind the proposed policy measures. Apparently, however, the problem of the high costs had still higher priority than the limited knowledge on IFNβ’s effectiveness. As a result, policy makers left it to the medical profession to implement policy measures. Neurologists, however, considered cost-containment not their problem and not their responsibility.

This also explains why the initial questions about the long-term effectiveness remained unanswered, without this being perceived as a problem. The effectiveness of IFNβ and the possible role of the industry was an issue of international debate. Clinical trials have shown that IFNβ decreased the number and severity of exacerbations in patients with relapsing-remitting MS [16, 23]. Probably, this is enough reason for neurologists to claim the right to prescribe IFNβ. That evidence of long-term effectiveness, in terms of preventing disability, and safety was still limited [26] is not of great concern to them. Neither is the possible influence of the industry on prescription practices through the financing of clinical trials that had been discussed [13, 21, 22, 28]. Some authors have even challenged the hypothesis of MS as an inflammatory auto-immune disease [1]. A national registry could have been relevant from this point of view. In the UK, the Department of Health initially refused to reimburse IFNβ use for reasons of costs [17]. In 2002, however, agreements were made with the industry concerning the funding of IFNβ. The Department of Health announced that it would reimburse IFNβ for MS patients who agreed to participate in a monitoring program on the effectiveness of IFNβ.

Secondly, the policy development process from the beginning centred around two specific policy measures, whereas it could have started with a broader scope, actors involved first eliciting the problems they perceive and adequately structuring the problem. Now, the proposed measures appeared not to be the most optimal solutions for the problems as perceived by neurologists, and therefore represented right solutions to the wrong problems. This typically concerns what Hischemuller and Dunn have coined an ‘error of the third kind’ [6, 14]. We have found that on a first order level, all actors agreed that a national guideline and a registration could be meaningful. But as it was, on a second order level they disagreed as to what goals such policy measures should serve. For example, neurologists considered a national guideline relevant, as long it was a broad guideline on the diagnosis and treatment of MS in general. All actors agreed that a national registry could be relevant, but defined different goals. Each goal came with different variables to be included in the registry.

Thirdly, during the workshops, the proposed policy measures remained rather vague and were not elaborated in detail. As a result, participants could easily consent without violating their background theories and preferences. They had not felt the urge to (re)consider these theories and preferences. This, however, was necessary, as is clearly illustrated by the respondents who questioned whether participants would have come to agreement on specific recommendations, such as which data should be collected and who should become the owner of the registry. Such an agreement requires that policy measures are made sufficiently substantive for every actor to understand what are the consequences. Moreover, it requires that background theories and normative preferences are elicited and scrutinised.

Fourthly, changes in the socio-political context affected policy development. Opinions with respect to which kind of policy measures are the responsibility of the HCIB had changed over time. The PAM staff considered conducting a clinical trial to obtain relevant information that would not be provided by the industry. Simultaneously, their position towards another Dutch institute involved with subsidising clinical research changed. Putting out clinical trials was then no longer their responsibility. Furthermore, the relationship between the HCIB and the Ministry of Health changed over the course of this project. These changes strongly affected ideas with respect to the Board’s responsibility on initiating a national registry of patient data. On this basis we conclude that the workshops failed to meet the objectives set for interactive processes in policy development, namely to prevent policy implementation problems as a result of diverging views amongst target groups. Moreover, we concede that they could not have met those objectives. From the case of IFNβ, we infer that an interactive process should meet the following criteria:

1. The interaction should cover the whole process from problem structuring to policy implementation, in order to be able to deal with problem shifts and changes in the socio-political context. It is not enough to reduce interactivity to workshops at one or two moments in time.

2. Actors involved should resist the temptation to think that policy problems can be understood at a first order level. That is, background theories and normative preferences should be explicated in the problem definition phase. Only if problem structuring is taken seriously in this sense can one think of developing solutions that could meet with the approval of all target groups.

3. Developing widely endorsed solutions requires that actors involved are willing to learn from one another and adapt their views if necessary. A process of interactive policy development should include room for such learning processes [12, 24, 25, 29]. Grin and Van de Graaf [10] have argued that learning is likely to occur only if external events urge a revision of background theories and preferences, or if repeated failures show actors that their background theories are not functional.

4. Proposed policy measures should be sufficiently elaborated, as to enable target groups to assess their consequences and constructively engage in the interactive process. As the case of IFNβ shows, actors involved are not willing to reconsider their background theories and preferences, should proposed policy measures not be sufficiently elaborated and actors involved not understand what is at stake.

In sum, this study has shown that the organisation of workshops in which target populations participate does not qualify as an appropriate process of interactive policy development. From the beginning, emphasis had been put on a limited number of interventions aimed to control the expected increase in treatment costs. Although target populations participated in policy development, perceived problems and which interventions could provide a solution had been discussed insufficiently. For policy development to be successful, interactive methods are needed, in which problem definitions and assumptions are explicated and discussed, providing an opportunity for mutual learning between actors involved.