Skip to main content
PMC home page
Cost Effectiveness and Resource Allocation : C/E logoLink to Cost Effectiveness and Resource Allocation : C/E
. 2012 Feb 29;10:2. doi: 10.1186/1478-7547-10-2

Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa

Jacqui Miot 1,✉, Monika Wagner 2, Hanane Khoury 2, Donna Rindress 2, Mireille M Goetghebeur 2
PMCID: PMC3330006  PMID: 22376143

Abstract

Background

Systematic and transparent approaches to priority setting are needed, particularly in low-resource settings, to produce decisions that are sound and acceptable to stakeholders. The EVIDEM framework brings together Health Technology Assessment (HTA) and multi-criteria decision analysis (MCDA) by proposing a comprehensive set of decision criteria together with standardized processes to support decisionmaking. The objective of the study was to field test the framework for decisionmaking on a screening test by a private health plan in South Africa.

Methods

Liquid-based cytology (LBC) for cervical cancer screening was selected by the health plan for this field test. An HTA report structured by decision criterion (14 criteria organized in the MCDA matrix and 4 contextual criteria) was produced based on a literature review and input from the health plan. During workshop sessions, committee members 1) weighted each MCDA decision criterion to express their individual perspectives, and 2) to appraise LBC, assigned scores to each MCDA criterion on the basis of the by-criterion HTA report.

Committee members then considered the potential impacts of four contextual criteria on the use of LBC in the context of their health plan. Feedback on the framework and process was collected through discussion and from a questionnaire.

Results

For 9 of the MCDA matrix decision criteria, 89% or more of committee members thought they should always be considered in decisionmaking. Greatest weights were given to the criteria "Budget impact", "Cost-effectiveness" and "Completeness and consistency of reporting evidence". When appraising LBC for cervical cancer screening, the committee assigned the highest scores to "Relevance and validity of evidence" and "Disease severity". Combination of weights and scores yielded a mean MCDA value estimate of 46% (SD 7%) of the potential maximum value. Overall, the committee felt the framework brought greater clarity to the decisionmaking process and was easily adaptable to different types of health interventions.

Conclusions

The EVIDEM framework was easily adapted to evaluating a screening technology in South Africa, thereby broadening its applicability in healthcare decision making.

Background

In any country, healthcare resource allocation decisions are complex and involve assessment of the available scientific evidence, clarification of priorities, value judgments and ethical considerations [1,2]. In developing countries, which are generally low-resource settings, priority setting for healthcare becomes even more important. Not only are resources limited, but there are also other factors such as poor information, lack of policy, barriers to implementation, and political agendas, to name but a few [3]. The result is that priority setting is inconsistent and unstructured [1]. Transparent and explicit approaches to decisionmaking help produce decisions that are sound and acceptable to stakeholders [4-7].

In the healthcare sector of South Africa, current decisionmaking approaches are centered around evidence-based medicine, affordability and, where available, cost-effectiveness/costutility analysis (CEA/CUA) [8]. Increasingly CEA/CUA is being used for priority setting at all levels: the patient; the healthcare service; and within populations. This is evident in the field of HIV/AIDS where cost-effectiveness analyses of antiretroviral (ARV) medicines has enhanced access to treatment and reduced drug prices [9,10]. In many instances, a simple cost-minimization approach is all that is attempted. However, this approach has shortcomings, as there are a number of additional important dimensions, such as budget impact, equity, availability of alternatives, disease severity, etc. [11,12], that are not incorporated. Where these are taken into consideration, they are often assessed in an ad hoc manner and there is a lack of transparency as to how they impact the final decision [1].

Thus, there is a need for a process that supports consideration of all dimensions impacting a decision in a systematic and explicit fashion, and increases transparency and access to the evidence upon which decisions are based. Multi-criteria decision analysis (MCDA) is a tool to support complex decisionmaking which allows a structured, objective consideration of factors that are both measurable and value-based in an open and transparent manner [1,13,14].

While MCDA has been used historically in sectors such as transport or agriculture, there is a growing interest in using and applying the principles of MCDA, and similar approaches based on multiple decision criteria, to resource allocation decisionmaking in health care [15-21]. The EVIDEM framework has been developed to bring together Health Technology Assessment (HTA) and MCDA by proposing a comprehensive set of decision criteria together with standardized processes/methods to develop HTA reports that are structured on these criteria [22]. The aims of the framework are to facilitate concurrent consideration of multiple decision criteria, to stimulate reflection on priorities and values, and to promote transparency and communication within the decisionmaking committee as well as with outside stakeholders. The application of the EVIDEM framework is postulated to be wide ranging from decisionmaking by the healthcare provider, to coverage decisions by the funders or government policy setting.

A proof-of-concept study of the EVIDEM framework was performed in Canada involving a diverse panel of stakeholders appraising 10 medicines [23]. The core framework was also further developed to include standardized contextual criteria and tested for clinical decisionmaking by a panel of pediatric endocrinologists and other stakeholders who applied it to appraise growth hormone for children with Turner syndrome in Canada [24]. The framework was also tested as a support for drug formulary decisionmaking by a public healthcare payer in Canada [25].

The objective of the current study was to expand the scope of field testing both geographically and with respect to type of intervention: i.e., to field test the framework as a support for coverage decisionmaking on a cervical cancer screening test (liquid-based cytology, LBC) by a private health plan in South Africa.

The healthcare system in South Africa is dichotomous with a small (approximately 7 million lives) but resource-rich private healthcare sector and a growing (approximately 45 million lives) resource-scarce public sector. The private healthcare sector is largely funded by medical insurance companies (health plans), while the public sector is currently funded out-of-pocket or by the government. With respect to cervical cancer screening, the Department of Health guidelines stipulate that all women should be screened three times in their lifetime from the age of 30 years [26]. In the private healthcare sector, cervical cancer screening is often recommended and carried out on an annual basis. Screening uptake is low in South Africa with poor accessibility to healthcare facilities, poorly trained staff, long turnaround times between laboratory and healthcare facility, as well as lack of education cited as some of the reasons [27]. LBC for cervical cancer screening was selected by the health plan as a relevant case study due to its recent introduction as an alternative to conventional Pap smears.

Methods

Study design

The EVIDEM framework was field-tested in the private healthcare sector of South Africa with a clinical policy and decisionmaking committee of a major health plan. This panel of experts included doctors (specialists and general practitioners), pharmacists and nurses with at least 12 months experience in decision-making at a health policy level for the health plan. At the time of the study, the committee was already using evidence-based medicine, cost-effectiveness, affordability and some epidemiological principles in a multicriteria-based decisionmaking process.

The study design is presented in Figure 1. Based on a literature review and input from the clinical committee of the health plan, a structured HTA report on LBC for cervical cancer screening was produced and tailored to investigate each of the 14 MCDA decision criteria of the framework organized in the MCDA matrix [22]. (Note that the framework contains 15 MCDA decision criteria; however, the criterion "Adherence to requirements of decisionmaking body" was not considered for appraisal in this field test.) Four contextual criteria were proposed by the healthcare funder: "Impact on future decisions", "Relationship with pathology providers", "Impact on screening intervals" and "Patient expectation"; these criteria were appraised qualitatively. The contents of the report were tailored to reflect the local context (i.e., South African private health plan). Each committee member was familiarised with the LBC technology through prior evaluation for clinical decisionmaking and/or review of the HTA report.

Figure 1.

Figure 1

Open in a new tab

Study plan.

During workshop sessions, committee members assigned: 1) weights to each criterion of the MCDA matrix to express their perspectives; 2) scores for LBC for each criterion of the MCDA matrix based on data from the by-criterion HTA report; and 3) the qualitative impact of system-related criteria on the appraisal. The adoptability and utility of the EVIDEM approach were explored through a post-testing survey.

Health technology assessment report

Relevant data for each decision criterion of the framework was identified by searching PubMed, websites of HTA agencies (including the Canadian Agency for Drugs and Technologies in Health [CADTH], National Institute for Health and Clinical Excellence [NICE], US Preventative Services Task Force and Danish Centre for Evaluation and Health Technology Assessment) and the website of South Africa's Department of Health.

Data on the projected budget impact of LBC as well as local costing data was obtained from the health plan. The health plan also provided input for the four contextual criteria, which were appraised qualitatively. Collected data was extracted, analyzed and synthesized to produce a by-criterion HTA report.

The quality of the clinical evidence and the economic evaluation was assessed using previously developed instruments according to two quality criteria: "Completeness and consistency or reporting" and "Relevance and validity of evidence" [22]. The HTA report was entered into an interactive web prototype (Tikiwiki v2.2) that was hyperlinked to the full-text sources from which data was extracted. The web prototype was designed to allow online appraisal of the proposed intervention and collect feedback from the committee.

Field testing by the committee

To explore their individual values, committee members were asked to assign weights to each MCDA decision criterion on a scale of 1 to 5, where 5 represents the greatest importance to their decisionmaking. Weights were to be assigned from individual perspectives in the context of the health plan, independent of the intervention to be appraised.

All committee members were asked to familiarise themselves with the study intervention (e.g. clinical trials, health economic studies, disease characteristics and treatments etc.) using the EVIDEM Synthesised evidence report prior to carrying out the evaluation.

To appraise the proposed intervention, committee members were then asked to score (on a 4-point scale from 0 to 3) LBC for cervical cancer screening with respect to each MCDA criterion of decision, based on synthesized data from the by-criterion HTA report. Committee members then considered what type of impact each of the four qualitative criteria might have on use of LBC in the context of the health plan.

Feedback on the framework and process was elicited during discussion periods at the first workshop and at a follow-up workshop, and from a questionnaire administered during the follow-up workshop. Committee members were surveyed about whether each of the framework's decision criteria should always, sometimes or never be considered in decisionmaking. They were also surveyed about how the EVIDEM approach compared to their current process regarding data provided, deliberative process and communication of decision (improved, same or worse).

Data collection and analyses

Weights, scores and impact obtained from committee members were entered into Excel software. Descriptive statistics were used and mean ± standard deviations (SD) were calculated for weights and scores. The MCDA value estimate for the use of LBC in cervical cancer screening was then obtained by applying an MCDA linear additive model [22]. Thus, the value contribution of each decision criterion was calculated by multiplying its mean weight (normalized across the criteria to add up to 1 for each committee member) and score (standardized by division by the maximum score 3). The MCDA value estimate was then calculated as the sum of the value contributions of the 14 applicable criteria of the MCDA matrix.

Results

Decision criteria and committee values

When surveyed on whether each of the criteria of the framework should always, sometimes or never be considered in the decisionmaking process, all committee members agreed that "Budget impact on health plan" should always be considered (Table 1). For eight further criteria ("Completeness and consistency of reporting evidence", "Relevance and validity of evidence", "Disease severity", "Clinical guidelines", "Improvement of efficacy/effectiveness", "Type of medical service", "Cost-effectiveness" and "Appropriate use"), eight out of nine (89%) committee members felt that these should always be considered. Among the 15 MCDA matrix criteria, "Improvement of PRO, convenience & adherence" and "Adherence to requirements of decisionmaking body" received the lowest acceptance from committee members, with only 44% and 33%, respectively, indicating that these should always be considered. "Political context" received least approval among all criteria included in the survey, with only one (11%) committee member expressing the opinion that it should always be considered. No committee member indicated that any criterion should never be considered.

Table 1.

Committee member responses to whether specific decision criteria should be considered in decision making

Criteria Total number of responses Distribution of responses
Always Sometimes Never
MCDA matrix criteria
Quality of evidence
Adherence to requirements of decision making body 9 33 67 0
Completeness and consistency of reporting evidence 9 89 11 0
Relevance and validity of evidence 9 89 11 0
Disease impact
Disease severity 9 89 11 0
Size of population affected by disease 9 67 33 0
Intervention
Current clinical guidelines 9 89 11 0
Current interventions limitations 9 78 22 0
Improvement of efficacy/effectiveness 9 89 11 0
Improvement of safety & tolerability 9 78 22 0
Improvement of PRO, convenience & adherence 9 44 56 0
Public health interest (prevention & risk reduction) 9 78 22 0
Type of medical service 9 89 11 0
Economics
Budget impact on health plan 9 100 0 0
Cost-effectiveness of intervention 9 89 11 0
Impact on other spending 9 78 22 0
Other criteria
Appropriate use 9 89 11 0
Opportunity costs 9 78 22 0
Organizational structure 9 22 78 0
Stakeholder pressures 9 33 67 0
Political context 9 11 89 0
Population priorities and access 9 22 78 0
Regulatory status of intervention 9 56 44 0
Open in a new tab

Committee members were asked to weight each of the 14 MCDA decision criteria included in this field test according to their importance to the appraisal of an intervention, independently of the intervention that was to be appraised. (The criterion "Adherence to requirements of decisionmaking body" was not considered for appraisal in this field test.) The greatest weights (mean 4.7, on a scale of 1 to 5) were assigned to the criteria "Budget impact on health plan", "Cost-effectiveness of intervention" and "Completeness and consistency of reporting evidence", followed by "Improvement of efficacy/effectiveness" and "Relevance and validity of evidence" (mean 4.4) (Figure 2). The lowest weights were given to "Improvement of safety and tolerability" (mean 3.9) and "Public health interest" (mean 3.8). Weights for "Budget impact on health plan", "Cost-effectiveness of intervention" and "Completeness and consistency of reporting evidence" showed the least variation among committee members (SD 0.5). The largest divergence of weights was recorded for the criteria "Size of population affected" (SD 1.0) and "Public health interest" (SD 1.0).

Figure 2.

Figure 2

Open in a new tab

Mean weights assigned to each decision criterion of the MCDA matrix by committee members. A five point weighting scale was used with 1 for lowest weight and 5 for highest weight.

Appraisal of liquid based cytology for cervical cancer screening

By-criterion Health Technology Assessment report

The HTA report for LBC was based on 14 references (Table 2, 'lite' version). (The full report is available online at http://www.evidem.org/tiki/?page=CCLBC-RecordMenu). Key items included in the assessment are reported here. In the context of cervical cancer screening, LBC refers to a technique in which the cervical sample is suspended into a preservative liquid, rather than, as in conventional cytology (CC, Pap test), being smeared onto a slide. Cervical cancer causes high morbidity and mortality [28-30]. With annual incidence rates of 9 per 100,000 among white and 40 per 100,000 among black women [31], cervical cancer is the second most common cause of cancer morbidity and mortality in South African women [32-34]. Current South African guidelines recommend screening with at least three cervical Pap smears (CC) per lifetime to detect precancerous lesions; they do not mention LBC [26]. In addition to the occurrence of unsatisfactory samples, the chief limitations of CC are its relatively low specificity and sensitivity and its low uptake by the population [27,35,36]. There is considerable public health interest in improving screening methods and uptake to reduce the high burden of cervical cancer in South Africa [32,33]. As a screening tool, LBC allows early detection which, with appropriate treatment, improves outcomes [30]. A meta-analysis of RCTs found that LBC had 6.4% higher sensitivity but 4.0% lower specificity than CC, coupled with a slight reduction in the percentage of unsatisfactory samples [35]. The most relevant outcomes, impact on morbidity and mortality due to cervical cancer, have not been measured. The incremental budget impact of LBC is estimated to be ZAR76.90 per patients screened and the incremental colposcopy cost per patient lifetime ZAR8 [36,37]. The incremental cost per QALY gained was projected to be ZAR758,000 with a cost of ZAR764,000 per life-year gained [35]. However, since these data are based on a Canadian economic evaluation model they may have limited relevance to the South African setting.

Table 2.

HTA report with data for each decision criterion of the EVIDEM framework ('Lite' highly synthesized version)*

Overview
Disease: Type of intervention: Cervical cytology test
Liquid Based Cytology (LBC)
Intervention: Indication: Screening for cervical cancer in women
Cervical cancer
Setting: Administration: Liquid based cytology (LBC) requires physician to obtain
Discovery Health, South Africa cells from the cervix and to place the head of the spatula/brush or rinse it into a vial with preservative fluid. The pathologist then extracts the cells into a microscopy slide and the cells are examined in the usual way
Comparator(s): Conventional pap smears (i.e., conventional cytology, CC) where the brush containing the sample is smeared onto the slide, thereby transferring the cells
Data included:Data available from public domain
MCDA matrix criteria Highly synthesized information Scoring of intervention
Disease impact Not severe Very severe
D1 Disease severity Disease symptomatic only when spreading from cervix: vaginal bleeding, post-coital spotting, vaginal discharge, pelvic or low back pain. 0 1 2 3
Late stages: severe anemia, weight loss and uncontrolled release of urine and feces through the vagina
Survival: < 18 months in 50% of untreated patients
D2 Size of population Incidence in South Africa: 9/100,000 white women; 40/100,000 black women Very rare disease Very common disease
0 1 2 3
Intervention
I1 Clinical guidelines LBC is not included in any guidelines.South African guidelines recommend at least 3 pap smears test per lifetime Not recommended Strong recommendation
0 1 2 3
I2 Comparative interventions limitations Sensitivity: 74% - relatively low; Specificity: 87%;Unsatisfactory sample: 3%Low screening uptake: 2.8% black women; 18.8% white women No or very minor limitations Major limitations
0 1 2 3
I3 Improvement of efficacy/effectiveness Meta-analyses of RCTs (n = 28,736 vs 39,377): Sensitivity: 80% - difference with CC = 6.4% (95% CI: -6.5 to 18.8%)Specificity: 82% - difference with CC = - 4.0% (95% CI: -19.9 to 10.6%) Unsatisfactory samples: ThinPrep = 2.2% and Sure Path = 0.82%; difference with CC: ThinPrep = -0.8% and Sure Path = -2.5% Lower than comparators Major improvement
0 1 2 3
I4 Improvement of safety & tolerability Safety and tolerability do not differ between LBC and conventional cytology Lower than comparators Major improvement
0 1 2 3
I5 Improvement of patient reported outcomes Patient reported outcomes: No data - fewer recalls and fewer inadequate specimens with LBC may improve quality of lifeConvenience: no need to return for HPV testing in case of a positive result Lower than comparators Major improvement
0 1 2 3
I6 Public health interest Screening programs for cervical cancer in South Africa: mortality decreased by 50% for white women, 40% for Asian women but rose for black women between 1960's 1990's. and Considerable interest to improve screening methods and uptake. No risk reduction Major risk reduction
0 1 2 3
I7 Type of medical service Goal of intervention: improve outcome of the disease due to early detection; 5-years survival ranging between close to 100% for early stage and 5% to 15% for late stage Minor Service Major Service (e.g. cure)
Economics
E1 Budget impact on health plan Intervention price: LBC - R180.00; CC - R103.10Incremental cost per pt screened with LBC: R76.90Annual projected budget impact per 1000 women: R76,900 Substantial additional spending Substantial savings
0 1 2 3
E2 Cost-effectiveness of intervention Cost per life-year gained: R764,000Cost per QALY gained: R758,000 Not cost-effective Highly cost-effective
0 1 2 3
E3 Impact on other spending Incremental colposcopy cost per patient over lifetime horizon: R 8
(excludes drug cost, see E1)No available data on other spending (including treatment of cancer and pre-cancer)		 Substantial additional spending Substantial Savings
0 1 2 3
Quality of evidence
Q1 Adherence to requirements of decisionmaking body Not applicable for case study Low adherence High adherence
Q2 Completeness and consistency of reporting evidence Quality score: Clinical data:75% - primary and secondary outcome measures as well as sensitivity analyses not clearly specified; Economic evaluation: 50% - disaggregated cost not reported; incomplete reporting of effectiveness outcomes Many gaps/inconsistent Complete & consistent
0 1 2 3
Q3 Relevance and validity of evidence Quality score: Clinical data:75% -most relevant outcome not assessed (morbidity and mortality due to cervical cancer);Economic evaluation: 75% - Canadian screening coverage and HPV epidemiology not completely applicable for South African private payer setting Low relevance/validity High relevance/validity
0 1 2 3
Open in a new tab

* The full HTA report is available online at http://www.evidem.org/tiki/?page=CCLBC-RecordMenu

Quantitative considerations - scores

Using synthesized data from the HTA report in the MCDA matrix (Table 1), committee members assigned scores to appraise LBC for each criterion (Figure 3). The highest scores were assigned to the criteria "Relevance and validity of evidence" (mean 2.5, SD 0.3, on a scale of 0 to 3) and "Disease severity" (mean 2.2, SD 0.4). The lowest scores were given to "Cost-effectiveness of intervention" (mean 0.3, SD 0.7), "Improvement of safety & tolerability" (mean 1.0, SD 0.0) and "Completeness and consistency of reporting evidence" (mean 1.0, SD 0.0). The committee expressed unanimous agreement (SD 0.0) when scoring the criteria "Improvement of safety & tolerability" (mean 1.0), "Completeness and consistency of reporting evidence" (mean 1.0), and "Type of medical service" (mean 2.0). In contrast, 3-point differences across the full scoring scale (0 to 3) were seen for the criteria "Clinical guidelines" (mean 2.0, SD 1.2) and "Public health interest" (mean 2.0, SD 0.9).

Figure 3.

Figure 3

Open in a new tab

Mean scores assigned to each decision criterion of the MCDA matrix by committee members for the appraisal of liquid based cytology. A four point scoring scale was used with 0 for lowest score and 3 for highest score.

MCDA value estimate

Combination of the committee's weights and scores yielded a mean MCDA value estimate for using LBC in cervical cancer screening of 46% (SD 7%) of the potential maximum value (Figure 4). Individual MCDA value estimates ranged from 36% to 61% among the nine committee members; 7 out of 9 estimates were in the range between 39% and 48%. The criterion "Relevance and validity of evidence" made the greatest contribution to the mean MCDA value estimate (13% of the total), followed by "Disease severity", "Clinical guidelines" and "Type of medical service" (11% each). The criterion "Cost-effectiveness" made the smallest contribution (2%) to the MCDA value estimate.

Figure 4.

Figure 4

Open in a new tab

MCDA value estimate of liquid based cytology for cervical cancer screening based on weights and scores assigned by committee members. Weights were normalized across the 14 criteria and scores are presented on a scale of 0 to 1. *MCDA estimate was obtained using a linear model combining normalized weights and scores for each decision criteria. For an intervention to achieve close to 1 on this scale, it would have to cure an endemic disease, demonstrate major improvement in safety, efficacy, and patient reported outcomes compared to limited existing approaches and result in major healthcare savings.

Qualitative considerations - impacts

Background information for the four contextual criteria proposed by the health plan was collected in the context of the health plan and provided to the committee to stimulate reflection and discussion (Table 3). Committee members considered that a decision to cover LBC may have a negative impact on future decisions (setting a precedent). When considering relationships with pathology providers, covering LBC was considered to have a positive impact. The possibility of extending screening intervals would also have a positive impact on the value of LBC. When considering patient expectations, committee members considered that patients would likely resist if co-payments were implemented for LBC.

Table 3.

Qualitative considerations and their impact on the appraisal of LBC by committee members

Contextual criteria* Should this be considered?
Would it impact positively or negatively on value of intervention?
Impact on future decisions Decision to fund LBC coverage although not considered cost-effective will be setting a precedent as LBC technology will be expanded to other cytology tests Considered as negative impact on value
Relationship with pathology providers Ongoing negotiations with the pathology groups in other areas may be impacted Considered as a positive impact on value
Impact on screening intervals Screening interval may be extended to 2 to 3 years than annually Considered as a positive impact on value
Patient expectation Health plan members expect that their pap smears will be paid in full and are likely to resist any benefit design which implements a co-payment for a more expensive technology Considered as a negative impact on value
Open in a new tab

*Contextual criteria were proposed by the healthcare funder. Specific contextual criteria may be defined on the basis of the following generic criteria/themes: mission and scope of the healthcare system, opportunity costs; population priorities & access; system capacity and appropriate use; political context; and stakeholder pressures.

Feedback from committee on overall approach

A feedback questionnaire was completed by the committee members following the evaluation. The elements of the questionnaire were pre-determined and part of the standardised set of tools available for the EVIDEM process. When surveyed on how the EVIDEM approach compared to existing process that committee members were familiar with, 50% of the committee members felt that it improved understanding of the intervention being appraised, access to quality assessment of the evidence on the intervention, and consideration of all key elements of the decision (Table 4). Fifty-six percent of committee members felt that the framework improved the transparency of the decision and the communication of the decision to stakeholders. No committee member rated the EVIDEM process as "worse". Concern was raised regarding the time and resources required to generate the HTA report, but on the whole, the committee felt it was a positive step forward in improving decisionmaking by bringing greater clarity to the decisionmaking process. The committee also thought that the approach could easily be applied to evaluation of other health interventions such as medicines and devices.

Table 4.

Feedback from committee members on the EVIDEM process compared to current approaches

Component Total number of responses Distribution responses (%)
Improved Same Worse
Intervention under scrutiny
Understanding of intervention 8 50 50 0
Access to evidence on intervention 8 25 75 0
Access to quality assessment of evidence on intervention 8 50 50 0
Deliberative process
Considering all key elements of decision 8 50 50 0
Expressing personal/expert opinion 9 11 89 0
Sharing & discussing values among committee members 9 22 78 0
Communication of decision
Transparency of decision 9 56 44 0
Understandability of decision by stakeholders 9 56 44 0
Acceptability of decision by stakeholders 7 43 57 0
Open in a new tab

The EVIDEM process for LBC resulted in a consideration by the health plan to only fund for LBC up to the value of conventional pap smears. A negotiation process was started with the pathology laboratories to review their tariffs for this diagnostic. The fee for LBC was reduced to an amount which was considered appropriate for full funding.

Discussion

The EVIDEM framework can be applied to assess the value of a screening intervention in the South African private health plan context, providing a practical tool for integrating HTA with an MCDA approach to support decisionmaking.

The use of the by-criterion HTA report allowed the clinical decisionmaking panel to assess all the criteria easily and from a single source. Where greater detail was required, hyperlinks embedded in the web-prototype provided access to the more detailed synthesized information as well as full source data. Available clinical evidence was fairly relevant and valid. Partial reporting of disaggregated data in the published economic evaluation limited its usefulness in the South African context although local costs were used to improve transferability. The fact that the committee gave the highest weighting to "Budget impact on health plan", "Cost-effectiveness of intervention" and "Completeness and consistency of reporting evidence" reflects its focus on issues of affordability and cost-effectiveness as well as quality of clinical evidence. This was also reflected by the low score (mean 0.3) the committee assigned to the cost-effectiveness of LBC (Figure 3), given an ICER of ZAR764,000 per QALY.

The MCDA estimate resulting from a combination of weights and scores, provided a comprehensive measure of the perceived value of LBC in the current context, i.e., relative to existing technology (conventional cytology), reflecting minor improvement at a significant cost, and also captured the importance of absolute elements of value such as disease severity, type of medical service and quality of evidence.

Non-quantifiable, contextual criteria of decision were also identified and considered. Given the lack of cost-effectiveness and high additional cost of LBC compared to conventional Pap smears, the budget impact would be substantial and therefore a co-payment would be considered. However, health plan members and many other South Africans are used to receiving Pap smears for free as part of health screening programmes and the implementation of co-payments would meet with strong resistance from both patients and healthcare providers. The use of the MCDA framework as a communication device to convey the structure and transparency of the decisionmaking process was thought to be an important tool to improve acceptability of the committee's decision.

Adaptation of the EVIDEM framework from medicines evaluation to a screening technology required minimal change to the structure of the framework. Since devices, procedures and technologies form a growing portion of the basket of healthcare interventions that require evaluation it was useful to assess the adaptability of the framework to these types of interventions.

Currently, there is no single HTA body (such as NICE or CADTH) in South Africa with the result that evaluations of new and existing health interventions are scattered and often duplicated. Skilled resources to carry out a proper HTA are scarce and a need is recognized for pooling these resources, allowing greater availability to all sectors in the country [38,39]. Health technology assessments and reports are freely available from a number of sources such as NICE (UK), CADTH (Canada), AHRQ (USA), IQWIG (Germany), etc., as well as through subscription to knowledge bases such as the Cochrane Collaboration. However, none of these reports offers the opportunity to rationally and transparently measure the impact of the various criteria on the final clinical decision. This is left to clinical decisionmaking members who are intrinsically subjective based on their own priorities and knowledge. Using MCDA in healthcare decisionmaking adopts an existing, well established methodology based on sound, scientific principles [40].

On the whole, the committee members favoured the use of the EVIDEM process and were positive about its use. Some committee members rated their feedback of the EVIDEM process as "same" as existing process in the questionnaire, which may be explained by the fact that a more crude and simpler multicriteria approach was already being used by the committee.

This field test indicates that the EVIDEM framework may be useful for the evaluation of health technologies in the South African private healthcare sector context. However, further adaptation to a health plan's specific mission, scope and priorities is required to create a "custom-made" decisionmaking framework fully aligned with the local context. The framework's contextual tool ("extrinsic value tool") -- a result of a further development of this framework [24] -- facilitates this customization by providing generic criteria/themes on the basis of which contextual criteria can be identified, and if so desired, integrated into the MCDA model. Further development and validation of the framework's weighting and scoring methodologies is also currently underway.

Conclusions

The EVIDEM framework was easily adapted to evaluating a screening technology thereby broadening its applicability in healthcare evaluation. It was found to improve: understanding of intervention; access to quality assessment of evidence; consideration of key elements of decision; transparency of decision; and understandability of decision by stakeholders. Further field testing and instrument validation and development are ongoing to collaboratively advance MCDA approaches and contribute to more transparent and efficient healthcare decisionmaking.

Abbreviations

AEs: Adverse events; ANOVA: Analysis of variance; CE: Cost-effectiveness; EVIDEM: Evidence and Value: Impact on DEcisionMaking; HTA: Health technology assessment; ICC: Intra-rater correlation coefficients; INAHTA: International Network of Agencies for Heath Technology Assessment; MCDA: Multicriteria decision analysis; PRO: Patient reported outcomes; QALY: Quality-adjusted life-year; QM: Quality Matrix; RCT: Randomized controlled trial; SD: Standard deviation.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

JKM, MMG, and MW designed the study and HK collected and analysed the data. JKM, MMG and MW drafted the manuscript. DR reviewed the manuscript and is one of the developers of the EVIDEM framework. All authors read and approved the final manuscript.

Contributor Information

Jacqui Miot, Email: jacqui.miot@gmail.com.

Monika Wagner, Email: monika_wagner@biomedcom.org.

Hanane Khoury, Email: hanane_khoury@biomedcom.org.

Donna Rindress, Email: donna_rindress@biomedcom.org.

Mireille M Goetghebeur, Email: mireille_goetghebeur@biomedcom.org.

Acknowledgements

We wish to acknowledge the contributions of the members of the Clinical Policy Unit at Discovery Health who participated in this beta-testing. Internal sources of support for the study were provided by Discovery Health and BioMedCom Consultants inc.

References

  1. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006;4:14. doi: 10.1186/1478-7547-4-14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Tunis SR. Reflections on science, judgment, and value in evidence-based decision making: a conversation with David Eddy. Health Aff (Millwood) 2007;26:w500–w515. doi: 10.1377/hlthaff.26.4.w500. [DOI] [PubMed] [Google Scholar]
  3. Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health. 2009;14:930–939. doi: 10.1111/j.1365-3156.2009.02311.x. [DOI] [PubMed] [Google Scholar]
  4. Bryan S, Williams I, McIver S. Seeing the NICE side of cost-effectiveness analysis: a qualitative investigation of the use of CEA in NICE technology appraisals. Health Econ. 2007;16:179–193. doi: 10.1002/hec.1133. [DOI] [PubMed] [Google Scholar]
  5. Dhalla I, Laupacis A. Moving from opacity to transparency in pharmaceutical policy. CMAJ. 2008;178:428–431. doi: 10.1503/cmaj.070799. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Vuorenkoski L, Toiviainen H, Hemminki E. Decision-making in priority setting for medicines-a review of empirical studies. Health Policy. 2008;86:1–9. doi: 10.1016/j.healthpol.2007.09.007. [DOI] [PubMed] [Google Scholar]
  7. Daniels N, Sabin J. Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philos Public Aff. 1997;26:303–350. doi: 10.1111/j.1088-4963.1997.tb00082.x. [DOI] [PubMed] [Google Scholar]
  8. Doherty J. Cost-effectiveness analysis for priority setting in South Africa - what are the possibilities? S Afr Med J. 2010;100:816–821. doi: 10.7196/samj.4440. [DOI] [PubMed] [Google Scholar]
  9. Cleary SM, McIntyre D, Boulle AM. The cost-effectiveness of antiretroviral treatment in Khayelitsha, South Africa-a primary data analysis. Cost Eff Resour Alloc. 2006;4:20. doi: 10.1186/1478-7547-4-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cleary SM, McIntyre D, Boulle AM. Assessing efficiency and costs of scaling up HIV treatment. AIDS. 2008;22(Suppl 1):S35–S42. doi: 10.1097/01.aids.0000327621.24232.71. [DOI] [PubMed] [Google Scholar]
  11. Camidge DR, Oliver JJ, Skinner C, Attwood B, Nussey F, Jodrell D. et al. The impact of prognosis without treatment on doctors' and patients' resource allocation decisions and its relevance to new drug recommendation processes. Br J Clin Pharmacol. 2008;65:224–229. doi: 10.1111/j.1365-2125.2007.02996.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Schlander M. The use of cost-effectiveness by the National Institute for Health and Clinical Excellence (NICE): no(t yet an) exemplar of a deliberative process. J Med Ethics. 2008;34:534–539. doi: 10.1136/jme.2007.021683. [DOI] [PubMed] [Google Scholar]
  13. Baltussen R, Youngkong S, Paolucci F, Niessen L. Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences. Health Policy. 2010;86:262–264. doi: 10.1016/j.healthpol.2010.01.009. [DOI] [PubMed] [Google Scholar]
  14. Dolan JG. Multi-Criteria clinical decision support. A primer on the use of multiple-criteria decision-making methods to promote evidence-based, patient-centered healthcare. Patient. 2010;3:229–248. doi: 10.2165/11539470-000000000-00000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Browman GP, Manns B, Hagen N, Chambers CR, Simon A, Sinclair S. 6-STEPPPs: A modular tool to facilitate clinician participation in fair decisions for funding new cancer drugs. J Oncol Practice. 2008;4:2–7. doi: 10.1200/JOP.0812001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Martin DK, Pater JL, Singer PA. Priority-setting decisions for new cancer drugs: a qualitative case study. Lancet. 2001;358:1676–1681. doi: 10.1016/S0140-6736(01)06714-9. [DOI] [PubMed] [Google Scholar]
  17. Mullen PM. Quantifying priorities in healthcare: transparency or illusion? Health Serv Manage Res. 2004;17:47–58. doi: 10.1258/095148404322772723. [DOI] [PubMed] [Google Scholar]
  18. Wilson E, Sussex J, Macleod C, Fordham R. Prioritizing health technologies in a Primary Care Trust. J Health Serv Res Policy. 2007;12:80–85. doi: 10.1258/135581907780279495. [DOI] [PubMed] [Google Scholar]
  19. Wilson EC, Peacock SJ, Ruta D. Priority setting in practice: what is the best way to compare costs and benefits? Health Econ. 2009;18:467–478. doi: 10.1002/hec.1380. [DOI] [PubMed] [Google Scholar]
  20. Baltussen R, Ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan. 2007;22:178–185. doi: 10.1093/heapol/czm010. [DOI] [PubMed] [Google Scholar]
  21. Nobre FF, Trotta LT, Gomes LF. Multi-criteria decision making-an approach to setting priorities in health care. Stat Med. 1999;18:3345–3354. doi: 10.1002/(SICI)1097-0258(19991215)18:23&#x0003c;3345::AID-SIM321&#x0003e;3.0.CO;2-7. [DOI] [PubMed] [Google Scholar]
  22. Goetghebeur M, Wagner M, Khoury H, Levitt R, Erickson LJ, Rindress D. Evidence and Value: Impact on DEcisionMaking - the EVIDEM framework and potential applications. BMC Health Services Research. 2008;8:270. doi: 10.1186/1472-6963-8-270. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): Applying the EVIDEM framework to medicines appraisal (ahead of print) Med Decis Making. 2011. in press . [DOI] [PubMed]
  24. Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decisionmaking framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc. 2010;8:4. doi: 10.1186/1478-7547-8-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P. et al. Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res. 2011;11:329. doi: 10.1186/1472-6963-11-329. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Department of Health, South Africa. National guideline on: cervical cancer screening programme. http://www.doh.gov.za/docs/factsheets/guidelines/cancer.pdf
  27. Cronje HS, Beyer E. Screening for cervical cancer in an African setting. Int J Gynaecol Obstet. 2007;98:168–171. doi: 10.1016/j.ijgo.2007.05.005. [DOI] [PubMed] [Google Scholar]
  28. Alliance for Cervical Cancer Prevention. Planning and implementing cervical cancer prevention and control programs: a manual for managers. Chapter 10: Cancer treatment and palliative care. http://screening.iarc.fr/planningmanual.php?lang=1&PHPSESSID=6c8ded0f31c8336a3ed5e 9f6a4d59e63
  29. Adriano E, Jagoe JM, Harrison T, Riffenburgh RH, Johnstone PA. Survival of patients with untreated cervical carcinoma. Am J Clin Oncol. 2003;26:369–373. doi: 10.1097/01.COC.0000026909.73608.88. [DOI] [PubMed] [Google Scholar]
  30. Waggoner SE. Cervical cancer. Lancet. 2003;361:2217–2225. doi: 10.1016/S0140-6736(03)13778-6. [DOI] [PubMed] [Google Scholar]
  31. Department of Health. Statistical Notes. Focus of the Month: Cervical Cancer. http://www.doh.gov.za/docs/facts/2006/countryhealth2006.pdf
  32. Franco EL, Duarte-Franco E, Ferenczy A. Cervical cancer: epidemiology, prevention and the role of human papillomavirus infection. CMAJ. 2001;164:1017–1025. [PMC free article] [PubMed] [Google Scholar]
  33. Bailie RS, Selvey CE, Bourne D, Bradshaw D. Trends in cervical cancer mortality in South Africa. Int J Epidemiol. 1996;25:488–493. doi: 10.1093/ije/25.3.488. [DOI] [PubMed] [Google Scholar]
  34. Department of Health. Research update 2003. http://www.doh.gov.za/docs/reports/2004/nwhsr.pdf
  35. Canadian Agency for Drugs and Technologies in Health. Liquid-based techniques for cervical cancer screening: systematic review and cost-effectiveness analysis. Issue 103. http://cadth.ca/index.php/en/publication/799
  36. Anderson N, Discovery Health: Internal Report prepared for Discovery Health. Summary of liquid-based cytology. 2008.
  37. Discovery Health: Internal Report prepared for Discovery Health. Pap smear analysis updated 02. 2008.
  38. Parrish A, Blockman M. Clinical excellence and the NICEties of value-based priority setting. S Afr Med J. 2008;98:758–761. [PubMed] [Google Scholar]
  39. Kenyon C, Ford N, Boulle A. When best practice is bad medicine: a new approach to rationing tertiary health services in South Africa. S Afr Med J. 2008;98:350. 352, 354. [PubMed] [Google Scholar]
  40. Nutt DJ, King LA, Phillips LD. Drug harms in the UK: a multicriteria decision analysis. Lancet. 2010;376:1558–1565. doi: 10.1016/S0140-6736(10)61462-6. [DOI] [PubMed] [Google Scholar]

Articles from Cost Effectiveness and Resource Allocation : C/E are provided here courtesy of BMC

RESOURCES