Prioritization of capacities for the elimination of dog-mediated human rabies in the Americas: building the framework

Abstract

The region of the Americas pledged to eliminate dog-transmitted human rabies by 2015. After 30 years of sustained efforts, regional elimination appears possible as dog-mediated human rabies cases are at an all-time low, and a number of countries and territories have already eliminated the disease. In this setting, there is an opportunity to generate a framework to support countries strategies in the achievement and maintenance of rabies-free status (RFS). To this end, we describe the development of a multi-criteria decision analysis (MCDA) model to help the evaluation of rabies programmes and the identification of the best investment strategy for countries and territories to improve and efficiently maintain their rabies status. The model contemplates human and animal related capacities, six in each area, to comprehensively assess the wide scope of rabies programmes. An initial elicitation of expert opinion of values and weights for the MCDA model was performed via a web-based questionnaire. Even at this pilot stage, the model produces comparable capacity-scores, and overall (combined for public and animal health areas) as well as area-specific investment strategies. The model is being developed by the Pan American Health Organization (PAHO) as part of the regional efforts towards dog-mediated human rabies elimination and will be presented to the countries for review, refinement, contextualization, and testing. The aspiration is that countries use the model to identify the best allocation of resources towards the elimination of dog-mediated human rabies.

Introduction

The regional elimination of human rabies transmitted by dogs is a 30-year old goal of the Americas. Regional efforts started in 1983 coordinated by the Pan American Health Organization (PAHO) and have led to the elimination of the disease in a number of countries and territories.¹ The commitment to regional elimination by 2105 was reiterated by the countries at the 19th Session of PAHO Directing Council through CD49.R19 Resolution referring to the elimination of neglected diseases and other poverty related infections,² as well as at the 16th Inter-American Meeting at Ministerial Level on Health and Agriculture (RIMSA 16) held in Santiago, Chile, in July 2012.³ This commitment was restated at the 14th meeting of rabies programme directors (REDIPRA) held in Lima (Peru) in the summer of 2013.⁴ Among other recommendations targeted at the regional elimination of dog-mediated human rabies, programme directors highlighted the strategic relevance of a dog-mediated human rabies free declaration of countries and territories as a mechanism to motivate political support for rabies control, and agreed on the added value that a regional free declaration framework managed and championed by PAHO would bring to national schemes and certifications. To ensure regional consistency, programme managers identified the need for regional standards for a rabies-free declaration, developed in a collaborative manner by a dedicated working group to that effect, and embedded within a quantitative model to allow comparisons, trend analyses, and the identification of optimal investment strategies for rabies control. Addressing this latter requirement would add to the existing national schemes that mostly focus on epidemiological indicators. The demand by programme directors for optimal investment portfolio approaches is a consequence of tighter resources, alternative competing health priorities, and the diminishing relevance of dog-transmitted rabies as a public health problem in the region, and follows the increasing public demand for greater transparency and accountability in the use of government budgets.⁵ Quantification of the capacities of a health programme, via the development of partial or capacity-specific scores that can be aggregated to a country-level score is not common, and to the best of our knowledge, novel for rabies. In what follows, we describe the development of the model which aims to evaluate national rabies programme capacities. The work is still ongoing and hence results are presented purely for illustrative purposes. Refinements to the current theoretical framework and extensions to the work are suggested in the discussion.

Materials and Methods

National programmes for the efficient control of dog-transmitted human rabies require the development of capacities at the animal and human level, independent of the agents or stakeholders involved in their delivery. For this exercise, we considered the following essential capacities relating to public health: risk analysis, pre-exposure prophylaxis, post-exposure prophylaxis, case control activities, education, and surveillance. Relating to animal health, the capacities considered were: risk analysis, vaccination, dog population management, import controls, surveillance, and case control activities. Furthermore, we defined three levels of performance or investment options within each capacity (Fig. 1). The complexity of deciding the best rabies investment strategy, given competing objectives, possible alternate scenarios, and their combinations, is obvious. Decision analysis techniques are well suited to help decision making in these situations.

Figure 1.

Levels of performance (n  =  3) within the capacities (n  =  6) for each area, public and animal health.

One decision analysis technique is multi-criteria decision analysis (MCDA), as used here. Multi-criteria decision analysis is widely used in multifaceted problems in several disciplines, permits integration of several streams of information, and gives decision-makers an insight into the values that underlie choices leading to more transparent and rational decisions.⁶ Multi-criteria decision analysis models were used in this setting to (i) develop a model to allow scoring of rabies capacities, and (ii) to inform the best investment strategy leading to overall rabies capacity improvements in a country or territory.

Multi-criteria decision analysis model building generally comprises of eight stages: definition of objectives, definition of criteria, identification of alternatives or options, identification of experts, evaluation of alternatives, definition of weights, prioritization of alternatives, and aggregation of results. The first four steps contribute to problem structuring, and it is normally conducted at expert workshops. However, for this pilot study, problem structuring was based on previous conceptualizations conducted by different countries and within PAHO, and did not involve expert elicitation at dedicated workshops. The main objective was defined as the elimination of dog-mediated human rabies or the achievement of a rabies-free status (RFS). Second-level objectives or means objectives were then defined, and from these, the criteria against which the levels of performance within the capacities will be evaluated upon. Figure 2 shows the value tree for the achievement of RFS.

Figure 2.

Value tree for the achievement of dog-mediated human RFS. End objective is shown at the top of the tree. Means objectives constitute the second hierarchy. Criteria are at the bottom of the tree.

Evaluation of options and elicitation of weights were conducted by means of an online questionnaire to a reduced group of experts (rabies country programme managers in the region) and PAHO staff with rabies responsibilities. Online elicitation was chosen because of the need to develop a sustainable mechanism that allowed regular updating of the models and evaluation of the rabies programmes, without the need for face-to-face meetings. Two questionnaires were developed, one for the public health component and one for the animal health component, with complete definitions for the four criteria (impact, time, public opinion, and cost) and for the levels of performance within each capacity. Questionnaires are available on request to the corresponding author.*

Experts and PAHO staff scored the competing options, i.e. the levels of performance in our setting, against the four criteria by a direct rating technique.⁷ The scores provided by this exercise represent the relative improvement of moving between levels, against each criterion. Next, an indirect ranking procedure, considered easier for the online elicitation, the rank order centroid (ROC) technique,⁸ allowed the elicitation of capacities and criteria weights from the experts. For capacities, experts ranked the maximum improvements or swings of every capacity, i.e. from the lowest to the highest level of performance, against each criterion. Relative weights w_j are then assigned according to the capacity rank position. Relative weights for the criteria were also derived from the ranking of the preferred swings.

As capacities for the two areas, public and animal health, were assessed separately, purely to facilitate response by experts, they need to be merged to allow the comprehensive evaluation of rabies programmes. The use of the same criteria for their evaluation permits this. To this end, experts were asked to weigh the two areas by first allocating a score of 100 to the most important area, and then score the second area relative to the first.

The final stage in model building was the calculation of the overall score by means of a linear additive, multi-criteria value function.⁹ For this calculation, the criteria ‘Impact’ and ‘Public Opinion’ were considered as benefits, i.e. higher scores against these two criteria result in better effects, whereas ‘Cost’ and ‘Time’ had a negative effect and consequently lower scores were preferred. The overall value score for the level of performance y_i, given m decision criteria, n capacities, and p areas, is given by the weighted benefit assessed against each criterion as follows:

(1)

where, b_i is the overall score of the level of performance i; a_k is the weight of the area k for k = 1…p (in this case p = 2) and ; c_j is the weight of criterion j for j = 1…m (in this case m = 4) and ; w_l is the weight of capacity l, where l = 1…n (in this case n = 6) and ; and v(y_i,j,l) is the partial value of the level of performance i in capacity l with respect to criterion j.

Each function v(⋅) is a single-dimensional measurable value function and its value derives from the scoring procedure explained above. Each improvement between levels of performance is assessed by b_i, which will be used to prioritize investment options. The optimal investment strategy is found by simply ranking improvements. It is worth noticing that overall scores with negative sign may occur as some improvements may bear large costs (due to large scores against the criteria ‘Time’ and ‘Cost’) and exceed the benefits (represented by the criteria ‘Impact’ and ‘Public opinion’). Options with a negative value would suggest that investment in that capacity would not be worthy.

Each expert contributing to the model will produce an individual rank order of investment options. However, we seek aggregation of multiple expert opinions. Combination of responses from a diverse pool of experts is no trivial task. Two approaches to expert opinion aggregation prevail in the literature: averaging and pooling (see Ref. 10 for a comparison of methods). Here we proceeded with the simplest approach and average the un-weighted scores from each expert. This is equivalent to creating a new set of responses for an average expert on which weights were applied, as described above, to create a new ranking.

Results

Three PAHO staff members contributed to both the public and animal health questionnaires. Two additional experts, responsible for national rabies programmes in the region, also completed the animal health questionnaire. We present results for this area only, purely for illustrative purposes. Seven more experts contributed answers to the questionnaires, two for animal health and five for public health, but they were incomplete and could not be used for analyses.

The two areas, six capacities, and two possible improvements in the levels of performance (from 0 to 1, and from 1 to 2), presented 24 options to the experts for scoring. Experts that completed one area scored 12 options. Experts that completed two areas scored 24 options. For example, Table 1 shows the un-weighted scores by PAHO staff member 1 for both areas. Un-weighted scores were then combined with the capacity, criteria, and area weights using equation 1 to produce final scores for each option (Table 2). In general, it can be seen that negative scores are predominant for the public health area. This is due to the larger weight allocated to the criteria ‘Time’, which has a negative effect. The last column of Table 2 also shows the option-specific rank order as assessed by PAHO staff member 1.

Table 1. Un-weighted scores by expert 1, by level of performance (rows) against the four criteria (2nd–5th columns) for the two areas of interest (public health and animal health).

Public health	Criteria
	Impact	Time	Public opinion	Cost
Risk analysis from level [0 to 1]	50	54	70	33
Risk analysis from level [1 to 2]	50	46	30	67
Pre-exposure prophylaxis from level [0 to 1]	97	38	52	52
Pre-exposure prophylaxis from level [1 to 2]	3	62	48	48
Post-exposure prophylaxis from level [0 to 1]	92	42	52	35
Post-exposure prophylaxis from level [1 to 2]	8	58	48	65
Case control from level [0 to 1]	96	54	96	49
Case control from level [1 to 2]	4	46	4	51
Education from level [0 to 1]	97	98	96	53
Education from level [1 to 2]	3	2	4	47
Surveillance from level [0 to 1]	64	64	34	68
Surveillance from level [1 to 2]	36	36	66	32
Animal health
Risk analysis from level [0 to 1]	81	52	55	43
Risk analysis from level [1 to 2]	19	48	45	57
Vaccination from level [0 to 1]	95	11	97	52
Vaccination from level [1 to 2]	5	89	3	48
Population control from level [0 to 1]	95	99	95	94
Population control from level [1 to 2]	5	1	5	6
Import control from level [0 to 1]	95	35	25	23
Import control from level [1 to 2]	5	65	75	77
Surveillance from level [0 to 1]	85	58	20	67
Surveillance from level [1 to 2]	15	42	80	33
Case control act from level [0 to 1]	76	53	44	53
Case control act from level [1 to 2]	24	47	56	47

Table 2. Weighted scores by expert 1 for the public health and animal health areas, against the four criteria.

Public health	Criteria				Overall score	Rank
	Impact	Time	Public opinion	Cost
Risk analysis from level [0 to 1]	2.1	6.8	0.1	2.0	−5.7	21
Risk analysis from level [1 to 2]	2.1	5.8	0.1	4.0	−6.6	23
Pre-exposure prophylaxis from level [0 to 1]	0.7	1.2	0.3	1.2	−1.2	16
Pre-exposure prophylaxis from level [1 to 2]	0.0	2.0	0.3	1.1	−2.4	19
Post-exposure prophylaxis from level [0 to 1]	10.2	2.2	0.8	0.5	7.2	1
Post-exposure prophylaxis from level [1 to 2]	0.9	3.1	0.7	1.0	−2.2	18
Case control from level [0 to 1]	6.3	4.5	1.0	0.4	2.0	4
Case control from level [1 to 2]	0.3	3.8	0.0	0.5	−3.4	20
Education from level [0 to 1]	1.6	1.4	2.5	0.2	2.1	3
Education from level [1 to 2]	0.0	0.0	0.1	0.2	−0.1	10
Surveillance from level [0 to 1]	1.8	13.6	0.1	2.4	−12.3	24
Surveillance from level [1 to 2]	1.0	7.7	0.3	1.1	−6.6	22
Animal health
Risk analysis from level [0 to 1]	6.7	1.2	0.5	1.8	0.5	6
Risk analysis from level [1 to 2]	1.6	1.1	0.4	2.4	−0.2	11
Vaccination from level [0 to 1]	20.2	0.2	1.5	1.4	2.5	2
Vaccination from level [1 to 2]	1.1	1.3	0.0	1.3	−0.2	12
Population control from level [0 to 1]	12.0	0.4	2.4	0.7	1.7	5
Population control from level [1 to 2]	0.6	0.0	0.1	0.0	0.1	8
Import control from level [0 to 1]	1.4	1.2	0.0	2.5	−0.3	13
Import control from level [1 to 2]	0.1	2.3	0.1	8.5	−1.3	17
Surveillance from level [0 to 1]	2.7	3.5	0.1	4.4	−0.6	15
Surveillance from level [1 to 2]	0.5	2.5	0.3	2.2	−0.5	14
Case control from level [0 to 1]	4.1	0.5	0.3	0.9	0.4	7
Case control from level [1 to 2]	1.3	0.4	0.4	0.8	0.1	9

After aggregation of the weighted scores by the five experts that completed the questionnaire on animal health, the rank order of investment options was as follows: (1) vaccination I, (2) population control I, (3) risk analysis I, (4) case control I, (5) population control II, (6) risk analysis II, (7) vaccination II, (8) surveillance I, (9) case control II, (10) surveillance II, (11) import controls I, (12) import controls II (the investment options are indexed ‘I’ to show improvements from level of performance 0 to level of performance 1, and ‘II’ to show improvements from level of performance 1–2).

Discussion

The results are presented aggregated, assuming that the budget for rabies control is one, finite and that activities in the two areas, public and animal health, compete for the same resources. Alternatively, the model can be area-specific, as shown for animal health. Albeit provisional and based on few observations, the rank order suggested by the aggregation of the experts’ opinions on the animal health area merit some discussion. As expected, given the large evidence of its impact in the control of canine rabies,¹¹ vaccination appeared as the highest priority. Perhaps the most salient feature is the little relevance of the surveillance capacity. This is mostly due to the high score of surveillance against the criterion ‘time’, with negative effects, and the relative low scores against ‘impact’ and ‘public opinion’, of positive sign (results not shown). In general, the value or the impact of surveillance efforts is of difficult characterization.¹² Questions like ‘How many cases of disease x have we prevented thanks to a timely, representative and sensitive surveillance?’ regularly demand complex methodological approaches. Moreover, the scope of these approaches is restricted and does not include the contribution of surveillance to many other health interventions such as vaccination and risk analyses (such consideration would possibly require a different modelling of these capacities in our setting due to dependencies). A more comprehensive and demanding exercise would be required to capture these complexities. This was outside the scope of this pilot study and will be considered for future developments.

The model presented here merely introduces a systematic platform for the evaluation of rabies capacities. Once better informed by a larger number of experts, the application of the model by a country should produce two outputs: (i) a snapshot of the country rabies programme capacities with overall and capacity-specific scores, and (ii) a ranked list of what levels of performance return the overall best investment strategy. Furthermore, the regular application of the model by a country or territory should allow trend analysis of rabies capacities, if weights are kept the same in time, or the observation of how differing constraints and rabies risks led to a variable prioritization of capacities. For example, countries may replace government-led dog mass vaccination campaigns for enhanced surveillance given different risk attitudes and public perceptions. The model should also facilitate comparisons between those countries or territories with similar weights across the model, and lead to efficient investment strategies not only for countries seeking dog-mediated human rabies elimination, but also for those countries and territories that have already reached that status and still sustain sizeable rabies programmes. In addition, given the neglect that rabies programmes suffer relative to other diseases, by presenting a transparent and efficient investment strategy, the model might give a competitive advantage to rabies over other conditions contending for the same finite resource.

Methodological improvements and following steps

Improvements to the model are many. We describe just a few. First and foremost is the need to review the identification of objectives and capacities, and definitions across the model by a larger pool of experts. This can only add robustness to the model and increase its acceptability by the countries that are its potential users, as experience has shown that strategic-level decisions are better accepted and implemented when MCDA does not start with a pre-defined set of options.¹³

Online elicitation was a convenient and inexpensive option to gather expert opinion. However, it is not free from problems. The questionnaires were large (on average, an hour was required for their completion). In addition, online collection of judgements such as scoring and criteria comparison uses abstract concepts and imposes a heavy cognitive burden on the experts. The combination of these two requirements could well explain why seven experts failed to complete the questionnaires. Similar low responses to online MCDA exercises are reported elsewhere.¹⁴ Facilitated meetings with the working group on rabies-free certification, recommended by rabies programme managers at the REDIPRA 14 meeting, can be planned as part of the working group agenda.

The model assumes preferential independence among criteria. However, this may be questionable for the ‘cost’ and ‘impact’ criteria as alternatives leading to greater impact may lead to greater costs too. In the future, model alterations will be needed to account for this effect.

As a result of the current characterization of the problem, options are cumulative and hence it is not possible to move from level of performance 1 to 2 without first going from level of performance 0 to 1. Future developments will allow evaluations to start from any level. In addition, greater granularity within the capacities will be considered by the addition of a fourth level of performance.

Approaches to the aggregation of expert responses need formal evaluation by means of sensitivity analyses. To this end, we assessed the impact of varying the stage at which expert averaging is conducted on the rank order of options. By averaging the weighted scores, instead of the un-weighted ones, we observed minor variations in the middle of the rank order for animal health options. However, more observations will be required to asses the adequacy of any aggregating method.

In its current format, the model does not include investment amounts, as costs are evaluated by qualitative judgements. However, actual budgets or costs could be allocated to the individual capacities to allow selection of the best ranked options up to an overall budget ceiling. Other components can be added to the model to increase the scope of the evaluation, on wildlife rabies for example.

Successful application of MCDA models in regular policy contexts have been reported elsewhere,¹⁵ and have shown the benefits of systematic approaches to capacity prioritization. Indirectly, as countries or territories align their evidence needs to the framework requirements, we will be able to assemble a repository of comparable data on process-related indicators across the region, such as response time to outbreaks, is of critical importance for future modeling, operational and policy needs towards the elimination of dog mediated rabies in the Americas.